Production Postmortems & RCAs
Transparent, blameless postmortems documenting past real-world incidents, 5 Whys root cause breakdowns, and permanent automated mitigations.
Production Postmortems & RCA Repository
Transparent, blameless postmortems detailing incident timelines, 5 Whys root cause investigations, and permanent automated mitigations.
GeoFusion Vector Tile Cache Invalidation Race Condition on Spatial Ingestion
0.08% transient tile render mismatch on North Sea seismic exploration map layers during active vector batch ingest. Zero data loss; all vector embeddings persisted in PostgreSQL/PostGIS.
Prometheus alert: vector_tile_render_latency_ms spike > 1200ms.
Abdlrrahman triaged: write worker evicted local Redis cache before Cloudflare purge hook completed.
Hotfix deployed: added transactional Postgres advisory lock and atomic versioned ETags.
All edge CDN caches verified in sync. Incident closed.
- W1:Why did users observe outdated vector tiles? Edge CDN served stale tiles because the purge token had not finished propagation.
- W2:Why was the purge token delayed? Cloudflare API rate limit backoff triggered during high-volume batch ingest.
- W3:Why did the API trigger so many purge requests? The worker purged per-well-log instead of per-bounding-box batch.
- W4:Why did the worker operate per-record? Spatial tile clustering was decoupled from the ingestion queue.
- W5:Root Cause: Ingestion worker lacked bounding-box aggregation and atomic ETag versioning.
- Always bind cache invalidations to immutable content hashes (ETags) rather than URL path purges.
- Aggregate spatial invalidations by GeoJSON bounding boxes before triggering CDN webhooks.
- Automate end-to-end chaos tests for CDN rate-limiting under burst ingestion.
Implement atomic versioned ETags for all vector tile endpoints
Add GeoJSON bounding-box cache aggregator in Celery worker