The story of an incident
34 minutes, one page, zero manual cleanup.
eu-west-k8s-prod · GKEOpen
T+00:00pod/payments-7f9c CrashLoopBackOff · restart 1
T+00:09docker orders-worker-3 cpu 97% · saturation
T+00:21pod/payments-a81b CrashLoopBackOff · restart 2
T+00:34pod/payments-c3d9 OOMKilled · exit 137
14alerts a traditional stack would page on in the next minute
VIGILsilence_detected
payments-api stopped emitting logs for 90s. The surrounding buffer is frozen before the pods are rescheduled and the evidence disappears.
14:41:52 WARN pool=pg-payments wait 1840ms
14:41:58 WARN heap 241Mi / 256Mi
14:42:03 ERROR alloc failed · runtime: out of memory
14:42:03 ·· no output 90s ··
14:43:33 VIGIL snapshot sealed · 50 lines
Fingerprint engineInvestigating
14→1persistent incident, one page to on-call
compose_servicepayments-api
container ids7f9c · a81b · c3d9 · …
contributorsk8s.pod.restart · docker.cpu_spike
Stability windowResolved
6/6 promoter checks · every 5 min30m stable
# PIR — INC-2291 payments-api
duration 34m · auto-resolved
root cause OOMKilled · limit 256Mi
evidence s3://forensics/payments-api/14-42.log
pages sent 1 (would have been 14)