checkout-prod · 2026-10-11
run 2026-10-11 03:00 UTC against staging · 12 failure modes · blast radius within limits
| # | Outcome | Failure mode | Observed | First impact | Recovery |
|---|---|---|---|---|---|
| 01 | survived | Pod / Container Kill critical | recovered in 2.1s | 1.2s | 2.1s |
| 02 | survived | Memory Pressure 90% high | degraded cleanly, no errors | — | — |
| 03 | failed | Disk Fill 95% high | writes blocked, no alert fired
Report: impact and remediation → |
11s | — |
| 04 | survived | Network Latency +500ms medium | timeouts held, retries bounded | — | — |
| 05 | failed | DNS Resolution Failure critical | cascade: auth at 4s, gateway 502 at 8s
Report: impact and remediation → |
4s | 30s |
| 06 | failed | Certificate Expiry critical | TLS handshake errors at the edge
Report: impact and remediation → |
0.4s | — |
| 07 | survived | Dependency Unavailable high | circuit breaker opened | 0.9s | 0.9s |
| 08 | survived | Network Partition critical | leader re-elected in 6.4s | 2.8s | 6.4s |
| 09 | degraded | CPU Saturation medium | p99 up 1.9s until the autoscaler added a replica at 48s
Report: impact and remediation → |
12s | 48s |
| 10 | survived | Credential Revocation high | token refreshed, one request retried | 0.2s | 0.3s |
| 11 | degraded | Clock Skew +5min medium | token validation errors for 40s
Report: impact and remediation → |
1s | 40s |
| 12 | survived | Config Corruption high | startup validation refused the bad value | — | — |
survived 7 degraded 2 failed 3 previous run: 7 · 2 · 3
Suggested remediation
03 · Disk Fill 95% failed
- Alert on volume usage well before it is full, and on write errors, not only on crashes. — from: observed “writes blocked”
- Cap what fills the volume — log rotation, retention limits, a quota — so usage falls back without a person. — from: no recovery inside the injection window
- Alert on the symptom this drill caused; nothing watched it this time. — from: outcome failed; observed “no alert”
05 · DNS Resolution Failure failed
- Add a DNS caching sidecar, such as dnsmasq or CoreDNS with the cache plugin. — from: outcome failed
- Retry DNS lookups with bounded exponential backoff instead of failing the health check on the first miss. — from: outcome failed; first impact 4s after injection
- Put a circuit breaker between the caller and the dependency whose name stopped resolving, so one unreachable dependency degrades a path instead of the gateway. — from: observed “cascade”
06 · Certificate Expiry failed
- Watch certificate expiry ahead of time and alert on the TLS handshake error rate at the edge. — from: observed “TLS handshake errors at the edge”
- Automate renewal and reload (cert-manager or equivalent) so a replaced certificate is picked up without a restart. — from: no recovery inside the injection window
09 · CPU Saturation degraded
- Lower the autoscaler's CPU target or its stabilisation window, so capacity arrives before latency degrades. — from: recovery took 48s
11 · Clock Skew +5min degraded
- Allow a bounded leeway (tens of seconds) on token `exp` and `nbf` checks. — from: observed “token validation errors for 40s”
- Run time synchronisation on every node and alert on drift. — from: outcome degraded
record: evidence/checkout-prod/2026-10-11. What was injected, what was observed and what broke, dated. This run has no sealed record. Every failed or degraded row links to its written report — impact and remediation — which the record carries too.