checkout-prod · 2026-07-26
run 2026-07-26 03:00 UTC against staging · 12 failure modes · blast radius within limits
| # | Outcome | Failure mode | Observed | First impact | Recovery |
|---|---|---|---|---|---|
| 01 | survived | Pod / Container Kill critical | recovered in 2.1s | 1.2s | 2.1s |
| 02 | survived | Memory Pressure 90% high | degraded cleanly, no errors | — | — |
| 03 | failed | Disk Fill 95% high | writes blocked, no alert fired
Report: impact and remediation → |
11s | — |
| 04 | survived | Network Latency +500ms medium | timeouts held, retries bounded | — | — |
| 05 | failed | DNS Resolution Failure critical | cascade: auth at 4s, gateway 502 at 8s
Report: impact and remediation → |
4s | 30s |
| 06 | failed | Certificate Expiry critical | TLS handshake errors at the edge
Report: impact and remediation → |
0.4s | — |
| 07 | failed | Dependency Unavailable high | no circuit breaker; gateway queued until timeout
Report: impact and remediation → |
0.8s | — |
| 08 | survived | Network Partition critical | leader re-elected in 6.4s | 2.8s | 6.4s |
| 09 | degraded | CPU Saturation medium | p99 up 1.9s until the autoscaler added a replica at 48s
Report: impact and remediation → |
12s | 48s |
| 10 | failed | Credential Revocation high | refresh loop retried on a revoked token until rate-limited
Report: impact and remediation → |
0.2s | — |
| 11 | degraded | Clock Skew +5min medium | token validation errors for 40s
Report: impact and remediation → |
1s | 40s |
| 12 | survived | Config Corruption high | startup validation refused the bad value | — | — |
survived 5 degraded 2 failed 5 previous run: 4 · 2 · 6
Suggested remediation
03 · Disk Fill 95% failed
- Alert on volume usage well before it is full, and on write errors, not only on crashes. — from: observed “writes blocked”
- Cap what fills the volume — log rotation, retention limits, a quota — so usage falls back without a person. — from: no recovery inside the injection window
- Alert on the symptom this drill caused; nothing watched it this time. — from: outcome failed; observed “no alert”
05 · DNS Resolution Failure failed
- Add a DNS caching sidecar, such as dnsmasq or CoreDNS with the cache plugin. — from: outcome failed
- Retry DNS lookups with bounded exponential backoff instead of failing the health check on the first miss. — from: outcome failed; first impact 4s after injection
- Put a circuit breaker between the caller and the dependency whose name stopped resolving, so one unreachable dependency degrades a path instead of the gateway. — from: observed “cascade”
06 · Certificate Expiry failed
- Watch certificate expiry ahead of time and alert on the TLS handshake error rate at the edge. — from: observed “TLS handshake errors at the edge”
- Automate renewal and reload (cert-manager or equivalent) so a replaced certificate is picked up without a restart. — from: no recovery inside the injection window
07 · Dependency Unavailable failed
- Add a circuit breaker with a fallback and a timeout below the caller's own, so requests fail fast instead of queueing. — from: observed “no circuit breaker”
- Degrade the paths that need the dependency rather than the whole gateway. — from: observed “gateway”
09 · CPU Saturation degraded
- Lower the autoscaler's CPU target or its stabilisation window, so capacity arrives before latency degrades. — from: recovery took 48s
10 · Credential Revocation failed
- Stop retrying on an authorisation failure with the same token: re-fetch the credential once, then fail and alert. — from: observed “loop retried on a revoked token until rate-limited”
11 · Clock Skew +5min degraded
- Allow a bounded leeway (tens of seconds) on token `exp` and `nbf` checks. — from: observed “token validation errors for 40s”
- Run time synchronisation on every node and alert on drift. — from: outcome degraded
record: evidence/checkout-prod/2026-07-26. What was injected, what was observed and what broke, dated. This run has no sealed record. Every failed or degraded row links to its written report — impact and remediation — which the record carries too.