identity-prod · 2026-07-01
run 2026-07-01 04:00 UTC against staging · 12 failure modes · blast radius within limits
| # | Outcome | Failure mode | Observed | First impact | Recovery |
|---|---|---|---|---|---|
| 01 | survived | Pod / Container Kill critical | replica replaced in 1.8s; sessions kept in the shared store | 0.5s | 1.8s |
| 02 | degraded | Memory Pressure 90% high | GC pauses; token issuance slowed for 28s
Report: impact and remediation → |
15s | 28s |
| 03 | survived | Disk Fill 95% high | alert fired at 90%; the audit log rotated | — | — |
| 04 | survived | Network Latency +500ms medium | token issuance slowed by 510ms, no timeouts | — | — |
| 05 | failed | DNS Resolution Failure critical | directory lookups failed; sign-in down until DNS returned
Report: impact and remediation → |
2s | — |
| 06 | survived | Certificate Expiry critical | fell back to the secondary signing certificate | 0.3s | 0.3s |
| 07 | survived | Dependency Unavailable high | directory breaker opened; cached profiles served | 1.1s | 1.1s |
| 08 | survived | Network Partition critical | leader re-elected in 7.9s | 3.2s | 7.9s |
| 09 | survived | CPU Saturation medium | autoscaler added a replica at 30s; p99 under 600ms | — | — |
| 10 | survived | Credential Revocation high | signing key rotated; clients refetched the key set | 0.1s | 0.2s |
| 11 | degraded | Clock Skew +5min medium | assertions rejected as not yet valid for 12s
Report: impact and remediation → |
0.8s | 12s |
| 12 | survived | Config Corruption high | admission webhook refused the bad config map | — | — |
survived 9 degraded 2 failed 1 previous run: 9 · 2 · 1
Suggested remediation
02 · Memory Pressure 90% degraded
No rule matches what was observed for this mode, so nothing is suggested rather than a generic paragraph.
05 · DNS Resolution Failure failed
- Add a DNS caching sidecar, such as dnsmasq or CoreDNS with the cache plugin. — from: outcome failed
- Retry DNS lookups with bounded exponential backoff instead of failing the health check on the first miss. — from: outcome failed; first impact 2s after injection
11 · Clock Skew +5min degraded
- Run time synchronisation on every node and alert on drift. — from: outcome degraded
record: evidence/identity-prod/2026-07-01. What was injected, what was observed and what broke, dated. This run has no sealed record. Every failed or degraded row links to its written report — impact and remediation — which the record carries too.