ledger-staging · 2026-07-08
run 2026-07-08 02:00 UTC against staging · 12 failure modes · blast radius within limits
| # | Outcome | Failure mode | Observed | First impact | Recovery |
|---|---|---|---|---|---|
| 01 | survived | Pod / Container Kill critical | container restarted by compose in 3.4s | 0.7s | 3.4s |
| 02 | degraded | Memory Pressure 90% high | swap thrash; requests slowed for 25s
Report: impact and remediation → |
9s | 25s |
| 03 | degraded | Disk Fill 95% high | write errors on ledger-db for 40s until log rotation freed space
Report: impact and remediation → |
6s | 40s |
| 04 | survived | Network Latency +500ms medium | timeouts held at 2s; no retries needed | — | — |
| 05 | survived | DNS Resolution Failure critical | resolver cache served the last good records | — | — |
| 06 | failed | Certificate Expiry critical | ledger-api refused every client handshake; no alert
Report: impact and remediation → |
0.3s | — |
| 07 | degraded | Dependency Unavailable high | queue backed up for 60s while the payments stub was down
Report: impact and remediation → |
1.5s | 1m |
| 08 | survived | Network Partition critical | api reconnected to ledger-db 4.5s after the partition healed | 2s | 4.5s |
| 09 | survived | CPU Saturation medium | one core pinned; p99 held under 400ms | — | — |
| 10 | failed | Credential Revocation high | db password rotated; api kept the old pool until restarted
Report: impact and remediation → |
0.4s | — |
| 11 | survived | Clock Skew +5min medium | nothing on this path reads the clock | — | — |
| 12 | degraded | Config Corruption high | restart loop for 22s; the old container kept serving
Report: impact and remediation → |
1s | 22s |
survived 6 degraded 4 failed 2 previous run: 6 · 4 · 2
Suggested remediation
02 · Memory Pressure 90% degraded
- Set memory limits with headroom over the working set, and shed load before the limit. — from: observed “swap thrash”
03 · Disk Fill 95% degraded
- Alert on volume usage well before it is full, and on write errors, not only on crashes. — from: observed “write errors”
06 · Certificate Expiry failed
- Watch certificate expiry ahead of time and alert on the TLS handshake error rate at the edge. — from: observed “handshake”
- Automate renewal and reload (cert-manager or equivalent) so a replaced certificate is picked up without a restart. — from: no recovery inside the injection window
- Alert on the symptom this drill caused; nothing watched it this time. — from: outcome failed; observed “no alert”
07 · Dependency Unavailable degraded
No rule matches what was observed for this mode, so nothing is suggested rather than a generic paragraph.
10 · Credential Revocation failed
No rule matches what was observed for this mode, so nothing is suggested rather than a generic paragraph.
12 · Config Corruption degraded
- Validate configuration at startup and refuse to start on an invalid value, rather than running on it. — from: outcome degraded
record: evidence/ledger-staging/2026-07-08. What was injected, what was observed and what broke, dated. This run has no sealed record. Every failed or degraded row links to its written report — impact and remediation — which the record carries too.