ledger-staging · 2026-10-14
run 2026-10-14 02:00 UTC against staging · 12 failure modes · halted by a stop condition
disk fill 95% crossed its stop condition (write errors above 5% on ledger-db); the run halted there and the modes after it were not reached
| # | Outcome | Failure mode | Observed | First impact | Recovery |
|---|---|---|---|---|---|
| 01 | survived | Pod / Container Kill critical | container restarted by compose in 3.4s | 0.7s | 3.4s |
| 02 | degraded | Memory Pressure 90% high | swap thrash; requests slowed for 25s
Report: impact and remediation → |
9s | 25s |
| 03 | failed | Disk Fill 95% high | write errors on ledger-db; stop condition reached
Report: impact and remediation → |
6s | — |
| 04 | not reached | Network Latency +500ms medium | — | — | — |
| 05 | not reached | DNS Resolution Failure critical | — | — | — |
| 06 | not reached | Certificate Expiry critical | — | — | — |
| 07 | not reached | Dependency Unavailable high | — | — | — |
| 08 | not reached | Network Partition critical | — | — | — |
| 09 | not reached | CPU Saturation medium | — | — | — |
| 10 | not reached | Credential Revocation high | — | — | — |
| 11 | not reached | Clock Skew +5min medium | — | — | — |
| 12 | not reached | Config Corruption high | — | — | — |
survived 1 degraded 1 failed 1 not reached 9 previous run: 1 · 1 · 1
Suggested remediation
02 · Memory Pressure 90% degraded
- Set memory limits with headroom over the working set, and shed load before the limit. — from: observed “swap thrash”
03 · Disk Fill 95% failed
- Alert on volume usage well before it is full, and on write errors, not only on crashes. — from: observed “write errors”
- Cap what fills the volume — log rotation, retention limits, a quota — so usage falls back without a person. — from: no recovery inside the injection window
record: evidence/ledger-staging/2026-10-14. What was injected, what was observed and what broke, dated. This run has no sealed record. Every failed or degraded row links to its written report — impact and remediation — which the record carries too.