Operations
Skeleton. Alert rules land in M2-T4. The cutover runbook is finalised in M3-T5. Runtime switches (ADR-0012) land in M2-T1.
SLO
Decided within 3 s: proportion of POST /v1/payments that return a final decision within 3 s,
out of all valid requests. Target 99.9 % over 28 days. Alert on multi-window burn rate (1 h / 5 m fast
burn, 6 h / 30 m slow burn).
Alerts (planned)
| Alert | Condition | First action |
|---|---|---|
| SLI fast burn | burn rate > 14.4 | Check the wallet breaker and the decline-reason split |
| Wallet breaker open | open > 2 min | Contact the Wallet team. Consider stand-in only with business sign-off. |
| Orphaned money | UNKNOWN / REVERSAL_PENDING / HELD older than 15 min > 0 | Check the wallet-async bulkhead and DEAD items |
| Work item DEAD | > 0 | Inspect last_error, fix, replay (reset to PENDING) |
| Reconciliation drift | > 0 | Manual investigation. Never auto-resolve. |
Switches
| Switch | Property | Effect |
|---|---|---|
| Phase 2 cohort | runtime switch rollout.hold-capture-percent (POST /actuator/switches) |
Share of accounts on HOLD_CAPTURE for new payments, on every instance within 5 s |
| Kill switch | runtime switch rollout.kill-switch |
All new payments use DIRECT_DEBIT within 5 s |
| Stand-in | runtime switch stand-in.enabled; floor and caps in application.yml |
Approve under the floor limit while the wallet is unavailable. Needs Head of Risk sign-off (ADR-0009). |
Runbook: cutover (draft)
- Confirm wallet v2 invariants H1–H5 against their sandbox (contract tests).
- Set the percentage to 1. Watch for 24 h: decline rate per mode, p99, orphaned holds, drift.
- Go 10 → 50 → 100, holding 24–72 h at each step.
- Rollback:
POST /actuator/switches {"name":"rollout.kill-switch","value":"true"}. No other action is needed, because in-flight holds finish on their own.