Product requirements: resilient POS payment authorisation
| Status | Baseline v1.1, agreed 2026-10-19. No open questions: each one is resolved by a written assumption and an ADR (§7). |
| Owner | Transaction Service team |
| Depends on | Wallet Service team (Phase 2 interface), Risk/Business (stand-in policy) |
| Related | Problem statement · Architecture · Roadmap |
1. Goals
| ID | Goal | Measured by |
|---|---|---|
| G1 | No overdraft caused by our check-then-act race, including against concurrent legacy debiters | Concurrency regression test; zero race-caused negative balances in reconciliation |
| G2 | Tills always get a decision inside their patience, even when the wallet is slow or down | SLI: % of POST /payments answered with a final decision within 3 s (SLO 99.9 %) |
| G3 | No double-debit and no lost money across retries, timeouts and restarts | Idempotency test matrix; reconciliation drift = 0 |
| G4 | Wallet degradation is contained: it never exhausts Transaction Service resources | Pool and bulkhead saturation stay bounded under fault injection |
| G5 | A clean path to hold/capture without a big-bang cutover | Cohort rollout 0 → 100 % with kill switch, mixed-mode tests green |
2. Non-goals
- Changing Wallet Service ourselves. Phase 2 is a proposal the Wallet team implements.
- Risk/fraud scoring. It is the same problem class (a synchronous external dependency) and gets the same primitives later, but it is out of scope here.
- Card-network messaging (ISO 8583), multi-currency FX, partial or incremental authorisation, chargebacks.
- Notifications to payers. The outbox records events; a publisher is deferred.
- An asynchronous (
202 + poll) payment contract. It is deliberately rejected for till callers (C1).
3. Users and callers
| Caller | Needs |
|---|---|
| POS terminal / till (primary) | One synchronous call, a decision in under 3 s, safe retries, a way to reverse when it timed out without a paymentId |
| Operations / on-call | To tell "wallet is down" apart from "customers are broke", to see orphaned money, and a kill switch |
| Business / risk owner | Control over the stand-in floor limit and exposure caps, and a record of every stand-in decision |
| Wallet team | A precise, testable interface request for holds |
4. Functional requirements
Payments
- FR-1
POST /v1/paymentsreturns201withstatus ∈ {APPROVED, DECLINED}within the deadline. It never returns a pending status to the till. - FR-2
Idempotency-Keyis required. Same key and same body → the stored response (same status, equal JSON body) withIdempotent-Replayed: true. Same key and different body →422 IDEMPOTENCY_KEY_REUSED. Same key while the first attempt is still running →409withRetry-After. A key older than its 24 h retention →422 IDEMPOTENCY_KEY_EXPIRED, never re-executed (ADR-0010). - FR-3 Phase 1 decides by calling wallet
POST /debitswithreference = paymentId.201→ APPROVED.409 INSUFFICIENT_FUNDS→ DECLINED, not retryable.409 DUPLICATE_REFERENCE→ the original debit counts as success. - FR-4 If the decision call is ambiguous (timeout after send), look it up by reference within the remaining budget. Found → APPROVED. Not found → we still cannot be sure, so go to FR-5.
- FR-5 If the outcome is still unknown at the deadline → reply DECLINED
WALLET_TIMEOUT, retryable, and persistREVERSAL_PENDING. A worker resolves it later: if a debit is found, it posts a compensating credit. The customer is never charged for goods they did not receive. - FR-6 Wallet unavailable (breaker open, bulkhead full, 503, connection refused) → DECLINED
WALLET_UNAVAILABLE, retryable, withRetry-After, in milliseconds rather than seconds. Nothing is queued. - FR-7
GET /v1/payments/{id}returns the current external status. A payment that is still being decided returns404 PAYMENT_NOT_YET_DECIDEDwithRetry-After: 1.
Reversals
- FR-8
POST /v1/reversalsis addressed byoriginalIdempotencyKey, because a till that timed out never learned thepaymentId. It always returns202. - FR-9 Reversal before the original arrives → a tombstone is stored. A late original with that key is
declined
REVERSED_BEFORE_AUTHORISATION. Payment intake and reversal intake serialise on the key. - FR-10 Reversal of an APPROVED payment → a compensating credit (Phase 1) or a release/refund (Phase 2), asynchronous, idempotent, and retried with backoff until done or dead-lettered.
Phase 2 and cutover
- FR-11 When the wallet v2 holds API is available,
HOLD_CAPTUREmode places an atomic hold (decision), then captures asynchronously. A hold that cannot be captured is released or left to expire by TTL. - FR-12 The decision mode is chosen per new payment by a stable account-hash cohort
(
hold-capture-percent0–100). The kill switch forcesDIRECT_DEBITfor new payments. The mode is persisted, and in-flight payments complete with their original mode. - FR-13 Hold TTL > max capture retry window + reaper period. Expiry uses the wallet's
expiresAt, never a locally computed time.
Degradation policy
- FR-14 The degradation policy is a separate, table-driven component, consulted only when the
decision call is
Unavailable: (switches, amount, currency, account standing, exposure) → (stand-in approve | decline retryable). The rules are in ADR-0009. - FR-15 Stand-in (approve below a floor limit when the wallet is unavailable) is implemented and tested but disabled by default. Enabling it requires: floor limit > 0, per-account and global exposure caps, every stand-in decision flagged and reconciled, and a kill switch. Because of C2, a stand-in approval is real credit risk, and the business must own it explicitly.
5. Non-functional requirements
| ID | Requirement |
|---|---|
| NFR-1 | Server-side budget 2,400 ms (till ~3,000 ms). Every downstream timeout is derived from the remaining deadline (min(own limit, remaining)), never configured independently. |
| NFR-2 | Wallet calls on the decision path use a dedicated bulkhead (wallet-decision). Worker calls use a separate one (wallet-async), so a backlog of reversals or captures cannot starve live decisions. |
| NFR-3 | The breaker trips on slow-call rate as well as error rate. It is count-based, has a minimum number of calls, an exponentially increasing open duration, and needs several half-open successes. |
| NFR-4 | At most one retry layer: the layer that owns the idempotency key. There is a retry budget, and backoff uses full jitter. |
| NFR-5 | Money is bigint minor units plus ISO-4217 currency, everywhere. |
| NFR-6 | Every state change and its outbox event commit in one local transaction. |
| NFR-7 | Background work uses a Postgres work queue (FOR UPDATE SKIP LOCKED) with bounded attempts, a dead-letter state, and alerts on depth and age. |
| NFR-8 | Observability: per-dependency RED metrics, pool and bulkhead saturation, breaker time-open, decline reasons split by cause, orphaned and unknown counts, reconciliation drift, stand-in exposure. |
6. Assumptions
Each assumption is a working assumption the design is built on. It is verified by a contract test against the stub where possible, and confirmed later by its owner. If it turns out false, the "If false" column and the linked ADR's revisit trigger say what changes.
| ID | Assumption | Owner | If false |
|---|---|---|---|
| A1 | Wallet v1 POST /debits is atomic and funds-checked. It never overdraws. |
Wallet team | Phase 1 falls back to per-account serialisation plus a risk-capped read. The documented residual risk is the legacy-debit race. |
| A2 | Wallet v1 rejects a duplicate reference with DUPLICATE_REFERENCE and the original id. |
Wallet team | Retries are unsafe, so retry count drops to 0 and resolution relies on lookup only. |
| A3 | Wallet v1 supports GET /debits?reference=. |
Wallet team | Ambiguous outcomes can only be resolved by the daily statement reconciliation. |
| A4 | A request the wallet has accepted is processed, or abandoned, within T_settle = 30 s. After that, "not found" is final (ADR-0008). | Wallet team | A late landing is reported as drift (ADR-0011) and repaired manually. Raise T_settle. |
| A5 | Wallet v2: available = ledger − holds, and all debit paths, legacy included, check available. |
Wallet team | A capture can fail with NSF, which needs a forced-capture or overdraft flow. That is a Phase 2 blocker. |
| A6 | The till sends a reversal when it does not get a response, keyed by the original idempotency key. | POS team | Ambiguous declines are still auto-reversed by FR-5. This only affects how quickly they close. |
| A7 | Load: 200 rps peak, wallet p99 ~50 ms healthy. | Platform | Retune the bulkhead sizes and the Hikari pool. |
| A8 | Wallet v1 credits dedup on reference exactly like debits (DUPLICATE_REFERENCE + existingCreditId), so a compensating credit is safely retryable. |
Wallet team | Compensation must look up before every credit. That needs a credit lookup API. |
| A9 | The Wallet team ships v2 as specified, including H1–H5. M3 is built and tested against the stub regardless. Production enablement (hold-capture-percent > 0) is gated on contract tests against their sandbox. |
Wallet team | Phase 1 stays the steady state (ADR-0003, "If rejected"). |
| A10 | One settlement currency per deployment (NGN). Stand-in limits and caps are in that currency. |
Business | Stand-in config becomes per currency. |
| A11 | A healthy wallet answers debit/hold with p99.9 ≤ 400 ms (ADR-0007). | Wallet team | Retune the limits from measurement (ADR-0007, revisit trigger). |
| A12 | A terminal never resends a payment more than 24 h after its first attempt (ADR-0010). | POS team | Raise the idempotency retention. |
| A13 | No consumer needs outbox events within M1–M3 (ADR-0011). | Platform | Schedule the relay task. |
| A14 | ≥ 2 service instances, one region, sharing one Postgres. Breakers and bulkheads are per instance. | Platform | — |
| A15 | Management (actuator) endpoints are reachable only from the internal network (ADR-0012). | Platform | Add authentication to /actuator/switches. |
7. Questions log
There are no open questions. Every question was resolved by a working assumption (§6) and an ADR.
The assumption's owner confirms it later. If the answer differs, the ADR's revisit trigger says what
changes. A new question is raised as a GitHub issue labelled question, and any task it affects is
not ready until the question is closed.
| ID | Question | Resolution | Assumption | ADR |
|---|---|---|---|---|
| OQ-1 | What is the wallet's p99.9 for POST /debits? |
Assume ≤ 400 ms. Decision call 800 ms, lookup 300 ms, server budget 2.4 s. One lookup and no retries on the decision path. | A11 | 0007 |
| OQ-2 | Does the wallet abandon a request on client disconnect? | Does not matter. "Not found" is final only after T_settle = 30 s, and compensation reads the wallet's truth. | A4 | 0008 |
| OQ-3 | Floor limit, caps, and who signs off on stand-in? | Off by default. The Head of Risk signs off. A fixed rule set (known account, caps reserved atomically). Test values are not defaults. | A10 | 0009 |
| OQ-4 | Is 24 h of idempotency retention enough? | 24 h. An expired key returns 422 IDEMPOTENCY_KEY_EXPIRED and is never re-executed. |
A12 | 0010 |
| OQ-5 | Who consumes outbox events? When is the relay needed? | No consumer yet. A fixed event catalogue is written now, and the relay is deferred. | A13 | 0011 |
| — | Will the Wallet team accept the v2 holds API? | Build M3 against the stub now. Enabling Phase 2 in production is gated on sandbox contract tests. | A9 | 0003 |
| — | How do the kill switch and cohort change without a deploy, on every instance? | Postgres-backed runtime switches, refreshed every 5 s | A14, A15 | 0012 |
8. Success metrics
- Till-facing SLI ≥ 99.9 % decided within 3 s, measured over a 28-day window with multi-window burn-rate alerts.
- Zero double-debits and zero race-caused overdrafts in reconciliation.
- Orphaned
UNKNOWN/REVERSAL_PENDINGolder than 15 min: 0 (alert). - Phase 2: 100 % cohort on
HOLD_CAPTUREwith the decline rate unchanged and no increase in drift.