Problem statement
The situation
Transaction Service (our team) authorises card-present and till payments. For every payment it depends on an external Wallet Service that owns customer balances. Today the integration is:
GET /{accountNo}/balances → read the balance
if balance ≥ amount → approve and debit, else decline
The balance cannot be cached: it is volatile, and other legacy systems debit the same accounts. When Wallet Service becomes slow or goes down, Transaction Service goes down with it. Threads block on the wallet call, the connection pool drains, and tills time out.
The ask: keep Transaction Service available and correct while Wallet Service degrades.
The constraints that shape the answer
| # | Constraint | Consequence |
|---|---|---|
| C1 | The caller is synchronous. A person is standing at the till, and the terminal's own timeout is a few seconds (~3 s). | We can never queue the decision. Queuing would silently turn a synchronous approve/decline into an asynchronous one. A fast, honest decline is a better product than a correct "pending". |
| C2 | We are not the only party debiting the account. Legacy systems debit directly. | A local reservation ledger ("last balance − my in-flight amounts") is not a safe, conservative estimate. Any decision taken off a read is exposed to concurrent legacy debits. |
| C3 | Wallet Service is owned by another team. We cannot change it unilaterally. | We have to fix it now with the API that exists, and propose a better interface for the long term. |
| C4 | Money must never be lost or double-moved. | Idempotency on every boundary, plus a local ledger that can reconcile without the wallet being reachable. |
The two problems hiding in the prompt
1. A correctness defect that happens on a perfectly healthy system
Read the balance, then decide is a time-of-check-to-time-of-use race. Balance ₦100. Two ₦80 payments arrive close together, from two tills, or from one till and a legacy standing order. Both read ₦100, both approve, both debit. Nothing failed: no timeout, no outage, and a circuit breaker never fires. No resilience mechanism can reach this bug. Only moving the decision inside the system that owns the money fixes it.
2. An availability defect when the wallet is slow or down
By Little's Law (L = λW), when the wallet goes from 50 ms to 1 s at 200 rps, the number of calls in
flight goes from 10 to 200. A pool of 20 saturates, the observed latency grows without bound, and tills
time out while the wallet is still returning 200 OK. Slow, failing and flapping are three different
modes, and each needs a different defence.
What "solved" looks like
The solution has two phases, because that is how this lands in real life:
- Phase 1, now: fix it with the API we already have. The existing wallet
POST /debitsis atomic and funds-checked. We make the debit itself the authorisation (debit-as-authorisation). The wallet enforces the balance invariant, which removes the race even with legacy debiters. Around it we add the deadline budget, bulkheads, a slow-call-aware breaker, inbound and outbound idempotency, a fast retryable decline when the wallet is degraded, and automatic reversal of ambiguous outcomes. - Phase 2, long term: propose a hold/capture API to the Wallet team. Once it ships, an atomic hold decides the payment and capture happens asynchronously. Cancellation becomes a clean release rather than a compensating credit. We switch over by account cohort behind a flag with a kill switch, and each payment finishes with the mechanism it started with.
Details: PRD · Architecture · ADRs · Roadmap