Skip to content

Architecture

Read problem-statement.md first. This document describes the design. The why behind each major choice is in an ADR.

1. Principles

  1. Decide where the money lives. The approve/decline decision is a single atomic call to the system that owns the balance: a debit in Phase 1, a hold in Phase 2. We never read, then decide.
  2. Never queue the decision. A person is waiting. We queue only follow-ups that nobody waits for: capture, release, compensating credit.
  3. Ambiguity is a state, and it is bounded. A timeout after sending is UNKNOWN, not a failure or a success. It always resolves, by lookup, by reversal or by TTL.
  4. One deadline, propagated. Every timeout is a slice of the till's ~3 s budget.
  5. Self-sufficient local ledger. Every consistency question can be answered without the wallet being up. The wallet is needed eventually for repair, never immediately for safety.
  6. Policy is configuration. Degradation tiers and the cutover cohort are configuration, so they can be tested as tables and changed without a deploy.

2. Context (C4 level 1)

flowchart LR
  till["POS terminal / till<br/><i>synchronous, ~3 s patience</i>"]
  ts["<b>Transaction Service</b><br/>(this repo)"]
  wallet["Wallet Service<br/><i>external, other team</i><br/>v1: debits/credits<br/>v2: holds (proposed)"]
  legacy["Legacy systems<br/><i>also debit accounts</i>"]
  ops["On-call / Ops"]
  biz["Business / Risk owner"]

  till -- "POST /v1/payments<br/>POST /v1/reversals" --> ts
  ts -- "debit · lookup · credit<br/>(v2: hold · capture · release)" --> wallet
  legacy -- "direct debits" --> wallet
  ops -. "metrics, kill switch" .-> ts
  biz -. "stand-in floor limit, caps" .-> ts

Legacy debiters are the reason Phase 1 cannot use a local reservation ledger. Our in-flight amounts are not the only thing reducing the balance, so last balance − my reservations is not a conservative estimate.

3. Containers and components (C4 level 2–3)

flowchart TB
  subgraph TS["Transaction Service (Spring Boot 4, Java 25)"]
    web["Web adapter<br/>generated from api/transaction-service/v1<br/>Idempotency-Key · Request-Deadline-Ms"]
    idem["Idempotency gate<br/>INSERT-first, fingerprint, in-flight"]
    uc["Authorise use case"]
    router["DecisionModeRouter<br/>cohort % + kill switch"]
    policy["Degradation policy<br/>table-driven, stand-in OFF"]
    dd["DirectDebitStrategy<br/>(Phase 1)"]
    hc["HoldCaptureStrategy<br/>(Phase 2)"]
    wc["Wallet clients (libs/wallet-client)<br/>deadline-capped RestClient<br/>Resilience4j: breaker 'wallet',<br/>bulkheads 'wallet-decision' / 'wallet-async'"]
    workers["Workers (SKIP LOCKED)<br/>RESOLVE_UNKNOWN · REVERSE · CAPTURE · RELEASE"]
    recon["Reconciler / hold reaper<br/>drift → metric"]
    db[("Postgres local ledger<br/>idempotency_key · payment · reversal<br/>work_item · outbox")]
  end
  wallet["Wallet Service"]

  web --> idem --> uc
  uc --> router
  uc --> policy
  uc --> dd & hc
  dd & hc --> wc --> wallet
  uc --> db
  workers --> dd & hc
  workers --> db
  recon --> wc
  recon --> db
Component Responsibility Task
Idempotency gate Insert key IN_FLIGHT first; the PK violation is the duplicate signal. Compare the sha256 fingerprint (mismatch → 422). In flight → 409. Complete → replay the stored response. M1-T1
Authorise use case Persist intent → pick mode → strategy.decide(deadline) → persist decision + outbox in one transaction → respond M1-T4
DecisionModeRouter Stable CRC32(accountNo) % 100 < hold-capture-percent, with kill switch ✅ baseline
Runtime switches Kill switch, cohort %, stand-in on/off. Postgres-backed, refreshed every 5 s (ADR-0012) M2-T1
Degradation policy On Unavailable: switches / amount / standing / exposure → stand-in approve or decline retryable (ADR-0009). Stand-in disabled by default. M2-T2, M2-T3
Wallet clients One call = one WalletOutcome (Success, Rejected, Ambiguous, Unavailable), classified per ADR-0007. No exceptions on the decision path. M1-T2, M1-T3, M3-T1
Workers Claim work_item with FOR UPDATE SKIP LOCKED. Policy, compensation and the stale sweeper are in ADR-0008 M1-T6, M1-T7, M2-T3, M3-T3
Reconciler / hold reaper Finds orphaned open payments and expired holds. It re-enqueues what it can, and never resolves drift silently: it publishes it (ADR-0011). M2-T4, M3-T4

4. Phase 1: debit-as-authorisation (existing wallet API)

sequenceDiagram
  autonumber
  participant T as Till
  participant S as Transaction Service
  participant D as Local ledger
  participant W as Wallet v1
  T->>S: POST /v1/payments (Idempotency-Key K, deadline 3 s)
  S->>D: INSERT idempotency_key K IN_FLIGHT · INSERT payment PENDING
  S->>W: POST /{acct}/debits {reference: paymentId} (timeout = min(800 ms, remaining))
  alt 201 posted / 409 DUPLICATE_REFERENCE
    W-->>S: debitId
    S->>D: payment DEBITED + outbox PaymentApproved (one tx)
    S-->>T: 201 APPROVED
  else 409 INSUFFICIENT_FUNDS / 423 / 404
    S->>D: payment DECLINED + outbox
    S-->>T: 201 DECLINED (not retryable)
  else breaker open / bulkhead full / 503 / refused
    S->>D: payment DECLINED WALLET_UNAVAILABLE
    S-->>T: 201 DECLINED retryable + Retry-After
  else timeout after send (ambiguous)
    S->>W: GET /{acct}/debits?reference=paymentId (min(300 ms, remaining))
    alt found
      S->>D: payment DEBITED
      S-->>T: 201 APPROVED
    else not found / no budget left
      S->>D: payment REVERSAL_PENDING + work_item RESOLVE_UNKNOWN (one tx)
      S-->>T: 201 DECLINED WALLET_TIMEOUT retryable
      Note over S,W: Worker, after T_settle: lookup → if debit exists, POST /credits {reference: rev-paymentId}
    end
  end

Why this is correct even with legacy debiters: the wallet checks the balance and posts the debit atomically, so there is no window between check and act on our side. A concurrent legacy debit either lands first, and our debit gets NSF, or lands second and gets NSF itself.

What Phase 1 costs compared with holds: cancelling an approved payment means a compensating credit, not a release. The customer sees a debit and a credit on their statement, and money really left the account for a moment. That cost is why Phase 2 exists.

5. Phase 2: hold/capture (proposed wallet v2)

sequenceDiagram
  autonumber
  participant T as Till
  participant S as Transaction Service
  participant D as Local ledger
  participant W as Wallet v2
  T->>S: POST /v1/payments (K)
  S->>D: payment PENDING (decision_mode = HOLD_CAPTURE)
  S->>W: POST /{acct}/holds (Idempotency-Key hold:{paymentId}, ttl)
  alt 201 HELD
    S->>D: payment HELD (+ hold_id, wallet expiresAt) + work_item CAPTURE (one tx)
    S-->>T: 201 APPROVED
    Note over S,W: async: POST /holds/{id}/capture (Idempotency-Key capture:{paymentId}) → CAPTURED
  else 409 INSUFFICIENT_FUNDS
    S-->>T: 201 DECLINED
  else ambiguous
    S->>W: GET /{acct}/holds?reference=paymentId
    Note over S: found → HELD/APPROVED · otherwise decline WALLET_TIMEOUT + RESOLVE_UNKNOWN<br/>(releases the hold if it landed, ADR-0008 · never relies on TTL alone)
  end

TTL rule (FR-13): hold TTL > max capture retry window + reaper period. If this is backwards, a hold can expire while a legitimate capture is still retrying. The wallet then rejects the capture (HOLD_NOT_ACTIVE, invariant H5), and the payment needs manual repair. We alert on it and never silently re-debit.

6. Payment state machine

stateDiagram-v2
  [*] --> PENDING: intent persisted
  PENDING --> DEBITED: P1 debit posted
  PENDING --> HELD: P2 hold placed
  PENDING --> DECLINED: NSF / blocked / wallet unavailable / reversed-before-auth
  PENDING --> STANDIN_APPROVED: wallet unavailable + policy (ADR-0009, off by default)
  STANDIN_APPROVED --> DEBITED: post-recovery debit
  STANDIN_APPROVED --> STANDIN_UNRECOVERED: post-recovery debit rejected
  STANDIN_APPROVED --> REVERSAL_PENDING: till reversal
  HELD --> CAPTURE_FAILED: hold expired / not active (alert)
  CAPTURING --> REVERSAL_PENDING: till reversal (compensator reads hold status)
  PENDING --> REVERSAL_PENDING: stale request sweeper
  PENDING --> UNKNOWN: timeout after send
  UNKNOWN --> DEBITED: lookup found (within deadline)
  UNKNOWN --> HELD: lookup found (within deadline)
  UNKNOWN --> REVERSAL_PENDING: deadline reached, till told DECLINED
  HELD --> CAPTURING: capture issued
  CAPTURING --> CAPTURED: wallet confirmed
  CAPTURING --> CAPTURING: retry (idempotent)
  DEBITED --> REVERSAL_PENDING: till reversal
  HELD --> REVERSAL_PENDING: till reversal
  CAPTURED --> REVERSAL_PENDING: till reversal
  REVERSAL_PENDING --> REVERSED: credit posted / hold released / nothing had moved
  DECLINED --> [*]
  REVERSED --> [*]
  CAPTURED --> [*]
  DEBITED --> [*]

External mapping (PaymentState.externalStatus()): DEBITED/HELD/CAPTURING/CAPTURED/STANDIN_APPROVED/ STANDIN_UNRECOVERED/CAPTURE_FAILED → APPROVED, DECLINED → DECLINED, REVERSAL_PENDING → REVERSAL_PENDING, REVERSED → REVERSED. PENDING and UNKNOWN are never shown to a till.

UNKNOWN is an in-memory classification inside one request and is never persisted (ADR-0008). PENDING persists only while a request runs, or after a crash. The stale-request sweeper closes it. The STANDIN_* and CAPTURE_FAILED states are added to PaymentState by the tasks that introduce them (M2-T2, M2-T3, M3-T3).

7. Idempotency: two independent keys

Inbound: till → us Outbound: us → wallet
Generated by The till, once per business intent Us, derived deterministically from paymentId
v1 mechanism idempotency_key table, INSERT-first reference = paymentId (debit), rev-{paymentId} (credit)
v2 mechanism same Idempotency-Key: hold:/capture:/release:{paymentId}
Survives restart Stored Re-derivable, so nothing to store
Replay returns Stored response, byte for byte The original debit, credit or hold

A retryable technical decline is final for that key. The till retries the payment with a new Idempotency-Key. Re-executing under the same key would make a replay return a different outcome.

8. Resilience: deadline budget and configuration

Numbers, timeout classification and the retry rule are fixed by ADR-0007 (assumption A11).

Hop Budget
Till abandons ~3,000 ms (given)
Server total (safeledger.budget.server-total) 2,400 ms
Wallet decision call: debit or hold (wallet-decision-call) ≤ 800 ms, capped by remaining
Ambiguity lookup (wallet-lookup-call) ≤ 300 ms, capped by remaining
Local DB writes (Hikari connection-timeout 250 ms) ~100 ms
Headroom the rest. A budget with no slack fails on the first GC pause.
Breaker wallet Value Why
Window count-based, 100, min 20 calls Time windows misbehave at low traffic
Failure rate 50 % Standard
Slow-call rate 50 % over 500 ms Without it the breaker never trips on a slow wallet
Open 5 s → ×2 → 60 s Avoids synchronised retry storms against a flapping wallet
Half-open 5 permitted calls One success is not recovery
Scope per instance Shared breaker state would add a dependency to the failure path
Business rejections (NSF, DUPLICATE) not failures They are answers

Bulkheads: wallet-decision (40, no wait) and wallet-async (8). Workers can never consume the till path's permits.

9. Degradation policy (M2-T2, rules in ADR-0009)

Tier Condition (wallet unavailable) Action Default
0 stand-in enabled ∧ amount ≤ floor ∧ account and global exposure under caps APPROVE as STAND_IN, flag it, reconcile by posting the debit after recovery off (floor = 0)
1 otherwise DECLINE WALLET_UNAVAILABLE, retryable, Retry-After on

The policy runs only when the decision call was Unavailable. It never runs on Ambiguous. A tier-0 approval needs every condition in ADR-0009 (known account, no recent block, single currency, both caps reserved atomically).

Stand-in is real credit risk. Because of the legacy debiters (C2), we cannot even estimate the balance conservatively. It exists so the business can choose to buy availability with risk, explicitly, with a kill switch. It must never become an accidental default.

10. Cutover: Phase 1 → Phase 2 (ADR-0005)

  1. The Wallet team ships v2. Contract tests against our stub and their sandbox pass, and H1–H5 are verified.
  2. hold-capture-percent: 0 → 1 → 10 → 50 → 100. Hold at each step until the decline rate, p99, orphaned holds and drift look like the DIRECT_DEBIT control cohort.
  3. Kill switch (runtime switch rollout.kill-switch=true, ADR-0012) sends every new payment back to DIRECT_DEBIT within 5 s on every instance. In-flight HOLD_CAPTURE payments still capture or release through their own workers.
  4. After 100 % plus a soak period, Phase 1 code stays available as the kill-switch target until the Wallet team deprecates v1 debits for this use.

11. Data model

See V1__local_ledger.sql. Two indexes carry the design:

  • PRIMARY KEY (scope, idem_key) on idempotency_key is the concurrency control for double-submit.
  • The partial index ix_payment_open (state IN open states) keeps reaper and reconciler queries proportional to outstanding work, not to history.

12. Observability (M2-T4)

Signal Why
safeledger.payments.decided{within_deadline} The SLI: decided within 3 s
Decline count by reason Separates an outage (WALLET_*) from customer behaviour (INSUFFICIENT_FUNDS)
Wallet RED + slow-call rate, per operation Shows "the wallet is slow" before it becomes "we are down"
Hikari and bulkhead in-use / max Leading indicator of the Little's Law failure
Breaker state and time open Alert on duration, not on the transition
payments.open{state} and oldest age Orphaned UNKNOWN / REVERSAL_PENDING / HELD
work_item depth, oldest age, DEAD count Queue bound and DLQ ownership
Reconciliation drift Never zero just because nobody counted
Stand-in exposure (current, cumulative) What the kill switch is armed against

13. Repository layout

api/                         OpenAPI contracts: the source of truth (contract-first)
  transaction-service/v1/    what we serve to tills
  wallet-service/v1/         what the wallet offers today (consumer view)
  wallet-service/v2/         what we propose (holds)
build-logic/                 Gradle convention plugins (java, spring-service, openapi-models)
gradle/libs.versions.toml    single version policy
libs/wallet-client/          generated wallet models + hand-written deadline-aware transport
services/transaction-service hexagonal: domain / application / adapter.{in,out} / config
stubs/wallet-wiremock/       wallet stub: same mappings for docker-compose and tests
deploy/compose/              local environment
docs/                        this site (MkDocs Material → GitHub Pages)