A payment system that refuses to guess

The first plate in this series took the easiest system on the whiteboard and asked one question of it: which request must survive? The URL shortener's answer was a read, and the whole design fell out of protecting it. This time the specimen is a payment processor, the Stripe-shaped system that lets a merchant charge a customer's card without building card infrastructure themselves, and the question mutates in a way I find genuinely instructive.

Because here is the thing the interview write-ups bury: payments are not hard for the reasons candidates prepare for. Ten thousand transactions a second is not a scaling problem worth the name; the shortener shrugged off more. What makes a payment system hard is that roughly half of it lives in other people's buildings. Every charge is a border crossing into networks you do not operate, cannot observe, and may not hear back from, and the answers arrive on a schedule set by institutions that still think in batch files. The dominant failure mode is not a dead server. It is a silence you cannot interpret.

So this article is about designing for doubt. The resilience question from last time is still here, and so are the funerals. But the recurring test I will apply to every choice is different: when this component fails, what does the system now believe about money, and can it defend that belief to an auditor? A payment system that is down is having a bad day. A payment system that is guessing is committing a small fraud against someone, and it does not yet know whom.

The parts everyone agrees on

The consensus baseline, dispatched briskly. Merchants call us to charge customers; customers pay by card; merchants can ask how it went. The unit of truth is the payment intent: the merchant's declared wish to collect a specific amount, created before anything touches a card, carrying the state machine from created through processing to succeeded or failed. Charges hang off the intent as attempts, several if the first ones misfire. I have written before about why a payment is a state machine and not a boolean, and all of that applies here; this article takes the lifecycle as read and designs the building around it.

The API is three verbs: create an intent, attach a card to it, poll its status. The card arrives as a token minted in an iframe we host on the merchant's checkout, so the raw number never lands on their servers and their PCI scope stays the size of a postage stamp. Webhooks push status changes to merchants who would rather not poll; we will get to why webhooks are a courtesy and not a source of truth.

The numbers confirm the suspicion: this is not a big-data problem either. Ten thousand writes a second lands within reach of one well-run Postgres with sharding by merchant held in reserve, and the growth story is an archiving policy, not an architecture. The interesting property is elsewhere. Reads of payment status are frequent and forgiving; writes that record what happened to money are sacred. Hold that asymmetry; it is about to do all the work.

-- The intent owns the lifecycle; attempts record each conversation with the
-- network. Note what is missing: an UPDATE that sets status directly. State
-- moves through transition(), or it does not move.
create table payment_intents (
  id               text primary key,
  merchant_id      text not null,
  amount_minor     bigint not null,      -- pennies. money is integers.
  currency         char(3) not null,
  status           text not null default 'created',
  idempotency_key  text not null,
  created_at       timestamptz not null default now(),
  unique (merchant_id, idempotency_key)
);

create table attempts (
  id          text primary key,
  intent_id   text not null references payment_intents(id),
  status      text not null default 'pending',
  network_ref text,                      -- their name for our question
  created_at  timestamptz not null default now()
);

// The only door into a status change. An illegal transition is a bug
// surfacing, and a bug surfacing loudly is a feature.
const LEGAL: Record<string, string[]> = {
  created:    ["processing", "canceled"],
  processing: ["succeeded", "failed", "processing"],
  succeeded:  ["refunded", "disputed"],
  failed:     ["processing"],            // a retry is allowed to try again
};

export async function transition(intentId: string, to: string): Promise<void> {
  await db.tx(async (t) => {
    const intent = await t.intents.lock(intentId); // row lock; no races
    if (!LEGAL[intent.status]?.includes(to)) {
      throw new IllegalTransition(intent.status, to); // refuse, loudly
    }
    await t.intents.setStatus(intentId, to);
    await t.events.append(intentId, intent.status, to); // the paper trail
  });
}

Find the request that must survive

Last time the answer was the redirect: the read lane was the product and everything else was staff. Payments invert it. The reads here are status checks and dashboards, and every one of them can be seconds stale or briefly unavailable without anyone losing a penny. The request that must survive is a write: the one that records what we are about to do with money, and the one that records what actually happened to it. The ledger is the product. The payment is a side effect the outside world insists on.

That inversion flips the failure posture too, and this is the decision I most want on the whiteboard before any boxes get drawn. The shortener failed open: serve the stale mapping, keep the redirect moving, apologise to nobody. Money fails closed. If the system cannot durably record an attempt, it does not make the attempt; a charge we cannot write down is a charge we refuse to send. Failing open with reads costs you freshness. Failing open with writes about money costs you the one thing a payment company sells, which is being believed.

So the availability budget from last time gets spent differently. The durable write path gets the four nines and the engineering attention; the status endpoints get three nines, a replica, and an honest staleness label. Notice this is the same unequal-contract move as No. 01 with the lanes swapped, which is rather the point of a method: the question travels, the answer belongs to the system.

The record precedes the action. Everything else in this design is that sentence, applied.

Design the funerals before the features

Same drill as last time: every component gets its funeral rehearsed on paper before production schedules a real one. But payments add a twist to the exercise, because the component most likely to ruin your evening is one you cannot restart, patch, or even properly observe.

The network goes quiet first, and this is the funeral that defines the system, because nothing has actually died. An authorisation goes out and no answer comes back. The request might be queued behind someone's batch window; it might have been rejected by a system that saw no reason to say so; it might have succeeded, with the approval lost somewhere on the way home. You cannot tell, and no amount of infrastructure spend on your side of the border changes that. I play enough chess to recognise the shape: the positions that ruin you are rarely the ones where you know you are losing; they are the ones you cannot evaluate and must move anyway.

The system's move is the one guardrail I hold absolute: it is not allowed to guess. A timeout resolves to a third state, unknown, written down as honestly as success or failure would be. The attempt stays pending, the merchant is told the truth, and the retry, when it goes, carries the same idempotency key as the original so the network can recognise a question it has already been asked. The discipline travels well beyond payments, but this is the system it was invented for: a retry must be a question repeated, never a wish made twice.

// The mechanism the quiet-network funeral leans on. The unique constraint
// on (merchant, key) is the actual guardrail; this function is just good
// manners around it. Note the order: the attempt row exists before any
// packet leaves the building.
export async function charge(req: ChargeRequest): Promise<Attempt> {
  const existing = await db.attempts.byKey(req.merchantId, req.idempotencyKey);
  if (existing) return existing;         // a replay, not a new wish

  const attempt = await db.attempts.create({
    intentId: req.intentId,
    key: req.idempotencyKey,
    status: "pending",                   // the record precedes the action
  });

  try {
    const answer = await network.authorise(attempt, { timeoutMs: 3000 });
    await transition(req.intentId, answer.approved ? "succeeded" : "failed");
    return db.attempts.resolve(attempt.id, answer);
  } catch (err) {
    if (isTimeout(err)) return attempt;  // still pending. unknown is a state,
  }                                      // not an error. reconciliation will
  throw err;                             // collect what the network owes us.
}

The store dies second, and here the fail-closed posture earns its keep. New charges are declined at the door, fast and honestly, because the alternative is accepting money-shaped promises we cannot write down. Status reads degrade to a replica with a staleness label. The subtle part is the traffic that does not stop: the card network keeps answering about payments already in flight, on its own schedule, whether our database is present or not. Those callbacks queue durably at the edge and drain in order once a replica is promoted. The outside world does not pause for your incident, so the design's job is to make sure nothing it says during one goes unheard. The failover itself should be boring and rehearsed; that sentence survives from No. 01 unedited, because it is true of every system and practised in almost none.

The retry storm comes third, and it is barely a funeral because the earlier decisions already attended it. A merchant's checkout hits a timeout and retries with enthusiasm; multiply by every integration that copied the same three lines of retry logic. Idempotency keys collapse the storm back into single questions, and rate limits keyed by credential rather than IP keep one loud merchant from spending everyone else's capacity. The storm arrives, the ledger records one attempt, and the graphs barely notice.

Three funerals, one pattern: in each case the system's first duty was not to stay up. It was to stay honest, and to be certain afterwards about what it had and had not done.

The paper trail is load-bearing

In most systems the audit trail is exhaust: something you keep because compliance asked. In a payment system it is a structural member, because every capability that distinguishes a real processor from a demo turns out to be a consumer of the same stream of facts. So the design commits properly. Every state change lands in the ledger as an append, never an edit, double-entry style; change data capture lifts those changes into an event stream, keyed by intent so each payment's history replays in order; and everything downstream, webhooks, reconciliation, analytics, fraud review, reads the same stream rather than maintaining private versions of the truth.

Two consumers deserve their sentence. Webhooks are the courtesy copy: signed, retried with backoff, monitored, and still never the system of record, because a merchant's endpoint being down for an hour must never be able to change what happened to money. And reconciliation is the capability I would defend with the most budget, because it is the mechanism that pays off the quiet-network funeral. Every night the network sends its version of the day; every night we replay ours and compare, penny by penny. The pendings resolve, the lost responses surface, and the disagreements become correcting entries, new events on top of the old, never erasures. An audit trail with erasures is not one.

Reconciliation is also where the humans enter the design on purpose, rather than as a fallback. Small corrections apply automatically. Anything above a threshold waits for a person to look at it and sign it, and the threshold is a product decision made in daylight, not an incident-time improvisation. The system proposes; the human disposes. Designing that seam deliberately, with queues, deadlines, and an interface a tired person can use at 3am, is as much a part of the architecture as any box on the diagram.

// The nightly audit that assumes we are wrong. Matches resolve pendings;
// mismatches become correcting events with a human gate above a threshold.
// Nothing in this file has UPDATE privileges on history, by design.
export async function reconcile(window: Day, files: SettlementFile[]) {
  const theirs = index(files, (r) => r.networkRef);
  const ours = await ledger.attemptsIn(window);

  for (const attempt of ours) {
    const theirRow = theirs.get(attempt.networkRef);
    const verdict = compare(attempt, theirRow); // amount, status, currency

    if (verdict.agrees) continue;
    if (verdict.resolvesPending) {
      await transition(attempt.intentId, theirRow.settled ? "succeeded" : "failed");
      continue;                          // tuesday's silence, answered
    }

    const fix = correctionFor(attempt, theirRow);
    if (fix.amountMinor <= AUTO_APPLY_LIMIT) {
      await ledger.appendCorrection(fix); // small, boring, automatic
    } else {
      await reviewQueue.enqueue(fix);     // a person signs this one
    }
  }
}

Same discipline, newer weather

As with the shortener, none of the physics changes because a model showed up; what changes is the temperament of the traffic and the toolkit of the operators. But payments feel it sooner and harder, because agents do not browse: they buy.

Start with the traffic. A checkout driven by an agent retries faster than any human, abandons nothing, and will replay a charge it already made because a timeout ate the receipt. Everything above was built for exactly this client: idempotency keys are mandatory rather than encouraged, rate limits ride on credentials, and a 429 carries a Retry-After header that well-built agents honour to the second. There is a quieter design consequence too. An agent spending someone's money on their behalf wants scoped credentials: this merchant, this ceiling, this week, revocable in one call. The systems that treat delegation as a first-class object, rather than an API key in a prompt, are the ones that will take this traffic safely.

Fraud is where the last plate's absolute rule, the model stays off the synchronous path, meets its hardest test, because fraud scoring genuinely wants to happen before authorisation. The honest resolution is a budget, not an exception. A small, fast, boring classifier sits on the path with a hard latency ceiling and a deterministic answer ready for the day it exceeds it, and the decision of which way to fail, open for small amounts, closed for large ones, is made in a design review rather than discovered in an outage. The large models stay asynchronous, reviewing the borderline cases after the fact and distilling what they learn into the small model's next version. Capability on the path, intelligence behind it.

And then the operators, where I will simply confess the No. 02 version of the rule from No. 01. Models are superb at the reading half of running this system: watching pending-age curves nobody graphed, correlating a settlement anomaly with Tuesday's config deploy, drafting the incident timeline while the humans think. The paper trail we just built is, not coincidentally, the best possible substrate for that help: a system that writes everything down is a system a model can actually be useful about. But the ops model's grant is read-only and its edge in the diagram points one way, in. It does not retry charges, it does not touch the ledger, it does not approve corrections, and it never holds the pager. The operating model around the AI matters more than the model: automate the reading, keep the judgement, and be suspicious of any design where the second half is implicit.

// What an agent is allowed to hold: not our API key, a delegation. Scoped,
// ceilinged, expiring, revocable. The agent proves it can spend; it never
// learns to mint.
export interface SpendGrant {
  merchantId: string;
  ceilingMinor: bigint;      // per charge and per window, both enforced
  windowSpentMinor: bigint;
  expiresAt: Date;
  revoked: boolean;
}

export function admit(req: ChargeRequest, grant: SpendGrant): Admission {
  if (grant.revoked || grant.expiresAt < now()) return deny("expired");
  if (!req.idempotencyKey) return deny("agents retry; keys are mandatory");
  if (req.amountMinor + grant.windowSpentMinor > grant.ceilingMinor) {
    return deny("over ceiling", { retryAfter: grant.windowReset });
  }
  return allow();
}

// And the ops model's entire privilege surface. There is no second function.
export const OPS_MODEL_GRANTS = ["events:read", "metrics:read"] as const;

What travels to the next system

Strip away the payments and the method gains a second clause. Find the request that must survive, still, but then ask which way each lane fails when it cannot serve: open, with staleness, or closed, with an honest refusal, and choose per lane rather than per system. Write the record before you take the action. Treat unknown as a state to be stored, never a gap to be papered over with optimism in either direction. Make the audit trail a structural member and reconciliation a rehearsed capability, because two honest systems will still remember the same day differently. Put the human seam in the design on purpose, with a threshold chosen in daylight. And grant your models the reading, all of it, while keeping the writes, the money, and the pager on the other side of a line the topology itself enforces.

The shortener taught the method on a system where the worst outcome was a broken link. This one raised the stakes to money and the answer to every hard question turned out to be the same small sentence: when you do not know, say so, durably.

The next plates take the method somewhere less forgiving again: systems where the writes are the product and the volume is real, where backpressure stops being a politeness and starts being the design.

Further reading

• Hello Interview: Design a Payment System: the interview-facing treatment, done properly. I used it to sanity-check the baseline and the arithmetic, then spent this article on the convictions it files under deep dives.
• Pat Helland, "Memories, Guesses and Apologies": the 2007 paper that says the quiet part about distributed truth out loud; every reconciliation job is an apology with a schema.
• Stripe: Designing robust and predictable APIs with idempotency: the idempotency key discipline from the people who process the retries, including the ones their own docs caused.
• Amazon Builders' Library: Making retries safe with idempotent APIs: the same lesson from a different building, which is roughly how you know a lesson is real.