# Moving money is hard
> Most software engineers think a payment is a database transaction. The payments industry has spent decades proving otherwise.
By Matthew D. Webb · 2025-12-01 · 14 min read
Canonical: https://mdwebb.io/articles/moving-money-is-hard/
AI involvement: Updated with images and diagrams.
---
Most engineers think they understand payments. You click a button, money moves from one account to another, and the job is done. It is a clean little mental model, and it survives right up until the moment somebody actually has to build a payment system.
Then the questions start arriving. Why was the customer charged but the order failed? Why was the order created but the payment declined? Why does a refund take five days when the database write took four milliseconds? Why can a payment succeed and then fail later, and, more unsettlingly, why can a payment fail and then succeed later? Why does every payment provider on earth hand you an idempotency key and look at you meaningfully? And why does everyone who has worked in the industry seem to carry a faint air of someone who has seen things?
The answer is short. A payment is not a transaction. It is a distributed system wearing a transaction's clothing, and the gap between those two ideas is where every interesting bug lives.
## The great lie
Every ecommerce checkout sells you the same comforting story: customer taps a button, success appears, money has moved. It is clean and it is reassuring and it is mostly fiction. What actually sits behind the button is a relay of independent parties, each handing the request along: the merchant, then the gateway, the processor, the acquirer, the card scheme, and finally the issuer that holds the customer's money. Every one of those boxes is somebody else's company, with its own databases, its own networks, its own outages, its own retries, its own latency and its own creative failure modes.
The moment a payment leaves your application, you are participating in a distributed system whether you signed up for one or not.
That is the part the button is carefully not telling you.
## Buying trainers
Picture somebody buying a pair of trainers for a hundred pounds. They tap pay, and in their head the money goes straight from their bank account into the merchant's. What really happens is a small bureaucratic procession, and the cruel part is how spread out it is along the clock:
Those steps are not milliseconds apart. The authorisation happens in roughly a second, and that second is the entire transaction as far as the customer is concerned. Capture, settlement and the real transfer of money follow minutes, hours or days later, run by different institutions on different schedules. Which leads to the first genuinely counterintuitive thing about payments: a successful payment does not mean money has moved anywhere at all. It means everyone has agreed that it eventually should.
## Authorisation is not payment
This one surprises almost everybody. When you buy something online, the first thing that usually happens is an authorisation, where the bank essentially says: yes, this customer appears to have the money. Note the verb. The money is reserved, not transferred. The merchant has received precisely nothing. The customer still technically owns the funds; the bank has just earmarked them and promised not to let anyone else spend them.
Think of it as asking the library to hold a book behind the desk. Nobody else can take it. You still haven't checked it out, and you might never come back for it.
## Capture is not settlement
After authorisation comes capture, which is the merchant telling the network that yes, we really do want this money now, please begin the actual business of moving it. Only at that point does anything resembling a transfer start. And even then, the customer is looking at a cheerful "order successful" screen while the merchant may not see those funds land in their account for days.
Payments are full of these moments - everyone agrees something has happened, and the thing itself has not happened yet. The screen, the email and the bank are three clocks that are never quite in sync, and your code has to stay honest about which one it is actually reading.
## Distributed systems everywhere
My favourite thing about payments is how reliably [every problem decays into a distributed-systems problem](/articles/the-anatomy-of-a-resilient-system/) if you wait long enough. Consider a sequence that has happened to essentially every payment team alive, usually at the worst possible hour. The payment is approved. Your application receives the success response. Then, before the order can be written to your database, the write times out, so the order is never created. The customer, staring at a spinner that went nowhere, hits refresh.
Now what? Did they pay? Should you retry? Should you create another order? Should you charge them again? You are now holding two facts that disagree (the payment network thinks money is on the move, your database thinks nothing ever happened) and a customer who just wants their trainers. Welcome to payments. Every team learns this lesson exactly once, and never on a quiet afternoon.
The instinct is to fix this by trusting the synchronous response harder: wrap it in a transaction, retry it, log it more loudly. That instinct is the bug. The HTTP reply from a payment provider is a hint about what might have happened, not a record of what did. It can be lost on the wire after the charge already succeeded, or come back a cheerful `200` for a payment that fails downstream an hour later. The only thing that genuinely knows the state of a payment is the provider, and the only safe way to learn it is to ask in two directions at once.
So you consume the provider's webhooks (delivered at least once, which means your handler has to be idempotent or it will double-count), and you run a reconciliation job that periodically compares your ledger against theirs and flags anything that disagrees. Make the order creation idempotent on the payment intent rather than the click, and the refresh-the-page disaster stops being a disaster: the second attempt finds the first one's work and returns it, because you stopped treating a network round trip as the source of truth.
## The importance of idempotency
This is why payment systems are so quietly obsessive about idempotency. A customer taps pay, the network times out, nothing visibly happens, so they tap again. And again. And once more for luck. Without protection, a system will happily interpret four nervous taps as four separate hundred-pound charges, which is generally considered bad for the relationship.
An idempotency key is how a provider recognises a request it has seen before and returns the original result instead of doing the work twice. The interesting part is the assumption underneath it. Idempotency keys do not exist because duplicate requests are a rare edge case to be defended against.
They exist because duplicate requests are inevitable, and the system is built around expecting them.
Building it is less mystical than it sounds. The client generates one key per logical attempt (per checkout, not per click, so every retry of the same intent carries the same key), and the server stores that key next to the result of the first request it managed to complete. The enforcement lives in the database, not the application: a unique constraint on the key means two retries racing each other collide at `INSERT` rather than both sailing through to charge the card.
```ts
// the key names the attempt; the unique index does the actual enforcing
const existing = await db.idempotency.findByKey(key);
if (existing) return existing.response; // seen it: replay the stored result
const result = await provider.charge(request); // only ever runs once per key
await db.idempotency.insert({ key, fingerprint: hash(request), response: result });
return result;
```
It also pays to fingerprint the request body and reject a key that arrives attached to different parameters, because a key reused for a different charge is not a retry. It is a bug, and you would rather fail it loudly than quietly process it.
## The refund problem
Customers love refunds. Payment teams feel differently, and the reason is instructive. When a refund happens, everyone is in agreement almost immediately: the merchant wants to return the money, the processor is fine with it, the acquirer is fine with it, the scheme and the issuing bank are both fine with it. Unanimous consent, and it still takes three to five business days.
Why? Because the refund travels back through exactly the same ecosystem the original payment came through. It is not a tidy line from merchant to customer. It is merchant to acquirer to scheme to issuer to customer, and every one of those participants has its own processing windows, batch schedules, reconciliation jobs and operational rituals.
The money is not moving slowly because computers are slow. It is moving slowly because institutions are involved, and institutions move at the speed of institutions.
## Chargebacks: the reverse payment
Most software systems are built on the assumption that a completed transaction stays completed. Payments find this assumption adorable. Months after a purchase, a customer can dispute it, the issuing bank can side with them, and the money can simply move backwards. A done thing becomes an undone thing, long after everyone stopped thinking about it.
Imagine running an inventory system where stock you sold in March could spontaneously return itself to the shelf in July, without warning, and bill you for the inconvenience. That is roughly the emotional texture of a chargeback. A payment is not necessarily finished just because it succeeded; sometimes it has merely moved into a later, more litigious phase of its life.
## Model it as a state machine, not a boolean
If one design decision separates payment systems that age gracefully from the ones that page you at 3am, it is this: a payment is not a `paid` boolean, it is an explicit state machine, and every transition is a fact you persist rather than a value you infer.
The states are not subtle once you name them. A payment is initiated, then authorised, then captured, then settled, and along the way it can be voided, refunded or charged back. What gets people into trouble is collapsing all of that into a single flag and then reconstructing it on the fly from whatever happens to be in front of them: a redirect URL, a success page, the return value of one API call. None of those are the state. They are rumours about the state.
```ts
type PaymentState =
| "initiated" | "authorised" | "captured" | "settled"
| "voided" | "refunded" | "charged_back" | "failed";
// only these moves are legal; everything else is a bug or an attack
const allowed: Record = {
initiated: ["authorised", "failed"],
authorised: ["captured", "voided", "failed"],
captured: ["settled", "refunded"],
settled: ["refunded", "charged_back"],
voided: [], refunded: [], charged_back: [], failed: [],
};
```
Model it this way and a whole category of bugs simply cannot be expressed. A capture against a payment that was never authorised gets rejected because the transition is not in the table, not because someone remembered to write the right `if`. The state machine is the `if`, written down once and applied everywhere, which is exactly where you want that decision to live.
## Eventual consistency with real money
Engineers meet eventual consistency in databases all the time and make their peace with it. Payments offer you eventual consistency with actual money attached, which concentrates the mind. At any given instant one system can believe a payment is `paid`, another can believe it is `pending`, and a third can believe it is `settled`, and depending on timing all three can be correct at once.
The whole job becomes keeping several independent systems aligned despite the fact that they pointedly refuse to update at the same moment. If that sounds familiar, it should - it is the same problem distributed-systems engineers have been chewing on for decades. The only real difference is that when the inconsistency involves money, the finance team gets considerably more involved in the retrospective.
## A pound is not a float
A quick word on the least glamorous decision in the whole system, and one of the most consequential: never represent money as a floating-point number. `0.1 + 0.2` does not equal `0.3` in IEEE 754, and you do not want to meet that fact for the first time during a settlement run. Money is an integer count of the smallest unit the currency has, carried together with its currency code, all the way through the system.
```ts
// 1499 minor units, not 14.99; the currency rides along with the amount
type Money = { amount: number; currency: "GBP" | "USD" | "EUR" };
```
The currency is not decorative either. An amount without a currency is not a smaller bug than a wrong amount; it is the same bug, waiting patiently for someone to add two of them together. The first day you handle more than one currency, every sum, every comparison and every refund has to refuse to mix them, and the type is the cheapest place on earth to enforce that.
## Why reconciliation exists
Eventually somebody senior asks the obvious question: how do we actually know everything balances? The answer is reconciliation, and payment providers pour an astonishing amount of effort into it, endlessly comparing orders against payments against captures against settlements against refunds against chargebacks across systems that were never designed to agree in real time.
Not because something is definitely wrong, but because eventually something will be. Networks fail, messages arrive late, processes crash, people fat-finger a config. Reconciliation is the standing acknowledgement that reality refuses to cooperate cleanly, so you had better check the books rather than trust that they balanced themselves.
The thing that makes reconciliation tractable is an append-only ledger rather than a mutable balance. You never update a row to say an account is now worth forty pounds; you append an entry recording the movement and let the balance be a fold over the entries. In double-entry style every movement lands as paired entries that have to sum to zero, so a ledger that does not balance is shouting at you rather than quietly lying. The state of any account becomes a function of its full history, not a number somebody overwrote in place, which means you can replay it, audit it, and point at the exact entry where two systems started to disagree.
This is event sourcing by another name, and payments may be the domain where it stops being an architectural preference and becomes table stakes. When the auditors arrive, "we mutated a balance and trust that it was right" is not a sentence anyone enjoys finishing.
## The hidden complexity
The genuinely impressive thing is that customers see none of this, and absolutely should not have to. A good payment experience feels like nothing at all: tap the card, see the confirmation, walk out with the trainers. Behind that nothing sits one of the most complicated distributed systems most engineers will ever brush against, an entire stack of banks, processors, networks, fraud models, currencies, compliance regimes, settlement windows, reconciliation jobs, risk scoring, refunds and chargebacks.
All of it compressed behind a single button that says pay now, which is, when you sit with it, one of the more audacious abstractions in software: make the hard part invisible and let everyone assume it was easy.
## The real lesson
The lesson here is not really about payments. It is about engineering. A great many systems [look simple only because somebody worked extraordinarily hard to make the complexity disappear](/articles/building-a-trading-exchange-in-200-lines/), and payments are about the purest example of that going. The customer believes they are moving money. The engineer knows they are coordinating dozens of independent systems that have merely agreed, for now, that money ought to move.
Most of the time that coordination is flawless, and the button keeps its promise. Occasionally it doesn't, and that is the moment you finally see the thing for what it always was.
Not a transaction. A distributed system that has merely agreed to answer to the name, and like every distributed system, far weirder up close than it ever looked from the checkout page.
## Further reading
- [Stripe: idempotent requests](https://docs.stripe.com/api/idempotent_requests): the canonical writeup of why every serious payments API hands you a key and assumes your client will retry at the worst moment.
- [Designing Data-Intensive Applications, on transactions and consistency](https://dataintensive.net/): Kleppmann's chapters on weak isolation and eventual consistency are the best map of the territory payments forces you to walk anyway.
- [Adyen: the journey of a payment](https://www.adyen.com/knowledge-hub/guides/payments-101): a processor explaining authorise, capture and settle in its own words, which is useful precisely because it has skin in getting the model right.
- [The Stripe Press essay collection](https://press.stripe.com/): not strictly payments engineering, but a good reminder that the people moving the money also think unusually hard about why systems are the shape they are.
