Two million games, one running clock

Most write-ups of this system start with the matchmaker and finish with the leaderboard, because that is the order a product manager meets the features. I want to start somewhere else: with the observation that online chess is the purest consistency problem most engineers will ever be handed. Two people, one board, one truth, and a clock that punishes every millisecond of hesitation. I have given a year of my life to the playing side of this system, which makes it the one specimen in this series where I am both the architect and the complaint department. Every decision below is something I have personally been on the wrong end of at one in the morning with four seconds left.

The first plate took the URL shortener, a system whose whole personality is stateless reads, and asked which half of it was allowed to die. This is the opposite pole. Here the state is alive, it is shared between two players who do not trust each other, and it changes under a running clock. The method does not change. The answers do.

The parts everyone agrees on

Three requirements, briskly. Players are paired with an opponent of similar strength on the time control they asked for. They play a real-time game in which the server, never the client, owns the rules, the turn, and both clocks. And when a game ends, a rating moves and a leaderboard notices. Spectating, chat, puzzles and tournaments all exist in the real product and all stay below the line; they hang off this core rather than defining it.

The scale is worth pinning down first, because it drives everything and it is smaller than it looks.

Four million concurrent connections is a genuine fleet problem: a few hundred servers, each holding ten or twenty thousand sockets while validating moves and running clocks. But the state itself is tiny, and that asymmetry is the design. The difficulty here was never volume. It is that every one of those 800 megabytes is precious while its game runs, and worthless the moment it ends.

Matchmaking is a pending pool per time control, keyed by rating: a Redis sorted set answers "who is waiting near 1730" in one range query. The claim must be atomic, because two matchers will spot the same waiting player in the same instant, and a player double-booked into two games is a corrupted evening. The request itself is a long-poll held open until an opponent appears or a timer widens the acceptable band; players at the rating extremes wait longer for a fair game or accept a worse one, and that trade gets tuned from real wait-time data rather than taste. Identity comes from the session token and rating from the player record. Anything a client could lie about to get an easier opponent is not the client's to send.

The game itself rides a WebSocket scoped to the game id: the client sends squares and nothing else, the server answers with verdicts and both authoritative clock times, and the opponent hears the move the instant it is accepted. The leaderboard, finally, splits into an easy read and a nasty one. The top hundred is a btree walking a hundred index entries and stopping, cheap at any row count. "Where am I out of eighty million" is a count of everyone above you, O(rank) precisely for the mid-pack players who ask it most, so a sorted set keyed on rating answers rank in logarithmic time instead. We will earn the right to trust that sorted set in the funerals section.

That is the consensus design, dispatched. The diagram already shows the decision that matters: there are two halves behind that router, and only one of them holds anything that cannot be regenerated.

Find the request that must survive

Same opening question as every entry in this series: which single request is this system for? Rank the failures and it answers itself. A match request that dies costs a player five seconds and a re-queue. A leaderboard that lags costs nothing anyone can point at. But a move that is lost, applied twice, or shown to one player and not the other is a different species of failure. Two people are now looking at different boards, each certain they are right, in a game they care about winning. There is no apology page for that. The game is simply ruined, and so, more quietly, is the player's trust in every game after it.

So the move, mid-game, is the request that must survive, and notice the inversion from the shortener. There the product was a read, and the writes could have a bad day without anyone much minding. Here the product is a write: every move is a state change that two mutually suspicious parties must agree happened, in order, under time pressure. The consistency dial turns all the way up accordingly. When a game server cannot be sure of its state, the correct behaviour is to stop: freeze both clocks, hold both players, recover. A paused game resumes. A diverged game is a small betrayal you cannot take back. An availability purist will tell you the pause is downtime; in this system the pause is the feature.

That one conviction prices everything else. The move path gets the four nines and a 150ms end-to-end budget, because in bullet a slow move is indistinguishable from a broken one. Matchmaking gets three nines and an honest error, because "try again" has been an acceptable matchmaking experience since the dawn of lobbies. The leaderboard gets to be eventually right, with the emphasis on right. Three components, three deliberately unequal contracts, chosen rather than inherited.

The live game is the product. Everything else is staff.

Put the state where the clock is

Now the decision that shapes the whole live half: a game's board and clocks live in memory on a game server, and both players' sockets land on that box. A chess game is a few hundred bytes that lives for minutes. Offloading it to a shared store buys resilience you can get cheaper another way, and spends the latency budget on network hops to fetch a board you could simply have kept. Validating against local memory costs microseconds. Routing is a consistent hash on the game id over a small membership registry, so both seats resolve to the same server and a membership change moves only the games it must. Frameworks exist that will own this placement for you, and lichess famously runs its boards on one; at this size the hand-rolled router is genuinely fine, and the recovery story below is identical either way.

The load-bearing sentence in this section is about ordering. The server appends every accepted move to a durable log before either player is told. Not for audit, and not for the replay feature, though both come free: the move log is the recovery mechanism. If the acknowledgement or the opponent's push ever went out ahead of the append, a crash in that gap would recover a board missing a move two humans already saw, which is exactly the divergence we just swore to pause rather than permit. The synchronous write costs a few milliseconds inside a 150ms budget. Correctness at that price is not a trade-off; it is a rounding error.

// The order is the contract: nothing is broadcast that is not yet durable.
// A replayed log must never be missing a move somebody already saw.
async function onMove(game: Game, seat: Seat, mv: Move) {
  if (!legal(game.board, seat, mv)) {
    return send(seat, reject("not that piece, not that square, not your turn"));
  }

  const stamped = stampClocks(game, seat, mv);   // mover's clock stops here
  await moveLog.append(game.id, game.generation, stamped); // durable, fenced
  game.board = apply(game.board, stamped);       // now the live truth advances

  send(seat, ack(stamped, game.clocks));         // and only now does anyone hear
  send(other(seat), opponentMove(stamped, game.clocks));

  const end = verdict(game.board, game.clocks);  // mate, stalemate, or a flag
  if (end) await finish(game, end);              // the result is one more append
}

A chess board is a fold over its moves. Keep the moves safe and the board is a matter of arithmetic.

Design the funerals before the features

Here is the same system again, for reference while the funerals proceed. Every component gets one on paper before production schedules a real one.

The game server dies first, and this is the funeral the entire design was arranged around. A box holding five thousand live games disappears, taking five thousand boards and ten thousand sockets with it. What the design does next is almost anticlimactic. Every affected game is already frozen at its last durable move, because nothing was ever acknowledged that had not been appended. Both clients notice the dead socket and reconnect through the router, which now hashes their game to a healthy server. The successor bumps the game's generation with one compare-and-set in the membership registry, replays the move log, and reopens for business with both clocks exactly where they paused. Nobody flags during a funeral; the seconds the outage ate belong to the incident, not to either player.

The generation number is the part people skip, and it is the part that keeps the pause honest. The dead server may not be dead: a long GC pause or a network partition produces a machine that still believes it owns the game and will happily keep accepting moves. So the log itself enforces the fence. Every append carries the server's generation, and the log refuses anything stale. The zombie can shout into a closed door until its health checks catch up with reality. The trap at this point in the design is reaching for checkpoints or a snapshot table; a chess game is far too short to need either, and knowing when the fancy pattern is unnecessary is most of the job.

// Recovery is a replay plus a fence. The successor takes the game by
// bumping its generation once; the log then refuses appends from the past.
async function adopt(gameId: GameId): Promise<Game> {
  const gen = await registry.bumpGeneration(gameId); // one CAS, strongly consistent
  const moves = await moveLog.read(gameId);          // a few hundred bytes
  const game = replay(moves);                        // the board is a fold
  game.generation = gen;
  game.clocks = clocksAsOf(moves.at(-1));            // paused where the crash found them
  return game;
}

// Inside the log. The zombie server still thinks it owns the game;
// this is where it finds out otherwise.
async function append(gameId: GameId, gen: number, mv: StampedMove) {
  const current = await registry.generation(gameId);
  if (gen < current) throw new Fenced(); // declined, with prejudice
  await appendRow(gameId, gen, mv);
}

The matchmaking pool dies second, and the correct amount of drama is none. A pending match request is a wish, not state: lose the whole pool and every waiting client gets a fast, honest failure, re-submits, and refills the queue within seconds. Meanwhile the entire live half plays on, because no game server has ever had a reason to speak to the matchmaking Redis. The instructive contrast is that this is the same technology hosting two opposite contracts: the pool is Redis we can afford to lose, the sorted-set leaderboard is Redis we can rebuild, and the move log is the one store we treat as precious. What you protect is decided by what it costs to lose, never by what it cost to buy.

The leaderboard dies last and never really dies. A rating is a derived total over finished games; each game snapshots the ratings it started from, so its delta is self-contained, and the result's append to the log is the single commit point. From there an apply step fans the change out to the player record and the sorted set, keyed on the game id, so a crash mid-update is corrected by replay rather than double-counted. If drift creeps in anyway, a periodic reconciliation recomputes from the record and overwrites both views, and the worst case is rebuilding the sorted set from scratch during a quiet hour. A view can be late. It cannot stay wrong, because the truth it derives from never moved.

Add it up and the degradation order reads like a policy document, which is exactly what it is: matchmaking fails first and loudest, the leaderboard fails invisibly and heals itself, and the live game fails last, briefly, and as a pause.

Charge for thinking, not for distance

The server owning both clocks is non-negotiable; a clock a client can influence is not a clock, it is a suggestion. But the honest version of that design has a quiet unfairness baked in: the server can only stop your clock when your move arrives, so your network sits inside your thinking time. Two players, identical skill, different continents, and one of them is paying rent on every move.

Eleven seconds decides real blitz games; I have lost enough of them from hotel wifi to hold this conviction personally. The fix is lag compensation: measure each connection's latency continuously with socket-level pings, and credit the one-way transit back to the mover's clock on each move, capped. Both halves of that sentence are load-bearing. Measured, because a client asked to report its own lag will discover lag at the most convenient moments; this is the same rule that kept ratings out of the match request. And capped, because uncapped credit is free thinking time, and somewhere a player with a lag switch is already reaching for it. I spent years around trading systems where latency decided who got the price, and the discipline transfers whole: fairness is a budget you measure and allocate, not a property you hope for.

I put this section among the funerals deliberately, because unfairness is a failure mode. It just fails silently: no pager goes off when your platform is quietly running a rigged market for everyone far from the data centre. It surfaces months later, in the churn numbers, filed under product problems by people who never saw the eleven seconds.

Same discipline, newer weather

None of the physics changes because models arrived. Moves still validate in memory, the log still commits before anyone hears, and the pause is still the worst day. What changes is who sits at the board, and what lives in the walls.

Start with the traffic. A real share of players on a modern chess platform are not people: engine bots, research agents, someone's weekend LLM experiment that plays charming, illegal chess until the server corrects it. The right posture is lichess's, not a fortress: bots are first-class citizens with their own credentials, their own matchmaking pool, and opponents who opted in. What needs designing for is their temperament. Agents arrive in bursts, retry with terrible enthusiasm, and will resend a move whose acknowledgement a timeout ate. Chess hands us the cleanest idempotency story imaginable: a move is fully addressed by game, seat and move number, so a replay collects the ack it already earned instead of becoming an illegal-move argument. Rate limits key on the credential rather than the IP, and a 429 carries a Retry-After that well-built agents genuinely honour. Backpressure becomes a conversation, and the human in the other seat never feels the weather.

// A move is idempotent by address: (game, seat, move number) names it
// exactly once. Retries are answered, not re-applied; that one property
// makes agent traffic boring, which is the highest compliment.
async function submit(gameId: GameId, seat: Seat, mv: Move) {
  const game = await lookup(gameId);
  const seen = game.history[mv.moveNumber];
  if (seen && seen.seat === seat) {
    return sameMove(seen, mv)
      ? ack(seen, game.clocks)  // the retry collects its earlier answer
      : reject(seat, "that move number is already spoken for");
  }
  return onMove(game, seat, mv);
}

Then the walls. Engine assistance is the existential threat to this product, and it is nearly invisible in any single game; I have written about how little machinery it takes to outplay every human who has ever lived, and that machinery now runs in a phone browser. Detection is a genuinely hard ML problem: how often a player's moves match an engine's first choice, how their thinking time distributes across easy and critical positions, how their accuracy sits against their rating history. All of it is statistics over finished games, which hands us the architecture for free, and I hold two convictions about it absolutely. The model stays off the synchronous path: a model endpoint is the least available dependency you will ever own, and the move path waits for nothing that is not the move. And the model never holds the ban button. It flags, with evidence attached, onto a human desk, because a false cheating ban is the one failure this product cannot recover from. An outage costs you an evening of goodwill. A wrong ban costs you the player, publicly, forever, and deserves the same fear we reserved for diverged boards.

The same boundary governs the operations loop. A fleet of stateful servers throws off exactly the telemetry models read well: the reconnect spike that is actually one rack, the clock drift that correlates with one deploy, the draft timeline assembling itself while the humans debug. Take all of that help. The generation fence, the pause, the failover: those stay decisions a person makes, for the same reason the fair-play model does not ban. Capability was never the constraint. Accountability is.

What travels to the next system

Strip the chess away and the method is portable, which is the point of the series. Name the request that must survive, and accept the unpopular corollary that the others matter less. When two parties share one truth, choose the pause over the divergence, and say so out loud in the design review. Notice which log you are already writing, because it is probably your recovery plan; fence the zombie that will one day try to write behind it. Price fairness like latency, as a measured budget, because it fails silently and compounds. And let the models read everything while deciding nothing irreversible: off the hot path, off the pager, and a long way from the ban button.

The shortener asked us to decide which half of a stateless system was allowed to die. Chess asked what to do when the state is alive and two people are watching it. The next plates head somewhere less comfortable: systems where the precious state is too big for a replay, where the funerals overlap, and where the clock belongs to a regulator instead of a player. Every system ends up in time trouble eventually. The design decides how it plays when it gets there.

Further reading

• Hello Interview: Design Online Chess: the interview-facing treatment, done properly. I used it to sanity-check the arithmetic and the baseline, then spent this article on the convictions it has no room for.
• Lichess source code: the reference implementation, run in the open by roughly two people and a fundraiser, which is its own lesson in what this design actually requires.
• How to do distributed locking (Martin Kleppmann): the canonical argument for why the generation fence is not optional, told through a lock that lied.
• Ken Regan's fair-play research: decades of statistical cheat detection in chess, from long before it was rebranded as an ML problem.