Breaking production on purpose

I ended the anatomy of a resilient system on what I thought was the safest claim in the piece: recovery is a capability, not a document, and you only really know you have it once you have used it on purpose. It is the kind of sentence everyone nods along to. Then everyone goes back to not unplugging anything.

There is an entire discipline built around taking that sentence literally. It carries the most lurid name in software engineering and the most conservative temperament: chaos engineering, the practice of injecting failure into a system deliberately, under controlled conditions, to find out whether your beliefs about it survive contact with evidence. The name suggests arson. The practice is closer to an insurance audit run by people who genuinely enjoy their work.

The framing I find most useful is older than any of the tooling. Most offices test the fire alarm at a fixed time each week. Nobody calls this chaos. Nobody stands in the car park afterwards demanding a post-incident review of why the alarm went off. The building's worst failure mode gets rehearsed until it is boring, which is precisely the ambition. Software rarely receives the same courtesy: we write the evacuation plan, laminate it, and wait for the actual fire.

Chaos engineering is the fire drill. That is the whole idea.

A monkey loose in the estate

The origin story is well worn but worth telling properly, because the details carry most of the lesson. In August 2008, a database corruption stopped Netflix shipping DVDs for three days. That incident, and the years of cloud migration that followed, forced a change of worldview: in a big enough estate, instances die constantly, so you can either treat each death as an emergency or you can treat it as weather. Netflix chose weather, then went one step further and hired the weather.

Chaos Monkey, unleashed on production around 2010, did one thing: it terminated instances at random. The detail everyone forgets is the schedule. It ran during business hours, on working days, deliberately. Engineers were at their desks, dashboards were being watched, and anything the monkey exposed got fixed by somebody holding a coffee rather than somebody woken at 3am. Failure was moved from the worst possible moment to the most convenient one, and then repeated until nobody found it interesting.

A whole Simian Army followed: Latency Monkey degrading network calls, Chaos Gorilla taking out availability zones, Chaos Kong dropping entire regions. The names are silly and the escalation is not. Each new monkey was the same bet at a larger blast radius: we believe the system survives this, and we would rather be wrong on a Tuesday afternoon than during the Christmas surge.

The monkey never worked weekends. Fear of failure, it turns out, responds well to office hours.

The least chaotic people in the building

The name is marketing. The discipline underneath would satisfy a lab supervisor, and the principles that formalised it read more like experimental method than mischief. Define the steady state: what does healthy actually look like, in numbers? State a hypothesis: we believe that if this cache node dies, checkout latency stays under 300ms. Pick the smallest blast radius that can teach you anything. Keep a hand on the abort switch. Run it, measure it, widen it only as confidence earns the widening.

Every clause in that paragraph is doing safety work. An experiment you cannot abort is not an experiment; it is an incident you booked in advance. A test with no hypothesis is not research; it is vandalism with a Grafana tab open. The practitioners who do this well are the least chaotic people in the building, which is roughly how you can tell they are doing it well.

If you want to see what that discipline looks like at full scale, this is the talk I would start with: Netflix's chaos team on how deliberate failure became a platform, with guardrails, sign-off and a paper trail.

Your system is already broken

Here is the uncomfortable idea underneath all of it: chaos experiments do not really inject failure. They reveal the failure already resident in the system. The retry policy that turns one slow dependency into a self-inflicted denial of service. The failover that has never once been exercised in anger. The timeout set to thirty seconds in 2019 for reasons no one can now reconstruct. All of that is already in production, waiting patiently for the right Tuesday.

The best articulation of why comes from Richard Cook, a physician who spent his career studying failure in complex systems and gave the safety world most of its good sentences. His argument, compressed brutally: complex systems always run in degraded mode, they are kept alive by humans compensating at the edges, and catastrophe needs several small latent failures to line up rather than one big one. Chaos engineering, seen through Cook's lens, is simply choosing to find the latent failures before they find each other.

It is the same conviction that drives the system design series: the interesting question is never whether the system works, it is which half of it is allowed to fail, and whether anyone chose that on purpose. Cook's talk is nineteen minutes long and has aged better than most of the infrastructure it was delivered on.

Start embarrassingly small

You are not Netflix, and the fastest way to discredit chaos engineering in your organisation is to cosplay as them. You do not need a platform, a vendor, or a monkey. You need an afternoon, a hypothesis, and a system that has observability worth the name.

Book a game day. Pick one belief the team states with confidence: if the primary database fails over, we lose no writes; if the payments provider times out, orders queue instead of failing. Then, in staging, with everyone watching and one person's hand on the abort switch, make the bad thing happen. Kill the process. Block the dependency at the firewall. Let the certificate expire on a copy of the system instead of waiting for it to expire on the real one, on a bank holiday, as certificates prefer.

I will make one prediction about your first game day: the failure you engineer will not be the interesting result. The interesting result will be at the observability layer, where something is now demonstrably broken while every dashboard remains a serene and confident green. Discovering that your monitoring cannot see the exact failure you just caused, on a calm afternoon with everyone in the room, is worth the entire exercise on its own.

Chess players do not discover how they handle lost positions in the tournament hall. They set the position up on a training evening and sit in the discomfort deliberately, because the skill being trained is not avoiding the position, it is playing on from it. A game day is the same move: losing on purpose, cheaply, while it teaches.

If the thought of running the experiment even in staging makes you nervous, notice the nervousness. That is a result, and it arrived before you ran anything.

Break the people too

The most interesting recent turn in this discipline is pointing it away from the servers. Systems survive because humans compensate, and the human layer has single points of failure that no architecture diagram will ever show you. Every team has a load-bearing person, and most teams find out who it is by accident, during their fortnight in Portugal.

So run the experiment. Declare an engineer unavailable for the day: they sit in the room, they may say nothing, and the team discovers in real time which knowledge lives only in their head. Add latency to communications and watch which decisions stall. Google has rehearsed disasters this way for years in its DiRT programme, up to and including scenarios where the people who normally take charge are scripted out of the building. The talk below makes the case that your organisation deserves the same rigour as your infrastructure, and it is funnier than it has any right to be.

Servers became cattle a decade ago, so the industry keeps saying. The person who knows why the settlement job must run at 4pm is still very much a pet.

The objection is the result

Raise any of this in the wrong meeting and you will hear the standard objection: we have quite enough chaos already, thank you. It is meant as a refusal. It is actually the most informative sentence in the room, because what it usually means is: I believe the experiment would fail. Which means the experiment has, in a sense, already run. The team is carrying a known critical weakness, unticketed and unprioritised, and has just declined to look at the print-out.

The honest guardrails do exist. Do not run chaos experiments against a system everyone already knows is broken; that is not research, that is cruelty, and the backlog is the correct instrument. Do not run them without observability, because you will learn nothing and still have to clean up. Respect the error budget: a team mid-incident does not need a rehearsal. Earn production the way Netflix did, one blast radius at a time, starting somewhere so small it feels faintly embarrassing.

But run them. The ambition here was never drama. The ambition is the fire alarm on a Tuesday morning that nobody looks up from their coffee for: failure rehearsed until it is boring, recovery exercised until it is muscle memory. You can believe your system recovers, or you can know.

The difference is an afternoon, a hypothesis, and the nerve to press the button.

Further reading

• Principles of Chaos Engineering: the manifesto. A handful of principles that took a decade of outages to earn, delivered with the calm of people who have already had theirs.
• The Netflix Simian Army: the 2011 post that introduced the monkeys to everyone else. Reads like a zookeeper's inventory compiled by an SRE.
• Chaos Engineering (Basiri et al., IEEE Software): the Netflix team writing it up properly for the peer-reviewed record, steady states, hypotheses and all.
• Gremlin's chaos engineering guide: the practical end of the literature: what to break first, and how to explain the idea to the people who sign things.