# Caching is easy until it isn't
> Most caching strategies look brilliant during architecture reviews and become considerably less impressive the first time stale data reaches production.
By Matthew D. Webb · 2020-02-01 · 5 min read
Canonical: https://mdwebb.io/articles/caching-is-easy-until-it-isnt/
AI involvement: Updated with images, diagrams and the interactive demo.
---
Caching is one of the few ideas in software that can make a system dramatically faster and dramatically more complicated in the same afternoon. It is also one of the few where the bill for the first decision arrives months later, addressed to whoever happens to be on call.
Most caching conversations start the same optimistic way. A service is buckling under load, the database queries that used to be cheap have quietly become expensive, response times are creeping up, and someone in the review suggests adding a cache. They are almost always right. For a while.
The interesting problems never show up during implementation. They turn up later, when the requirements have moved, traffic looks nothing like it did at launch, and the team discovers that keeping a cache honest is a great deal harder than the textbook implied.
## The first cache is easy
The first cache nearly always works. You hold a little data in memory, database traffic drops, response times improve, and everyone is quietly pleased with themselves. The architecture diagram gains a new box, and the team moves on to the next fire.
It behaves because the assumptions underneath it are still simple. The data changes rarely, nobody is fussy about consistency, and the failure modes are the sort you can reason about over a coffee. Life is good.
Mechanically there is not much to it. A read checks the cache first, and only troubles the database when it has to:
This diagram is the whole story for exactly as long as the box marked "serve from cache" and the box marked "fetch from source" agree on the answer. Nothing in the picture guarantees they do. The diagram describes performance and says nothing about correctness, and correctness is where the bodies are buried.
## Then the business arrives
Software does not exist to flatter architecture diagrams. It exists to serve a business, and businesses change their minds. Requirements evolve, customers start expecting things to update the moment they happen, product introduces a workflow nobody modelled for, and operations would quite like to see what is actually going on.
Eventually somebody asks the awkward question - how fresh is this data, really? The instant that question is on the table, the conversation has changed register from performance to correctness. The cache has stopped being an optimisation you can take or leave. It is now part of the behaviour of the system, and it has been for some time; you simply had not been made to look at it.
## The myth of cache invalidation
There is an old line that there are only two hard problems in computer science: cache invalidation and naming things. It has survived this long because it is annoyingly accurate.
Reading from a cache is trivial. Knowing the precise moment a cached value stopped being true is the hard part, and every strategy that claims to have solved it is just choosing the way it would prefer to be wrong.
Time-based expiry looks simple, right up until a customer sees a price that changed five minutes ago. Event-driven invalidation looks elegant, right up until an event goes missing and one node spends the rest of the afternoon confidently serving a value nobody else believes in. Write-through looks robust, right up until you measure what it did to your write latency.
There is no option on the board that costs nothing. The job was never to dodge the trade-off; it is to know which trade-off you can afford for the system in front of you. It helps to stop arguing about that in the abstract and watch the two things you actually care about move in opposite directions. Here is the same read-through cache, with one dial for how long a value is allowed to live and another for how often the underlying data changes underneath it:
Push the time-to-live up and the hit ratio climbs, the database stops sweating, and the whole thing feels faster. Push it a little further and watch the stale reads climb right alongside it. The two move against each other, and no amount of cleverness makes them stop. Every caching strategy ever shipped is just a particular spot on that dial, given nicer vocabulary.
## Consistency is a business decision
The most useful thing I picked up working on large systems is that the consistency requirement is almost never a technical decision at all. It is a business decision that engineers have been left to make by accident.
A product catalogue can usually carry a few minutes of staleness without anyone noticing or caring. [A trading screen cannot](/articles/latency-budgets-trading-uis/); I spent long enough near front-office systems to know that "a few seconds behind" is just a politer phrase for "wrong". A customer profile will happily tolerate eventual consistency. [A payment authorisation almost certainly will not](/articles/moving-money-is-hard/), and the post-incident review will not accept "but the cache was faster" as a defence.
None of these are really questions about Redis or eviction policies or how clever your invalidation is. They are questions about what the business can afford to be wrong about, and for how long. Place that correctly and the technology mostly chooses itself. Place it wrong and no technology will save you.
The point of the line is that nothing sits at either end by default. You have to place each system deliberately, before the cache places it for you and picks whichever spot happened to be easiest to build.
## At three in the morning
Most caching discussions are really about speed, because speed is the part that demos well. The more interesting conversation, and the one that tends to get skipped, is about correctness. Performance problems announce themselves - you can watch a page crawl. Consistency problems sit quietly until a customer trips over one, usually in front of someone important.
The best caching strategies I have worked with were rarely the cleverest. They were the ones whose behaviour you could still hold in your head at three in the morning, half awake, during an incident, when the graphs are red and nobody can quite remember which layer is allowed to lie. That is not a bad test for a cache. It is not a bad test for most of the things we build.
## Further reading
- [Martin Fowler: Two Hard Things](https://martinfowler.com/bliki/TwoHardThings.html): the actual provenance of the cache-invalidation line, tracked back to Phil Karlton, plus the better jokes it spawned.
- [AWS: Caching challenges and strategies](https://aws.amazon.com/builders-library/caching-challenges-and-strategies/): Amazon's own account of where caches bite at scale, from cache coherence to the thundering herd.
