The traceability problem

Most engineering teams believe they have a traceability strategy. The belief tends to survive right up until the morning something goes wrong.

A production outage. A security incident. A deployment that failed quietly three hours before anyone noticed. A regulator with a list of questions. A customer who has stopped being polite. Whatever the trigger, the organisation abruptly loses interest in technical questions and starts asking a different kind. What changed. Who changed it. When, and why, and whether anybody looked at it before it went out.

This is the moment most teams discover something uncomfortable about themselves. They are not short of data. They are drowning in it. What they lack is a coherent story.

Every incident becomes an investigation

The first hour of a production incident is rarely spent fixing the incident. It is spent trying to work out what the incident is.

An engineer notices errors climbing. A dashboard turns an unpleasant colour. The first message in the channel is almost never a hypothesis about the fault. It is a question, and it is nearly always the same question: what changed? It sounds like it should be easy to answer, and in a small system it is. In a real one it is the start of an afternoon.

A modern service is not a service. It is a collection of repositories, pipelines, infrastructure, configuration and third-party dependencies owned by different teams and changing at different speeds. Somewhere in that system, something changed. Traceability is the process of turning that observation into an answer.

The problem with "nothing changed"

Every engineer who has worked an incident has heard the phrase, usually said with great confidence early on.

Nothing changed.

It is almost always wrong. Software does not, as a rule, decay while you sleep and break itself at nine in the morning for no reason. Something changed. A deployment went out. A feature flag flipped. A certificate that had been quietly counting down finally hit zero. A dependency updated itself two minor versions while nobody was reading the changelog. A cloud provider had a bad day in a region you forgot you depended on.

The system changed. "Nothing changed" almost never means nothing changed. It means the organisation cannot see what did, and has mistaken its own blindness for stability. Traceability is the unglamorous machinery that turns a suspicion into a fact you can point at.

Git solves less than people think

Ask an engineer where the traceability lives and a fair number will say Git, with the slight impatience of someone explaining the obvious. Git helps, genuinely. It tells you who committed what, and when, and it does so reliably enough that we have stopped being impressed by it.

That is also roughly where it stops. Git knows the commit. It does not know who approved the change, or when it was deployed, or which environment actually received it, or which ticket was supposed to justify it in the first place. It has nothing to say about the feature flag someone toggled in a console, or the configuration value edited in a dashboard, or the access granted for "just five minutes" last Tuesday. Most of what moves in a production system never passes through a repository at all. Treating source history as the whole record is how you end up confidently reconstructing half the story and guessing the rest.

The missing chain

A mature organisation can take a single production change and walk it backwards through the entire sequence that produced it. Business requirement, ticket, pull request, review, deployment, production. Each link answers one part of the question, and because the links are joined, the answer arrives as a chain rather than a hunch.

Where that chain does not exist, it gets reassembled by hand, in the moment, under pressure. People go searching through Slack (or Teams if you are that unfortunate), half-remembered standups, a meeting note that may or may not be current, and the collective memory of whoever happens to be online. It works, after a fashion, for a while. It does not survive scale, it does not survive staff turnover, and it has a particular habit of falling apart on the one incident where the answer actually mattered.

Infrastructure changed too, and it used to be worse

For most of the industry's history, infrastructure was the part of the system with no traceability at all. A change to production infrastructure looked, in practice, like this - Dave changed something. Nobody knew precisely what. Nobody knew when. Nobody, including Dave by the following week, knew why.

Infrastructure as code dragged that whole category into the daylight. Infrastructure changes can now be reviewed, versioned, diffed, audited and reverted, on exactly the same footing as application code. That shift gets sold as an operational win, and it is one, but the deeper benefit is evidential. The most important thing infrastructure as code gave us was not repeatable environments. It was a record of who did what, and the end of the Dave era.

The audit log nobody appreciates

Audit logs are among the least loved artefacts in engineering. Nobody arrives at work hoping to spend the day reading them. They generate steadily, cost a little to keep, and earn precisely no admiration in the good times.

Then a bad time arrives and they become the most valuable thing in the building. Who reached production. Who changed a permission. Who rotated the key, or disabled MFA, or edited the policy, and at what hour. With a trustworthy audit trail those questions have answers, and the answers take seconds. Without one they collapse into a single sentence that no security team enjoys hearing... we're not entirely sure. That sentence is expensive, and it is expensive exactly when you can least afford it.

Traceability is not a compliance exercise

Here is where a lot of organisations talk themselves into the wrong shape. Traceability gets filed as a compliance concern, something demanded by auditors and risk teams and grudgingly provided to keep them quiet. Build it for the audit, treat it as a cost, resent it mildly.

That reading has cause and effect reversed. Compliance benefits from traceability, certainly. But traceability does not exist because compliance asked for it. It exists because human beings are dreadful at reconstructing complex events from memory, and the bigger the system grows, the more dreadful the gap between what we think happened and what did. Traceability is organisational memory, written down at the time rather than reconstructed afterwards by people with a stake in the answer. The audit is just the day you are grateful you kept it.

The provenance problem AI brings

AI bends this in an interesting direction. A change used to have a simple lineage - an engineer thought of it, an engineer wrote it. You could put a name to the work and mostly stop there.

The chain is getting longer. An engineer prompts a model, the model generates code, the engineer edits it, something downstream transforms it again, and the result ships. Each hop is a place where the question of provenance gets murkier. Which model produced this. Against which prompt. Who actually reviewed the output, as opposed to skimming a diff and trusting the green tick. What was validated and what was waved through because it looked plausible and plausible is what these systems are built to be. As generating code gets cheaper, understanding where a given line came from gets more costly, which is the whole argument for keeping a human firmly in the loop. The need for traceability does not soften as the typing gets easier. It sharpens.

When the change is a leaked credential

A security incident is a traceability problem on a shorter clock. The control was usually there. The visibility was not.

Take a leaked credential, the kind of thing that turns out to be lurking in more places than anyone budgeted for. The questions come fast and they all need answering at once: where was it used, who accessed it, what systems lean on it, when was it exposed, and how long was it live before anyone noticed?

An organisation with traceability spends the incident solving the problem. An organisation without it spends the incident assembling the facts it should already have had, while the clock that actually matters keeps running. The difference is not competence. It is whether the evidence was quietly captured before anyone needed it.

Build it into the platform, not the process

The organisations that get this right eventually stop treating traceability as something engineers do and start treating it as something the platform provides. Deployments are traceable because the deployment system records them, not because someone remembered to fill in a form. Access, configuration and infrastructure changes carry their own provenance because the paved path generates it as a side effect of being used.

This is the same logic behind every good golden path: the safe behaviour has to be the default behaviour, or it does not happen under load. Ask people to produce metadata by hand and they will, right up until the week they are busy, which is the week the incident lands. Bake the record into the road itself and you get the evidence whether anyone was thinking about it or not. Traceability you have to remember to do is traceability you will not have on the day you need it.

What good actually looks like

A healthy organisation can answer a short list of questions quickly. What changed, who changed it, why, who approved it, when it was deployed, what it touched, and whether it can be reversed. Not eventually. Not after three days and a war room. Quickly, because the answers were recorded as the change happened rather than excavated afterwards.

That speed is not a property of having clever engineers. Plenty of clever engineers work in organisations that cannot answer a single one of those questions without a half-day archaeological dig. It is a property of having decided, in advance, that the questions were worth answering. The teams who respond with confidence are almost never the ones with the highest raw talent in the room. They are the ones who paid for visibility long before anything forced them to.

The real lesson

Most of what we talk about in engineering is the building of systems. Traceability is about the understanding of systems, and the two are not the same discipline. You can be excellent at the first and helpless at the second, and most organisations are, right up until the first time the gap costs them.

Because every outage becomes a detective story. Every security incident becomes an investigation. Every audit becomes a hunt for evidence that either exists or does not. Traceability is the quiet decision, made long beforehand, about which of those it is going to be. And at scale, that decision is most of the difference between an organisation that stays calm when something breaks and one that simply finds out, again, how little it could see.

Further reading

• Building Secure and Reliable Systems (Google SRE): the chapters on logging and access justification are the clearest argument going for capturing provenance before you need it rather than after.
• Supply-chain Levels for Software Artifacts (SLSA): a framework for build and deployment provenance, written by people who got tired of taking "it came from CI" on trust.
• How Complex Systems Fail (Richard Cook): eighteen short points on why incidents are never as simple as one cause, and why reconstruction is so much harder than anyone expects.