# Secrets are everywhere > Year after year the breach reports tell the same story: not sophisticated attackers, just credentials sitting somewhere they shouldn't. By Matthew D. Webb · 2025-11-01 · 8 min read Canonical: https://mdwebb.io/articles/secrets-are-everywhere/ AI involvement: Updated with images. --- Most engineers think of secrets as passwords. They aren't, or at least they aren't only that. A secret is anything that proves identity. Database credentials, API keys, OAuth tokens, private keys, certificates, service account credentials, signing keys, webhook secrets, encryption keys, cloud access tokens. The list keeps going, and the moment you start counting properly it gets uncomfortable. Modern systems are full of them, far more than anyone ever signed up for. Most organisations have considerably more secrets than they realise. And the annual breach reports keep naming the same culprit: one of these, traced back far enough, sitting somewhere along the path. Not a sophisticated attacker bending the laws of cryptography. Just a credential in the wrong place. ## The database password that never changed Every engineer has seen something like this: ```javascript const connection = { host: "prod-db.company.com", user: "admin", password: "SuperSecret123" }; ``` Hopefully not in production. Often in production. The problem isn't really that the password exists; passwords have to live somewhere. The problem is where it goes next. By the time anyone thinks to worry about it, that one string has copied itself into git history, developer laptops, CI systems, build logs, backups and three open pull requests. A secret rarely stays where it started. Secrets spread. That is the whole difficulty, in a word. ## The lifecycle nobody thinks about Most engineers think hard about creating a secret and almost nothing about what happens to it afterwards. A credential has a whole life ahead of it: ```text Create → Store → Access → Rotate → Revoke → Delete ``` Every secret should eventually move through all six. Most organisations get as far as store and then quietly lose interest, which is roughly like building a house with a front door and no plumbing. The exciting part is over. The part that actually keeps the place habitable never gets done. ## Environment variables are better, not perfect The usual first improvement is to drag the password out of the source and into the environment: ```bash DATABASE_PASSWORD=super-secret ``` ```javascript process.env.DATABASE_PASSWORD ``` This is better. It is not secure. It is just better, and the distinction matters, because a lot of teams stop here convinced the job is done. Environment variables still surface in crash dumps, debug tooling, container inspection, process listings and the CI system that injected them in the first place. They reduce exposure. They don't eliminate it. Anything you hand to a running process is reachable by anyone who can reach the process, and in the browser the situation is worse still, which is a whole separate category of pain. ## Secret managers exist for a reason Eventually an organisation finds the dedicated tools, all of which solve roughly the same problem: AWS Secrets Manager, HashiCorp Vault, Azure Key Vault, Google Secret Manager. Pick one. The idea underneath all of them is to separate the secret from the code. Credentials live in one central place, applications ask for them at runtime, and the things that were previously impossible become routine. Rotation becomes manageable. Audit trails become possible. You can finally answer the question of who read what, and when. This is the point where secret management stops being an engineering problem and starts becoming a governance one. Hold that thought; we come back to it. ## The service account explosion Humans are rarely the biggest problem. Machines are. A modern platform is a crowd of services all talking to each other: a frontend, an API, payments, loyalty, inventory, search, analytics. Every service talks to something. Every interaction requires trust, and trust, in practice, means a credential. So the arithmetic gets strange very quickly: ```text 500 engineers 3,000 service accounts ``` The humans are comprehensively outnumbered by the machines acting on their behalf, and every one of those machine identities needs creating, storing, rotating and revoking exactly like a human one, except there are six times as many and not one of them will ever notice it's been locked out. Identity stops being an HR-shaped problem and becomes an infrastructure one. This is the same machine-to-machine trust question that sits underneath [authentication](/articles/trust-but-verify/), and it shares almost none of the failure modes of the human kind. ## Long-lived credentials are technical debt One of the most dangerous phrases in security is also one of the most reasonable-sounding: > We'll rotate it later. Later rarely arrives. Years pass, teams change, the person who set it up leaves, the documentation rots, and eventually nobody can say with any confidence where the key is even used. At which point the organisation is no longer keeping the credential because it's safe. It's keeping it because it's frightened to touch it. A credential that cannot be rotated is already a liability. It just hasn't sent the invoice yet. ## Dynamic credentials This is where it gets genuinely interesting. The old model issued a credential and assumed it would live forever: ```text credential valid: forever ``` The modern model assumes the opposite: ```text credential valid: 15 minutes ``` The application requests a credential, uses it, and lets it expire. A fresh one is issued when it's next needed. Nothing is meant to last. And because nothing lasts, the window in which a stolen credential is worth anything shrinks from "indefinitely" to "about as long as it takes to make a cup of tea". The secret becomes disposable, which turns out to be one of the most important shifts in modern security architecture. You stop guarding the credential and start trusting the process that mints it. ## The GitHub problem Every organisation goes through this particular rite of passage. Someone commits a key: ```javascript const apiKey = "sk_live_xxxxxxxxx"; ``` Maybe to a private repository. Maybe not. It barely matters, because the safe assumption is identical either way: the secret is compromised. The instinctive response is to delete the commit and pretend it never happened. This is almost always the wrong move, and worse, it feels productive. The right move is to rotate the credential. Assume exposure, act accordingly, get on with your day. Git remembers everything, and the people you're worried about know exactly how to read it. You cannot un-commit a secret. You can only retire it. ## Encryption keys are different Not all secrets carry the same blast radius. If an API key leaks, the fix is annoying but bounded: replace the key, update the callers, move on. If an encryption key leaks, the fix might be to re-encrypt everything it ever touched, which is a rather different kind of afternoon. The damage from a leaked encryption key scales with everything it has ever protected, not with the key itself. That is why key management deserves its own architecture rather than being filed in the same drawer as everything else. Treating all secrets as equal is a tidy assumption and an expensive one. ## The governance problem Sooner or later every large organisation reaches the same unwelcome realisation. Secret management isn't really a security problem at all. It's a governance problem. The questions that actually decide whether you're safe have very little to do with cryptography. Who can create secrets. Who can access them. Who can rotate them. Who approves that access. Who audits the usage afterwards. The technology to do all of this is, by now, fairly boring and well understood. The process around it is where things fall apart, because process requires people to agree on who is responsible, and getting people to agree on responsibility is the genuinely hard part of almost everything. ## What good looks like A mature setup is almost aggressively unglamorous. Centralised secret storage. Audit logging. Short-lived credentials. Automated rotation. Least-privilege access. No secrets in source control. Regular access reviews. None of that gets you invited to give a conference talk. Most real security engineering doesn't, because the goal was never elegance. The goal is to lower the probability of a very bad day, and very bad days are headed off by dull, consistent habits rather than clever ones. ## The real lesson The interesting thing about secrets is that they behave almost exactly like [technical debt](/articles/why-most-technical-debt-is-really-organisational-debt/). Every individual shortcut feels harmless. One hard-coded password. One shared account. One key nobody rotates. One exception, just this once, because there's a deadline and it's Friday. Then those decisions accumulate, the way they always do, until you're running an organisation full of credentials nobody understands, nobody owns, and nobody is brave enough to touch. Good secret management was never really about protecting passwords. It's about controlling trust. Every secret is, in the end, the stored answer to a single question: > Who is allowed to do what? Modern systems ask that question thousands of times a second, mostly with nobody watching. The hard part, the part that decides whether you get a quiet year or a catastrophic one, is making sure the answer stays correct long after everyone who set it up has moved on. ## Further reading - [The OWASP Secrets Management Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html): exhaustive, unglamorous, and worth more than most things written on the subject. Read it before you decide your `.env` file counts as a strategy. - [The Twelve-Factor App: config](https://12factor.net/config): the essay that talked a generation into environment variables. Still good advice, provided you remember it was the floor, not the ceiling. - [GitHub, "Removing sensitive data from a repository"](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/removing-sensitive-data-from-a-repository): the official guide to scrubbing a leaked secret, which opens by telling you to rotate the credential first. The tool quietly admits its own limits, which is more than most do. - [Verizon Data Breach Investigations Report](https://www.verizon.com/business/resources/reports/dbir/): the annual census of how breaches actually happen. Stolen credentials have sat at or near the top of it for years, which is this post's argument in a bar chart.