Securing GenAI at enterprise scale

The first question most organisations ask about generative AI is whether the model is secure. It usually is, which makes it a short conversation. The frontier models are hosted by vendors whose security teams outnumber most companies' entire engineering departments, and attacking the model weights directly is nobody's idea of a productive afternoon.

The more interesting question is whether everything around it is.

A language model in production is one component in a much larger distributed system. Before a prompt gets anywhere near it, a user has authenticated, permissions have been evaluated, documents have been retrieved, tools have been selected and external APIs are waiting on the far side. After it responds, a human or another system will decide how much of the output to trust. Every one of those stages carries its own attack surface, and not one of them is new.

That's the quietly reassuring part. Securing GenAI is much closer to securing a modern distributed application than to inventing a new discipline. The vocabulary has changed considerably more than the fundamentals have.

Think beyond the model

Most enterprise GenAI architectures end up looking something like this:

Notice where the model sits: at the far end of the chain, in someone else's building, well after the decisions that matter. By the time it generates a single token, identity has been established, policy has been evaluated, and the retrieval layer has already decided what it is allowed to see. The real question is rarely whether the model can answer.

It's whether this user should ever have been able to ask.

Identity first

Every GenAI request should begin the way every other production request begins: with identity. Who is making the request, which business unit they belong to, which environments they can touch, which documents they are authorised to retrieve. If two employees ask the same question, they should not necessarily get the same answer. Access control did not stop existing because natural language replaced SQL, and the question of where trust should live is exactly the same one every authentication system has always had to answer.

This is where proofs of concept quietly fall apart. During experimentation everyone has access to everything, because it's a demo and the data is harmless, right up until somebody points it at the real corpus and forgets to revisit the assumption.

Six months later, someone is asking why an intern can summarise board papers.

Treat retrieval as a database query

Retrieval-augmented generation is usually described as "adding context", which is accurate and rather too gentle. I prefer to think of it as executing a database query, because the retrieved documents become part of the prompt, and the prompt is the thing the model acts on. Whatever retrieval returns, the model has effectively read.

So every document should carry exactly the same permissions inside the AI system that it carries outside it. The retrieval layer must never become an alternative route around the authorisation model your organisation already spent years building.

If your search engine wouldn't return a document to this user, neither should your vector search.

Every tool is a privilege

Modern agents don't just answer questions. They create tickets, deploy software, transfer money, approve changes and reset passwords, and every tool connected to a model is another permission the model can exercise on behalf of a user. That framing does most of the work for you: treat every tool the way you would treat any production API. Define who can invoke it. Validate every parameter. Audit every request.

Never assume the model produced sensible input just because it arrived sounding confident. Confidence is the one resource a language model never runs short of.

The model is not your policy engine

One of the easiest mistakes to make is asking the model to decide whether something should happen. Should this deployment be approved? Should this payment be processed? Should this customer receive a refund?

Those are business rules, and business rules belong in deterministic software, where the same inputs produce the same answer on a quiet Tuesday and at five o'clock on a Friday. The model can explain a policy fluently. It must not be allowed to improvise one. Let software decide whether an action is permitted, and let the model explain why.

Expect prompt injection

Prompt injection is often presented as a mysterious new class of AI attack. Conceptually it isn't: you are allowing untrusted input to influence system behaviour, which is a problem the industry has been defending against for decades under other names. What's new is the delivery mechanism. Assume every document, email, web page and PDF your system retrieves may contain instructions written for the model rather than for the human, because some of them already do.

Your application should decide which instructions matter. Not the document.

Log everything

Enterprise AI systems deserve the same operational maturity as every other production service, which means recording who made the request, which documents were retrieved, which tools were called, the model and prompt versions in play, response times, token usage, safety decisions and approval events. None of this is glamorous, and all of it is the difference between an investigation and a shrug.

When something goes wrong, "the AI did it" is not an incident report.

Build for human approval

Not every action deserves automation, and the highest-value enterprise systems often keep humans exactly where they add the most value. Generate the deployment plan, and let an engineer approve it. Generate the payment recommendation, and let finance authorise it. Generate the incident summary, and let the incident commander publish it.

The goal was never removing humans. It's removing the repetitive work while keeping accountability precisely where it belongs.

Governance is architecture

Many organisations treat AI governance as a committee. I think it is primarily an architectural concern, and I've argued elsewhere that the operating model around the AI matters more than anything inside it. Identity, least privilege, policy enforcement, auditability, observability, versioning, deterministic workflows: these are not governance documents. They are engineering decisions, and good ones limit what an AI system can do long before anyone drafts a policy about it.

By the time governance reaches a steering committee, the platform should already have made most of the important decisions.

The boring parts win

Every conference wants to talk about autonomous agents. Far fewer talks cover authentication, audit trails, approval workflows or permission models, yet those unfashionable components are what determine whether a GenAI platform can safely operate inside a bank, a retailer or a healthcare provider.

The model will be replaced by a better one within months. Your security architecture has to survive for years.

That is where the engineering effort belongs.

Further reading

• OWASP Top 10 for Large Language Model Applications: the industry's list of ways this goes wrong, most of which turn out to be old failures with new stationery.
• Guidelines for secure AI system development: the NCSC's take, co-signed by agencies from eighteen countries, which is more consensus than the industry manages about anything else.
• Simon Willison on prompt injection: the person who named the attack, patiently explaining for several years why it still isn't solved.
• NIST AI Risk Management Framework: drier than the Atacama, and the document your risk team will quote at you either way. Better to have read it first.