The LLM security checklist we run before any launch

Prompt injection: treat every input as hostile, including your own documents

Prompt injection works because a model cannot reliably distinguish instructions from data. A user typing "ignore your previous instructions" and an uploaded PDF containing the same sentence are the same attack at different addresses: both are text your system will read, and either can steer it. Treat everything the model reads — user input, retrieved documents, web pages, tool responses — as potentially adversarial. Concretely: state in the system prompt that retrieved content is data, not commands; require confirmation outside the model for any sensitive action its output proposes; and make every tool safe to call with the worst plausible arguments. Check before launch: can a sentence buried in an uploaded document make your assistant exfiltrate data or misuse a tool?

Least privilege for a machine: scoped keys, rate limits, audit logs

An agent with tools is a machine holding credentials, and it exercises them with less judgment than any employee. Every key it can reach gets the minimum scope: a tool that only reads orders holds a read-only key, and nothing an LLM touches ever carries admin rights. Put spend and rate limits at the API layer, so a retry loop or an attacker cannot drain an account overnight. And log every tool call — arguments, outcome, caller — because when something goes wrong, that log is the only record of what the machine actually did. Check before launch: list every credential the model can reach, the exact permissions each one grants, and the blast radius if the model is fully compromised. Any answer of "everything" is the work.

Insecure output handling: the model's words are untrusted input

Model output is untrusted input to the rest of your system — the same status as a form field on a public website. If the model generates SQL, it passes through parameterisation, never string concatenation. If it generates HTML shown to users, it gets sanitised or rendered as text. If it produces a file path, URL or command, that goes through an allowlist before anything executes. Teams that validate input everywhere mysteriously skip validation when the text arrives from a clever model; attackers will not extend the same courtesy. Check before launch: trace every path where model output crosses into a database query, a browser, a file system or a network call, and confirm each crossing has validation that assumes nothing about where the text came from.

Supply chain and change control: models, packages, prompts

You are shipping other people's weights and libraries into your infrastructure, so treat them as dependencies: models and packages from verifiable sources, versions pinned, upgrades deliberate rather than automatic — a silent provider-side model change can alter behaviour overnight, which is a security event as much as a quality one. Keep system prompts and instructions in version control with review, because they are executable code in the only language your system reliably obeys. Check before launch: can you name the exact model version serving traffic, the package versions behind it, and who changed the system prompt last — with a diff? If any answer requires archaeology, close that gap before the first real user closes it for you.

Frequently asked questions

Is the OWASP LLM Top 10 enough for compliance?

It is the right starting map, but compliance regimes care about outcomes — data protection, incident handling, access control — not checklists. Use the Top 10 to structure your engineering, then map what you built onto your actual obligations, GDPR included. Passing an audit is a consequence of the work, not a substitute for it.

Do we need a red team before launch?

For a first launch, a focused internal adversarial pass beats an expensive engagement: try to make your own system leak its system prompt, exfiltrate data through a tool, or bypass its filters. Commission a professional red team once the product handles sensitive data at scale, or when a breach would be a headline rather than an inconvenience.

Where do LLM features most commonly get breached?

Not through the model — through the conveniences around it: an over-privileged API key, model output concatenated straight into a database query, or an unlogged action nobody could reconstruct afterwards. The model is rarely the weak link; the plumbing it was bolted onto is. Audit the plumbing first.