LLM security is not a prompt problem

Every month there's another headline about someone 'jailbreaking' a chatbot. And every month, the systems we're asked to secure are not failing at the prompt layer. They're failing at the architecture layer: a model with write access, a tool that can reach sensitive data, a response rendered without escaping. The prompt is the attacker's easiest trick and the builder's least important defense.

Threat-model like it's a service, because it is

An LLM application is a web service whose 'input parser' is a probabilistic model. Every principle that applies to a normal service applies double here: least privilege, input validation, output encoding, audit logs, rate limits. The model is not a trusted component. It's an untrusted interpreter sitting in the middle of your data flow.

• Prompt injection is a code-injection class: untrusted content reaches the instruction layer. Mitigate by separating instructions from data, treating retrieved documents as untrusted input, and validating the model's tool calls — not by 'stronger system prompts', which are a speed bump, not a boundary.
• Excessive agency is the big one. A model with every tool and a broad instruction will do more than you asked. Scope tools to the task, require confirmation for anything destructive, and time-box every run.
• Insecure output handling: model output rendered into HTML or SQL is untrusted input. Escape it everywhere it lands.
• Data leakage: any data that reaches the model can be exfiltrated through it. Route data by classification, and don't put secrets in a context window that a tool result can also touch.
• Sensitive information disclosure: the model will happily reproduce personal data from retrieval. Filter and redact at the boundary, not in a prompt.

The agent security checklist

• Every tool has a typed schema and an allowlist. No free-form 'run this'. The model proposes tool calls; your code validates them against the schema before executing.
• Destructive actions require a human. Deletes, writes, money movement, external sends — approval gate, always.
• Separate the instruction layer from the data layer. Retrieved content and tool results are untrusted and cannot change the agent's goal.
• Model output is data, never code. Render it escaped; never eval it; never concatenate it into a query.
• Log every tool call, every input, every output hash. When something goes wrong you need the exact sequence, and 'the model did something' is not an audit trail.

// Untrusted: retrieved document text
const doc = await retrieve(query);
// Instruction layer stays fixed; doc is data.
const messages = [
  { role: "system", content: SYSTEM_GOAL },  // trusted
  { role: "user", content: doc.text },        // untrusted data
];
const toolCall = await model.tools(messages);
if (!ALLOWED_TOOLS.has(toolCall.name)) throw new ForbiddenTool();
if (toolCall.name === "transfer_money") await approve(toolCall);

The model doesn't need to be safe. It needs to be powerless. Put the boundaries in the tools and the data flow, not in the prompt, and the jailbreaks become a demo, not a breach.

— Security review notes, Kodex

Secure LLM systems look unglamorous from the outside: scoped tools, validated calls, escaped output, and a good log. That's the point. The glamour is in the demo; the security is in the plumbing.