AI agents in production: the parts nobody demos

There's a specific kind of demo I've seen a dozen times this year: a single prompt, a clean input, and an agent that cheerfully completes a multi-step task in front of an impressed room. Every one of those demos is true. And none of them is production.

Production is a user pasting a garbled document. It's a tool that returns a 500. It's a model confidently citing a source that doesn't exist. The gap between 'works in a demo' and 'safe on real traffic' isn't a tweak — it's an architecture.

Design for the failure modes, not the happy path

Before writing a line of agent code, write down everything that can go wrong and decide, per failure, whether the system degrades gracefully or stops loudly. LLMs are non-deterministic: the same input produces different output on different runs. That's fine when the output is a draft. It's dangerous when the output is an invoice or a deletion.

• Missing context — the model fills the gap with a confident guess. Detect low-confidence retrieval and ask instead of inventing.
• Tool failures — a search API times out. The agent should retry with backoff, then re-plan, then surface the problem to a human.
• Infinite loops — agents that re-plan forever burn tokens and money. Cap steps and budget per run.
• Latent harmful actions — the model's natural-language intent gets translated to an API call. That translation is where you enforce guardrails, not in the prompt.

Evals are the testing framework

Unit tests assert on fixed inputs. Agents need evals: a curated set of tasks with known-good outcomes, run on every change. Start with fifty cases that capture your real traffic — not the demos. Add every production failure as a regression case, the same way you'd add a failing unit test.

const task = {
  input: "move the invoice for AMX-204 to paid",
  expected: { tool: "invoices.update", args: { id: "AMX-204", status: "paid" } },
  guards: ["no other record modified", "audit event emitted"],
};

An eval suite that catches regressions is the difference between shipping an agent and shipping a fire. Budget for it the way you'd budget for a test harness — it's not a nice-to-have, it's the contract that makes the non-determinism safe.

The boring architecture wins

The most reliable agents we've put into production have a structure that looks almost boring: a planner that decomposes a task, a set of narrow tools with typed inputs, an executor with step caps, and an approval gate for anything destructive. The cleverness lives in the tools and the evals, not in the loop.

The best agent is the one you forget is an agent. It's just software that does its job, quietly, and files a ticket when it can't.

— Kodex Labs

If you remember one thing: production agents fail on the 2% of inputs you didn't think about. Find them early with evals, contain them with guardrails, and make 'ask a human' the default when confidence is low. Do that, and the demo becomes the product.