Observing LLM systems: beyond tokens and cost

A few years ago, observability meant uptime, latency, and error rates. Those still matter. But an LLM product can have perfect uptime and be quietly getting worse, because the input — the model, the prompt, the retrieval data — changed, and no dashboard measured what actually changed.

The quality problem

LLM systems fail in ways that don't look like failures. A support answer can be wrong, confident, and under the latency budget. Your pager stays silent. The only way to catch it is to measure quality directly — and that means making judgments about outputs part of your pipeline, not an afterthought.

What to capture per request

• The full prompt: system, retrieved context, tool results. You can't debug what you can't replay.
• The full output, and a hash of it, plus the model, temperature, and prompt version. Reproducibility lives here.
• Token counts and latency per step — but also which tools were called, in what order, and what they returned.
• The retrieval receipts: which chunks were used and their scores. Retrieval regressions are invisible without this.
• A quality signal: a judge score, a user rating, or at minimum the 'did the user have to rephrase?' proxy.

Evals in production, not just CI

Offline evals catch regressions before you ship. Production needs a second layer: sample a percentage of live traffic, run an automated judge against each sample, and alert when the quality score drifts. It's a canary for your model's behavior — the same idea as checking your service health, but for judgment.

{
  "request_id": "req_8f2k…",
  "model": "claude-sonnet-4-5",
  "prompt_version": "2026-04-12",
  "context": [{ "chunk": "doc-41", "score": 0.87 }],
  "tool_calls": [{ "name": "tickets.search", "ok": true, "ms": 180 }],
  "tokens": { "in": 4120, "out": 380 },
  "quality": { "judge_score": 0.91, "rated": true },
  "latency_ms": 1900
}

Trace the agent, not just the call

An agent run is a graph: plan, tool calls, retries, re-plans. Logging one API call tells you nothing. Trace the whole episode as a single unit with an ID you can correlate back to the user. When a run goes sideways, you want to replay the sequence, not guess from four separate log streams.

If you can't replay a request exactly, you can't fix it — and with non-deterministic models, 'exactly' is the whole game.

— On-call notes, Kodex

Observing an LLM system means treating outputs as data: capture everything, judge a sample, trace the episode, and alert on drift. Do that, and you'll know — before your users do — that the model changed for the worse.