Operations
AI observability: what to log, trace and measure
A practical observability model for LLM applications: traces, quality signals, cost and latency, drift, and what to do with sensitive data in logs.
2 min readLast reviewed
Traditional monitoring tells you whether a service is up and how fast it responds. AI systems also need to tell you whether they are right. Observability for language model applications combines familiar telemetry with quality signals that require deliberate design.
Trace every request end to end
A single user request may involve retrieval, several model calls and tool calls. Record each as a span in a trace: inputs, outputs, model and version, parameters, token counts, latency and errors. OpenTelemetry now includes semantic conventions for generative AI operations, which helps keep this data portable between tools.
Four families of signals
| Family | Examples |
|---|---|
| Operational | Latency per step, error and timeout rates, retries, rate-limit hits |
| Cost | Tokens and spend per request, per feature and per customer |
| Quality | Automated evaluation scores, user feedback, edit rates on drafts, escalation rates |
| Safety | Guardrail triggers, blocked tool calls, prompt-injection detections, policy violations |
Quality needs ground truth
Most quality signals in production are proxies. Thumbs-up ratings are sparse and biased; edit rates on drafts are more informative. Combine them with periodic human review of a sample and with automated scoring by a model where that scoring has been checked against human judgement. Research on using strong models as judges shows useful agreement with human preferences on some tasks, along with known biases, so calibrate before relying on it.
Watch for drift
Behaviour changes when a provider updates a model, when your content changes or when users start asking different questions. Track the distribution of request types and evaluation scores over time, and rerun your offline evaluation suite on a schedule even when nothing has been deployed.
Handle sensitive data in telemetry
- Redact or pseudonymise personal data before it is written to logs.
- Set retention periods for traces and enforce them automatically.
- Restrict access to raw prompts and outputs to the people who need them.
- Remember that evaluation datasets built from production data are themselves sensitive.