Agentic AI
What makes an AI agent reliable in production
Reliable agents come from narrow scope, constrained tools, explicit stopping rules and continuous evaluation, not from a more capable model alone.
3 min readLast reviewed
An AI agent is a system in which a language model decides which steps to take, calls tools to take them and uses the results to decide what to do next. The pattern, popularised in research such as ReAct, is powerful because the path through a task does not have to be known in advance. It is also why agents fail in ways that ordinary software does not.
Most agent failures we see in reviews are not caused by a model being insufficiently clever. They come from the engineering around it: too many tools, vague goals, no limit on how long the agent may keep trying, and no tests for the situations that actually occur. This article sets out the practices that make the difference.
Start by asking whether you need an agent at all
Anthropic’s engineering guidance on agents makes a distinction worth adopting: workflows follow predefined code paths in which models are called at specific steps, while agents direct their own process. Workflows are more predictable and cheaper. Agents earn their cost only when the sequence of steps genuinely cannot be fixed in advance.
A practical test: write down the steps a skilled person follows. If the list is stable for most cases, build a workflow with AI steps inside it. If the steps change substantially depending on what is discovered along the way, an agent may be justified for that part of the task.
Constrain the action space
An agent can only do what its tools allow. Each tool should be narrow, typed and described precisely, because the description is effectively the instruction manual the model reads. Broad tools such as “run SQL” or “send email to anyone” turn every model error into a potential incident.
- Prefer several small tools with clear names over one general-purpose tool.
- Validate every tool call against a schema before execution and return useful errors.
- Separate read tools from write tools, and gate writes behind policy or approval.
- Give the agent credentials scoped to the task, never a shared administrator account.
The OWASP Top 10 for LLM Applications lists “Excessive Agency” as a distinct risk for exactly this reason: damage results from an agent having more functionality, permissions or autonomy than the task requires.
Treat everything the agent reads as untrusted
Agents read emails, web pages, tickets and documents written by other people. Research on indirect prompt injection showed that instructions hidden in such content can redirect an LLM-integrated application. Reliable agents keep untrusted content clearly separated from instructions, never let retrieved text expand their permissions, and require confirmation before acting on anything that came from outside.
Define when to stop
Every agent needs explicit stopping conditions: a maximum number of steps, a budget, a timeout, and rules for handing the task to a person. Without them, agents loop, retry failing tools indefinitely or produce an answer just to finish. A well-designed agent treats “I could not complete this, here is what I found” as a successful outcome.
Evaluate on scenarios, not demos
An evaluation suite for an agent is a set of realistic scenarios with expected outcomes: the normal cases, the awkward ones, the malicious ones and the ones where the right answer is to escalate. Run it on every change to prompts, tools or models. Score the final result and also the path: did the agent use the right tools, avoid unnecessary calls and stop when it should?
| Measure | What it tells you |
|---|---|
| Task success rate | Does the agent reach the correct outcome on the scenario set? |
| Escalation precision | When it hands over, was that the right call? |
| Unsafe action attempts | How often does it try a disallowed or unapproved action? |
| Steps and cost per task | Is it efficient, or wandering? |
Make every run inspectable
When an agent gets something wrong, you need to see what it saw. Log the inputs, each model call, each tool call with arguments and results, and the final output, with sensitive fields redacted. Traces turn vague complaints into specific fixes and supply new test cases for the evaluation suite.
Sources and further reading
- Anthropic, “Building effective agents” (2024)
- Yao et al., “ReAct: Synergizing Reasoning and Acting in Language Models” (2022)
- OWASP Top 10 for Large Language Model Applications
- Greshake et al., “Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection” (2023)