What does agent reliability mean?
Reliability is how often your agent completes its task correctly across many runs and real conditions, not how impressive it looks once in a demo.
Capability and reliability are different things. Capability is whether the agent can do the task. Reliability is whether it does the task correctly again and again, including on the runs you never see. A model can be brilliant and still be unreliable, because brilliance once is not the same as correctness every time.
Reliability is a rate, not a yes or no
Because agents are non-deterministic, the honest way to talk about reliability is as a rate: out of a hundred realistic attempts, how many finished correctly. An agent that succeeds 70 percent of the time and one that succeeds 99 percent of the time can look identical in a single demo. The gap only appears at volume, which is exactly where your users live.
Correct means the end state, not the transcript
For an agent that takes actions, correctness is about what actually happened: the right record updated, exactly one charge made, nothing left half-done. A confident summary of success is not success. Throughout this handbook, reliability means the world ended up the way it should, measured over many runs.
Key takeaways
- Judge reliability as a success rate over many runs, not a single result.
- Separate capability (can it) from reliability (does it, every time).
- Define correctness as the final state, not the agent's own account of it.
Further reading
- Building effective agents AnthropicAnthropic's guide to how effective agents are structured and where they commonly break down.