Improve reliability over time

Turn every production failure into a regression test and re-measure, so reliability compounds instead of decaying.

Close the loop

The most valuable source of test cases is production itself. Every real failure the agent hits should become a new scenario in your suite, so the same problem is caught automatically the next time. This is how a reliability program compounds: the agent gets steadily harder to break.

Guard against regressions

Re-run the full suite on every meaningful change, a prompt, a model version, a tool, and compare success rates against the last accepted baseline. A change that quietly lowers reliability should fail the build, not surprise a user. Running this in continuous integration against a realistic environment keeps every change honest, with something like Promptfoo driving the scenarios on each commit.

Reliability is a practice, not a project

There is no point where an agent is finished being made reliable. The teams that succeed treat it as an ongoing loop: measure, find the weakest failure mode, fix it, and measure again.