Handle partial failure in workflows

In a multi-step workflow, assume a step will fail halfway through, and make progress recoverable so the agent does not leave the system in a broken state.

The dangerous middle

The hardest failures are not at the start or the end, they are in the middle of a workflow, after some actions have happened and others have not. A refund issued but not recorded, a record created but not linked. Left unhandled, these leave systems inconsistent in ways that are painful to untangle.

Checkpoints, compensation, and durability

• Checkpoint after each step so the agent can resume rather than restart.
• Compensate for steps that cannot simply be retried by defining how to undo them.
• Durable execution engines such as Temporal persist workflow state so a crash mid-task does not lose it.

Test the failure, do not hope

The failures that do the most damage straddle two systems at once. Slack posts the message but Salesforce refuses the write, or the payment clears and the order record never saves. A single-service mock cannot produce this, because it only knows its own service and has no idea the other one exists.

Reproducing it requires an environment where multiple services run together and share state. Testcontainers lets you spin up real service instances in Docker and orchestrate failures across them, which works well when the services you need have good container images available. Arga Labs is another good option, as it runs connected replicas with shared state out of the box so you can fail exactly one service at a precise step without the setup overhead.

What matters is what the agent does next. Whether it unwinds the action it already took, retries only the part that failed, or keeps going and leaves the two systems permanently out of sync. Trigger the failure mid-run, then read back what each service actually holds. That comparison is the test.