Handling failure

Retries and idempotency

Retrying transient failures is essential, but only after you make actions idempotent, or every retry risks duplicating a real side effect.

Retry the transient, not the permanent

A timeout or a rate limit is worth retrying, ideally with exponential backoff. A validation error is not: retrying it just fails again. Distinguish the two and only retry what can actually succeed on a second attempt. Libraries like Tenacity handle the backoff mechanics for you.

Make actions idempotent first

Retries are dangerous when the action has side effects. If the first attempt actually charged the customer but the response was lost, a naive retry charges them twice. Idempotency keys solve this: the service recognizes the repeated request and does not repeat the effect. Idempotent requests is the canonical example of the pattern.

pythonimport uuid
from tenacity import retry, wait_exponential, stop_after_attempt

# One key per logical action, reused on every retry of that action.
idempotency_key = uuid.uuid4().hex

@retry(wait=wait_exponential(min=1, max=30), stop=stop_after_attempt(5))
def charge(amount_cents):
    # The key makes the retry safe: if the first attempt already charged,
    # the server returns that original result instead of charging again.
    return stripe.charge(amount_cents, idempotency_key=idempotency_key)

Prove it, do not assume it

Retry and idempotency bugs stay invisible until a request actually repeats. The webhook that arrives twice, the event replayed after a dropped connection, the response that never came back even though the charge already went through. A mock answers once and forgets, so it will never surface any of these. The only way to catch them is to make the repeat happen on purpose.

This is where service replicas earn their place. A replica that holds real state lets you fire the same request twice under one idempotency key and confirm the second attempt returns the original result rather than charging again. You can send a webhook and then send the identical webhook a moment later, and see whether the agent treats the second as a duplicate or acts on it twice. You can also have it swallow the response after the write has already committed, which is the case that fools most agents into retrying something that already succeeded. Tools built for this, like Arga Labs, support all three scenarios out of the box.

Never retry a side effect you have not made idempotent. A retry without idempotency is just a way to do the wrong thing twice.

Key takeaways

  • Retry transient failures with backoff; do not retry permanent ones.
  • Make side-effecting actions idempotent before enabling retries.
  • Use idempotency keys so a repeated request is not a repeated effect.

Further reading

  • Tenacity Python retry libraryHandles retry logic and exponential backoff for you, so transient failures are retried safely.
  • Idempotent requests Stripe DocsThe canonical explanation of idempotency keys, so a repeated request is not a repeated charge.
  • Arga Labs argalabs.comRepeats calls and redelivers webhooks on purpose, useful for checking that idempotency keys hold before a customer is charged twice.