Idempotency and Retries

A shop's app sends "charge $40" to its payments API. The API charges the card, but the response is lost on the way back. After 2 seconds the app gives up and tries again, and the card is charged a second time. If 1 in 1,000 calls times out like this and the shop takes 100 payments a second, that is 100 × 0.001 = 0.1 double charges a second: 8,640 angry customers a day.

The retry was right: most failures are brief. The server just had no way to recognise "this is the same charge again".

#The idea

An operation is idempotent if doing it twice has the same effect as doing it once. "Set the status to shipped" is idempotent. "Add $40" is not. Over a network you cannot tell a lost request from a lost response, so every client that retries is sending some operations twice. Make those operations idempotent, then retry freely.

Retries need three companions: a timeout, so you stop waiting; backoff with jitter, so retries do not pile on a struggling server; and a limit, so a failure does not become an endless loop.

#Timeouts and backoff

A timeout means "I do not know what happened", never "it failed". Set it a little above the callee's p99 (if p99 is 300 ms, try 500 ms), not a round 30 seconds that ties up a thread.

Retry only what can succeed next time: timeouts, connection resets, 503, 429. Never retry a 400; the same bad request fails the same way.

Wait longer after each failure: 100 ms, 200 ms, 400 ms, doubling up to a cap. That is exponential backoff. Then add jitter: wait a random time between 0 and that ceiling. Without jitter, 1,000 clients that failed together retry together, at exactly 100 ms, then 300 ms, and hit the recovering server in synchronised waves.

async function withRetries(call, maxAttempts = 4, baseMs = 100, capMs = 2000) {
  for (let attempt = 1; ; attempt++) {
    try {
      return await call();
    } catch (err) {
      if (!err.retryable || attempt === maxAttempts) throw err;
      const ceiling = Math.min(capMs, baseMs * 2 ** (attempt - 1));
      await sleep(Math.random() * ceiling); // full jitter
    }
  }
}

def with_retries(call, max_attempts=4, base_ms=100, cap_ms=2000):
    for attempt in range(1, max_attempts + 1):
        try:
            return call()
        except RetryableError:
            if attempt == max_attempts:
                raise
            ceiling = min(cap_ms, base_ms * 2 ** (attempt - 1))
            time.sleep(random.uniform(0, ceiling) / 1000)  # full jitter

With 4 attempts, the waits are drawn from at most 100, 200 and 400 ms, so the whole retry adds at most 700 ms on top of the attempts themselves.

#Retries multiply

If the app, the API and the service behind it each make up to 3 attempts, one user request can reach the database behind them 3 × 3 × 3 = 27 times. When that service is overloaded, retries turn trouble into an outage. Retry at one layer, usually the one closest to the user, and give each client a retry budget: retries may add at most 10% to normal traffic; beyond that, failures are returned.

#Idempotency keys

For operations that are not naturally idempotent, the client makes them so. It creates a unique idempotency key (a random UUID) for each logical action, such as one press of "Pay", and sends it with every attempt. The server remembers keys it has seen:

1. Insert the key with status "in progress". A unique constraint means only one attempt can do this.
2. If the key already exists and is done, return the saved response without doing the work again. If it is still in progress, return 409 and let the client retry later.
3. Do the work, and pass the key on to any downstream system that accepts one (the card processor).
4. Save the response with the key. Keep keys for about 24 hours, longer than any client keeps retrying.

Also store a hash of the request body. The same key with a different amount is a client bug: reject it with 422 instead of returning the old result.

One gap remains: if the server crashes after charging but before saving "done", the key is stuck "in progress" and every retry gets 409. So the claim holds a lease, say 30 seconds. Once it expires, a retry takes over and calls the processor again with the same key, and the processor returns the original charge instead of making a new one. A processor without keys needs a sweeper that checks stuck keys against it first.

#Ways to make work safe to repeat

#Numbers worth remembering

• Timeout: a little above the callee's p99, not its average.
• Backoff: start around 100 ms, double, cap at a few seconds, 3 to 5 attempts.
• Retry budget: about 10% extra traffic. Retry at one layer, not every layer.

#In the interview

Raise this whenever a client can retry an action with side effects: payments, orders, sending a message, any consumer of an at-least-once queue. The phrase that shows judgement: "A timeout is not a failure, it is an unknown. So the create call takes an idempotency key."

Follow-ups you should expect:

• "Who generates the key, client or server?" The client: only it knows that two requests are the same action.
• "Should every layer retry?" No. Retries multiply; pick one layer and use a budget.

#Common mistakes

#What to practise next

Retries are one part of staying up; circuit breakers and load shedding are in reliability and failure. To use idempotency keys where they matter most, work through Design a Payment System.