Timeouts, retries, jitter, and backoff strategies

A remote call that hangs forever is worse than a failed one. It holds threads, sockets, and memory while the rest of the system waits. Retries can hide short blips — or multiply load until a dependency collapses. Amazon’s builders treat timeouts, retries, and backoff with jitter as one toolkit for surviving partial and transient failures without turning a small error rate into an outage.

This article is a practical reference for that toolkit: what each piece does, how the pieces interact, when to use them, and the mistakes that turn “resilience” into a thundering herd.

What these patterns are

Four ideas work together:

• Timeout — cap how long you wait for a remote call so resources are released
• Retry — send the same logical request again after a transient failure
• Backoff — wait longer between attempts so you do not hammer a struggling dependency
• Jitter — add randomness so many clients do not retry in lockstep

Amazon’s Builders’ Library frames the problem clearly: systems rarely fail as a single unit. They suffer partial failures (some requests succeed) and transient failures (the problem is short-lived). Timeouts bound the wait. Retries buy another chance at success. Backoff and jitter keep that second chance from becoming a synchronized flood.

How timeouts work

A timeout is the maximum time a client waits for a request to complete. Without one, a slow or stuck dependency can exhaust memory, threads, connections, or ephemeral ports on the caller.

Connection timeout vs request timeout

A robust client usually needs both:

• Connection timeout — how long to wait to establish a TCP (and often TLS) connection
• Request / read timeout — how long to wait for the response after the connection is up

Prefer timeouts built into well-tested HTTP/RPC clients. Low-level socket options such as Linux SO_RCVTIMEO are easy to misapply: they may not cover DNS, TLS handshakes, or the full end-to-end wait the way a higher-level client deadline does.

Choosing a value

Amazon’s guidance for calls within a region is metric-driven: pick an acceptable false-timeout rate (for example 0.1%), then set the timeout near the matching latency percentile of the dependency (for 0.1%, that is roughly p99.9). That keeps most healthy calls inside the window while still cutting off pathological waits.

Pitfalls they call out:

• Internet clients — add realistic network latency; clients may be worldwide
• Tight latency distributions — when p99.9 is close to p50, add padding so a small latency bump does not timeout almost everything
• Cold connections — a timeout that includes TLS setup can false-fire after deploys when new hosts establish connections; warm pools or slightly higher timeouts on first connect help

Timeouts that are too high waste resources. Timeouts that are too low create fake failures, trigger retries, and can turn a mild latency spike into a self-inflicted outage.

How retries and backoff work

Retries assume the next attempt might succeed. That is often true for network blips, brief overload on one replica, or a load balancer that closed an idle connection. It is false for validation errors, auth failures, and many “not found” responses — those should fail fast.

Retries are selfish

Marc Brooker’s Builders’ Library essay puts it bluntly: a retry spends more of the server’s time to improve your chance of success. When failures are rare, that trade-off raises client availability. When the dependency is already overloaded, retries add load and can delay recovery.

Exponential backoff (capped)

Instead of retrying immediately, wait between attempts. Exponential backoff multiplies the wait after each failure, usually with a cap so delays do not grow without bound:

temp = min(cap, base * 2^attempt)
sleep = temp   // no jitter yet — do not ship this alone

Capping alone creates another failure mode: once every client hits the cap, they all retry at the same steady rate. Limit attempt count (and overall deadlines) so you fail early in the call stack instead of retrying forever.

Full jitter, equal jitter, decorrelated jitter

Backoff without randomness still clusters. If every client waits exactly 100 ms, then 200 ms, then 400 ms, you have replaced one spike with a series of synchronized spikes — the classic thundering herd.

Marc Brooker’s 2015 AWS Architecture Blog post compared formulas with a contention simulator. The important variants:

// Full jitter (strong default)
sleep = random(0, min(cap, base * 2 ** attempt));

// Equal jitter (keeps a minimum wait)
const temp = min(cap, base * 2 ** attempt);
sleep = temp / 2 + random(0, temp / 2);

// Decorrelated jitter (depends on previous sleep)
sleep = min(cap, random(base, prevSleep * 3));

In that simulation, no-jitter exponential backoff did the most work and took the longest. Equal jitter was worse than full jitter on both work and completion time. Full jitter and decorrelated jitter were both strong; full jitter tended to do less total work, decorrelated finished a bit faster at the cost of more work. For most clients, full jitter is the right default: simple and kind to the server.

AWS SDKs in standard retry mode use exponential backoff with full jitter. They also use different base delays by error class — about 50 ms for transient errors and 1,000 ms for throttling — with an individual delay cap of 20 seconds.

Retry budgets and quotas

Per-request attempt limits are not enough under a widespread outage. Google’s SRE book recommends a process-wide retry budget (for example, only allow 60 retries per minute in a process; once exhausted, fail without retrying). AWS SDKs implement a related idea as a retry quota (token bucket): retries consume tokens; successes refill them; when the budget is empty, the SDK fails fast instead of waiting through hopeless retries.

That combination — few attempts per call plus a shared budget across calls — is what keeps “helpful” retries from becoming a cascading failure.

Why it matters

Without timeouts, one slow dependency can pin a large fraction of your concurrency. Without careful retries, a 5% error rate can become far more traffic than the dependency can handle. Google’s SRE book on cascading failures gives the multiplication story: if a database is overloaded and frontend, backend, and client each retry three times (four attempts), one user action can produce 4³ = 64 database attempts.

Correct timeout + retry + jitter policy is not polish. It is load control under stress.

When to use it (and when not to)

Use timeouts on every remote call — including cross-process calls on the same machine. Prefer client libraries with explicit deadlines that cover the whole operation.

Retry when:

• The failure looks transient (timeouts, connection resets, HTTP 502/503/504, explicit throttling with guidance to try later)
• The operation is idempotent, or the API gives you an idempotency key / client token
• You still have attempt budget and retry budget left
• You are retrying at one well-chosen layer, not every hop

Do not retry when:

• The error is permanent for this request (400 validation, 401/403, malformed payloads)
• A write already may have succeeded and the API is not idempotent (duplicate charges, double creates)
• The dependency is signalling sustained overload and your retry budget is empty — fail fast and shed load upstream
• Multiple layers already retry the same logical call

Amazon’s preference for control-plane and many data-plane operations: retry at a single point in the stack.

Advantages and trade-offs

• Aggressive retries, no backoff — masks short blips quickly, but amplifies overload and creates thundering herds
• Exponential backoff, no jitter — reduces average retry rate, yet correlated spikes remain
• Full jitter + capped backoff — spreads load and lowered server work in Brooker’s simulations, with slightly more latency variance
• Retry quotas / budgets — protects recovery during outages; some clients see errors sooner (usually desirable)
• Circuit breakers — hard stop when unhealthy, but modal behavior is harder to test and can slow recovery if mis-tuned

Amazon’s Builders’ Library notes that circuit breakers are popular but introduce modal behavior that is hard to test. Limiting retries with a token bucket often buys much of the protection with simpler local behavior — and that pattern shipped in the AWS SDK in 2016.

Practical example

A small TypeScript helper that applies capped exponential backoff with full jitter, classifies retryable failures, and stops after a budget of attempts:

type RetryOptions = {
  maxAttempts?: number; // includes the first try
  baseMs?: number;
  capMs?: number;
  isRetryable?: (error: unknown) => boolean;
};

function fullJitterDelay(attempt: number, baseMs: number, capMs: number) {
  const ceiling = Math.min(capMs, baseMs * 2 ** attempt);
  return Math.floor(Math.random() * (ceiling + 1));
}

async function withRetry<T>(
  operation: (signal: AbortSignal) => Promise<T>,
  {
    maxAttempts = 3,
    baseMs = 50,
    capMs = 20_000,
    isRetryable = () => true,
  }: RetryOptions = {},
): Promise<T> {
  let lastError: unknown;

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const controller = new AbortController();
    const timeout = setTimeout(() => controller.abort(), 2_000);

    try {
      return await operation(controller.signal);
    } catch (error) {
      lastError = error;
      const isLast = attempt === maxAttempts - 1;
      if (isLast || !isRetryable(error)) throw error;

      const delayMs = fullJitterDelay(attempt, baseMs, capMs);
      await new Promise((resolve) => setTimeout(resolve, delayMs));
    } finally {
      clearTimeout(timeout);
    }
  }

  throw lastError;
}

// Example: only retry likely-transient HTTP failures
await withRetry(
  (signal) => fetch("https://api.example.com/items", { signal }).then(async (res) => {
    if (res.status === 429 || res.status >= 500) {
      throw Object.assign(new Error(res.statusText), { status: res.status });
    }
    if (!res.ok) {
      throw Object.assign(new Error(res.statusText), { status: res.status, fatal: true });
    }
    return res.json();
  }),
  {
    isRetryable: (error) =>
      !(error && typeof error === "object" && "fatal" in error && error.fatal),
  },
);

Production code should also honour Retry-After / server-directed delays when present, share a process-wide retry budget, and treat writes as unsafe until the API is idempotent.

Idempotency makes retries safe

A timeout does not mean the server did nothing. The response may have been lost after a successful write. Amazon’s preferred pattern is a caller-provided client request identifier (for example EC2’s ClientToken): the same token on a retry returns a semantically equivalent result instead of creating a second resource. Design APIs for idempotency before you turn on aggressive client retries.

Best practices

1. Timeout every remote call — connection and request; prefer deadlines that cover DNS/TLS when relevant.
2. Derive timeouts from dependency latency — false-timeout budget first, then the matching percentile; pad for the internet and for cold connections.
3. Retry only transient, classified errors — separate throttling from other failures when you can (longer base delay for throttle).
4. Use capped exponential backoff with full jitter by default.
5. Cap attempts and use a retry budget so outages fail fast instead of retry-storming.
6. Retry at one layer in the stack to avoid multiplicative amplification.
7. Make side-effecting APIs idempotent (client tokens / idempotency keys) before enabling retries.
8. Jitter periodic work too — cron-aligned storms are real; Amazon reports spreading minute/hour jobs reduced required capacity for the same work.
9. Prefer proven SDK retry modes (for example AWS SDK standard) over hand-rolled loops when calling managed APIs.
10. Exercise retry paths in tests and production — untested retry code is how you discover storms during incidents.

Common mistakes

• No timeout — hung calls silently consume the fleet
• Timeout too aggressive — latency blip → mass timeout → mass retry → outage
• Retrying 4xx validation/auth errors — wasted work that can never succeed
• Retrying non-idempotent POSTs — duplicate side effects
• Retries at every microservice hop — 3×3×3 becomes a traffic multiplier
• Exponential backoff without jitter — synchronized herds
• Unlimited retries — threads blocked on hopeless work during a blackout
• Ignoring server Retry-After / overload signals — fighting backpressure instead of cooperating

Alternatives and comparisons

• Fail fast, no retry — best for non-idempotent writes and permanent errors; surface the error to a higher policy
• Immediate retry (1×) — for extremely short connection flakes; still pair with a timeout and avoid on overload
• Full-jitter exponential backoff — default for general RPC/HTTP clients, matching Brooker’s comparisons
• Adaptive / client-side rate limiting — for single-resource, throttle-heavy workloads; AWS SDK adaptive can delay initial calls and is not a universal default
• Hedging (parallel speculative requests) — targets read tail latency; a different tool that can also amplify load
• Circuit breaker — hard isolation of a bad dependency; complements budgets, and you should test open/half-open modes

FAQ

What is the difference between backoff and jitter?

Backoff decides how large the wait window grows after failures (often exponentially). Jitter randomizes where inside that window you sleep so clients do not align. You usually want both.

Is full jitter always better than decorrelated jitter?

Not always. Brooker’s simulations found full jitter used less work while decorrelated completed slightly faster with more work. For typical service clients protecting a shared backend, full jitter is the safer default. Prefer data from your own load tests when latency-to-completion dominates.

How many retries should I configure?

Start small. AWS SDK standard mode defaults to 3 max attempts (one try plus two retries) for most services. Google SRE advises limiting retries per request and adding a process-wide budget. More retries only help if failures are short-lived and the dependency has spare capacity.

Should mobile or browser clients retry the same way as servers?

Be more conservative. Millions of clients with slow update cycles can prolong a bad retry policy. Prefer server-side retries closer to the failing dependency, clear error codes, and smaller client attempt counts.

Do timeouts replace retries?

No. Timeouts stop unbounded waits; retries recover from transient failures after a timed-out or failed attempt. A timeout without a retry policy fails fast. Retries without timeouts can pile up concurrent attempts. Use them together with backoff, jitter, and budgets.

Conclusion

Treat every remote call as potentially slow, partial, or lying about whether it finished. Put a real timeout on it. Retry only when the failure is transient and the operation is safe. Space attempts with capped exponential backoff, break synchronization with full jitter, and protect the fleet with attempt limits plus a retry budget. Retry in one place, not everywhere.

If you change only one thing in a client this week: add full jitter to your backoff — and stop retrying at three layers of the same call path.

References

1. Timeouts, retries, and backoff with jitter — Amazon Builders’ Library, Marc Brooker (2019)
2. Exponential Backoff And Jitter — AWS Architecture Blog, Marc Brooker (2015; updated 2023)
3. Making retries safe with idempotent APIs — Amazon Builders’ Library
4. Retry behavior — AWS SDKs and Tools Reference Guide
5. Announcing updated retry behavior for AWS SDKs and Tools — AWS Developer Tools Blog
6. REL05-BP03 Control and limit retry calls — AWS Well-Architected Framework
7. Addressing Cascading Failures — Google SRE Book
8. What is Backoff For? — Marc Brooker (2022)