The circuit breaker pattern

A dependency that is already failing does not need more traffic. Timeouts free your threads. Retries give transient blips a second chance. Neither stops a stampede of callers from hammering a service that will not recover for minutes. That is the job of a circuit breaker: fail fast locally, give the dependency room to heal, then probe carefully before restoring full traffic.

Michael Nygard popularized the pattern in Release It! (2007; 2nd ed. 2018). Martin Fowler’s widely cited write-up and Microsoft’s Azure Architecture Center docs formalized the three-state machine most libraries still ship today. This article is a practical reference: how the states work, how to tune failure detection, how to combine breakers with retries, and when Amazon’s builders prefer a retry quota instead.

What it is

A circuit breaker is a proxy around a remote call (or any operation that can fail). It watches recent outcomes. When failures cross a threshold, it trips: further calls return an error (or a fallback) without invoking the dependency. After a cool-down, it lets a limited number of probe requests through. Success closes the circuit; failure opens it again.

The electrical metaphor is deliberate. An electrical breaker opens under fault current so wiring and equipment are not destroyed. A software breaker opens under fault rate (or consecutive failures) so your thread pools, connection pools, and upstream callers are not destroyed by waiting on doomed work.

It complements — it does not replace — timeouts, retries, jitter, and backoff:

How it works

Microsoft’s Cloud Design Patterns description matches the machine almost every library implements.

Closed

Normal operation. Requests go to the dependency. The breaker records successes and failures in a window (count-based or time-based). Occasional errors stay in Closed. When the failure signal crosses the configured threshold and enough calls have been observed, the breaker transitions to Open.

Open

Fast fail. Calls do not reach the dependency. Callers get an immediate error (or a configured fallback: cached data, default, queue-for-later). A wait / reset / break duration timer starts. When it expires, the breaker moves to Half-Open — not straight back to Closed.

Open is the point of the pattern: you stop spending latency and concurrency on work that is likely to fail, and you stop adding load to a struggling service.

Half-Open

Cautious recovery. A limited number of probe requests are allowed. If they succeed (by count or by failure-rate rules), the breaker returns to Closed and resets failure metrics. If a probe fails, it returns to Open and the cool-down starts again.

Half-Open exists because you cannot know recovery without trying — but flooding a half-recovered service with full traffic can knock it over again. Microsoft calls this out explicitly: a recovering dependency may only handle limited volume until it is healthy.

        failures ≥ threshold
  Closed ───────────────────► Open
     ▲                         │
     │                         │ wait duration expires
     │      probes succeed     ▼
     └─────────────── Half-Open
                         │
                         │ probe fails
                         └──► Open

What counts as a failure?

Not every error should trip the breaker. Fowler notes that some failures are normal business outcomes and should not open the circuit. Typical production rules:

• Count as failure: timeouts, connection errors, HTTP 5xx, explicit “dependency unavailable”
• Often ignore: 4xx validation / auth (caller fault), expected “not found”
• Sometimes special-case: HTTP 429 / 503 with Retry-After — Microsoft describes accelerated breaking: trip immediately and stay open at least as long as the dependency asked you to wait

Libraries also treat slow calls as a separate signal. Resilience4j can open the circuit when the slow-call rate exceeds a threshold (calls slower than slowCallDurationThreshold), even if they eventually succeed — useful when latency, not hard errors, is what exhausts your concurrency.

Failure detection in practice

Toy examples trip after N consecutive failures. Production breakers usually use a sliding window and a failure rate.

Resilience4j defaults (widely used as a mental model):

minimumNumberOfCalls prevents a cold process from opening after one or two unlucky errors. Polly’s circuit breaker strategy uses the same idea: failure ratio over a sampling duration, with minimum throughput before the ratio is trusted (example from Polly docs: 50% failures in any 10-second window, at least 8 actions, then break for 30 seconds).

Opossum (Node.js) exposes the same knobs with different names: errorThresholdPercentage, volumeThreshold, resetTimeout, plus an optional per-call timeout.

Why it matters

Without a breaker, a slow or dead dependency turns into resource exhaustion on the caller: threads blocked on timeouts, connection pools stuck, queue depth rising, then your callers time out and retry. Google’s SRE book on cascading failures frames the core risk as positive feedback — overload on one replica increases load on the rest until the whole tier fails. Client-side protection (failing fast, shedding work, not retrying forever) is how you cut that loop.

Breakers also improve user-visible latency. An open circuit returns in microseconds instead of waiting for a multi-second timeout. Azure’s docs stress that this protects response times and gives operations a clear signal: state transitions are excellent health events.

When to use it (and when not to)

Use a circuit breaker when:

• Synchronous calls to a remote service or shared resource can fail for minutes, not milliseconds
• Retries alone would amplify load during a sustained outage
• You need graceful degradation (cache, stub, queue, “try later”) while the dependency is dark
• You want an explicit, observable “we stopped calling X” mode for ops

Prefer something else when:

• Failures are mostly short and rare — timeouts + capped retries with jitter and a retry budget may be enough
• Amazon’s Builders’ Library caution applies: circuit breakers add modal behavior that is hard to test and can slow recovery if mis-tuned. They often mitigate retry storms with a token-bucket retry quota instead (built into the AWS SDK since 2016)
• Microsoft’s guidance: local in-memory work, pure business exceptions, message-driven flows with dead-letter queues, or platform-level isolation (mesh / load balancer health checks) may not need an app-level breaker
• Waiting for cool-down + probes would violate a hard latency SLO — fail the request another way and shed load upstream

AWS Prescriptive Guidance still recommends the pattern when a callee is down or high latency would otherwise cause retry pile-ups, network contention, and thread-pool burn.

Advantages and trade-offs

Advantages

• Fast failure instead of timeout piles
• Load relief for a struggling dependency
• Clear ops signal (Open / Half-Open)
• Natural place for fallbacks and degradation

Trade-offs

• Extra state machine to configure, monitor, and test (including concurrent callers)
• Mis-tuned thresholds: false Open (availability hit) or late Open (cascade already started)
• Shared breaker across independent backends (e.g. shards or tenants) can block healthy paths — Microsoft warns against merging unrelated failure domains
• Half-Open too aggressive → recovery flaps; too timid → prolonged degradation
• Does not fix missing timeouts: long dependency timeouts can still pin threads before the breaker records a failure

Practical example

A minimal TypeScript breaker matching the classic three-state model. Prefer a battle-tested library in production (opossum on Node, Polly on .NET, Resilience4j on the JVM); this sketch shows the control flow.

type State = "closed" | "open" | "half_open";

type Options = {
  failureThreshold: number; // consecutive failures to open (simple detector)
  resetTimeoutMs: number; // open → half-open cool-down
  successThreshold: number; // half-open successes to close
};

class CircuitBreaker {
  private state: State = "closed";
  private failures = 0;
  private successes = 0;
  private openedAt = 0;

  constructor(private readonly options: Options) {}

  async exec<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (Date.now() - this.openedAt >= this.options.resetTimeoutMs) {
        this.state = "half_open";
        this.successes = 0;
      } else {
        throw new Error("CircuitOpen");
      }
    }

    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      throw error;
    }
  }

  private onSuccess() {
    if (this.state === "half_open") {
      this.successes += 1;
      if (this.successes >= this.options.successThreshold) {
        this.state = "closed";
        this.failures = 0;
      }
      return;
    }
    this.failures = 0;
  }

  private onFailure() {
    this.failures += 1;
    if (
      this.state === "half_open" ||
      this.failures >= this.options.failureThreshold
    ) {
      this.state = "open";
      this.openedAt = Date.now();
      this.successes = 0;
    }
  }
}

const payment = new CircuitBreaker({
  failureThreshold: 5,
  resetTimeoutMs: 30_000,
  successThreshold: 2,
});

// Pair with a per-call timeout in real code
await payment.exec(() => chargeCard(orderId));

With opossum, the same idea is configuration rather than a hand-rolled state machine:

import CircuitBreaker from "opossum";

const breaker = new CircuitBreaker(chargeCard, {
  timeout: 3_000,
  errorThresholdPercentage: 50,
  resetTimeout: 30_000,
  volumeThreshold: 10,
});

breaker.fallback(() => ({ status: "queued" })); // degrade while open
breaker.on("open", () => console.warn("payment circuit open"));

await breaker.fire(orderId);

Combining with retries

Microsoft’s rule is the one to remember: retry through a breaker, but stop retrying when the breaker says the fault is not transient (open / broken-circuit exceptions).

A sane pipeline:

1. Timeout on each attempt
2. Circuit breaker around the dependency
3. Retry with full jitter only for retryable errors and only while the circuit is closed (or for the probe itself)
4. Retry budget so the whole process cannot amplify traffic unboundedly

Do not retry open-circuit errors with the same aggressiveness as a single 503. That recreates the stampede the breaker just stopped.

Best practices

1. One breaker per failure domain — per dependency, and often per critical endpoint or shard, not one global switch for the whole process
2. Require minimum volume before evaluating failure rate
3. Classify errors — do not open on caller bugs (400 / 401)
4. Emit state changes — log and metric every Closed → Open → Half-Open transition; alert on Open duration
5. Provide fallbacks where the product allows (stale cache, async queue, reduced UI)
6. Allow manual isolate / close — Polly’s CircuitBreakerManualControl and Resilience4j’s forced states exist so ops can drain traffic during incidents
7. Load-test the modes — Closed success path, Open fast-fail, Half-Open probe limits, and recovery under rising traffic
8. Tune cool-down to recovery reality — database failovers and cold caches need longer opens than a single overloaded replica

Common mistakes

• Breaker without timeouts — threads still pile up waiting for the first failure sample
• Retries ignoring open circuit — multiplies load the breaker tried to cut
• One breaker for many backends — one bad shard blacks out healthy shards
• Threshold too low / no minimum calls — flapping on sparse traffic
• Threshold too high / window too wide — cascade completes before Open
• Unlimited Half-Open traffic — classic Hystrix used a single probe; modern libs allow a small probe set — keep it small
• Treating the breaker as business logic — it is load and failure control, not a substitute for domain error handling
• No fallback story — Open without a product response is just a faster 500

Alternatives and comparisons

Use breakers when you need a hard stop and an explicit recovery handshake. Use budgets and shedding when you mainly need to bound amplification without a sharp mode change.

FAQ

Is a circuit breaker the same as a timeout?

No. A timeout bounds one call. A breaker decides whether to attempt the call at all based on recent health.

Should every microservice call use a circuit breaker?

No. Start with deadlines, idempotent retries, jitter, and a retry budget. Add a breaker where sustained dependency failure would cascade or pin resources — typically critical synchronous paths.

Where should the breaker live — client, library, or mesh?

Anywhere you can observe outcomes and fail fast. In-process libraries (opossum, Polly, Resilience4j) are easy to reason about per service. Meshes and gateways help when many languages share one policy plane. Distributed shared state (e.g. DynamoDB-backed status in AWS samples) helps when many compute instances must share one view of “payment is down.”

What should clients do when the circuit is open?

Handle it like a known outage: degrade, queue work, show a clear UI, or fail the user journey quickly. Do not spin on retries against CircuitOpen / BrokenCircuitException.

How do I choose failure rate and cool-down?

Start from dependency SLOs and recovery times. If p99 latency and error budgets say the service is unhealthy above ~X% errors for Y seconds, align the window and threshold nearby — then validate under load. Prefer conservative minimum throughput so sparse traffic cannot flap the breaker.

Conclusion

A circuit breaker is how you stop calling a dependency that has stopped being useful. Closed monitors, Open protects, Half-Open verifies. Pair it with timeouts and careful retries; respect open-circuit errors; isolate failure domains; and measure state transitions like any other production control plane.

If you only need to tame retry storms, try a retry budget first. If you need a hard stop and graceful degradation while something is dark for minutes, install a breaker — then test the Open and Half-Open paths as carefully as the happy path.

References

1. Circuit Breaker — Martin Fowler (bliki; credits Michael Nygard, Release It!)
2. Circuit Breaker pattern — Microsoft Azure Architecture Center (updated 2025)
3. Circuit breaker pattern — AWS Prescriptive Guidance
4. Timeouts, retries, and backoff with jitter — Amazon Builders’ Library (Marc Brooker; notes circuit-breaker trade-offs and token-bucket retry quotas)
5. Addressing Cascading Failures — Google SRE Book, Chapter 22
6. Handling Overload — Google SRE Book (client-side / adaptive throttling)
7. CircuitBreaker — Resilience4j documentation (sliding window defaults)
8. Circuit breaker resilience strategy — Polly documentation
9. opossum — Node.js circuit breaker (nodeshift)
10. Implement the Circuit Breaker pattern — .NET microservices guidance (Polly + IHttpClientFactory)
11. Nygard, Michael. Release It! Design and Deploy Production-Ready Software — Pragmatic Bookshelf (1st ed. 2007; 2nd ed. 2018)