Circuit Breaker

Circuit Breaker

ℹ️ This documentation describes the previous Polly v7 API. If you are using the new v8 API, please refer to pollydocs.org.

This page describes the operation of the original Polly CircuitBreaker and general circuit-breaker concepts. An AdvancedCircuitBreaker is also available from Polly v4.2+, described here.

For rationale - why use a circuit breaker? - see the general discussion of transient fault handling.

Syntax

CircuitBreakerPolicy breaker = Policy
  .Handle<HttpRequestException>()
  .CircuitBreaker(
    exceptionsAllowedBeforeBreaking: 2,
    durationOfBreak: TimeSpan.FromMinutes(1)
  );

The above example will create a circuit-breaker which would break after two consecutive exceptions of the handled type (HttpRequestException) are thrown by actions executed through the policy. The circuit would remain broken for 1 minute.

For full circuit-breaker syntax and overloads, see https://github.com/App-vNext/Polly/tree/7.2.4#circuit-breaker.

Syntax examples given are sync; comparable async overloads exist for asynchronous operation - see readme and wiki.

How the Polly CircuitBreaker works

Circuit-breaker as state machine

A circuit-breaker is best thought of as a state machine, with three main states.

Closed

The circuit initially starts closed. When the circuit is closed:

• The circuit-breaker executes actions placed through it, measuring the faults and successes of those actions.
• If the faults exceed a certain threshold, the circuit will break (open).
  • The original Polly CircuitBreaker will break after N consecutive actions executed through the policy have thrown any handled exception, where N is the int exceptionsAllowedBeforeBreaking the policy was configured with.
  • The AdvancedCircuitBreaker breaks on proportion of failures: see Advanced Circuit Breaker.
• For the action causing the circuit to trip, the original exception is rethrown, but the circuit state transitions to:

Open

While the circuit is in an open state:

• Any action placed for execution through the policy will not be executed.
• Instead, the call will fail fast with a BrokenCircuitException.
  • This BrokenCircuitException contains the last exception (the one which caused the circuit to break) as the InnerException.
• The circuit remains open for the configured durationOfBreak. After that timespan, when the next action is placed through the circuit or if CircuitState is queried, the circuit transitions to:

Half-Open

When the circuit is half-open:

• The next action will be treated as a trial, to determine the circuit's health: the action delegate passed to the .Execute(...) call will be attempted.
  • (One additional attempt will be permitted per durationOfBreak. All other attempts during half-open state are rejected, throwing BrokenCircuitException.)
• In this trial period:
  • If a handled exception is received, that exception is rethrown, and the circuit transitions immediately back to open, and remains open again for the configured timespan.
  • If a success result is received, the circuit transitions back to closed.
  • If an unhandled exception is received, the circuit remains in half-open.

Open/closed semantics

Note that the semantics of open/closed for circuit-breakers are opposite to those of a gate. This is best remembered by considering how a software circuit-breaker is analogous to an electrical switch:

• a closed circuit-breaker allows operations to flow
• an open circuit-breaker prevents.

Exception handling

A circuit-breaker exists as a measuring-and-breaking device: to measure handled exceptions thrown by actions you place through it, and to break when the configured failure threshold is exceeded.

• A circuit-breaker does not orchestrate retries.
• A circuit-breaker does not (unlike retry) absorb exceptions. All exceptions thrown by actions executed through the policy (both exceptions handled by the policy and not) are intentionally rethrown. Exceptions handled by the policy update metrics governing circuit state; exceptions not handled by the policy do not.

For a powerful combination, consider using a circuit-breaker nested within a retry policy (or vice versa), using PolicyWrap.

Scoping CircuitBreaker instances

An instance of CircuitBreakerPolicy maintains internal state to track failures across multiple calls through the policy: you must re-use the same CircuitBreakerPolicy instance for each execution through a call site, not create a fresh instance on each traversal of the code.

You may, further, share the same CircuitBreakerPolicy instance across multiple call sites, to cause them to break in common.

Thread-safety and locking

A CircuitBreakerPolicy instance maintains internal state across calls to track failures, as described above. To do this in a thread-safe manner, it uses locking. Locks are held for the minimum time possible: while the circuit-breaker reads or recalculates state, but not while the action delegate is executing.

The internal operation of the policy is thread-safe, but this does not magically make delegates you execute through the policy thread-safe: if delegates you execute through the policy are not thread-safe, they remain not thread-safe.

Circuit state (for health reporting) (Polly v4.0+)

CircuitState state = breaker.CircuitState;

/*
CircuitState.Closed
CircuitState.Open
CircuitState.HalfOpen
CircuitState.Isolated
*/

Closed: The circuit is operating normally and accepting calls.

Open: The automated circuit-breaker has broken the circuit (ie due to exceeding the configured threshold).

HalfOpen: Prior to executing the first action requested after an automated break timespan has expired.

Isolated: The circuit has been manually broken (see below).

Reducing thrown exceptions when the circuit is broken

A code pattern such as below can be used to reduce the number of BrokenCircuitExceptions thrown while the circuit is open, if those exceptions are a performance concern:

if (breaker.CircuitState != CircuitState.Open && breaker.CircuitState != CircuitState.Isolated)
{
  breaker.Execute(...) // place call
}

Note that code such as this is not necessary; it is an option for high-performance scenarios. In general, it is sufficient to place the call breaker.Execute(...), and the breaker will decide for itself whether the action can be executed. Additionally, the above code does not guarantee the breaker will not block the call. In a highly concurrent environment, the breaker state could change between evaluating the if condition and executing the action. Equally, in the half-open state, only one execution will be permitted per break duration.

Manual control (Polly v4.0+)

breaker.Isolate();

will place the circuit in to a manually open state. This can be used, for example, to isolate a downstream system known to be struggling, or to take it offline for maintenance.

Any action executed through the policy in this state will be blocked (not executed); instead, the call will fail fast with an IsolatedCircuitException. This IsolatedCircuitException extends BrokenCircuitException but does not contain any InnerException.

The circuit remains in the isolated state until a call to:

breaker.Reset();

Delegates on transitions of circuit state (Polly v4.0+)

The circuit-breaker can be configured with delegates on transition of circuit state (for example for logging, or other purposes).

onBreak: The delegate is executed immediately after the circuit transitions automatically to open. Parameters passed include the exception causing the break, duration of break, and (where relevant) context.

The delegate is also executed if .Isolate() is called. In this instance, the duration of break will be TimeSpan.MaxValue; the Exception value passed is indeterminate.

onHalfOpen: The delegate is executed immediately after the circuit transitions to half-open. Note: the delegate does not execute automatically after the automated break timespan has expired. It executes when state is next queried - for example, at the next attempt to execute an action, or when the state is next queried manually.

onReset: The delegate is executed immediately after the circuit transitions automatically to closed, after a successful call placed through the half-open state.

The delegate is also executed if a manual call to .Reset() is made.

Note: All state-transition delegates are executed within the lock held by the circuit-breaker during transitions of state. Without this, in a multi-threaded environment, the state-change represented by the delegate could fail to hold (it could be superseded by other events while the delegate is executing). For this reason, it is recommended to avoid long-running/potentially-blocking operations within a state-transition delegate. If you do execute blocking operations within a state-transition delegate, be aware that any blocking will block other actions through the policy.

Note: The state-transition delegates onBreak, onReset and onHalfOpen are (as at Polly v4.2.1) expected to be synchronous Actions. However, the C# compiler will let you assign async void lambdas to an Action without warning. This can have unexpected runtime consequences, as described by Stephen Cleary in this MSDN article. Calling code will not wait for an async void lambda to complete, before continuing.

Putting it all together

The detailed flow of an individual call through the circuit-breaker is:

Further reading

Further circuit-breaker articles may be found at the foot of the circuit-breaker section in the main readme.