Circuit Breaker

Circuit Breaker

This page describes the operation of the original Polly CircuitBreaker and general circuit-breaker concepts. An AdvancedCircuitBreaker is also available from Polly v4.2+, described here.

For rationale - why use a circuit breaker? - see the general discussion of transient fault handling.

Syntax

CircuitBreakerPolicy breaker = Policy
  .Handle<HttpException>()
  .CircuitBreaker(
    exceptionsAllowedBeforeBreaking: 2, 
    durationOfBreak: TimeSpan.FromMinutes(1)
  );

The above example will create a circuit-breaker which would break after two consecutive exceptions of the handled type (HttpException) are thrown by actions executed through the policy. The circuit would remain broken for 1 minute.

For full circuit-breaker syntax and overloads, see https://github.com/App-vNext/Polly#circuit-breaker.

How the Polly CircuitBreaker works

Circuit-breaker as state machine

A circuit-breaker is best thought of as a state machine, with three main states.

Closed

The circuit initially starts closed. When the circuit is closed:

• The circuit-breaker executes actions, measuring the faults and successes of those actions.
• If the faults exceed a certain threshold, the circuit will break (open).
  • The original Polly CircuitBreaker will break after N consecutive actions executed through the policy have thrown a handled exception, where N is the int exceptionsAllowedBeforeBreaking the policy was configured with.
  • The AdvancedCircuitBreaker breaks on proportion of failures: see Advanced Circuit Breaker.
• For the action causing the circuit to trip, the original exception is rethrown, but the circuit state transitions to:

Open

While the circuit is in an open state:

• Any action placed for execution through the policy will not be executed.
• Instead, the call will fail fast with a BrokenCircuitException.
  • This BrokenCircuitException contains the last exception (the one which caused the circuit to break) as the InnerException.
• The circuit remains open for the configured durationOfBreak. After that timespan, but before the next action, the circuit transitions to:

Half-Open

When the circuit is half-open:

• the next action will be treated as a trial, to determine the circuit's health.
• The action delegate passed to the .Execute(...) (or similar) call will be attempted.
  • If this call throws a handled exception, the circuit transitions immediately back to open, and remains open again for the configured timespan.
  • If the call throws no exception, the circuit transitions back to closed.

Exception handling

A circuit-breaker exists as a measuring-and-breaking device: to measure handled exceptions thrown by actions, so as to introduce protective breaking. A circuit-breaker does not (unlike retry) absorb exceptions. All exceptions thrown by actions executed through the policy (both handled exceptions and unhandled exceptions) are intentionally rethrown. For a powerful combination, consider using a circuit-breaker nested within a retry policy (or vice versa), as described in Policy Patterns.

Unhandled exceptions rethrown by the policy have no effect on the metrics governing circuit state.

Thread-safety and locking

As the circuit-breaker must track failures across multiple calls to .Execute(...), it must maintain state across calls. For thread-safety it therefore uses locking. Locks are held for the minimum time possible: while the circuit-breaker reads or recalculates state, but not while the action delegate is executing.

The internal operation of the policy is thread-safe, but this does not magically make delegates you execute through the policy thread-safe: if delegates you execute through the policy are not thread-safe, they remain not thread-safe.

Circuit state (for health reporting) (Polly v4.0+)

CircuitState state = breaker.CircuitState;

/*
CircuitState.Closed
CircuitState.Open
CircuitState.HalfOpen
CircuitState.Isolated
*/

Closed: The circuit is operating normally and accepting calls.

Open: The automated circuit-breaker has broken the circuit (ie due to exceeding the configured threshold).

HalfOpen: Prior to executing the first action requested after an automated break timespan has expired.

Isolated: The circuit has been manually broken (see below).

Note: Code such as this is not necessary:

if (breaker.State != breakerState.Open && breaker.State != breakerState.Isolated)
{
  breaker.Execute(...) // place call
}

It is sufficient to place the call breaker.Execute(...), and the breaker will decide for itself whether the action can be executed. Additionally, users should be clear that the above code does not guarantee the breaker will not block the call: in a highly concurrent environment, the breaker state could change between evaluating the if condition and executing the action. However, a code pattern such as above can be used to reduce the number of BrokenCircuitExceptions thrown while the circuit is open, if this is a performance concern.

Manual control (Polly v4.0+)

breaker.Isolate();

will place the circuit in to a manually open state. This can be used, for example, to isolate a downstream system known to be struggling, or to take it offline for maintenance.

Any action executed through the policy in this state will be blocked (not executed); instead, the call will fail fast with an IsolatedCircuitException. This IsolatedCircuitException extends BrokenCircuitException but does not contain any InnerException.

The circuit remains in the isolated state until a call to:

breaker.Reset();

Delegates on transitions of circuit state (Polly v4.0+)

The circuit-breaker can be configured with delegates on transition of circuit state (for example for logging, or other purposes).

onBreak: The delegate is executed immediately after the circuit transitions automatically to open. Parameters passed include the exception causing the break, duration of break, and (where relevant) context.

The delegate is also executed if .Isolate() is called. In this instance, the duration of break will be TimeSpan.MaxValue; the Exception value passed is indeterminate.

onHalfOpen: The delegate is executed immediately after the circuit transitions to half-open. Note: the delegate does not execute automatically after the automated break timespan has expired. It executes when state is next queried - for example, at the next attempt to execute an action, or when the state is next queried manually.

onReset: The delegate is executed immediately after the circuit transitions automatically to closed, after a successful call placed through the half-open state.

The delegate is also executed if a manual call to .Reset() is made.

Note: All state-transition delegates are executed within the lock held by the circuit-breaker during transitions of state. Without this, in a multi-threaded environment, the state-change represented by the delegate could fail to hold (it could be superseded by other events while the delegate is executing). For this reason, it is recommended to avoid long-running/potentially-blocking operations within a state-transition delegate. If you do execute blocking operations within a state-transition delegate, be aware that any blocking will block other actions through the policy.

Note: The state-transition delegates onBreak, onReset and onHalfOpen are (as at Polly v4.2.1) expected to be synchronous Actions. However, the C# compiler will let you assign async void lambdas to an Action without warning. This can have unexpected runtime consequences, as described by Stephen Cleary in this MSDN article. Calling code will not wait for an async void lambda to complete, before continuing.

Putting it all together ...

The detailed flow of an individual call through the circuit-breaker is:

Further reading

Further circuit-breaker articles may be found at the foot of the circuit-breaker section in the main readme.