From c8e44c308b89c6d5b7ec8310bec287183d51e434 Mon Sep 17 00:00:00 2001 From: Sean Cribbs Date: Mon, 2 Oct 2023 16:45:05 -0500 Subject: [PATCH] More README documentation and fix a type in Hardhat.Defaults --- README.md | 193 +++++++++++++++++++++++++++++++++++++--- lib/hardhat/defaults.ex | 2 +- 2 files changed, 184 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index eb3c8f7..89d0bd0 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,10 @@ An opinionated, production-ready HTTP client for Elixir services. - [X] Automatic retries for failed requests - [X] Timeout and `deadline` support +## Why Hardhat? + +TODO + ## Installation Add `hardhat` to the dependencies in your `mix.exs`: @@ -58,14 +62,12 @@ end SocialMediaAPI.get("http://media-api.social/posts") ``` -As mentioned in the example above, it is imperative for you to supervise the client module that includes the `use Hardhat` macro. Without starting the client under supervision, you will not be able to make requests. See [Connection pools](#module-connection-pools) below for more information. +As mentioned in the example above, it is **imperative** for you to supervise the client module that includes the `use Hardhat` macro. Without starting the client under supervision, you will not be able to make requests. See [Connection pools](#module-connection-pools) below for more information. ## General behavior -TODO: Contents of the default middleware, link to other sections - `Hardhat` is built on top of `Tesla`, and uses `Finch` as the adapter. Because -`Tesla` is the foundation, you are welcome to utilize publicly available +`Tesla` is the foundation, you are welcome to use publicly available `Tesla.Middleware` modules in your `Hardhat`-based client (with the exception that [we recommend](#module-timeouts-and-deadlines) you use `Hardhat.Middleware.Timeout` instead of `Tesla.Middleware.Timeout`). @@ -83,28 +85,199 @@ In addition to the adapter selection and default `Tesla` behavior, `use Hardhat` will inject the common functionality [listed above](#module-what-s-included) *after* any middleware that you supply via [`plug`](`Tesla.Builder.plug/2`). The current list is as follows: * `Hardhat.Middleware.DeadlinePropagation` -* `Tesla.Middleware.Retry` +* `Tesla.Middleware.Retry` ([see below](#module-retries)) * Either `Tesla.Middleware.Fuse` or `Hardhat.Middleware.Regulator` ([See below](#module-failure-detection) -* `Tesla.Middleware.Telemetry` -* `Tesla.Middleware.OpenTelemetry` +* `Tesla.Middleware.Telemetry` ([see below](#module-telemetry-and-tracing)) +* `Tesla.Middleware.OpenTelemetry` ([see below](#module-telemetry-and-tracing)) * `Hardhat.Middleware.PathParams` ## Connection pools -TODO: Sizing the pool, `Finch` options +As mentioned above, `Hardhat` uses `Finch` as the adapter. By [default](`Hardhat.Defaults.pool_options/1`), `Hardhat` specifies a connection pool of size `10` but sets no [other options](`Finch.start_link/1`) on the adapter. The name of the `Finch` process is proscribed by the `use Hardhat` macro, but you can set any other options for the pool that you like, including creating more than one pool or setting the HTTP protocol or TLS options by overriding the `pool_options/1` function. + +```elixir +defmodule H2Client do + use Hardhat + + # This function overrides the default coming from `Hardhat.Defaults`. + # The `overrides` will be passed from your process supervision initial + # arguments. + def pool_options(_overrides \\ []) do + %{ + # By default we'll use HTTP/2, with 3 pools of one connection each + :default => [ + protocol: :http2, + count: 3 + ], + # For this host only, we're using HTTP/1.1 and a single pool of 20 + # connections + "https://some-http1-only-host.com/" => [ + size: 20 + ] + } + end +end +``` ## Telemetry and tracing -TODO: `:telemetry` events, `OpenTelemetry` spans and events, propagation +`Hardhat` includes the stock `Tesla.Middleware.Telemetry` for injecting your own metrics and monitoring systems into its operation. The events emitted by this middleware are: + +* `[:tesla, :request, :start]` - at the beginning of the request +* `[:tesla, :request, :stop]` - at the completion of the request +* `[:tesla, :request, :exception]` - when a non-HTTP-status exception occurs + +```elixir +defmodule TelemetryClient do + use Hardhat +end + +:telemetry.attach( + "my handler", + [:tesla, :request, :stop], + fn _event, measures, _metadata, _config -> + # Don't do this, attach to your metrics system instead + IO.puts("Made a request in #{measures.duration}") + end, + nil +) +``` + +`Hardhat` wraps each request in an [OpenTelemetry](`Tesla.Middleware.OpenTelemetry`) span and propagates the trace context to the destination host. It does not currently expose the ability to change the span name in the trace, but it will observe any [path parameters](`Hardhat.Middleware.PathParams`) you interpolate into the URL so that similar spans can be easily aggregated. ## Failure detection -TODO: `:fuse` mode and configuration, `Regulator` mode and configuration +`Hardhat` provides two different failure detection and backoff strategies: + +* Static circuit breaking with `:fuse` (`Tesla.Middleware.Fuse`) +* Dynamic request rate regulation ([AIMD](`Regulator.Limit.AIMD`) with `Regulator` (`Hardhat.Middleware.Regulator`) + +These strategies cannot be used together safely, so you must choose one when defining your client. If your needs are simple and hard failures are relatively rare, `:fuse` is an easier strategy to comprehend and implement because it uses a straightforward failure-counting algorithm, and completely turns off requests when the configured threshold is reached. If you have a more complicated setup or high traffic, and do not want to spend as much time tuning your failure behavior, the `:regulator` strategy might be for you. `Regulator` allows your client to adapt to rapidly changing conditions by reducing the amount of concurrent work in the presence of failure, without causing a hard stop to activity. On the other hand, if your concurrent utilization is low, it might also bound your maximum concurrency even when requests are not failing. + +### The Fuse strategy + +The `:fuse` failure detection strategy is configured with two functions in your client which have default implementations that are injected at compile-time: + +* [`fuse_opts()`](`Hardhat.Defaults.fuse_opts/1`) - configuration for the middleware +* [`should_melt(result)`](`Hardhat.Defaults.should_melt/1`) - whether the result of the request is considered a failure + +You can override their default behavior by redefining the functions: + +```elixir +# This module uses `:fuse` for failure detection and backoff. +defmodule HardCutoffClient do + use Hardhat # defaults to `:fuse` + + # This is also valid: + # use Hardhat, strategy: :fuse + + # Customize fuse's circuit-breaking behavior + def fuse_opts do + [ + # 10 failed requests in 0.5sec flips the breaker, which resets after 1sec + opts: {{:standard, 10, 500}, {:reset, 1_000}}, + # Return the error that caused the fuse to break + keep_original_error: true, + # Use our custom failure test + should_melt: &__MODULE__.should_melt/1 + # Go fast and a little loose + mode: :async_dirty + ] + end + # Customize how responses are determined to be failures, + # in this case only TCP/adapter-type errors are considered + # failures, any valid response is fine. + def should_melt(result) do + case result do + {:error, _} -> true + {:ok, %Tesla.Env{}} -> false + end + end +end +``` + +### The Regulator strategy + +The `:regulator` failure detection strategy is configured with two functions in your client which have default implementations that are injected at compile-time: + +* [`regulator_opts()`](`Hardhat.Defaults.regulator_opts/1`) - configuration for the middleware +* [`should_regulate(result)`](`Hardhat.Defaults.should_regulate/1`) - whether the result of the request is considered a failure + +You can override their default behavior by redefining the functions: + +```elixir +# This module uses `Regulator` for failure detection and backoff +defmodule DynamicRegulationClient do + use Hardhat, strategy: :regulator # overrides default of `:fuse` + + # Customize Regulator's ramp-up and backoff strategy + def regulator_opts do + [ + # Backoff on failure by half instead of 90% + backoff_ratio: 0.5, + should_regulate: &__MODULE__.should_regulate/1 + ] + end + + # Customize how responses are determined to be failures, + # in this case TCP/adapter-level errors are considered failures, + # as well as HTTP `429 Too Many Requests` responses. + def should_regulate(result) do + case result do + {:error, _} -> true + {:ok, %Tesla.Env{status: 429}} -> true + {:ok, %Tesla.Env{}} -> false + end + end +end +``` ## Retries TODO: default options, note about interactions with failure detection +`Hardhat` injects automatic retries on your requests using `Tesla.Middleware.Retry`. Retries are configured with two functions in your client which have default implementations that are injected at compile-time: + +* [`retry_opts()`](`Hardhat.Defaults.retry_opts/1`) - configuration for the middleware +* [`should_retry(result)`](`Hardhat.Defaults.should_retry/1`) - whether the result of the request can be retried + +You can override their default behavior by redefining the functions: + +```elixir +# This client retries requests +defmodule SomeRetriesClient do + use Hardhat + + def retry_opts do + [ + # Retry up to 5 times + max_retries: 5, + # Delay at least 75ms between attempts + delay: 75, + # Delay at most 500ms between any attempts + max_delay: 500, + # Use the default retry rubric + should_retry: &__MODULE__.should_retry/1, + # Add jitter of up to 20% to delays + jitter_factor: 0.2 + ] + end +end + +# This client disables retries entirely! +defmodule NoRetriesClient do + use Hardhat + + # Override should_retry to disable retries + def should_retry(_), do: false +end +``` + +> ### Interaction with failure detection {: .warning} +> Retries can interact very negatively with [failure detection](#module-failure-detection), potentially triggering backoff behavior too quickly. Be sure to avoid retrying when the failure detector returns `{:error, :unavailable}`, which indicates that the circuit breaker has blown in the `:fuse` strategy, or the limiter is out of capacity in the `:regulator` strategy. +> +> The default implementation of `should_retry/1` implements this behavior. + ## Timeouts and deadlines TODO: Custom timeout middleware, `Deadline` support diff --git a/lib/hardhat/defaults.ex b/lib/hardhat/defaults.ex index 27170c8..3f9bcaf 100644 --- a/lib/hardhat/defaults.ex +++ b/lib/hardhat/defaults.ex @@ -113,7 +113,7 @@ defmodule Hardhat.Defaults do @doc """ Default options for the `Regulator` middleware, which can be used as an - alternative circuit-breaking strategy to `fuse`. + alternative circuit-breaking strategy to `:fuse`. The defaults include: * `:min_limit` - The minimum concurrency limit (defaults to 5)