A circuit breaker per {endpoint, bucket} for the HTTP object stores, shared
by every instance and every store on the node that talks to that bucket.
It opens on either of two signals:
- failures:
breaker_failures(default 5) consecutive requests that could not be answered (a transport error,408,500,502,504). A throttled answer (429,503) is an answer from a reachable store: it resets the count and isAnkusa.BlobStore.Retry's to back off from. - latency: the p99 of the last 100 metadata-sized requests
(
breaker_slow_ms, default 5 s) once at least 20 are known. A bucket that still answers, but slowly, would otherwise hold every caller's connection for the full timeout until the pool is spent.
Open, it answers {:error, :breaker_open} without sending anything, for
breaker_open_ms (default 30 s). Then one request goes through as a probe:
an answer closes the breaker, a failure opens it again. Each transition emits
[:ankusa, :blob_store, :breaker] (see Ankusa.Telemetry).
The state is one ETS row per {endpoint, bucket}, in a table
the application owns. Concurrent requests update it without a lock: the
worst a race does is lose a latency sample or let a second probe through.
breaker_failures: 0 turns the breaker off, the latency trip with it.
Summary
Functions
May a request to scope be sent? An open breaker past its time lets one probe through.
The breaker settings of a store's options.
Record the outcome of one request to scope: :ok (answered, latency_ms
a sample unless nil), :answered (a throttle: reachable, no sample) or
:failure.
Types
@type config() :: %{ failures: non_neg_integer(), open_ms: pos_integer(), slow_ms: pos_integer() }
@type outcome() :: :ok | :answered | :failure
Functions
May a request to scope be sent? An open breaker past its time lets one probe through.
The breaker settings of a store's options.
@spec record(atom(), scope(), config(), outcome(), non_neg_integer() | nil) :: :ok
Record the outcome of one request to scope: :ok (answered, latency_ms
a sample unless nil), :answered (a throttle: reachable, no sample) or
:failure.