Ankusa.BlobStore.Breaker (ankusa v0.7.0)

Copy Markdown View Source

A circuit breaker per {endpoint, bucket} for the HTTP object stores, shared by every instance and every store on the node that talks to that bucket.

It opens on either of two signals:

  • failures: breaker_failures (default 5) consecutive requests that could not be answered (a transport error, 408, 500, 502, 504). A throttled answer (429, 503) is an answer from a reachable store: it resets the count and is Ankusa.BlobStore.Retry's to back off from.
  • latency: the p99 of the last 100 metadata-sized requests (breaker_slow_ms, default 5 s) once at least 20 are known. A bucket that still answers, but slowly, would otherwise hold every caller's connection for the full timeout until the pool is spent.

Open, it answers {:error, :breaker_open} without sending anything, for breaker_open_ms (default 30 s). Then one request goes through as a probe: an answer closes the breaker, a failure opens it again. Each transition emits [:ankusa, :blob_store, :breaker] (see Ankusa.Telemetry).

The state is one ETS row per {endpoint, bucket}, in a table the application owns. Concurrent requests update it without a lock: the worst a race does is lose a latency sample or let a second probe through. breaker_failures: 0 turns the breaker off, the latency trip with it.

Summary

Functions

May a request to scope be sent? An open breaker past its time lets one probe through.

The breaker settings of a store's options.

Record the outcome of one request to scope: :ok (answered, latency_ms a sample unless nil), :answered (a throttle: reachable, no sample) or :failure.

Types

config()

@type config() :: %{
  failures: non_neg_integer(),
  open_ms: pos_integer(),
  slow_ms: pos_integer()
}

outcome()

@type outcome() :: :ok | :answered | :failure

scope()

@type scope() :: {endpoint :: String.t(), bucket :: String.t()}

Functions

allow?(scope, map)

@spec allow?(scope(), config()) :: boolean()

May a request to scope be sent? An open breaker past its time lets one probe through.

config(opts)

@spec config(keyword()) :: config()

The breaker settings of a store's options.

record(instance, scope, cfg, outcome, latency)

@spec record(atom(), scope(), config(), outcome(), non_neg_integer() | nil) :: :ok

Record the outcome of one request to scope: :ok (answered, latency_ms a sample unless nil), :answered (a throttle: reachable, no sample) or :failure.