# `Ankusa.BlobStore.Retry`
[🔗](https://github.com/jamescarr/ankusa/blob/ankusa-v0.7.0/packages/ankusa/lib/ankusa/blob_store/retry.ex#L1)

Retries one object-store request after a transient failure, for the HTTP
blob stores (`Ankusa.BlobStore.S3`, `GCS`, `Azure`, `OCI`), under one
deadline and a per-`{endpoint, bucket}` breaker (`Ankusa.BlobStore.Breaker`).

A transient failure is a transport error (refused, reset, timed out) or a
`408`, `429`, `500`, `502`, `503` or `504`. Anything else — success, `404`, a
`403` — is returned at once. The adapter option `:retries` (default `2`) is
the number of extra attempts, `0` for a single attempt. Each retry waits a
**full-jitter** delay: a uniformly random time between 0 and 200 ms, then 1 s,
then 5 s for every later one, so nodes throttled together do not retry in
lockstep. Each retry emits `[:ankusa, :blob_store, :retry]` (see
`Ankusa.Telemetry`).

## One deadline

`opts[:deadline]` (an absolute `System.monotonic_time(:millisecond)`; see
`Ankusa.BlobStore.Options.with_deadline/1`) bounds the whole operation. Each
attempt runs under the smaller of its timeout (`:timeout_ms` for a `:small`
request, `:upload_timeout_ms` for a `:bulk` one) and the time left; nothing
is sent, and no retry is slept, once it has passed (`{:error, :deadline}`), and
a retry whose wait would not fit returns the last answer instead.

## `Retry-After`

A `429` or `503` with a `Retry-After` (seconds, or an HTTP date) makes the
next wait at least that long, **if it fits in the time left**. If it does not,
nothing sleeps: the call returns `{:error, {:throttled, seconds}}` at once, for
the caller (dispatch, which has its own retry and backoff) to schedule. The
same is returned when the attempts are spent and the last answer was such a
throttle. A throttle with no `Retry-After` is an ordinary `{:status, 429 | 503,
body}`.

## Breaker

An open breaker answers `{:error, :breaker_open}` without a request, and is
not retried; every answered or failed attempt is recorded in it. A failure of
this node (the connection pool gone, or out of free connections) is retried but
not recorded: it says nothing about the bucket.

Every object-store call is idempotent (a `PUT` writes the same bytes to the
same key; a conditional one is settled by `Ankusa.BlobStore.same_object/5`),
so a timed-out request that did land is safe to send again. The adapters
build and sign the request inside `fun`, so each attempt carries a fresh date
and signature. `Ankusa.HttpClient` itself never retries (`retry: false`): the
sinks' retries are the dispatch pipeline's.

# `http_result`

```elixir
@type http_result() ::
  {:ok, non_neg_integer(), %{required(String.t()) => [String.t()]}, term()}
  | {:error, term()}
```

# `run`

```elixir
@spec run(
  atom(),
  keyword(),
  Ankusa.BlobStore.Breaker.scope(),
  :small | :bulk,
  (pos_integer() ->
     http_result())
) ::
  http_result()
  | {:error, :deadline | :breaker_open | {:throttled, non_neg_integer()}}
```

Run `fun` — one `Ankusa.HttpClient.request_with_headers/6`, given the timeout
of this attempt — for `instance` against the breaker `scope`
(`{endpoint, bucket}`), retrying it per `opts[:retries]`. `kind` is `:small`
or `:bulk` (see `Ankusa.BlobStore.Options.kind/1`). Raises `ArgumentError` on
a `:retries` that is not a non-negative integer.

---

*Consult [api-reference.md](api-reference.md) for complete listing*