The object stores claim packs live in — claim_check.blob_stores, else
[storage.blob_store] — and which of them a pack is written to and read
from. Stores are named by their index in that list. Claim keys are never
prefixed, so one gateway reads every node's claims.
Placement
claim_check.placement picks a new pack's home store:
:ordered— the first store; the others take packs only while it is down.:hash— the pack id's 64 random bits modulo the number of stores, so packs spread evenly over every store; the next stores in list order take over while a pack's home is down.
A write tries the home store, then each next one in list order, wrapping around, and stops at the first that takes the pack. The store a pack landed in is not recorded anywhere: a ref names only a tenant and a claim id.
Breaker
A store whose write or read fails (could not answer: not a 403, not a
missing key) is skipped by later writes, and searched last by later reads,
for claim_check.store_cooldown_ms, then tried again; any answer from it
closes it. When every store is down, every store is tried anyway — a write
is never refused untried. The cooldown is fixed: the stores back each other
up, and the dispatch retry policy backs the whole write. Each open and close
emits [:ankusa, :claim_check, :store_breaker]. Breakers live in an ETS
table this process owns; a restart closes them all. A throttled answer
(429, 503) is a reachable store: it never opens the breaker;
Ankusa.ClaimCheck.Reader's per-store limit is what backs off.
Reads
A read searches every store at the configured claim_check.key_layout,
then every store again at the other layout — so a pack a failover put
elsewhere, or one written before key_layout or placement changed, is
still found. Stores whose breaker is open are searched after every healthy
store has been asked at both layouts, never skipped: a read that cannot
reach a store opens its breaker too, so on a gateway that never writes, a
hung store costs one read its timeout and later reads go to the healthy
stores first; an answer from a store (found or not) closes it. A store that
could not answer is not asked again within the same read, at the other
layout: that would cost a hung store's timeout twice, and double the
requests to one that throttles.
A 403 is asked again (a credential without s3:ListBucket answers 403
for a missing key). A miss is reported as :not_found only when every
store answered; if any store could not, the read is {:unavailable, reason} (or :forbidden), the first such error, so a claim is never
declared gone while it may be in a store that did not answer.
Adding a store moves :hash homes (the search still finds older packs).
Removing a store makes the packs in it unreadable: remove one only after
its packs have aged out of retention.
Summary
Functions
Returns a specification to start this module under a supervisor.
length bytes at offset of key in store index: one ranged read. A
store index past the end of the list (one removed since the claim was
located) is :not_found.
The index of the store a new pack pack_id goes to first (see Placement).
The claim stores, indexed in config order.
Find pack pack_id under the first of keys any store has, and read
length bytes of it at offset: each key in turn, across every store in
the pack's write order, healthy stores first (see Reads). Returns the bytes
with the store index and key they came from.
Whether a read error is the store throttling this node (429, 503, or the
adapter's {:throttled, seconds} for one that carried a Retry-After): a
reachable store asking for less load, not one that is down.
Write a pack's bytes under key, only if nothing is there
(Ankusa.BlobStore.put_if_absent/4): the home store first, then the next ones,
skipping stores whose breaker is open (unless all are). Returns the index of
the store that took it. A store that already holds an object under key is
{:error, :already_exists} at once, with no failover: the pack id collides,
and the caller mints a fresh one. When every attempt fails:
{:unavailable, reason} after a single attempt, else
{:unavailable, [{index, reason}, ...]} in attempt order; a store that
throttled ({:throttled, seconds}) is a reachable store and does not open
its breaker.
The absolute deadline (System.monotonic_time(:millisecond)) of one check-in:
the largest op_deadline_ms among the claim stores
(Ankusa.BlobStore.Options) from now. It covers every store tried, every
attempt and multipart part of each, and every pack-id retry of
Ankusa.ClaimCheck.check_in/4 together.
Types
@type read_error() :: :not_found | :forbidden | {:unavailable, term()}
@type store() :: {index :: non_neg_integer(), module(), keyword()}
A claim store: its index in claim_check.blob_stores, module and options.
Functions
@spec child_spec(keyword()) :: Supervisor.child_spec()
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec get_range( atom(), Ankusa.Config.t(), non_neg_integer(), String.t(), non_neg_integer(), pos_integer() ) :: {:ok, binary()} | {:error, read_error()}
length bytes at offset of key in store index: one ranged read. A
store index past the end of the list (one removed since the claim was
located) is :not_found.
@spec home(Ankusa.Config.t(), String.t()) :: non_neg_integer()
The index of the store a new pack pack_id goes to first (see Placement).
@spec list(Ankusa.Config.t()) :: [store()]
The claim stores, indexed in config order.
@spec read_range( atom(), Ankusa.Config.t(), String.t(), [String.t()], non_neg_integer(), pos_integer() ) :: {:ok, binary(), non_neg_integer(), String.t()} | {:error, read_error()}
Find pack pack_id under the first of keys any store has, and read
length bytes of it at offset: each key in turn, across every store in
the pack's write order, healthy stores first (see Reads). Returns the bytes
with the store index and key they came from.
@spec start_link(keyword()) :: GenServer.on_start()
Whether a read error is the store throttling this node (429, 503, or the
adapter's {:throttled, seconds} for one that carried a Retry-After): a
reachable store asking for less load, not one that is down.
@spec write(atom(), Ankusa.Config.t(), String.t(), String.t(), iodata(), keyword()) :: {:ok, non_neg_integer()} | {:error, :already_exists | {:unavailable, term()}}
Write a pack's bytes under key, only if nothing is there
(Ankusa.BlobStore.put_if_absent/4): the home store first, then the next ones,
skipping stores whose breaker is open (unless all are). Returns the index of
the store that took it. A store that already holds an object under key is
{:error, :already_exists} at once, with no failover: the pack id collides,
and the caller mints a fresh one. When every attempt fails:
{:unavailable, reason} after a single attempt, else
{:unavailable, [{index, reason}, ...]} in attempt order; a store that
throttled ({:throttled, seconds}) is a reachable store and does not open
its breaker.
opts: :deadline, the absolute deadline every store's operation stays under
(see write_deadline/1).
@spec write_deadline(Ankusa.Config.t()) :: integer()
The absolute deadline (System.monotonic_time(:millisecond)) of one check-in:
the largest op_deadline_ms among the claim stores
(Ankusa.BlobStore.Options) from now. It covers every store tried, every
attempt and multipart part of each, and every pack-id retry of
Ankusa.ClaimCheck.check_in/4 together.