Where archive objects live in the object store.
A segment written by this release is
<storage.key_prefix>archive/v1/dt=<yyyy-mm-dd>/<node>/seg-<first_seq>-<last_seq>.seg
<storage.key_prefix>archive/v1/dt=<yyyy-mm-dd>/<node>/seg-<first_seq>-<last_seq>.idx
<storage.key_prefix>archive/v1/dt=<yyyy-mm-dd>/<node>/_manifest.jsondt is the UTC day of the segment's oldest hook (the hook id is a UUIDv7, so its
time is in the id; a segment of ids that carry no time goes under the day it was
written), <node> is intake.node_id or, without one, the host name, and the
seqs are zero-padded to 20 digits. The day comes before the node so a node that
has lost its catalogue can list one day's prefix and find every node's segments
for it (Queue.Archive.reconcile/2, the Ankusa.Storage.fetch/2
fallback) instead of needing the catalogue to say where to look.
_manifest.json is a constant-size summary of one node's day: how many segments,
their seq and id range and total bytes. It is rewritten after each segment and
lets a reader skip a node whose ids cannot hold the one it wants.
Segments an earlier release wrote, seg/<first>-<last>.seg (and .idx) under
the prefix, are still found by the catalogue and by reconcile.
Summary
Functions
The manifest with one more segment (a catalogue row) counted in.
The UTC day (yyyy-mm-dd) of an event id's time; today when the id carries none.
The prefix of one day's objects, every node's.
The days a fetch looks in for an id: its own and the day before; :error for an id with no time.
Decode a manifest object; {:error, :bad_manifest} for anything that is not one.
The full key of the manifest beside the dated segment seg_key.
Object name (no prefix) of the day manifest of node on dt.
An empty manifest for node on dt.
The name this node writes under: intake.node_id, else the host name, as a safe path part.
Parse a listed key: a dated segment, a legacy one, or nil for anything else.
The prefix every dated object is under (before storage.key_prefix).
Object name (no prefix, no extension) of the segment first..last of node on dt.
Types
@type parsed() :: %{ kind: :dated | :legacy, key: String.t(), idx_key: String.t(), first: pos_integer(), last: pos_integer(), dt: String.t() | nil, node: String.t() | nil }
Functions
The manifest with one more segment (a catalogue row) counted in.
The UTC day (yyyy-mm-dd) of an event id's time; today when the id carries none.
The prefix of one day's objects, every node's.
The days a fetch looks in for an id: its own and the day before; :error for an id with no time.
Decode a manifest object; {:error, :bad_manifest} for anything that is not one.
The full key of the manifest beside the dated segment seg_key.
Object name (no prefix) of the day manifest of node on dt.
An empty manifest for node on dt.
@spec node_name(Ankusa.Config.t()) :: String.t()
The name this node writes under: intake.node_id, else the host name, as a safe path part.
Parse a listed key: a dated segment, a legacy one, or nil for anything else.
@spec root() :: String.t()
The prefix every dated object is under (before storage.key_prefix).
@spec segment_name(String.t(), String.t(), pos_integer(), pos_integer()) :: String.t()
Object name (no prefix, no extension) of the segment first..last of node on dt.