Ankusa.Queue.Writer (ankusa v0.7.0)

Copy Markdown View Source

The only seq assigner: commits batches of hooks to the store, and answers a caller only once its hooks are on disk (Ankusa.Queue.Syncer fsyncs the WAL), so a 2xx to the provider always means the bytes are on disk.

One process per instance, under Ankusa.via(instance, :queue_writer). The commit is a single atomic store batch: the hook, one pending delivery row and due key (with its active-tenant marker) per sink, the archive obligation when the :storage role runs, the counter deltas of all of them, the m:next_seq marker and, for a tenant seen for the first time, its ordinal (this process is the only one that allocates them, see queue/tenants.ex). A failed commit consumes its seqs and ordinals (gaps are allowed) but acks nothing. A hook whose tenant would pass tenants.max is refused with {:error, :tenant_limit} in its place of the reply, and nothing of it is written.

Group commit

A call carries what needs no seq, prepared by its caller (Ankusa.Queue: the hook's binary, its encoded delivery rows, the dedupe keys read from the store). The writer takes the call that woke it and every :enqueue call queued behind it, from every batcher partition, up to batcher.group_commit_max_bytes of bodies and batcher.group_commit_max_records hooks, and commits them as one group: one tenant admission, one dedupe validation, dense seqs across all of it, one store write (applied with sync: false) and, under wal: :dual, one publish. The writer hands every caller's reply to Ankusa.Queue.Syncer, which fsyncs the WAL once for all the groups applied so far and then answers them: while group N syncs, group N+1 is assembled and applied here. A group's rows are not visible to dispatch, the archive or a retry's 201 duplicate before that fsync (Syncer.visible/2, ADR 0002 amendment 17).

The dedupe values a caller read are a prefetch: the first occurrence of a key in the group wins, and a key this process wrote since the caller read (in an earlier group, or in flight under wal: :dual) is answered from recent_* and the pending set, never from the stale read, so two partitions that saw the same key as absent still commit one hook.

Under wal: :dual (ADR 0002) the local commit runs in parallel with a confirmed publish to the intake log (Ankusa.Intake), and both are required before the 201. The writer does not wait: it records the batch in the pending set, returns {:noreply, _}, and answers the batcher's call when the publish task's result and the syncer's confirm of the local half have both arrived, so the next batch's local write overlaps the previous batch's fsync and publish wait. A half that fails is carried to completion by Ankusa.Intake.Reconciler, never undone; the caller's reply for such a hook is {:error, {:half, reason, hook_id}}.

Summary

Functions

Returns a specification to start this module under a supervisor.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

start_link(opts)