Ankusa.Intake.Recovery (ankusa v0.7.0)

Copy Markdown View Source

Boot recovery of wal: :dual (ADR 0002, section 8), and the same code offline as mix ankusa.recover.

A replacement node starts with the same intake.node_id as the lost one, restores from the backup (Ankusa.Store.Backup), then run/2:

  1. Identity. m:intake_node is stamped on first use; a different id refuses ({:intake_node_mismatch, stamped, configured}): restoring node a's backup into node b would interleave two seq spaces.
  2. Topology. Ankusa.Intake.verify/1 (skipped with intake.verify_topic: false); the node waits until the log answers.
  3. Replay the tail from m:intake_offset + 1 (:earliest when the store has no offset), in chunks of 500 in one synced batch each, the offset advancing in the same batch. A record is applied through one rule, keyed by seq and hook id: no row at hooks/<seq> writes the hook under its original seq (delivery rows from the source's current config, dedupe keys, counters, m:next_seq raised); the same id is skipped; a different id is a seq collision, written under a fresh seq, never over the other hook. A duplicate, reordered or twice-replayed record changes nothing.
  4. How far back. If the log's earliest position is above offset + 1 and the store was restored, {:intake_truncated, wanted, earliest} refuses (allow_gap: true goes on and says so); on an existing store it is a warning, because every hook in the gap is in the local store.
  5. Republish the local tail: every hooks row above m:intake_seq (a crash can leave local hooks the log never confirmed), then the watermark moves to next_seq - 1.

It runs as an Ankusa.Instance child after the source store (replay resolves sinks through it) and before the writer: the node serves nothing until it is done, and Health.ready/1 says intake: "failing" (reason recovering) meanwhile.

Summary

Functions

The boot sequence: identity, then (intake.recover_on_boot) topology, replay and republish. A log that cannot be reached is waited for, never a crash.

Returns a specification to start this module under a supervisor.

The one line mix ankusa.recover and the image's recover command print.

The current m:incarnation id, "0" for a store that never restored.

The options run_offline/2 takes, from the command line of mix ankusa.recover and of the server image's recover command: --node, --from (backup | earliest | a position), --dry-run, --allow-gap. {:ok, opts} or {:error, message}.

Steps 1-5 against an open store. Options: origin (:existing | :restored | :fresh), node (asserts the id), from (:backup, the default, :earliest or a position), dry_run, allow_gap. Returns the summary.

mix ankusa.recover: steps 1-5 against a stopped store, by the same code as boot. Opens the store (restoring it from the backup first when its directory is empty, as boot does), the intake adapter and the source store, runs run/2, and closes them. opts are run/2's; origin is derived.

Types

summary()

@type summary() :: %{
  records: non_neg_integer(),
  applied: non_neg_integer(),
  present: non_neg_integer(),
  collided: non_neg_integer(),
  unroutable: non_neg_integer(),
  skipped: non_neg_integer(),
  republished: non_neg_integer()
}

Functions

boot(instance, origin)

@spec boot(atom(), :existing | :restored | :fresh) :: :ok | {:error, term()}

The boot sequence: identity, then (intake.recover_on_boot) topology, replay and republish. A log that cannot be reached is waited for, never a crash.

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

format_summary(summary)

@spec format_summary(summary()) :: String.t()

The one line mix ankusa.recover and the image's recover command print.

incarnation(instance)

@spec incarnation(atom()) :: String.t()

The current m:incarnation id, "0" for a store that never restored.

parse_args(args)

@spec parse_args([String.t()]) :: {:ok, keyword()} | {:error, String.t()}

The options run_offline/2 takes, from the command line of mix ankusa.recover and of the server image's recover command: --node, --from (backup | earliest | a position), --dry-run, --allow-gap. {:ok, opts} or {:error, message}.

run(instance, opts \\ [])

@spec run(atom(), keyword()) :: {:ok, summary()} | {:error, term()}

Steps 1-5 against an open store. Options: origin (:existing | :restored | :fresh), node (asserts the id), from (:backup, the default, :earliest or a position), dry_run, allow_gap. Returns the summary.

run_offline(config, opts \\ [])

@spec run_offline(Ankusa.Config.t(), keyword()) :: {:ok, summary()} | {:error, term()}

mix ankusa.recover: steps 1-5 against a stopped store, by the same code as boot. Opens the store (restoring it from the backup first when its directory is empty, as boot does), the intake adapter and the source store, runs run/2, and closes them. opts are run/2's; origin is derived.

start_link(opts)