# `Ankusa.Store.Backup`
[🔗](https://github.com/jamescarr/ankusa/blob/ankusa-v0.7.0/packages/ankusa/lib/ankusa/store/backup.ex#L1)

Continuous backup of the node's store to an object store, and its restore
into an empty `data_dir` at boot. Off unless `backup.enabled`.

Every `backup.interval_ms` this process asks the store for a checkpoint
(`Ankusa.Store.checkpoint/2`: memtables flushed, files hard-linked) and
uploads it through the `:backup` scope of `Ankusa.BlobStore` (the
`backup.blob_store`, else the segment store, under `storage.key_prefix`):

  * `backup/sst/<name>` — every `.sst` and `.blob` file. RocksDB never
    rewrites one, so each is uploaded once and shared by every backup that
    lists it.
  * `backup/<id>/<name>` — every other file (`MANIFEST-*`, `CURRENT`,
    `OPTIONS-*`, the WAL), per backup.
  * `backup/<id>/manifest.json` — every file with its key, size and sha256,
    the store's format version and intake offset, and (with
    `backup.signing_key`) an HMAC-SHA256 signature over all of it; written
    after all the files.
  * `backup/LATEST` — the id, written last. A backup is the one `LATEST`
    names; anything an interrupted attempt left is ignored, then deleted.

With `backup.secondary` every one of those objects is written to a second
object store as well (another region or provider), one target after the
other: the attempt succeeds only when both hold the new backup, a failed
secondary is retried with the backoff below, and each target is purged
right after its own success so a down secondary never makes the primary
grow. Each object-store call is bounded by `backup.timeout_ms`.

After each successful upload, backups beyond `backup.keep` and every shared
file no kept backup lists are deleted. A failed attempt is logged and
retried after the interval doubled per consecutive failure (jittered,
capped at a minute), like the compactor.

One store per prefix: the first backup gives the store an id (the
`m:store_id` row, so every checkpoint and every restore carries it) and
every manifest records it. When `LATEST` names another store's backup, the
attempt fails with `{:foreign_backup, theirs, ours}` — nothing uploaded,
nothing purged — because two stores on one prefix would delete each
other's backups as stale. Two live nodes can also share one id (an old
host back beside the replacement that restored its backup): once this
process has written a backup, `LATEST` naming one it did not write fails
the attempt with `{:prefix_shared, latest}`, so one of the two stops.

## Restore

At boot, `Ankusa.Store` calls `restore/3` when its directory holds no
database: the `LATEST` backup is downloaded, its manifest checked (the
signature against every `backup.signing_key`, the store format against what
this release supports), every file checked against its size and sha256,
`CURRENT` written last (a restore that fails part-way leaves no database,
and the next boot starts it over). With signing keys configured a manifest
that is unsigned or signed by no configured key is refused
(`{:manifest_invalid, id, :unsigned | :bad_signature}`); a backup of a newer
store format than this release writes is refused
(`{:store_format_newer, found, supported}`), the same guard the store's own
open applies. When the primary cannot give a backup (unreachable, invalid,
failing a check, or empty) and a secondary is configured, the secondary is
tried the same way. No backup anywhere starts an empty store; a backup
location that cannot be read refuses the boot instead.
`reconcile_archive/1` then squares the new store with the segments in the
bucket. A restore (or a fresh start) leaves a `RESTORE-IN-PROGRESS` marker in
the directory until that reconcile has succeeded, so a node that dies in
between reconciles on its next boot (`reconcile_pending?/1`) rather than
booting as an existing store.

The manifest is signed, `LATEST` is not: whoever can write the bucket cannot
inject content a restore accepts, but can still point `LATEST` at an older
backup this node signed.

## Verify

`verify/2` reads `LATEST` back from an object store without writing
anything: the manifest (signature included) and every file it lists, one at
a time, against their size and sha256. With `backup.verify_interval_ms` this
process runs it for every target on that interval and emits
`[:ankusa, :backup, :verify]`; a failure is the counter
`ankusa_backup_verify_failed_total`.

## Stopping

A clean stop (`:normal`, `:shutdown`) takes one last backup before the
process exits, bounded by `backup.final_timeout_ms`, so the recovery point
of a node that is drained and stopped is the stop, not the last periodic
backup. The supervisor is told to wait that long (`child_spec/1`). The store
closes after this process, and the edge and dispatch before it.

## Recovery point and freshness

The recovery point is the last backup: hooks acked after it are lost with
the host, except for the archive copies `reconcile_archive/1` finds.
`age_ms/1` is what `Ankusa.Health` compares to `backup.max_age_ms`.

Emits `[:ankusa, :backup, :stop]` per attempt, `[:ankusa, :backup, :state]`
(`age_seconds` since the last successful backup) after it, and
`[:ankusa, :backup, :verify]` per verified target.

# `target`

```elixir
@type target() :: %{
  name: :primary | :secondary,
  mod: module(),
  opts: keyword(),
  prefix: String.t(),
  timeout: pos_integer()
}
```

One object store a backup is written to and read from.

# `age_ms`

```elixir
@spec age_ms(atom()) :: non_neg_integer() | :unknown
```

Milliseconds since the last successful backup, or since this process started
while it has not made one yet; `:unknown` when it is not running.

# `child_spec`

Returns a specification to start this module under a supervisor.

See `Supervisor`.

# `clear_restore_marker`

```elixir
@spec clear_restore_marker(Path.t()) :: :ok | {:error, term()}
```

Remove the marker `reconcile_pending?/1` looks for, once the reconcile succeeded.

# `reconcile_archive`

```elixir
@spec reconcile_archive(Ankusa.Config.t()) :: :ok | {:error, term()}
```

Square a store that started over (restored, or created empty with a backup
configured) with the archive segments already in the segment store. Called
by `Ankusa.Store` once the database is open, before the queue writer starts.
It is `Queue.Archive.reconcile/2` over this node's own segments (the
dated layout's `<node>` directory and the legacy `seg/` prefix):

  * `m:next_seq` moves past the highest archived seq, so no new segment key
    overwrites an archived one;
  * a segment the store has no catalogue row for is catalogued from its
    `.idx` object and its header, and the archive obligations it settles are
    cleared, so the compactor does not write those hooks again under the same
    first seq.

A segment whose index object is missing or does not decode (its writer died
between the two PUTs) is logged and skipped. Any other read failure is an
error: the store must not start with seqs it may hand out twice.

# `reconcile_pending?`

```elixir
@spec reconcile_pending?(Path.t()) :: boolean()
```

Whether the store directory `path` holds a database that a restore (or a
fresh start) created and the archive has not been reconciled with yet: the
node died between the two. `Ankusa.Store` then reconciles before serving.

# `restore`

```elixir
@spec restore(Ankusa.Config.t(), Path.t(), keyword()) ::
  {:ok, :restored | :fresh} | {:error, term()}
```

Restore the latest backup into `path`, a store directory that holds no
database. `{:ok, :restored}`; `{:ok, :fresh}` when the backup location is
readable and holds no backup; `{:error, reason}` otherwise
(`{:backup_unreachable, _}`, `{:manifest_invalid, id, _}`,
`{:store_format_newer, found, supported}`, `{:download_failed, key, _}`,
`{:size_mismatch, name}`, `{:checksum_mismatch, name}`, `{:write_failed, name, _}`,
`{:unrecognized_store_files, names}`).

The primary object store is tried first. With `backup.secondary`, any
outcome of the primary but a restored store sends the restore on to the
secondary, and an empty primary does not make the store fresh while the
secondary holds a backup. When neither gives a backup the error is the
primary's.

`:format` (default: the store format this release writes) is the newest
store format to restore: a backup of a newer one is refused.

A directory that holds database files but no `CURRENT`, and was not left by
an interrupted restore, is refused rather than cleared: it may be a damaged
store someone wants to recover, and deleting it is not this function's call.

Called by `Ankusa.Store` before it opens the database, with the config it
was started with: the instance need not be running.

# `run`

```elixir
@spec run(atom()) ::
  {:ok,
   %{id: String.t(), uploaded: non_neg_integer(), bytes: non_neg_integer()}}
  | {:error, term()}
```

Take one backup now, synchronously: the number of files uploaded (shared
files already in the object store are not) and their bytes, over every
target.

# `start_link`

# `validate_config!`

```elixir
@spec validate_config!(Ankusa.Config.t()) :: :ok
```

The checks `Ankusa.Config.new/1` leaves to this module.

# `verify`

```elixir
@spec verify(atom(), keyword()) ::
  {:ok, %{id: String.t(), files: non_neg_integer(), bytes: non_neg_integer()}}
  | :none
  | {:error, term()}
```

Check the latest backup of one object store without writing anything: the
`LATEST` object, its manifest (signature included, against every
`backup.signing_key`), and every file the manifest lists, downloaded one at
a time and compared to its size and sha256.

`:target` is `:primary` (default) or `:secondary`. `:none` when the store
holds no backup; otherwise `{:ok, %{id: id, files: n, bytes: b}}` or
`{:error, reason}` with the reasons of `restore/3`
(`{:manifest_invalid, id, _}`, `{:download_failed, key, _}`,
`{:size_mismatch, name}`, `{:checksum_mismatch, name}`,
`{:backup_unreachable, _}`). A `LATEST` that moved while the files were
read (a newer backup purged the one being checked) is read again once.

---

*Consult [api-reference.md](api-reference.md) for complete listing*