Ankusa.Instance.Drain (ankusa v0.7.0)

Copy Markdown View Source

What a node does before its supervision tree stops, so a stop never turns a request that is being answered, or a hook that is being committed, into a connection reset. Both applications run it from prep_stop/1 (core's for every instance in its own tree, AnkusaServer.Application's for the one it owns), which the VM calls when it is told to stop: SIGTERM, which is what docker stop and a Kubernetes pod deletion send.

In order:

  1. Ankusa.Health.drain/1: /ready answers 503 with "status": "draining" from this moment, so a load balancer stops choosing the node.
  2. Wait listener.drain_delay_ms (5 s by default), still serving: a load balancer needs a probe interval or two to notice, and a request it already routed must find the listener open.
  3. On a node with the :edge role, stop the ingress listener alone. It stops accepting, then waits Ankusa.Listener.ingress_shutdown_timeout_ms/1 (the batcher's record deadline, the store commit and a margin) for the requests it is serving, so a commit that starts late is not killed mid-fsync, and closes what is left.
  4. Wait until every batcher holds nothing (Ankusa.Edge.Batcher.queued/2), bounded by listener.drain_timeout_ms (30 s) from the end of the delay. Under wal: :none there are no batchers: a request waits for its sinks inside the listener's own wait.

It returns, and the tree stops in its own order: dispatch writes the outcomes it still holds when it terminates, the backup takes its final checkpoint (Ankusa.Store.Backup, bounded by backup.final_timeout_ms), and the store closes last. A node's grace period therefore has to cover the delay, the drain timeout and the stop itself: set terminationGracePeriodSeconds to at least drain_delay_ms + drain_timeout_ms, in seconds, plus 10 (and backup.final_timeout_ms when the node backs up).

run/1 never raises or exits: a drain that cannot finish logs why and lets the stop go on.

Summary

Functions

Drain instance (see the moduledoc). Blocks until it is done, and answers :ok whatever happened; an instance that never started has nothing to drain.

Functions

run(instance)

@spec run(atom()) :: :ok

Drain instance (see the moduledoc). Blocks until it is done, and answers :ok whatever happened; an instance that never started has nothing to drain.