What a node does before its supervision tree stops, so a stop never turns a
request that is being answered, or a hook that is being committed, into a
connection reset. Both applications run it from prep_stop/1 (core's for every
instance in its own tree, AnkusaServer.Application's for the one it owns),
which the VM calls when it is told to stop: SIGTERM, which is what docker stop
and a Kubernetes pod deletion send.
In order:
Ankusa.Health.drain/1:/readyanswers503with"status": "draining"from this moment, so a load balancer stops choosing the node.- Wait
listener.drain_delay_ms(5 s by default), still serving: a load balancer needs a probe interval or two to notice, and a request it already routed must find the listener open. - On a node with the
:edgerole, stop the ingress listener alone. It stops accepting, then waitsAnkusa.Listener.ingress_shutdown_timeout_ms/1(the batcher's record deadline, the store commit and a margin) for the requests it is serving, so a commit that starts late is not killed mid-fsync, and closes what is left. - Wait until every batcher holds nothing (
Ankusa.Edge.Batcher.queued/2), bounded bylistener.drain_timeout_ms(30 s) from the end of the delay. Underwal: :nonethere are no batchers: a request waits for its sinks inside the listener's own wait.
It returns, and the tree stops in its own order: dispatch writes the outcomes it
still holds when it terminates, the backup takes its final
checkpoint (Ankusa.Store.Backup, bounded by backup.final_timeout_ms), and the
store closes last. A node's grace period therefore has to cover the delay, the
drain timeout and the stop itself: set terminationGracePeriodSeconds to at
least drain_delay_ms + drain_timeout_ms, in seconds, plus 10 (and
backup.final_timeout_ms when the node backs up).
run/1 never raises or exits: a drain that cannot finish logs why and lets the
stop go on.
Summary
Functions
Drain instance (see the moduledoc). Blocks until it is done, and answers :ok
whatever happened; an instance that never started has nothing to drain.
Functions
@spec run(atom()) :: :ok
Drain instance (see the moduledoc). Blocks until it is done, and answers :ok
whatever happened; an instance that never started has nothing to drain.