The built-in Prometheus mapping of Ankusa.Telemetry's events, and the
reporter that serves it.
Every deployment wants metrics, so this ships in core rather than in an
adapter package: one TelemetryMetricsPrometheus.Core reporter per instance,
named :"ankusa_metrics_#{instance}" (bounded — one atom per configured
instance) and scraped by the admin API's GET /metrics, or by the metrics
listener's (Ankusa.Metrics.Router, metrics.enabled), which serves nothing
else.
:telemetry handlers are global to the VM, so a reporter sees only its own
instance: every definition keeps just the events whose :instance metadata
is the reporter's, and two instances in one VM never count each other's
traffic. A reporter also covers only the events its own node emits, so an
edge node's scrape has ingest series and a dispatch node's has delivery
series. There is no cross-node aggregation here; point Prometheus at every
node, or at every node's admin port through a proxy.
Unit conversion
core's Prometheus reporter ignores :unit, so durations are converted to
seconds by the measurement function itself (duration_seconds/1) rather than
exported in System.monotonic_time/0's native units. The :unit options
below therefore describe the value that actually reaches Prometheus.
Cardinality policy
A scrape must not grow with the number of tenants or sources: ten thousand
tenants each sending one hook must not be
ten thousand series. metrics.detail decides which identity labels a
per-hook metric carries, and metrics.max_series_per_metric caps what is kept
of them.
:tenant(the default):tenant_idon the ingest, verify-failure, quarantine, rate-limit and delivery counters, and the two per-tenant gauges. Duration histograms carry no identity.:source:tenant_idandsource_idon all of those, and on the ingest duration and delivery latency histograms.:none: neither label, and no per-tenant gauges: the series set is the same for one tenant and for a million.
Under :tenant and :source, a metric keeps its first
max_series_per_metric - 1 label sets that carry an identity; every later one
is reported with tenant_id/source_id/tenant set to "other" (see
Ankusa.Metrics.Cardinality). An identity is never an unauthenticated
value: tenant_id and source_id are only set after the source has been
looked up, and a request for an unknown one is a refusal
(ankusa.ingest.refused.total) that carries neither.
The gauges that sample state (ankusa.tenant.pending,
ankusa.dispatch.tenant.inflight) report the top 100 tenants and, past the
ceiling, drop the tenants that were not admitted rather than overwrite a
shared "other" gauge with one tenant's value.
Bounded labels
Every other label value comes from telemetry metadata the framework controls,
but failure reasons are terms ({:unavailable, {:status, 503, body}}) and
some tags are modules. Both would be an unbounded or unusable label, so every
tag value goes through normalize/1: tuples collapse to their leading atom
(:unavailable), modules become strings ("Ankusa.Sink.Http"), integers are
rendered as numbers (only ever a store index or an HTTP status, both
bounded), and anything else becomes :other. :outcome on ingest is already
one of a fixed set (:committed, :duplicate, :quarantined, :rejected,
:rate_limited, :tenant_overload, :quarantine_rate_limited,
:dedupe_pending, :quarantine_full, :overload, :tenant_limit,
:dedupe_quota, :tenant_erasing, :store_unavailable, and under wal: :dual
:intake_unavailable or :intake_pending) because Ankusa.Edge.Ingest tags it before emitting.
A request for a source that does not exist is counted on
ankusa.ingest.refused.total, whose :reason is one of :unknown_source,
:payload_too_large, :body_read_failed, :invalid_header,
:store_unavailable, :ip_rate_limited, :node_rate_limited or
:source_rate_limited, never on a series that carries the id from the URL.
Summary
Functions
Child spec for this instance's metrics: a supervisor over the cardinality
policy's table (Ankusa.Metrics.Cardinality) and the reporter.
The metric definitions for instance's reporter. Each keeps only events whose
:instance metadata is instance.
Registered name of this instance's reporter.
Render the current metrics in Prometheus text exposition format.
Functions
@spec child_spec(keyword()) :: Supervisor.child_spec()
Child spec for this instance's metrics: a supervisor over the cardinality
policy's table (Ankusa.Metrics.Cardinality) and the reporter.
start_async: false registers the metrics before the reporter's
start_link/1 returns, and Ankusa.Instance starts this child before any
other, so no sibling's events are missed. The two children restart together:
the reporter keeps every series it has seen, and the policy must keep the
record of which it admitted.
@spec metrics(atom()) :: [Telemetry.Metrics.t()]
The metric definitions for instance's reporter. Each keeps only events whose
:instance metadata is instance.
Registered name of this instance's reporter.
Render the current metrics in Prometheus text exposition format.