Ankusa.Metrics (ankusa v0.7.0)

Copy Markdown View Source

The built-in Prometheus mapping of Ankusa.Telemetry's events, and the reporter that serves it.

Every deployment wants metrics, so this ships in core rather than in an adapter package: one TelemetryMetricsPrometheus.Core reporter per instance, named :"ankusa_metrics_#{instance}" (bounded — one atom per configured instance) and scraped by the admin API's GET /metrics, or by the metrics listener's (Ankusa.Metrics.Router, metrics.enabled), which serves nothing else.

:telemetry handlers are global to the VM, so a reporter sees only its own instance: every definition keeps just the events whose :instance metadata is the reporter's, and two instances in one VM never count each other's traffic. A reporter also covers only the events its own node emits, so an edge node's scrape has ingest series and a dispatch node's has delivery series. There is no cross-node aggregation here; point Prometheus at every node, or at every node's admin port through a proxy.

Unit conversion

core's Prometheus reporter ignores :unit, so durations are converted to seconds by the measurement function itself (duration_seconds/1) rather than exported in System.monotonic_time/0's native units. The :unit options below therefore describe the value that actually reaches Prometheus.

Cardinality policy

A scrape must not grow with the number of tenants or sources: ten thousand tenants each sending one hook must not be ten thousand series. metrics.detail decides which identity labels a per-hook metric carries, and metrics.max_series_per_metric caps what is kept of them.

  • :tenant (the default): tenant_id on the ingest, verify-failure, quarantine, rate-limit and delivery counters, and the two per-tenant gauges. Duration histograms carry no identity.
  • :source: tenant_id and source_id on all of those, and on the ingest duration and delivery latency histograms.
  • :none: neither label, and no per-tenant gauges: the series set is the same for one tenant and for a million.

Under :tenant and :source, a metric keeps its first max_series_per_metric - 1 label sets that carry an identity; every later one is reported with tenant_id/source_id/tenant set to "other" (see Ankusa.Metrics.Cardinality). An identity is never an unauthenticated value: tenant_id and source_id are only set after the source has been looked up, and a request for an unknown one is a refusal (ankusa.ingest.refused.total) that carries neither.

The gauges that sample state (ankusa.tenant.pending, ankusa.dispatch.tenant.inflight) report the top 100 tenants and, past the ceiling, drop the tenants that were not admitted rather than overwrite a shared "other" gauge with one tenant's value.

Bounded labels

Every other label value comes from telemetry metadata the framework controls, but failure reasons are terms ({:unavailable, {:status, 503, body}}) and some tags are modules. Both would be an unbounded or unusable label, so every tag value goes through normalize/1: tuples collapse to their leading atom (:unavailable), modules become strings ("Ankusa.Sink.Http"), integers are rendered as numbers (only ever a store index or an HTTP status, both bounded), and anything else becomes :other. :outcome on ingest is already one of a fixed set (:committed, :duplicate, :quarantined, :rejected, :rate_limited, :tenant_overload, :quarantine_rate_limited, :dedupe_pending, :quarantine_full, :overload, :tenant_limit, :dedupe_quota, :tenant_erasing, :store_unavailable, and under wal: :dual :intake_unavailable or :intake_pending) because Ankusa.Edge.Ingest tags it before emitting. A request for a source that does not exist is counted on ankusa.ingest.refused.total, whose :reason is one of :unknown_source, :payload_too_large, :body_read_failed, :invalid_header, :store_unavailable, :ip_rate_limited, :node_rate_limited or :source_rate_limited, never on a series that carries the id from the URL.

Summary

Functions

Child spec for this instance's metrics: a supervisor over the cardinality policy's table (Ankusa.Metrics.Cardinality) and the reporter.

The metric definitions for instance's reporter. Each keeps only events whose :instance metadata is instance.

Registered name of this instance's reporter.

Render the current metrics in Prometheus text exposition format.

Functions

child_spec(opts)

@spec child_spec(keyword()) :: Supervisor.child_spec()

Child spec for this instance's metrics: a supervisor over the cardinality policy's table (Ankusa.Metrics.Cardinality) and the reporter.

start_async: false registers the metrics before the reporter's start_link/1 returns, and Ankusa.Instance starts this child before any other, so no sibling's events are missed. The two children restart together: the reporter keeps every series it has seen, and the policy must keep the record of which it admitted.

metrics(instance)

@spec metrics(atom()) :: [Telemetry.Metrics.t()]

The metric definitions for instance's reporter. Each keeps only events whose :instance metadata is instance.

reporter_name(instance)

@spec reporter_name(atom()) :: atom()

Registered name of this instance's reporter.

scrape(instance)

@spec scrape(atom()) :: String.t()

Render the current metrics in Prometheus text exposition format.