Alert Architecture#

This page describes how an alert moves through materialize-monitoring: evaluated by one of two rulers, sent to every replica of the bundled Alertmanager, routed by a configuration the chart renders from alerting, and delivered to a receiver. It covers why the notifier is shaped the way it is, what state it holds, and what happens to that state when things fail.

The other pages in this section cover each part in depth:

The production checklist for this component is Production Best Practices > Alertmanager.

End-to-end flow#

flowchart TB
  subgraph eval["Evaluation"]
    truler["Thanos Ruler<br/>PromQL"]
    lruler["Loki Ruler<br/>LogQL"]
  end

  hs["alertmanager-headless<br/>one address per ready replica"]

  subgraph am["Alertmanager — 2 replicas"]
    am0["replica 0<br/>zone A · own volume"]
    am1["replica 1<br/>zone B · own volume"]
    am0 <-->|"gossip :9094 TCP + UDP<br/>silences · notification log"| am1
  end

  cfg["alertmanager-config Secret<br/>rendered from alerting.*"]
  creds["alertmanager-receivers Secret<br/>receiver credentials"]
  recv["Receivers<br/>Slack · PagerDuty · webhook · email · …"]
  svc["alertmanager<br/>ClusterIP Service"]
  graf["Grafana<br/>Alertmanager datasource"]
  gw["alloy-gateway<br/>ServiceMonitor scrape"]

  truler -->|"dns+ lookup"| hs
  lruler -->|"SRV lookup"| hs
  hs -->|"every alert"| am0
  hs -->|"every alert"| am1
  cfg -.->|"mounted · reloaded in place"| am0
  cfg -.-> am1
  creds -.->|"*_file"| am0
  creds -.-> am1
  am0 --> recv
  am1 --> recv
  graf -->|"alerts · silences"| svc
  gw -->|"/metrics"| svc
  svc --- am0
  svc --- am1

Five properties of that shape are the design.

Alertmanager is the single notification surface. Both rulers notify it, so an operator configuring where alerts go configures one thing, whether the alert came from a metric or a log line.

Every replica receives every alert. The rulers resolve the headless Service to one address per replica and send each alert to all of them. See Why the rulers address every replica.

The replicas gossip, and deduplicate. Silences and the notification log replicate between them, so a notification goes out once and either replica can send it.

The configuration is rendered, not written. The chart builds Alertmanager’s configuration from alerting, checks it at render time, and reloads it in place. See Configuration.

Nothing about alerting lives in Grafana. Grafana reads and writes Alertmanager’s alerts and silences through a datasource, and keeps none of that state itself. It also lists the Loki ruler’s rules and their state, read-only, through the Loki datasource. That datasource points at the Loki Gateway, which routes Grafana’s rule-list requests to the ruler and refuses rule edits, because the alloy-gateway owns the rule definitions.

Why the rulers address every replica#

Alertmanager’s gossip replicates two things between replicas: silences and the notification log, which records what was sent to whom and when. It does not replicate alerts. A replica knows about an alert only if a ruler sent that alert to it.

That makes the address a ruler notifies part of the HA design rather than a detail.

So both rulers address the headless Service through DNS discovery, and re-resolve it as replicas come and go.

The headless Service publishes only ready replicas, so a replica that is starting or terminating drops out of both rulers’ target lists on its own.

The Loki ruler creates a tenant’s notifier only once that tenant has rule groups. Until then it has discovered nothing, and loki_prometheus_notifications_alertmanagers_discovered is absent rather than zero.

How two replicas send one notification#

Each replica evaluates the same routing tree against the same alerts and would notify independently. Gossip is what stops that.

A replica sends a group’s notification only after waiting its position in the cluster multiplied by --cluster.peer-timeout (15s by default), and only if the notification log does not already record that notification as sent. The first replica sends at once and gossips the log entry; the second finds it and stays quiet. If the first replica is down or cannot deliver, the second sends 15 seconds later.

This depends on the mesh port being open on both TCP and UDP. Memberlist joins peers and pushes full state over TCP, and gossips updates over UDP. With either one blocked the replicas do not converge, and every notification goes out once per replica. The failure presents as duplicate pages, which reads like a routing problem rather than a network one. The chart’s NetworkPolicy opens 9094 on both transports, and a validator warns when a narrowed policy does not.

High availability#

Two replicas is the default shape rather than a hardening step. A single Alertmanager is lost to an ordinary node drain, holds the only copy of every silence, and cannot report its own absence. A second replica survives the first and can report it gone.

Three is not the default. Gossip tolerates a partition by notifying from both sides rather than by electing a leader, so a third replica adds little availability. It adds a third copy of every notification whenever gossip is broken.

What each failure costs#

Zones and volumes#

The zone rule is whenUnsatisfiable: DoNotSchedule. A replica that cannot satisfy it goes Pending, which is the signal the cluster autoscaler reads to add a node in the deficient zone.

Each replica’s volume is zonal, which is compatible with a hard rule only under volumeBindingMode: WaitForFirstConsumer. That binding mode creates each volume in whichever zone the scheduler placed its pod, so each replica lands in its own zone and stays there. It is the default for every managed-cloud CSI class. Under Immediate binding both volumes can be provisioned in one zone before scheduling, and the second replica then stays Pending while the first keeps notifying.

Two differences from the zone rule on Thanos Receive are deliberate.

State and storage#

Alertmanager holds three kinds of state.

Gossip is what makes one replica’s loss invisible. The volume is what makes losing both survivable: silences exist nowhere else, and without the notification log the first evaluation after a total loss re-sends every notification that had already gone out.

Each replica keeps a 4Gi volume. Alertmanager needs kilobytes; 4Gi is the smallest disk GCP Hyperdisk and Azure managed disks provision. Both the volume’s existence and its size render into the StatefulSet’s volumeClaimTemplates, which Kubernetes refuses to change after install. Changing either requires deleting the StatefulSet with --cascade=orphan before the upgrade.

--data.retention (120h) is how long expired silences and notification-log entries are kept.

The cluster label#

Every alert carries a cluster label naming the Kubernetes cluster it came from. The value is clusterName, which Terraform’s cluster_name sets, and which is also stamped on every log line and every metric this stack collects.

Both fill the label only where an alert lacks it. An alert whose expression kept cluster from its series keeps that value, which is the same one wherever the series came from this cluster. Neither subchart can read pipeline.env, so the chart renders the name into a ruler-env ConfigMap in each ruler’s namespace, and each ruler reads it as an environment variable.

Within one cluster the label is constant, and it matters as soon as two clusters share anything downstream.

The default alerting.routes.root.group_by includes cluster for the second row. All three depend on clusterName being unique per cluster. Left at its default, default, on several clusters, the label is present and distinguishes nothing.

Configuration#

The chart renders Alertmanager’s configuration itself, from alerting, rather than passing it to the subchart.

The subchart renders its config block into a ConfigMap with toYaml, so nothing in it can be computed. The severity presets, the receiver classes and the render-time checks all have to be. So alertmanager.config.enabled is off, and the chart renders the alertmanager-config Secret and mounts it at /etc/alertmanager/config.

flowchart LR
  values["alerting.*<br/>(values)"] --> tpl["templates/alertmanager-config.yaml"]
  tpl --> val{"render-time checks"}
  val -->|"error"| fail["helm install fails"]
  val --> secret["alertmanager-config Secret"]
  secret -->|"kubelet refresh"| mount["/etc/alertmanager/config"]
  mount --> reloader["config-reloader sidecar"]
  reloader -->|"POST /-/reload"| am["Alertmanager"]

A change to alerting reaches the running replicas without a restart. The kubelet refreshes the mounted Secret, the config-reloader sidecar sees the change and asks Alertmanager to reload, and the whole path takes seconds to about a minute. The sidecar is needed because the subchart rolls its pods on a checksum of the ConfigMap it no longer renders.

The Secret also carries the chart’s notification templates, mzmon.gotmpl, ahead of the files under alerting.templates. They replace Alertmanager’s built-in subjects, Slack body and links, and the links point at Grafana’s alerting pages. The chart writes Grafana’s address into them at render time, from the same setting Grafana builds its own links from. Alert Channels describes them.

alertmanager.baseURL is left empty by default and is not pointed at Grafana: Alertmanager builds links into its own UI from it, and Grafana serves none of those paths. extraArgs.web.route-prefix: / keeps every endpoint at the root whatever baseURL is set to.

The routing tree#

The rendered tree has the same shape in every install.

The root route’s receiver is mzmon-null, which notifies nobody. While alerting.receivers is empty, rows 2 and 3 are left out and every alert lands there. Alerts on mzmon-null still fire and still show in Alertmanager and Grafana.

A class served by one receiver is a single route. A class served by several is a parent route with one child per receiver, each set to continue, because an Alertmanager route names exactly one receiver.

Alert Channels covers configuring each part.

What is checked, and when#

The chart does not model receiver types. A receiver’s config passes through verbatim, which is why every integration Alertmanager supports works, and why the last row exists.

Credentials#

Receiver credentials are referenced by path, never inlined. Nearly every Alertmanager credential field has a _file variant, and the chart mounts Secrets for those fields to read.

One Secret is mounted by default: alertmanager-receivers, at /etc/alertmanager/secrets/alertmanager-receivers/. It is optional, so the pods start before it exists. Alertmanager reads a *_file credential each time it sends, so creating or rotating the Secret takes effect without a restart.

The render fails when a receiver carries an inline credential, unless alerting.assertNoInlineCredentials is turned off. It also fails when a *_file path falls under no mounted volume, since that notification would fail at send time.

Amazon SNS is the one integration that uses the pod’s cloud identity instead, through IRSA or EKS Pod Identity on the alertmanager ServiceAccount. Cloud provider services lists the permissions for it and for the cloud email services.

TLS#

Alertmanager serves plaintext on its API port, 9093, by default. alerting.server.tls moves that port to TLS, and profiles/mtls.values.yaml turns it on along with the settings that have to move with it. The chart renders the listener settings into web.yml in the alertmanager-config Secret, which Alertmanager reads through --web.config.file.

Eight clients dial the port, and each one needs a matching change:

The render fails on each client left on the other scheme. Alertmanager reads its certificate, client CA and client-auth policy on every connection, so a renewal takes effect without a restart. Whether TLS is on at all is decided at startup, so turning it on or off rolls the replicas. It also rolls every Loki component, because the Loki ruler’s settings live in the configuration all of Loki shares.

Both rulers address the headless Service and reach each replica by pod IP (Thanos) or per-pod name (Loki). The certificate carries neither, so each ruler verifies Alertmanager as alertmanager, one of the Service names the certificate does carry. Each ruler presents its own component’s certificate from phase 1.

Phase 2 is as far as this port goes. The kubelet’s probes and the config reloader dial it and cannot present a client certificate. The render refuses RequireAndVerifyClientCert while either is in place. Gossip between replicas, on 9094, stays plaintext. Securing covers the phases for the whole stack.

Observing Alertmanager#

Alertmanager is scraped through a ServiceMonitor, once per replica. Both Services the subchart renders carry the same labels, so the monitor drops the headless Service’s targets to avoid scraping each replica twice.

Grafana’s Alertmanager datasource (connections.datasources.alertmanager) gives the alert list and the silence editor. It points at the ClusterIP Service, which is correct for a reader: every replica holds the same alerts and silences. Its scheme follows alerting.server.tls.

None of these series is alerted on yet. The deadman’s switch and the meta-alerts that read them arrive with the rule set.

Resource map#

The subchart’s names are pinned by alertmanager.fullnameOverride, like Loki’s, Thanos’s and Grafana’s, so they do not depend on the release name. That is what lets both rulers address alertmanager-headless literally, from inside subcharts where this chart’s helpers cannot run. The render warns when the rulers’ addresses and the pinned name disagree.

What is not built yet#