Architecture Overview#

This section provides an overview of the architecture of materialize-monitoring and its components.

materialize-monitoring Helm Umbrella Chart#

An Umbrella Helm Chart is a Helm Chart that orchestrates the installation of multiple dependent charts.

The materialize-monitoring Helm Chart is an Umbrella Chart that orchestrates the installation of the following dependent charts:

• alloy-agent (Grafana Alloy, Agent DaemonSet): o11y Pipelines
• alloy-gateway (Grafana Alloy, Gateway Deployment): o11y Pipelines
• metrics-server (metrics-server): cAdvisor/container runtime Metrics
• kube-state-metrics (kube-state-metrics): Kubernetes Metrics
• node-exporter (node-exporter): Node Metrics
• loki (Grafana Loki): Default Logging Infrastructure
• thanos (Thanos): Default Metrics Storage and Querying Infrastructure
• grafana (Grafana): Default Dashboarding and Visualization Infrastructure
• grafana-operator (Grafana Operator): Dashboards-as-Code Infrastructure
• alertmanager (Prometheus Alertmanager): Default Alerting Infrastructure

In addition to these dependent charts, materialize-monitoring also provides many opionated configurations such as o11y pipelines, Grafana dashboards, Scrape configurations, and Prometheus recording and alerting rules.

alloy-agent: Grafana Alloy Agent DaemonSet#

alloy-agent is a Grafana Alloy Agent DaemonSet that runs on every node in the cluster and is responsible for collecting logs from the node and forwarding them to the alloy-gateway.

The agent does not collect its own logs. Those arrive by a second, independent route: the gateway reads them straight from the Kubernetes API — see the agent log sidechannel below.

The agent log sidechannel#

Every other log in this stack reaches Loki because an agent tailed a file on its node and pushed it to the gateway. That makes the agent a single point of failure for exactly one set of logs: its own. When an agent OOMs, crash-loops on bad config, wedges on a full disk, or simply cannot reach the gateway, the logs explaining why are the ones it was about to send. The failure erases its own evidence, and the operator sees silence where they need a cause.

So the gateway tails the agent DaemonSet’s logs directly from the Kubernetes API, using loki.source.kubernetes against a discovery.kubernetes selector scoped to those pods. The two paths share nothing: a different process, on a different host, over a different transport, with a different failure domain. It also covers the windows where the agent legitimately is not running — a first install, a DaemonSet rollout, a node that has just come up — during which there is no file to tail yet.

Three properties are deliberate:

• Always on, not a failover. A path that only activates during an incident is a path nobody notices is broken. Running it constantly means its health is visible on an ordinary day, and there is no failure detection to get wrong.
• The agents do not also collect their own logs. With the sidechannel always on, an agent tailing its own log file would deliver every line twice. The agent’s discovery drops its own pods, so there is one path rather than two.
• Scoped to the agent DaemonSet only, server-side. The label selector goes to the API server, so the watch never returns other pods. The target set is one pod per node emitting alloy’s own modest output — which is what makes reading logs through the API server affordable here. This is not a general log-collection mechanism: pointed at application pods it would put the cluster’s entire log volume through the API server.

The gateway runs several replicas, so the tail is clustered — otherwise each replica would tail every agent pod and every line would land in Loki once per replica.

alloy-gateway: Grafana Alloy Gateway Deployment#

alloy-gateway is a Grafana Alloy Gateway Deployment that is responsible for the main observability pipeline processing and forwarding.

Logging responsibilities of alloy-gateway include:

• Receiving logs on multiple endpoints, so that the alloy-agent, an application, or even another alloy-gateway (a chained-gateway topology) can send to it:
  • A loki.source.api component receives Loki push traffic on alloy-gateway.$namespace:3100.
  • An otelcol.receiver.otlp component receives OTLP logs on alloy-gateway.$namespace:4317 (gRPC) and :4318 (HTTP).
• A loki.source.kubernetes_events component collects Kubernetes events and processes them as logs.
• A loki.process pipeline performs log processing — level normalization, cardinality reduction, and structured-metadata extraction.
• A loki.write component forwards logs to log storage (e.g., Grafana Loki) — or, in a remote-only topology, to an external OTLP or Loki destination.

Metrics responsibilities of alloy-gateway include:

• prometheus.operator.servicemonitors and prometheus.operator.podmonitors components read ServiceMonitors and PodMonitors in order to determine what targets to scrape for metrics and then scrapes those targets.
• A prometheus.enrich pipeline performs metric processing and enrichment on scraped metrics.
• A prometheus.remote_write component forwards metrics to metric storage (e.g., Thanos).
• An otelcol.exporter.otlp component supports forwarding to an external OTLP endpoint (e.g., Honeycomb, Datadog, New Relic, etc.) for metrics and logs.

Alloy supports further customization to integrate with an existing observability infrastructure.

metrics-server: Container Metrics API#

metrics-server is a Kubernetes Metrics API implementation that collects resource usage metrics from the kubelet on each node and exposes them via the Kubernetes Metrics API.

Do note that the metrics-server is primarily intended for decision-based components (like Horizontal Pod Autoscaler) and does not store historical metrics data. Nonetheless, Materialize relies on cluster-local metrics about its containers so this is required to not rely on external metrics sources for this data.

kube-state-metrics: Kubernetes Metrics#

kube-state-metrics is a service that listens to the Kubernetes API server and generates metrics about the state of the objects in the cluster (e.g., Deployments, Pods, Services, etc.).

This does not provide information about resource usage of individual containers.

node-exporter: Node Metrics#

node-exporter is a Prometheus exporter that collects hardware and OS metrics from the nodes in the cluster. It runs as a DaemonSet, is scraped by alloy-gateway through a ServiceMonitor, and covers the layer nothing else in the stack sees: swap, memory pressure, node-level network counters, disk I/O, and conntrack.

Two things about it are worth knowing before you change anything:

• It is a separate DaemonSet on purpose. Alloy’s prometheus.exporter.unix is node_exporter, and folding it into the agent would be one workload fewer. Keeping them apart keeps their resource limits apart, so a metrics regression cannot starve log collection — the signal you most need during the incident it caused. See the Terraform modules design doc for the full argument.
• Collectors are an allowlist, not the upstream defaults. The chart passes --collector.disable-defaults and names each collector it wants, so a new default-on collector in a future node_exporter release cannot silently join the cardinality budget. The list, and the reasoning behind every inclusion and exclusion, is in the values reference under Node Exporter.

The pods run with hostNetwork: true, which the network collectors require — netdev, netclass, netstat, sockstat and conntrack all read namespaced files under /proc/net, and in a pod network namespace they would report the pod’s traffic rather than the node’s. A consequence worth stating plainly: the exporter listens on the node’s own interfaces, so a NetworkPolicy is not what keeps port 9100 private. Most CNIs, Cilium and Calico among them, do not apply pod NetworkPolicy to host-networked pods. The chart ships one anyway — it is enforced where the CNI supports it, and it documents intent everywhere else — but the node firewall or security group is the real control.

loki: Grafana Loki#

Grafana Loki is a horizontally scalable log-aggregation system, included in materialize-monitoring as its default logging backend. It runs in microservice / distributed mode, where each component scales independently.

For the full component breakdown — diagrams, the read/write paths, the hash ring, day-2 operations, and alternative topologies — see Logs & Events > Logging Architecture. At a glance:

• Write path — alloy-gateway pushes processed logs to the Loki Distributor, which fans each stream across Loki Ingesters through a consistent hash ring (replication factor 3); ingesters flush chunks and the TSDB index to object storage.
• Read path — Grafana issues LogQL to the Loki Query Frontend, which queues and splits work for Loki Queriers (assisted by the Loki Query Scheduler and Loki Index Gateway); queriers read recent data from ingesters and historical data from object storage.
• Backend / maintenance — the Loki Compactor (singleton) compacts the index and enforces retention; the Loki Ruler evaluates LogQL alerting and recording rules, sending alerts to Alertmanager and recording-rule samples back through alloy-gateway to the long-term metric store (Thanos).

Loki writes all durable data to object storage (S3-compatible, GCS, or Azure Blob).

thanos: Thanos#

Thanos is a highly available Prometheus setup with long-term storage capabilities.

Thanos is included in materialize-monitoring as its default metrics storage and querying backend.

Refer to Thanos Design for more details on the architecture of Thanos.

The Thanos Receiver path includes:

• A thanos-receive statefulset that receive metrics in Prometheus Remote Write format.
  • The Shipper subcomponent writes metrics to the object storage layer.
  • The Store API subcomponent provides an API for querying recent metrics.

The Thanos Query path includes:

• An optional thanos-query-frontend deployment is an optional caching and fan-out layer for queries.
• A thanos-query scalable deployment that receives queries.
  • The Query API subcomponent handles PromQL queries.
  • The Store API component is used for gRPC internal communication between components.
• A thanos-storegateway deployment that serves metrics from the object storage layer.

Additional components include:

• A thanos-compactor singleton deployment that operates against the storage layer to compact, manage retention, and downsample metrics.
• A thanos-ruler deployment that runs the Ruler component for alerting and recording rules.
  • The Ruler subcomponent evaluates alerting and recording rules against incoming metrics.

grafana: Grafana#

Grafana is a multi-platform open source analytics and interactive visualization web application.

Grafana is included in materialize-monitoring as its main dashboarding and visualization tool.

Grafana is mainly deployed as a Deployment and is recommended to be backed with a compatible database for durability and scalability.

We use grafana-operator to manage resources on a Grafana deployment.

grafana-operator: Grafana Operator#

Grafana Operator is a Kubernetes Operator that manages Grafana instances and their resources (e.g., Dashboards, Datasources, etc.) as Kubernetes Custom Resources.

The operator itself is just a simple Kubernetes Deployment named grafana-operator that watches for Grafana Custom Resources and applies them to the Grafana instance.

It manages these kinds of resources:

• A Grafana defines how to set up a Grafana instance or connect to an existing Grafana instance.
• A GrafanaManifest defines a k8s-style (12+) Grafana Dashboard that can be applied to the Grafana instance.
• A GrafanaDashboard defines an old-style (<12) Grafana Dashboard that can be applied to the Grafana instance.
• A GrafanaDatasource defines a Grafana Datasource that can be applied to the Grafana instance. We typically configure a datasource for Thanos and Loki.

alertmanager: Prometheus Alertmanager#

Prometheus Alertmanager is a tool that handles alerts sent by Prometheus and other monitoring systems.

TODO: determine architecture and integration of Alertmanager in materialize-monitoring.