materialize-monitoring Documentation

Authoring

Mon, 01 Jan 0001 00:00:00 +0000

Authoring Pipelines#

This page covers the model and conventions for authoring alloy pipelines as YAML under packages/alloy-pipelines/. The runtime aspects (label families, retention) live in Logging and Metrics.

Pipeline model#

A pipeline is a YAML document that maps roughly 1:1 to an alloy config file:

description: |
 What this pipeline does, why it exists, what it forwards to.
logging:
 level: info
 format: logfmt
blocks:
 - loki.process:
 label: input_processor
 forward_to: [{ref: "loki.write.gateway.receiver"}]
 blocks:
 - stage.drop:
 older_than: "12h"
 drop_counter_reason: "backlog > 12hr"
 - stage.match:
 selector: '{app="alloy"}'
 blocks:
 - stage.logfmt:
 mapping: { msg: msg, level: level }

Each entry in blocks is a single-key object whose key is either:

Collecting

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Common Queries

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Configuring

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Grafana Operator

Mon, 01 Jan 0001 00:00:00 +0000

Managing Dashboards with Grafana Operator#

Configuring Grafana Operator via `materialize-monitoring` Helm Chart#

TODO

Connecting your Grafana Instance to Grafana Operator#

TODO

Importing Dashboards via Grafana Operator#

TODO

materialize-monitoring values

Mon, 01 Jan 0001 00:00:00 +0000

`materialize-monitoring` Helm Reference#

Unified observability stack for Materialize: Alloy-based pipeline, optional bundled backends (Loki / Thanos / Grafana / Alertmanager), and the dashboards / alerts / rules that consume the Materialize metrics surface.

Source Code#

https://github.com/MaterializeInc/materialize-monitoring/tree/main/charts/materialize-monitoring

Requirements#

Kubernetes: >=1.27.0-0

Storage Requirements#

Using Thanos or Loki (in the default bundled-stack profile) requires object storage (such as an AWS S3 bucket) for long-term storage.

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Production Best Practices

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Prometheus Remote Write

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Repo Layout

Mon, 01 Jan 0001 00:00:00 +0000

This page is a convenience cache of where things live in the repo. The repository is under active development, so this layout goes stale easily. If it disagrees with what is actually in the tree, trust the repo and update this page.

The repo is a polyglot monorepo: a Rust workspace, a uv-managed Python workspace, and a Go-tooled Hugo docsite all share the root.

materialize-monitoring/
- Cargo.toml / Cargo.lock: top-level Rust workspace (members under packages/)
- pyproject.toml / uv.lock / .python-version: Python workspace, managed by uv
- go.mod / go.sum: Go module that pins hugo and helm-docs via tool directives
- Makefile: top-level entrypoint (make all, charts, dashboards, helm-docs, serve-docs)
- .pre-commit-config.yaml: contributor-experience hooks (see Internal Development)
- bin/: bash dev/CI entrypoints (flat; no subdirectories)
  - check-lfs.sh: verify/repair Git LFS state
  - dashboard-sync.sh: regenerate and sync dashboards
  - grafonnet-render: grafonnet rendering helper
  - mz-monitoring-build / mz-monitoring-check: thin wrappers over the Rust binaries
- packages/: hand-authored, contributor-facing inputs
  - grafana-dashboards/: Grafana dashboards-as-code (Python + grafana-foundation-sdk); sources under dashboards/ (e.g. mz_environment/, render.py, palette.py)
  - py-mzmon-lib/: Python helper library imported by the dashboard packages; not consumed by customers
  - alloy-pipelines/: Alloy pipeline YAML inputs (agent.yaml, gateway.yaml)
  - ref-alloy-pipelines/: Python reference pipelines being ported into the typed Rust path (agent_config.py, gateway_config.py, processor.alloy, alloy/)
  - mzmon-lib/: Rust library — typed Alloy model plus embedded JSONSchemas under schemas/alloy/; not consumed by customers
  - mz-monitoring-build/: Rust CLI for artifact generation (gen_pipelines.rs, main.rs)
  - mz-monitoring-check/: Rust schema/consistency checks
- charts/
  - materialize-monitoring/: umbrella chart
    - Chart.yaml / Chart.lock: chart metadata; lock pins subchart versions
    - values.yaml / README.md / README.md.gotmpl: profile-driven defaults; README generated from the template via helm-docs
    - charts/: vendored subchart tarballs (LFS) — alloy, loki, thanos, alertmanager, grafana, grafana-operator, kube-state-metrics, metrics-server
    - pre-rendered/: generated artifacts loaded via {{ .Files.Get }}; never hand-edited
      - dashboards/: grafana/ and datadog/ JSON
      - pipelines/: rendered Alloy (agent.alloy, gateway.alloy)
      - rules/: prometheus/, loki/, thanos/
    - templates/: provided resources — alerts/, dashboards/, pipelines/, scrapers/, plus grafana-grafana.yaml and _helpers.tpl
    - examples/: example values overlays (e.g. existing-grafana, IRSA/S3)
    - profiles/: profile value sets (still being defined)
  - materialize-monitoring-crds/: CRDs chart (Chart.yaml, values.yaml, README.md)
- docs/: Hugo docsite (the source of this page)
  - hugo.toml: site config
  - content/: authored Markdown
    - top-level sections: getting-started/, metrics/ (incl. collecting/), logs-and-events/, dashboards/ (incl. grafana/), alerting/, operating/
    - reference/: helm/, stable-metrics/, and internal/ (this section — dashboard/, pipelines/, design-docs/, plus repo-layout.md, roadmap.md, releasing.md, skills.md, helm.md)
  - layouts/, static/, assets/, data/, i18n/, archetypes/, themes/: Hugo machinery
  - public/, resources/: generated output (not checked in)
- legacy/: preserved field-engineering assets — sql_exporter/, prometheus/, grafana/, datadog/, tests/, docker-compose.yml
- tools/: ancillary ecosystems kept out of bin/
  - chartlib/: helm-docs templates
  - grafonnet/: grafonnet/jsonnet vendoring (jsonnetfile.json + lock)
  - shlib/: shared bash helpers
- .claude/skills/: authoring conventions consumed by both contributors and AI agents
- .github/: GitHub Actions workflows

SDKs and Schemas

Mon, 01 Jan 0001 00:00:00 +0000

SDKs and Schemas#

The Grafana ecosystem has been undergoing major transitions in how dashboard configurations are managed circa 2025; web searches frequently turn up inconsistent or outdated documentation. This page pins down what we target, what we generate, and which SDKs we use to do it.

Targets#

Currently supported:

Grafana 13 (Dashboard v2 schema) — latest as of April 2026, primary target
Grafana 12 (Dashboard v2beta1 schema)

Planned (stubs are acceptable for now):

Scraping

Mon, 01 Jan 0001 00:00:00 +0000

Scraping Metrics with `materialize-monitoring`#

Work in Progress!

This content is a work in progress and is not yet ready for public consumption.

By default, materialize-monitoring is configured to scrape metrics from any ServiceMonitor or PodMonitor resources in the cluster. This allows you to easily add new metrics to your monitoring stack by simply creating a new ServiceMonitor resources.

Scrape Architecture with Grafana Alloy#

Work in Progress!

This content is a work in progress and is not yet ready for public consumption.

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Architecture Overview#

This section provides an overview of the architecture of materialize-monitoring and its components.

`materialize-monitoring` Helm Umbrella Chart#

An Umbrella Helm Chart is a Helm Chart that orchestrates the installation of multiple dependent charts.

The materialize-monitoring Helm Chart is an Umbrella Chart that orchestrates the installation of the following dependent charts:

alloy-agent (Grafana Alloy, Agent DaemonSet): o11y Pipelines
alloy-gateway (Grafana Alloy, Gateway Deployment): o11y Pipelines
metrics-server (metrics-server): cAdvisor/container runtime Metrics
kube-state-metrics (kube-state-metrics): Kubernetes Metrics
node-exporter (node-exporter): Node Metrics
loki (Grafana Loki): Default Logging Infrastructure
thanos (Thanos): Default Metrics Storage and Querying Infrastructure
grafana (Grafana): Default Dashboarding and Visualization Infrastructure
grafana-operator (Grafana Operator): Dashboards-as-Code Infrastructure
alertmanager (Prometheus Alertmanager): Default Alerting Infrastructure

In addition to these dependent charts, materialize-monitoring also provides many opionated configurations such as o11y pipelines, Grafana dashboards, Scrape configurations, and Prometheus recording and alerting rules.

Alert Channels

Mon, 01 Jan 0001 00:00:00 +0000

Alert Channels#

Datadog

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Dependencies

Mon, 01 Jan 0001 00:00:00 +0000

materialize-monitoring Dependencies#

Installing CRDs#

The following CRDs are required for materialize-monitoring to function properly.

A second materialize-monitoring-crds Helm chart is provided to install these CRDs separately from the main materialize-monitoring chart, which is recommended to manage the lifecycle of these CRDs separately from the main chart.

Configuring Storage#

Before you start, you need to be able to store your metrics and logs somewhere.

» I am running in a cloud environment with a managed Kubernetes service (EKS, GKE, AKS, etc.)#

TODO: setup bucket with IRSA

Logging

Mon, 01 Jan 0001 00:00:00 +0000

TODO

o11y Glossary

Mon, 01 Jan 0001 00:00:00 +0000

Glossary of Observability Terms#

A reference for the vocabulary you’ll see across these docs. The audience is a Materialize customer or operator: comfortable with SQL, conversant in DBA terminology, working knowledge of Kubernetes. We don’t define general database or Kubernetes terms (table, namespace, pod) — only what’s specific to monitoring, the Materialize internals you’ll meet in panels, or this stack’s deployment vocabulary.

Observability foundations#

Cardinality The number of unique time series (or log streams) a metric produces. The dominant cost-and-stability lever for any Prometheus-shaped stack: a single ill-chosen label that takes on millions of values can balloon storage and crash query backends. The materialize-monitoring pipeline applies a cardinality-reduction policy at the gateway layer before metrics egress to expensive backends — see Operating > Tuning.

o11y Troubleshooting

Mon, 01 Jan 0001 00:00:00 +0000

o11y Troubleshooting#

OpenTelemetry Endpoint

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Skills

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Storing

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Storing

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Style Guidelines

Mon, 01 Jan 0001 00:00:00 +0000

Dashboard Style Guidelines#

Conventions for building visually consistent, operator-friendly dashboards. The audience for the dashboards themselves is Materialize end users: database-literate operators with basic graph-reading fluency but minimal cloud / Kubernetes / observability expertise. SQL is fair game; jargon like “differential dataflow’s arrangement” needs a one-liner explanation when it appears.

Layouts#

Prefer automatic layouts over fixed positioning. Dashboard v2 provides more ergonomic options like Tabs and a formal automatic layout system.

Generating and Pushing Dashboards

Mon, 01 Jan 0001 00:00:00 +0000

Generating and Pushing Dashboards#

How dashboard code is structured, how we keep generation deterministic, and how dashboards flow from source code into a live Grafana instance.

Code structure#

Dashboards live in their respective packages within packages/. The current Python implementation lives in packages/grafana-dashboards as a uv workspace. Python helpers live in packages/py-mzmon-lib.

Within packages/grafana-dashboards, the top-level packages represent the family of concerns (e.g., mz_environment, infra). Within a family, each dashboard has its own sub-package (such as overview) with the main dashboard entrypoint suffixed with _dashboard.py.

Maintenance Windows

Mon, 01 Jan 0001 00:00:00 +0000

Maintenance Windows#

Meta Observability

Mon, 01 Jan 0001 00:00:00 +0000

Meta Observability#

Prometheus Scraper

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Querying

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Querying

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Terraform

Mon, 01 Jan 0001 00:00:00 +0000

Installing via Terraform#

Work in Progress!

This content is a work in progress and is not yet ready for public consumption.

This guide is for users who are using materialize-terraform-self-managed to set up their Materialize cluster and want to use Terraform to set up their monitoring infrastructure as well.

materialize-monitoring is not available in materialize-terraform-self-managed yet (As of May 2026), but it will be enabled by default in the future.

Compatibility

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Helm

Mon, 01 Jan 0001 00:00:00 +0000

Installing via Helm#

Work in Progress!

This content is a work in progress and is not yet ready for public consumption.

If you are not using the terraform module to install materialize-monitoring, you can use the provided Helm charts to install the monitoring stack in your Kubernetes cluster.

During early development (June 2026), the helm charts are the only way to install materialize-monitoring.

The terraform module is the recommended way to install materialize-monitoring.

Prometheus Operator

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Rules

Mon, 01 Jan 0001 00:00:00 +0000

Log and Event Rules#

Rules

Mon, 01 Jan 0001 00:00:00 +0000

Metric Rules#

Testing

Mon, 01 Jan 0001 00:00:00 +0000

Testing#

Dashboard code is tested with pytest. Unit tests are recommended to be placed next to their code with the _test.py suffix.

For broader code-quality tooling (ruff, pyright, configured linter rules), see Generating and Pushing Dashboards > Code quality.

TODO: This page is a stub. Expand with conventions for snapshot testing of dashboard JSON, fixture patterns, and integration testing against a live Grafana when those land.

Upgrading materialize-monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Upgrading materialize-monitoring#

Custom Resource Definitions (CRDs)

Mon, 01 Jan 0001 00:00:00 +0000

Custom Resource Definitions (CRDs)#

Kubernetes Custom Resource Definitions (CRDs) are a powerful way to extend the Kubernetes API with custom resources that can be managed using standard Kubernetes tools. materialize-monitoring relies on several CRDs to function properly, and this section provides an overview of these CRDs and how to install them.

ServiceMonitor (monitoring.coreos.com/v1)#

ServiceMonitors can be written by any application which indicates that it should have metrics scraped by Prometheus Operator / Alloy.

Helm

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Securing

Mon, 01 Jan 0001 00:00:00 +0000

Securing materialize-monitoring#

Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

Roadmap#

The goal of materialize-monitoring is first-class, opt-in observability for self-managed Materialize — logs, metrics, events, and alerts — for customers who want a one-stop-shop, without forcing our stack on customers who already run their own. This page is the current source of truth for what is built, what is in flight, and what is planned next. It supersedes the original Linear project (internal), which captured an earlier architecture that has since diverged (see below).

Tuning

Mon, 01 Jan 0001 00:00:00 +0000

Tuning your Observability Stack#

Releasing

Mon, 01 Jan 0001 00:00:00 +0000

Releasing#

Each artifact releases on its own version stream (see Versioning).

Changes-pending placeholders#

The top of CHANGELOG.md holds one unreleased placeholder per component that has changes, with the body _Changes Pending_. Placeholders are not populated on main — population happens in a version-update PR (below). A component with no changes since its last release has no placeholder, so the unreleased placeholders are not necessarily the very top entries and rarely-changed components are not churned every cycle.

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

Versioning in materialize-monitoring#

All artifacts of this repo have separate versioning streams. They generally follow semantic versioning (SemVer).

Components#

Components are defined in packages/components.yaml. Each component declares:

changelog — whether the component maintains its own changelog stream.
title — the human-readable name used in CHANGELOG.md headings.
version_paths — files whose version field is rewritten when the component is bumped (the write targets).
content_paths — the paths whose changes are attributed to this component (the attribution inputs).
content_exclude — paths to subtract from content_paths, typically generated outputs that belong to a dependency.
dependencies — other components whose bumps cascade into this one.
artifacts — glob patterns (repo-root-relative) for files attached as GitHub Release assets when the component is published; resolved file names must be unique.

Each changed file is attributed to the component with the longest matching content_paths entry, after dropping any component that excludes it. Generated outputs route to their source: the chart excludes its pre-rendered/ tree, and pre-rendered/dashboards and pre-rendered/pipelines are claimed by the dashboards and pipelines components, so a dashboard change appears under Dashboards (and rolls up into the chart via cascade) rather than as a first-class chart change.

Changelog

Mon, 01 Jan 0001 00:00:00 +0000

materialize-monitoring Changelog#

materialize-monitoring Optional CRDs v0.3.0 (Unreleased)#

Changes Pending

Dashboards v0.10.0 (Unreleased)#

Changes Pending

Prometheus Scrapers v0.1.1 (Unreleased)#

Changes Pending

materialize-monitoring Documentation

Authoring

Authoring Pipelines#

Pipeline model#

Collecting

Common Queries

Configuring

Grafana Operator

Managing Dashboards with Grafana Operator#

Configuring Grafana Operator via materialize-monitoring Helm Chart#

Connecting your Grafana Instance to Grafana Operator#

Importing Dashboards via Grafana Operator#

materialize-monitoring values

materialize-monitoring Helm Reference#

Source Code#

Requirements#

Storage Requirements#

Metrics

Production Best Practices

Prometheus Remote Write

Repo Layout

SDKs and Schemas

SDKs and Schemas#

Targets#

Scraping

Scraping Metrics with materialize-monitoring#

Scrape Architecture with Grafana Alloy#

Architecture

Architecture Overview#

materialize-monitoring Helm Umbrella Chart#

Alert Channels

Alert Channels#

Datadog

Dependencies

materialize-monitoring Dependencies#

Installing CRDs#

Configuring Storage#

» I am running in a cloud environment with a managed Kubernetes service (EKS, GKE, AKS, etc.)#

Logging

o11y Glossary

Glossary of Observability Terms#

Observability foundations#

o11y Troubleshooting

o11y Troubleshooting#

OpenTelemetry Endpoint

Skills

Storing

Storing

Style Guidelines

Dashboard Style Guidelines#

Layouts#

Generating and Pushing Dashboards

Generating and Pushing Dashboards#

Code structure#

Maintenance Windows

Maintenance Windows#

Meta Observability

Meta Observability#

Prometheus Scraper

Querying

Querying

Terraform

Installing via Terraform#

Compatibility

Helm

Installing via Helm#

Prometheus Operator

Rules

Log and Event Rules#

Rules

Metric Rules#

Testing

Testing#

Upgrading materialize-monitoring

Upgrading materialize-monitoring#

Custom Resource Definitions (CRDs)

Custom Resource Definitions (CRDs)#

ServiceMonitor (monitoring.coreos.com/v1)#

Helm

Securing

Configuring Grafana Operator via `materialize-monitoring` Helm Chart#

`materialize-monitoring` Helm Reference#

Scraping Metrics with `materialize-monitoring`#

`materialize-monitoring` Helm Umbrella Chart#