Roadmap#

The goal of materialize-monitoring is first-class, opt-in observability for self-managed Materialize — logs, metrics, events, and alerts — for customers who want a one-stop-shop, without forcing our stack on customers who already run their own. This page is the current source of truth for what is built, what is in flight, and what is planned next. This is generally synced with the internal Linear Project.

Cadence and milestones#

Releases track a monthly cadence aligned to the 15th.

Milestones are named by maturity stage; the date is a soft target.

Item tables below reference milestones by their short tag (M1–M4); names and target dates live in the table above.

Status legend#

• ✅ Done · 🔨 In progress · ⬜ Planned
• ⛓️ Blocked on an upstream metric-contract dependency (see Metrics contract)

Workstreams#

Dashboards#

The env-top overview is shipped and carries the cloud ↔ self-managed convergence work. Grafana 11 (dashboard v1) parity is a hard requirement for publicly hosted dashboards — the dashboard sources must continue to render against the v1 dashboard API, not only newer versions — so that the dashboards can be managed in the Grafana public dashboards gallery.

We weight Day 2 operations over Day 1: upgrades, resizing, changing sources, changing external destinations, and managing users are the operations that matter most for a running deployment. Upgrades is pulled into M2; the rest are M3. Day 1 dashboards (Dependencies, Sizing) are M4.

Change operation dashboards focus on new objects being added or removed and initially populated (rather than steady state metrics) with some error detection.

Pipelines (Alloy)#

Alloy carries both metrics and logs. The agent and gateway pipelines are in place; the OTLP export path is the near-term work.

Scraping (ScrapeConfigs & ServiceMonitors)#

Metric collection is configured through two surfaces: ScrapeConfigs (consumed manually, e.g. dropped into a Prometheus/Agent config) and ServiceMonitors / PodMonitors (consumed by prometheus-operator, or by Alloy via prometheus.operator.servicemonitor; GCP uses PodMonitoring). These ship as the released Prometheus Scrapers component and are bundled into the chart.

The cAdvisor and node-exporter rows are parity gaps against the stack the Terraform repo ships today, which collected both. They are functional gaps in the chart’s own default path, not Terraform-specific — the Terraform work only makes the bundled path everyone’s default.

Long term, ServiceMonitors belong in the materialize-operator Helm chart rather than here. This repo carries them now to fill the gap, with the intent to hand them off once the operator owns that surface.

Charts / Helm#

Helm is prioritized over Terraform. The umbrella chart loads pre-rendered artifacts and bundles the productionalized stack as subcharts.

Terraform#

Designed in Terraform Modules for materialize-monitoring. The common module lives in this repo (terraform/modules/materialize-monitoring), next to the chart whose value paths it encodes; per-cloud wrapper modules live in materialize-terraform-self-managed and wrap it. This replaces the hand-rolled Prometheus + Grafana modules that repo ships today, which vendor a point-in-time dashboard copy and a legacy scrape config.

The module ships as part of the materialize-monitoring component, not as a component of its own — one version stream covering two artifacts, so ?ref=materialize-monitoring/vX.Y.Z installs chart vX.Y.Z and there is no mapping to maintain between our own two surfaces. The module derives its chart version from the chart’s own Chart.yaml, so that coupling is structural rather than a convention someone maintains on each bump.

Qualification happens here, not downstream: the Terraform repo’s cloud integration tests consume released tags and assume our changes are already qualified. See Testing / CI.

Rules & alerts#

Profiles#

The profile set is deliberately not finalized — final shape is an M4 activity. The convention that has settled: the chart defaults target a medium install, and profiles are deltas away from it in both directions, each documenting the envelope it is sized for. Loki follows this today; Thanos has no sizing profiles at all (the chart sets no resources or replica counts for it).

Scheduling and storage class are profiles rather than a global.* block so the subchart fan-out map is inspectable data that snapshot tests can pin, instead of an unverified projection living in a downstream consumer.

Testing / CI & DevEx#

The E2E suite subsumes what was previously tracked as a synthetic-data smoke test. It asserts query success everywhere and non-empty results only on self-monitoring series — Materialize scrapers stay off, since those are integration-tested downstream, so env-top assertions are structural while the stack’s own telemetry provides real data.

Adoption / productionalization#

The M2 target is a productionalized deployment for Cloud, an internal team, and initial external adopters. (Specific adopter commitments are tracked out-of-band, not in this public roadmap.)

Metrics contract (upstream dependency)#

Several dashboards depend on metric instrumentation that lives upstream in the materialize repo, not in this repository. The metric/label contract is the public API for everything here, so this dependency shapes the dashboard roadmap directly.

The environmentd-native public metrics endpoint delivered Tier 1 (pre-aggregating clusterd counters into environmentd). The carry-over is Tier 2: roughly 39 signal families that today exist only via the SQL-on-scrape sources slated for deletion (legacy /metrics/mz_* and the v2_mz_* exporter). To retire those sources, environmentd must emit these natively. High-leverage asks, in priority order:

• ✅ mz_object_info (id → fully-qualified name → type) — the single highest-leverage item; delivered upstream. It gives every other metric a stable group_left join target for names.
• ✅ A family of _info metrics (mz_cluster_info, mz_replica_info, mz_source_info, mz_sink_info, …) carrying names and parent-id references; delivered upstream.
• ⬜ Native source/sink status metrics (no genuine source exists today).
• ⬜ Native hydration and frontier/freshness signals.
• ⬜ Label-family harmonization (short vs long vs very-long forms).

The _info family is now available, so name enrichment is unblocked for every panel. The remaining drilldowns are still ⛓️ gated on the items above: Sources / Sinks await native status metrics, and Hydration / Freshness await the hydration and frontier signals — which is why they stay M3 (stretch) rather than M2 (required).

Versioning, changelog, and releases#

Built. Each artifact has its own SemVer stream — the Helm chart, the optional CRDs chart, dashboards, pipelines, scrapers, and the shared lib — declared in packages/components.yaml. Full mechanics are in Versioning and Releasing; this replaces the earlier single-umbrella-chart framing.

• Per-component streams. ✅ Merged PRs are attributed to components by path; CHANGELOG.md is the source of truth, with cumulative Included <dep> @ vPREV..vNEW dependency rollups.
• Automation. ✅ mz-monitoring-build propose-bumps opens one version-update/<component> PR per changed component on each merge to main; publish-release tags <component>/vX.Y.Z and creates a GitHub Release (attaching each component’s artifacts) when such a PR merges.
• Deprecation policy. ⬜ Still to commit: at least one minor-release cycle for breaking changes to the label/metric contract, with a release-process check, and a called-out “customer-facing surface” changelog subsection. The label/metric contract is the public API; this discipline should land before broad adoption.
• Downstream pinning. ⬜ The Terraform modules (M3) pin a specific chart version, so Terraform never tracks a moving target. The common module ships from this repo inside the materialize-monitoring component rather than on a stream of its own: the chart version is the release version, and the module’s Git tag is that same version. Per-cloud wrappers downstream pin the module by Git ref, so a single number identifies both surfaces and there is no window where the pair is mismatched. The trade is that a Terraform-only change publishes a chart release, and a breaking module change bumps the chart’s major — both handled in the changelog rather than by splitting the stream.

Follow-up documentation#

• Releasing and Versioning are written, covering the release mechanics and the per-component model. ✅
• CHANGELOG.md exists and is maintained by the release tooling. ✅
• A customer-facing contract/deprecation-policy page (in customer terms, distinct from the internal versioning.md) is still to write. ⬜
• Repo Layout refreshed against the tree (August 2026), including terraform/ and test/. ✅ It goes stale easily by design — re-check it whenever a top-level directory moves.
• Uninstalling is written: the grafana-operator finalizer deadlock, the ordered teardown, and recovery. ✅
• Choosing the next version records the pre-1.0 bump policy and that the changelog placeholder heading is the decision. ✅
• Alloy’s rollout requirement is called out in Production Best Practices as an inversion of the normal chart guarantee — the one place the chart cannot own its own rollout. ✅