Roadmap#

The goal of materialize-monitoring is first-class, composable observability for self-managed Materialize — logs, metrics, events, and alerts — best-practices-by-default for customers who want a one-stop-shop, without forcing our stack on customers who already run their own. Composable is the operative word: the same stack is consumed through Helm with a lot of levers, through our Terraform module, and through the downstream per-cloud Terraform wrappers, and every component can be turned off in favour of one a customer already runs. This page is the current source of truth for what is built, what is in flight, and what is planned next.

The work spans two Linear projects, and this page tracks both:

• First-class Observability Infrastructure in Self-Managed — FCO, the platform. Building the thing.
• Operational Observability — OO, the successor. Hardening it, stamping 1.0, and making it answer operational questions.

Cadence and milestones#

Releases track a monthly cadence aligned to the 15th.

Milestones are named by maturity stage; the date is a soft target.

Linear restarts milestone numbering at M1 for every project, so milestones here carry a project prefix — FCO-M1–FCO-M4 and OO-M1–OO-M3. Item tables below reference milestones by those prefixed tags.

FCO — First-class Observability Infrastructure (the platform)#

OO — Operational Observability (the successor)#

Hardening and the 1.0 contract were briefly separate milestones and were merged back. They get worked in parallel, and the split was not carrying its weight: the hardening items are the reasons a 1.0 would be premature, and the contract items are what the number actually promises. Neither ships without the other.

Status legend#

• ✅ Done · 🔨 In progress · ⬜ Planned
• ⛓️ Blocked on an upstream metric-contract dependency (see Metrics contract)

Workstreams#

Dashboards#

The env-top overview is shipped and carries the cloud ↔ self-managed convergence work. Grafana 11 (dashboard v1) parity is a hard requirement for publicly hosted dashboards — the dashboard sources must continue to render against the v1 dashboard API, not only newer versions — so that the dashboards can be managed in the Grafana public dashboards gallery.

Infrastructure dashboards (infra-*)#

The table above is the product ask: dashboards a Materialize user needs, all of them scoped to an environment and all named env-*. An infrastructure admin running the cluster underneath has a different set.

The family is infra-*, scoped to the cluster rather than to an environment. infra-logs is the first of it and is built; the rest below are still proposals. Both questions the family raised are settled:

• infra-* ships by default, alongside env-*, in dashboards.selected. The alternative — useful to whoever operates a self-managed install, noise to whoever only runs Materialize on someone else’s platform — lost to the fact that a self-managed installation has no other operator.
• The two audiences are split by default rather than by a switch. env-logs opens on the Materialize namespaces and infra-logs subtracts them, so each dashboard answers for one audience while still reaching the other in a click.

None of the below is ticketed yet. The “collectable today” column is what a live self-managed install actually exposes, measured rather than assumed — it is the difference between a dashboard we could build this week and one that needs collection work first.

Ordering follows what is buildable and what an operator reaches for first. Nodes and Meta-monitoring need no collection work and answer the two questions that block everything else — is the platform healthy, and is the telemetry itself trustworthy. Pods is the natural drilldown target from both, and from env-logs. Autoscaling is buildable now as an events dashboard and becomes a real one when the controller is scraped. Networking and External components are gated on the collection gaps below.

Collection gaps these depend on#

Found while surveying a live install; each is a scrape-side gap in this repo rather than an upstream metric contract.

Materialize components beyond the environment#

env-top covers environmentd and clusterd. The rest of a Materialize deployment has no dashboard, and for the two public ones the reason is the same: they expose no metrics that reach Thanos. Both appear in env-logs and in env-upgrade’s reconciliation counters, so today they are observable only as logs and as the operator’s opinion of them.

⛓️ rather than ⬜ because the gap is upstream instrumentation, not collection: there is nothing to scrape. Both are listed in Metrics contract alongside the other asks.

A further set of Materialize cloud services would want dashboards too. They are deliberately not enumerated here: this repository is public, and neither their names nor their metric surfaces belong in it. Track them in the internal planning docs (internal), and if any of them ever ships in a self-managed deployment it earns a row above instead.

We weight Day 2 operations over Day 1: upgrades, resizing, changing sources, changing external destinations, and managing users are the operations that matter most for a running deployment. Day 1 dashboards (Dependencies, Sizing) stay last.

Upgrades is the sharpest of these and is tracked as Urgent. It is the Day 2 operation with the widest blast radius and the least visibility, and we have direct evidence of it blocking a production adoption decision — an operator watching a long upgrade with no way to tell whether it was progressing or stuck, and no idea what to do if it were. That framing is the requirement: the dashboard has to answer is it stuck and what do I do about it, not merely display version counts. “Is it stuck” is usually answered by orchestratord’s reconciliation state, which is why this dashboard and the controller instrumentation below should share signals.

Change operation dashboards focus on new objects being added or removed and initially populated (rather than steady state metrics) with some error detection.

Troubleshooting is the entry point rather than another sibling — it is symptom-first where env-top is subsystem-first, and every panel links onward into Logs & Events or the matching drilldown. That makes it dependent on those existing, so it sequences last within OO-M2.

Pipelines (Alloy)#

Alloy carries both metrics and logs. The agent and gateway pipelines are in place, the OTLP export path shipped with FCO-M2, and the metrics half now delivers — the gateway scrapes every kubelet’s cAdvisor endpoint. The near-term work is the OTLP/gRPC agent→gateway transport with a node-local WAL.

Scraping (ScrapeConfigs & ServiceMonitors)#

Metric collection is configured through two surfaces: ScrapeConfigs (consumed manually, e.g. dropped into a Prometheus/Agent config) and ServiceMonitors / PodMonitors (consumed by prometheus-operator, or by Alloy via prometheus.operator.servicemonitor; GCP uses PodMonitoring). These ship as the released Prometheus Scrapers component and are bundled into the chart.

The cAdvisor and node-exporter rows are parity gaps against the stack the Terraform repo shipped before the cutover, which collected both. They are functional gaps in the chart’s own default path, not Terraform-specific — the Terraform work only makes the bundled path everyone’s default, which is what moved them to OO-M1. Both have landed and both deliver: ~3.7k node_* and ~14k container_* series on the tier-2 cluster, and ~24.6k container_* on a real EKS cluster.

That closes the parity gap, and it makes container_* and node_* assertions writable in the E2E suite for the first time — worth adding, because the failure mode across this whole area is configured but returning nothing, which no render test can see. Note the trap the chart already documents: a distribution signing kubelet certs with an untrusted CA fails the scrape quietly, so up{job="cadvisor"} is the thing to check when bringing up a new one.

Long term, ServiceMonitors belong in the materialize-operator Helm chart rather than here. This repo carries them now to fill the gap, with the intent to hand them off once the operator owns that surface.

Charts / Helm#

Helm is prioritized over Terraform. The umbrella chart loads pre-rendered artifacts and bundles the productionalized stack as subcharts.

Grafana reachability and persistence are paired deliberately. Exposing Grafana without a durable backend turns a bundled extra nobody depended on into a primary interface that silently discards everything a user creates in it.

Terraform#

Designed in Terraform Modules for materialize-monitoring. The common module lives in this repo (terraform/modules/materialize-monitoring), next to the chart whose value paths it encodes; per-cloud wrapper modules live in materialize-terraform-self-managed and wrap it. This replaces the hand-rolled Prometheus + Grafana modules that repo shipped, which vendored a point-in-time dashboard copy and a legacy scrape config.

The module ships as part of the materialize-monitoring component, not as a component of its own — one version stream covering two artifacts, so ?ref=materialize-monitoring/vX.Y.Z installs chart vX.Y.Z and there is no mapping to maintain between our own two surfaces. The module derives its chart version from the chart’s own Chart.yaml, so that coupling is structural rather than a convention someone maintains on each bump.

Qualification happens here, not downstream: the Terraform repo’s cloud integration tests consume released tags and assume our changes are already qualified. See Testing / CI.

Rules & alerts#

The rule set ships; the routing that turns a firing rule into a page does not.

Alertmanager is bundled and the rules exist, but nothing routes them anywhere. Until that lands the alerting story is “we ship rules”, which is half a feature.

The two Alertmanager items split along “reaching a human” versus “surviving a bad day”, and are best worked together. Adoption is the higher-value half — until routing exists nobody is paged, which is why hardening is the lower priority of the pair despite Alertmanager being a single replica holding the only copy of its silences.

Profiles#

The profile set is deliberately not finalized — final shape is an OO-M1 activity, tracked as one issue (DEP-190) rather than four. The convention that has settled: the chart defaults target a medium install, and profiles are deltas away from it in both directions, each documenting the envelope it is sized for. Loki and Thanos both follow it now.

Scheduling and storage class are profiles rather than a global.* block so the subchart fan-out map is inspectable data that snapshot tests can pin, instead of an unverified projection living in a downstream consumer.

Testing / CI & DevEx#

Both scanners are scoped to findings this repository can act on, and both report everything else rather than gating on it. The upstream population is large and moves on someone else’s release schedule, so a gate that included it would turn CI red on a dashboard change — which is how scanners get switched off. Three follow-ups came out of building them: the suppressions are per-check rather than per-resource, so they also mask a finding if one of our own templates grows it (DEP-249); readOnlyRootFilesystem on grafana and alertmanager is probably fixable through values rather than permanently baselined (DEP-250); and SARIF findings point at temp render paths, so code scanning cannot map them back to a source line (DEP-251).

The E2E suite subsumes what was previously tracked as a synthetic-data smoke test (DEP-119, now closed as a duplicate). It asserts query success everywhere and non-empty results only on self-monitoring series — Materialize scrapers stay off, since those are integration-tested downstream, so env-top assertions are structural while the stack’s own telemetry provides real data.

The Rust suite deepens qualification rather than establishing it, which is why it sits in OO-M1 rather than gating the Terraform work. Tier 0 and the kind tiers are what the Terraform modules close against.

Observability for our own components#

The stack has better visibility into Materialize than into the things that run Materialize.

Reconciliation timing is the gap that matters most: today a stalled reconciliation is invisible until someone notices the effect of it downstream.

The two rows are substrate and projection. Reconciliation metrics describe the mechanics of the controller loop; environment lifecycle describes the objects that loop manages, which is what an operator actually asks about. Every other dashboard here is scoped to a single environment — nothing today answers “how are all of my environments doing”.

This section absorbed a separate project on orchestratord Day 1/2 metrics, closed as a duplicate. Everything else it scoped — upgrade progress, reconcile monitoring, error rates, installation prerequisites, Day 2 dashboards — was already covered by rows above.

Adoption / productionalization#

FCO’s target was a productionalized deployment for Cloud, an internal team, and initial external adopters. (Specific adopter commitments are tracked out-of-band, not in this public roadmap.)

Cloud deploys through Pulumi rather than Terraform or raw Helm, so neither existing consumption path reaches it. The two Cloud rows are complements: one is what Cloud deploys, the other is how it gets deployed.

BYOC#

A reduced copy of telemetry crosses into the control plane; the customer’s full-fidelity copy always stays with them. Metrics cross selected by importance tier — the tiers already exist and already work per-destination. Logs cross as an allowlisted, redacted, level-filtered subset, and log egress is opt-out for customers who will not permit it at any level of redaction, at a stated cost in support quality.

This revises an earlier position that logs never leave the customer network. Metrics alone do not make an escalation tractable: Upgrades above is the worked example of a question (“is it stuck, and what do I do”) that metrics describe and logs answer. What made the original position worth stating is preserved rather than dropped — the claim was never that logs are uniquely sensitive, but that nothing should cross unexamined.

The gateway pair is what enforces the boundary, rather than ad-hoc network configuration. Sanitization is what makes anything crossing safe, since the _info metrics that made dashboards legible are precisely the ones carrying customer names, and the same tension applies to log labels. Redaction attaches to the destination rather than to the pipeline, so the reduced copy is a fork of the customer’s stream and never a downgrade of it.

Call-home from self-managed#

The BYOC section above is about environments Materialize operates. This one is about the deployments Materialize cannot see at all: self-managed installs, which are invisible between escalations.

Nothing here is ticketed yet.

The channel is the BYOC channel; the feature is consent. A BYOC customer bought an operated service, so telemetry crossing the boundary is what they purchased. A self-managed customer bought software they run themselves, and every byte that leaves is a concession. The design is therefore a bounded, monotone, locally-visible ladder — off, heartbeat, alerts, metrics, diagnostics — rather than a pipeline, which already exists.

Two findings from drafting it belong on this page rather than only in the design doc.

The cheapest useful level is alerts, and nothing evaluates alerting rules today. That is the same gap the Rules & alerts row records, reached from the other direction: a call-home channel forwarding alert state from a stack that evaluates no rules forwards an empty set, which reads as good news. Alert state (the ALERTS series, over the existing metric fan-out) and alert notification (an Alertmanager webhook, after grouping and silences) are different signals, and the state series is the one that ships first because it reuses the channel.

A TLS-intercepting corporate forward proxy defeats mTLS outright, and that network shape is common in exactly this segment. That makes token exchange a reachability requirement for self-managed rather than the security refinement it is for BYOC, and it is the strongest argument for the oauth2.tls prerequisite already owed to the read path.

Receiving a signal creates an obligation. Collecting alerts nobody is paged on is worse than collecting nothing, because the customer took a disclosure risk on the assumption that someone is watching. A stated response model — and a stated non-response — is a prerequisite for the alerts level, not a follow-up.

Tenant-scoped read path#

Every workstream above is about collecting telemetry, and every consumer of it so far is a Grafana we deploy, reading backends it reaches over a ClusterIP Service. This one is about reading it from somewhere else: the Materialize console, and a customer’s own Grafana attached to our backends as a PromQL and LogQL datasource.

Nothing here is ticketed yet.

The read path is where this stack stops being optional. Console today renders environment metrics from SQL against the environment itself, which keeps a short window of history and is unavailable precisely when the environment is. Reading PromQL and LogQL instead makes a deployment without those endpoints a deployment with a broken Console, which is a change of posture for a repository whose stated goal is that every component can be turned off.

The design resolves that by mandating an interface rather than an implementation: a PromQL endpoint and a LogQL endpoint carrying the documented label contract, reachable through a tenant-scoped proxy. Thanos and Loki are the bundled implementation of it; a customer already running an equivalent points the proxy at theirs.

Two consequences reach other rows on this page. Cloud adoption becomes a prerequisite for Console dashboards in Cloud rather than a parallel track, and the metric and label contract under Metrics contract becomes load-bearing for a product surface rather than for dashboards alone.

This supersedes the earlier plan to expose a Prometheus endpoint for customers to scrape into their own environment. A scrape endpoint delivers current samples to whoever can reach inward and keeps no history across a gap. A federated or remote-read endpoint on the same proxy is strictly smaller than the query API, so federation survives as an option for customers with a dedicated Prometheus rather than as the integration story.

Two findings from drafting it belong on this page rather than only in the design doc. The audit log class is declared and unimplemented — pipeline.logging.tenancy.tenantMap.audit is a values key and GATEWAY_TENANT_MAP_AUDIT reaches the gateway’s ConfigMap, where nothing reads it, exactly as GATEWAY_UNFILTERED_PROM_METRICS did before DEP-232. The destination oauth2 block models no TLS for the token request, so a BYOC gateway cannot today exchange its license certificate for a short-lived token without falling back to a long-lived client secret.

The console row under Materialize components beyond the environment is a different subject with the same word in it. That row tracks Console as a component we cannot monitor; this section tracks Console as a consumer. Neither blocks the other.

Metrics contract (upstream dependency)#

Several dashboards depend on metric instrumentation that lives upstream in the materialize repo, not in this repository. The metric/label contract is the public API for everything here, so this dependency shapes the dashboard roadmap directly. Tracked as DEP-207.

The environmentd-native public metrics endpoint delivered Tier 1 (pre-aggregating clusterd counters into environmentd). The carry-over is Tier 2: roughly 39 signal families that today exist only via the SQL-on-scrape sources slated for deletion (legacy /metrics/mz_* and the v2_mz_* exporter). To retire those sources, environmentd must emit these natively. High-leverage asks, in priority order:

• ✅ mz_object_info (id → fully-qualified name → type) — the single highest-leverage item; delivered upstream. It gives every other metric a stable group_left join target for names.

• ✅ A family of _info metrics (mz_cluster_info, mz_replica_info, mz_source_info, mz_sink_info, …) carrying names and parent-id references; delivered upstream.

• ⬜ Native source/sink status metrics (no genuine source exists today).

• ⬜ Native hydration and frontier/freshness signals.

• ⬜ Label-family harmonization (short vs long vs very-long forms).

• ⬜ balancerd and console metrics — neither exposes anything that reaches Thanos, so the two components a user actually connects through are observable only as logs. Blocks the dashboards listed under Materialize components beyond the environment.

The operator-side instrumentation the env-upgrade dashboard reads is a separate upstream dependency. Tracked as CLO-188:

• 🔨 orchestratord_reconciliations_total, orchestratord_reconciliation_steps_total, and the two duration histograms. Plus Kubernetes events for reconciliation failures and Materialize lifecycle transitions. Built and verified against a real cluster, but not yet merged upstream — so env-upgrade ships with most of its panels empty until it lands. orchestratord_is_leader and environmentd_needs_update predate this and are available today.

The _info family is now available, so name enrichment is unblocked for every panel. The remaining drilldowns are still ⛓️ gated on the items above: Sources / Sinks await native status metrics, and Hydration / Freshness await the hydration and frontier signals.

Versioning, changelog, and releases#

Built. Each artifact has its own SemVer stream — the Helm chart, the optional CRDs chart, dashboards, pipelines, scrapers, and the shared lib — declared in packages/components.yaml. Full mechanics are in Versioning and Releasing; this replaces the earlier single-umbrella-chart framing.

• Per-component streams. ✅ Merged PRs are attributed to components by path; CHANGELOG.md is the source of truth, with cumulative Included <dep> @ vPREV..vNEW dependency rollups. Each PR’s entry also carries the author’s own release notes, read from the ### Release Notes section of its description, so the consumer-facing detail travels with the entry.
• Automation. ✅ mz-monitoring-build propose-bumps opens one version-update/<component> PR per changed component on each merge to main; publish-release tags <component>/vX.Y.Z and creates a GitHub Release (attaching each component’s artifacts) when such a PR merges.
• Downstream pinning. ✅ The Terraform modules pin a specific chart version, so Terraform never tracks a moving target. The common module ships from this repo inside the materialize-monitoring component rather than on a stream of its own: the chart version is the release version, and the module’s Git tag is that same version. Per-cloud wrappers downstream pin the module by Git ref, so a single number identifies both surfaces and there is no window where the pair is mismatched. The trade is that a Terraform-only change publishes a chart release, and a breaking module change bumps the chart’s major — both handled in the changelog rather than by splitting the stream.
• Deprecation policy. ✅ (DEP-127, OO-M1) Written up in Stability Guarantees and Deprecation Policy, landed as Stability guarantees (policy of record), Stability Guarantees and Deprecation Policy (customer-facing), and the committed-surface check. Surfaces are graded by how much control we have and by how a break presents — alerts fail silently and get the most care, metrics fail visibly and get the least — with a 30-day cooldown enforced by the generated artifacts we already commit rather than by anything new. Query IDs and chart value paths fall outside the customer-facing surface, since their consumers are our own dashboards and our own Terraform module. Still open: the alert and recording-rule naming decision, which is free only until the alerting path ships, and the upstream window for the load-bearing metric list.
• Stamping 1.0. ⬜ (DEP-205, OO-M1) The pre-1.0 bump policy lets breaking changes ride minors. At 1.0 that stops, and the label/metric contract, profile semantics, alert names, and chart value paths all acquire a deprecation cycle we owe. The window closes on its own: once enough customers have dashboards built on these labels the contract is frozen in practice whether or not it is frozen on paper, so the discipline should land before broad adoption rather than after.

Follow-up documentation#

• Releasing and Versioning are written, covering the release mechanics and the per-component model. ✅
• CHANGELOG.md exists and is maintained by the release tooling. ✅
• A customer-facing contract/deprecation-policy page (in customer terms, distinct from the internal versioning.md) is still to write. ⬜
• Repo Layout refreshed against the tree (August 2026), including terraform/ and test/. ✅ It goes stale easily by design — re-check it whenever a top-level directory moves.
• Uninstalling is written: the grafana-operator finalizer deadlock, the ordered teardown, and recovery. ✅
• Choosing the next version records the pre-1.0 bump policy and that the changelog placeholder heading is the decision. ✅
• Alloy’s rollout requirement is called out in Production Best Practices as an inversion of the normal chart guarantee — the one place the chart cannot own its own rollout. ✅
• A BYOC gateway-to-gateway design doc is owed under design-docs/, tracked as DEP-219. 🔨 Observability for Bring-Your-Own-Cloud is written and in review as a draft; it also covers DEP-124 and DEP-220. The BYOC section above is updated to match it: a reduced log subset crosses, where the earlier position was that logs never leave the customer network.
• A Tenant-Scoped Query API for Console and Customer Grafana is written and in review as a draft. 🔨 It proposes mandating a PromQL and LogQL read interface in self-managed and Cloud, and a JWT-authenticated single-tenant proxy in front of it. The Tenant-scoped read path section above is the roadmap position it establishes, including that it supersedes the customer-scraped Prometheus endpoint.
• A customer-facing read-endpoint page — how to obtain a token, the two Grafana datasource shapes, and what a tenant can and cannot read — is owed alongside it. ⬜
• Call-Home: Opt-In Telemetry from Self-Managed to the Control Plane is written and in review as a draft. 🔨 It proposes an opt-in consent ladder over the BYOC channel, with alerts as the lowest useful level and the bound made verifiable by a preview mode and a local egress meter. The Call-home from self-managed section above is the roadmap position it establishes, including that the alerts level is blocked on rule evaluation rather than on the pipeline.
• A customer-facing call-home page — the levels, the generated schedule for each, how to preview before enabling, how to read the local meter, and the retention, access and deletion commitments — is owed alongside it. ⬜