Roadmap#

The goal of materialize-monitoring is first-class, composable observability for self-managed Materialize: logs, metrics, events and alerts. It is best-practices-by-default for customers who want a one-stop shop, and it does not force this stack on customers who already run their own. The same stack is consumed through Helm, through the Terraform module in this repository, and through the per-cloud Terraform wrappers downstream. Every component can be turned off in favour of one a customer already runs.

This page lists the work still to do, by project and milestone. What has shipped is summarized once, in What 1.0 ships, and itemized in the changelog. Linear holds the status of each item; this page holds the reasoning that does not fit in a ticket.

Projects#

Two projects are closed. First-class Observability Infrastructure (FCO) built the platform and closed on August 31. Operational Observability (OO) hardened it, answered the operational questions, stamped 1.0, and closed on October 6.

Releases track a monthly cadence aligned to the 15th, and milestone dates are soft targets. A milestone is cited as its project tag and its Linear name, such as ADOPT · Loki cutover.

What 1.0 ships#

materialize-monitoring v1.0.0 stamps the chart and the Terraform module together, since the module’s Git tag is the chart version. From 1.0, a breaking change to the committed surface owes a deprecation cycle under the stability guarantees.

Four designs are fully shipped and now record why the stack is the way it is: Loki production infrastructure, Terraform modules, TLS authentication, and the stability and deprecation policy. The designs with work left are under Designs in flight.

Cloud adoption#

Tracked in Adopt materialize-monitoring in Cloud (ADOPT). Materialize Cloud moves its own monitoring onto this chart, on the same values.yaml surface self-managed customers consume. Cloud deploys through Pulumi, so a Pulumi component wraps the chart the way the Terraform module does (CLO-182). Cloud-only rules and configuration layer through values, with no privileged path (DEP-125). Cloud running the chart customers run is the strongest available test of the customer contract.

The migration is ordered by risk. Stateless components go first, because each can be replaced in place and rolled back. Each stateful component then gets a cutover of its own.

Grafana is not migrated. Cloud’s Grafana lives elsewhere and stays there; it gains datasources for Thanos and Loki.

The Thanos milestone has two chart gaps to close first, both in DEP-383. The destination map filters by importance tier alone, so nothing can send v2_mz_* to AMP and keep it out of Thanos. materialize.deploymentMode: cloud also makes every rule read v2_mz_, which a Thanos holding only mz_* cannot answer. Separately, a destination below the all tier drops every recorded series and ALERTS until DEP-368 gives them a tier.

Metrics for Cloud#

Tracked in Metrics for Cloud (CLOM): what Materialize emits, what it is called, and how it is read.

Tenant-scoped read path#

Designed in A Tenant-Scoped Query API and tracked as DEP-269. Every consumer of this stack so far is a Grafana the chart deploys, reading backends over a ClusterIP Service. The read path serves the Materialize console and a customer’s own Grafana through a tenant-scoped proxy instead. The proxy is a Rust service in this workspace, tested against prom-label-proxy as an oracle.

The read path is where this stack stops being optional. Console reads environment metrics from SQL today, which keeps a short window of history and is unavailable exactly when the environment is. Reading PromQL and LogQL instead makes a deployment without those endpoints a deployment with a broken Console. The design resolves that by mandating an interface rather than an implementation. The interface is a PromQL and a LogQL endpoint carrying the documented label contract, behind the proxy. Thanos and Loki are the bundled implementation, and a customer already running an equivalent points the proxy at theirs. This supersedes the earlier plan to expose a Prometheus endpoint for customers to scrape.

Metric label standardization#

One canonical label vocabulary for Materialize’s metrics, a reserved-label list, and an authoritative environment key. The key is environment_id from Materialize.spec.environmentId, with environment_name on mz_environment_info. Designed in A Label Vocabulary for Materialize Metrics, in review as #445. The work lands in two passes, ordered so that consumers never observe the second. This repository’s monitors first fill the canonical labels where they are empty, and Materialize then adopts the names upstream behind a label lint.

Metrics contract (upstream dependency)#

Tracked as DEP-207. The metric and label contract is the public API for everything here. Several dashboards depend on instrumentation that lives in MaterializeInc/materialize. About 39 signal families exist only through the SQL-on-scrape exporter path, which Cloud names v2_mz_*. v2_mz_ is not a newer variant; it names the exporter path, and the target is canonical mz_. environmentd has to emit those families natively before that path can retire. mz_object_info and the _info family are delivered, so name enrichment is unblocked everywhere.

balancerd does publish connection metrics, whose semantics CLO-282 questions. Its half of the last row may therefore be a scrape gap rather than an upstream one.

Materialize-side emission#

BYOC#

Tracked in BYOC Observability (BYOC). The observability control plane is where telemetry crosses from a customer-run cluster into Materialize’s. One channel serves two segments: BYOC environments, which Materialize operates, and self-managed installs, which opt in. Designed in Observability for Bring-Your-Own-Cloud.

Observability control plane#

A reduced copy of telemetry crosses into the control plane, and the customer’s full-fidelity copy always stays with them. Metrics cross selected by importance tier, which already works per destination. Logs cross as an allowlisted, redacted, level-filtered subset, and log egress is opt-out at a stated cost in support quality. Redaction attaches to the destination rather than the pipeline, so the reduced copy is a fork of the customer’s stream and never a downgrade of it. The gateway pair enforces the boundary, rather than ad-hoc network configuration.

Secret filtering#

Designed in Secret Filtering and tracked as DEP-306. Credentials are the one exception to redaction at the destination. They are filtered at ingest, before anything is stored, because a credential in the customer’s own Loki is their exposure. Each control-plane branch runs a second, stricter filter immediately before its writer, and a canary per layer proves both are in the path.

Call-home from self-managed#

Designed in Call-Home and tracked as DEP-270. BYOC environments are ones Materialize operates; self-managed installs are invisible to Materialize between escalations.

The channel is the BYOC channel; the feature is consent. A self-managed customer bought software they run themselves, so every byte that leaves is a concession. The design is a bounded, monotone, locally visible ladder (off, heartbeat, alerts, metrics, diagnostics) over the existing pipeline. The cheapest useful level is alerts, which has something to forward now that a default rule set installs. A TLS-intercepting corporate proxy defeats mTLS outright, which makes token exchange (DEP-287) a reachability requirement in this segment. Receiving a signal creates an obligation. A stated response model, and a stated non-response, is a prerequisite for the alerts level rather than a follow-up.

Backlog#

Tracked in Observability Backlog (Backlog): work with no committed date. Each milestone below is a candidate project of its own, and moves out of the backlog when it is prioritized.

External dependencies#

Designed in Monitoring Materialize’s External Dependencies and tracked as DEP-233. The client’s measurement of a dependency is the SLI, and the dependency’s own telemetry is the diagnosis. Several pieces shipped with 1.0: the client vantage point, the provider pulls, and the client and provider dashboards. So did the persist-failures split and the ext:consensus_* recording rules. What remains is the flavor-specific half. It stays opt-in behind the normalized ext:* contract. That contract keeps seven database flavors and five object stores from multiplying the dashboard and alert set.

No adapter flavor is the one that waits. Every shipped wrapper provisions managed PostgreSQL, while CockroachDB and CNPG both run in customer clusters. The on-premise shape has no Terraform wrapper, but terraform/test/generic-cloud already runs it. Its adapters are therefore the ones this repository can test end to end.

Dashboards#

Day 2 operations outweigh Day 1. Upgrades, resizing, changing sources and destinations, and managing users matter most to a running deployment, so the Day 1 dashboards come last. The change-operation dashboards focus on objects being added, removed and initially populated, with error detection, rather than on steady state. Troubleshooting is symptom-first where env-top is subsystem-first, so it sequences after the dashboards it links into. The drilldowns wait on the metrics contract, and Freshness is the one customers ask about most.

Alerting#

Designed in Alerting in Self-Managed. The evaluation and notification path shipped with 1.0, and what remains is breadth and operability.

Freshness and source rules exist but stay opt-in. Calibrated against a production fleet, an absolute threshold fires mostly on clusters that are behind by design. Only a declared tier can tell those apart. Sink health and correctness logs beyond the three Materialize log-derived defaults have no rule yet.

Collection#

Platform#

Designs in flight#

A design moves to Shipped when the rows it owns on this page are gone.