Common Queries#

These queries are used in various dashboards and may be a good starting point for a dashboard developer or for someone looking into particulars of a dashboard.

Most environments use mz_ for mzSqlPrefix and do not set mzEnvironmentFilter.

materialize-clusters#

Inventory and sizing of a Materialize deployment’s clusters and replicas. Adapted from the Overview dashboard’s “Cluster Objects / Replicas” tab.

materialize.clusters.count #

How many clusters exist, split into the Materialize-managed system clusters (mz_catalog_server, mz_system, mz_probe, …) that every environment has and the user clusters you created. The gap between the two is your own footprint.
count(
  group by (compute_cluster_id) (
    ${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id=~"${mzClusterList}"}
  )
)

count(
  group by (compute_cluster_id) (
    ${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id=~"${mzClusterList}", compute_cluster_id=~"^s.*"}
  )
)

materialize.clusters.replicas.count #

How many replicas back the selected clusters, and how many of those are redundancy beyond the first. Every cluster needs one replica to run; anything above that is capacity or availability headroom you’ve opted into, so a non-zero “additional” count is the quick check that HA is actually configured where you expect it.
count(
  group by (compute_cluster_id, compute_replica_id) (
    ${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id=~"${mzClusterList}", compute_replica_id=~"${mzReplicaList}"}
  )
)

count(
  group by (compute_cluster_id, compute_replica_id) (
    ${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id=~"${mzClusterList}", compute_replica_id=~"${mzReplicaList}", compute_replica_name!="r1"}
  )
)

materialize.clusters.replicas.sizes #

The replica fleet grouped by configured size. Most deployments settle on a handful of sizes; a long tail of one-off sizes usually means an experiment or a half-finished migration. The total agrees with the replica count.
count by (size) (
  ${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id=~"${mzClusterList}", compute_replica_id=~"${mzReplicaList}"}
)

materialize.clusters.info #

A reference row per (cluster, replica): ids, names, size, version, and scheduling metadata. The “what does my fleet actually look like” lookup — most useful for grabbing a cluster or replica id to scope the rest of a dashboard to.
${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id=~"${mzClusterList}", compute_replica_id=~"${mzReplicaList}"}

materialize-compute#

Queries for the compute side of a Materialize deployment — the indexes, materialized views, and subscribes that run as dataflows on cluster replicas, plus their freshness, hydration, and resource footprint.

materialize.compute.materialized_views.count #

Materialized views Materialize is actively maintaining. Each one is a query whose result is kept continuously up to date, so this tracks roughly how much standing compute the environment carries.
max(${mzSqlPrefix}mzd_views_count{${mzEnvironmentFilter}})

materialize.compute.indexes.count #

Indexes in the catalog. An index is an in-memory arrangement that makes reads against its relation effectively instant, in exchange for memory — so growth here is a leading indicator of cluster memory growth.
max(sum by (instance) (${mzSqlPrefix}indexes_count{${mzEnvironmentFilter}}))

materialize.compute.views.count #

Non-materialized views — query templates evaluated on demand. They cost nothing until something reads them, so this is a catalog-shape signal rather than a load one.
max(${mzSqlPrefix}views_count{${mzEnvironmentFilter}})

materialize.compute.subscribes.active #

Live SUBSCRIBE sessions — long-running queries that stream updates to a client as data changes. A handful of system subscribes are Materialize’s own internal probes; a persistently climbing user count is a classic leaked-connection signal.
sum by (session_type) (mz_active_subscribes{${mzEnvironmentFilter}})

materialize.compute.indexes.by_type #

Indexes split by the kind of relation they sit on. Workloads normally lean heavily on indexes over views (the standard “keep a query’s result hot” pattern); a large share of indexes on base tables is unusual and usually worth a second look.
sum by (relation_type) (${mzSqlPrefix}indexes_count{${mzEnvironmentFilter}})

materialize.compute.hydration.currently_hydrating #

Collections still rebuilding their in-memory state — a live hydration-queue proxy. After a restart, replica creation, or some DDL, a dataflow has to rebuild from persisted storage before it can serve, and until it does it has no output frontier.
count(
  max by (instance_id, collection_id) (
    mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}", instance_id!="", quantile="1"} > 1e15
  )
)

materialize.compute.hydration.queue_size #

Collections waiting in each replica’s hydration queue. environmentd schedules hydration in batches; a backlog means work is arriving faster than the replica can rebuild it.
sum by (instance_id, replica_id) (
  mz_compute_controller_hydration_queue_size{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}", replica_id=~"${mzReplicaList}"}
) > 0

materialize.compute.hydration.slowest_collections #

The 15 collections that took longest to finish hydrating. Hydration time scales with the size of the state being rebuilt, so large materialized views and indexes naturally top the list.
topk(15,
  ${mzSqlPrefix}compute_hydration_time_seconds{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}", replica_id=~"${mzReplicaList}", hydrated="1"}
)

materialize.compute.freshness.lag_by_cluster #

How far behind real time each cluster’s most-lagged collection is — the worst-case freshness across every index, materialized view, and source on the cluster.
max by (instance_id) (
  mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}", instance_id!="", quantile="1"} < 1e9
)

materialize.compute.freshness.top_collections #

The 15 collections whose output frontier is furthest behind real time — the per-collection breakdown behind the per-cluster freshness lag, labeled by object name.
topk(15,
  max by (instance_id, collection_id) (
    mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}", instance_id!="", replica_id=~"${mzReplicaList}", quantile="1"} < 1e9
  )
)

materialize.compute.dataflows.count #

Active dataflows on each replica. Every index, materialized view, and live SUBSCRIBE runs as one or more dataflows, so this count rises with DDL and subscribe activity.
max by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id) (
  mz_compute_replica_history_dataflow_count{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}
)

materialize.compute.dataflows.count_by_worker #

The dataflow count broken out per worker. Workers in a replica run in lockstep and should see exactly the same dataflows, so their series should overlap perfectly.
max by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, worker_id) (
  mz_compute_replica_history_dataflow_count{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}
)

materialize.compute.dataflows.elapsed_rate #

CPU-cores busy inside dataflows, per cluster — the whole of dataflow work: arrangement maintenance, query evaluation, and hydration. Capped by cluster size (a 400cc cluster can’t exceed 400 cores).
sum by (instance_id) (
  rate(
    ${mzSqlPrefix}dataflow_elapsed_seconds_total{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}", replica_id=~"${mzReplicaList}"}${interval}
  )
)

materialize.compute.arrangements.maintenance_rate #

CPU-cores spent maintaining arrangements — the in-memory indexed snapshots behind every index and materialized view — summed across a replica’s workers, so an N-worker replica can reach N.
sum by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id) (
  rate(
    mz_arrangement_maintenance_seconds_total{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}
  )
)

materialize.compute.arrangements.maintenance_rate_by_worker #

The same maintenance CPU, split per worker — each worker tops out at 1.0.
sum by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, worker_id) (
  rate(
    mz_arrangement_maintenance_seconds_total{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}
  )
)

materialize.compute.arrangements.records.system #

Row counts of arrangements for Materialize’s internal system collections (collection id starts with s). These back the catalog and internal probes, not user data, so they shouldn’t grow with your workload — unexpected growth here can point at a Materialize bug.
max by (collection_id) (
  ${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}", replica_id=~"${mzReplicaList}", collection_id=~"s.*"}
)

materialize.compute.arrangements.records.user #

Row counts of arrangements for your compute objects (collection id starts with u) — the row count of every user index and materialized view, and the primary driver of cluster memory. Growth on a collection tracks the size of its underlying data.
max by (collection_id) (
  ${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}", replica_id=~"${mzReplicaList}", collection_id=~"u.*"}
)

materialize.compute.arrangements.records.transient #

Row counts of transient (collection id t*) and uncategorized (none) arrangements — short-lived intermediates from query optimization and dataflow execution. Normally small and ephemeral.
max by (collection_id) (
  ${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}", replica_id=~"${mzReplicaList}", collection_id=~"t.*|none"}
)

materialize-connections#

Sessions, query activity, and SQL control-plane (adapter) traffic for a Materialize deployment. Adapted from the Overview dashboard’s “Connections / Activity” tab.

materialize.connections.sessions.active #

Open SQL sessions, split into system (Materialize’s internal probing — a few are always present) and user (client connections).
sum by (session_type) (mz_active_sessions{${mzEnvironmentFilter}})

materialize.connections.queries.rate #

Queries per second by session type — user tracks your client traffic, system is the steady single-digit baseline of internal health checks. Bursty is normal.
sum by (session_type) (rate(mz_query_total{${mzEnvironmentFilter}}${interval}))

materialize.connections.adapter.command_rate #

Commands per second through the adapter — the SQL protocol layer (parse, execute, prepare, fetch). Normally runs higher than the query rate, since one query is several commands.
sum(rate(mz_adapter_commands{${mzEnvironmentFilter}}${interval}))

materialize.connections.queries.distribution #

The mix of query kinds over the selected window — a workload-shape signal, not a rate. Heavy set_variable/fetch traffic is normal (that’s how Postgres clients manage session state); heavy insert/update/delete on something you think of as read-mostly is worth a look.
sum by (statement_type) (increase(mz_query_total{${mzEnvironmentFilter}}${range})) > 0

materialize.connections.queries.rate_by_statement #

Query rate broken down by statement type and session type, fully time-resolved — the moving picture behind the distribution donut. A spike in select/user is the thing to line up against peek latency to confirm the system kept pace.
sum by (statement_type, session_type) (rate(mz_query_total{${mzEnvironmentFilter}}${interval})) > 0

materialize.connections.peek_latency.p50 #

Median read-query latency — the typical time to look up the current state of an arrangement, which is the operation behind every SELECT against an index. Your “what does a normal query feel like” number.
histogram_quantile(0.50,
  sum by (le, instance_id) (
    rate(mz_compute_peek_duration_seconds_bucket{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}"}${interval})
  )
)

materialize.connections.peek_latency.p90 #

90th-percentile read-query latency — how slow the slowest 10% of queries feel. Catches the contention bursts and cold paths that the median hides.
histogram_quantile(0.90,
  sum by (le, instance_id) (
    rate(mz_compute_peek_duration_seconds_bucket{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}"}${interval})
  )
)

materialize.connections.peek_latency.p99 #

Tail read-query latency — the slowest 1% of queries, the ones users complain about.
histogram_quantile(0.99,
  sum by (le, instance_id) (
    rate(mz_compute_peek_duration_seconds_bucket{${mzEnvironmentFilter}, instance_id=~"${mzClusterList}"}${interval})
  )
)

materialize.connections.adapter.commands_by_application #

SQL control-plane command totals per client application_name over the window, so you can see which clients drive the adapter and which are failing. Most clients set application_name in their connection string; those that don’t bucket as unrecognized/unspecified (normal).
sum by (application_name, status) (increase(mz_adapter_commands{${mzEnvironmentFilter}}${range}))

materialize-health#

Common queries for checking the health of a Materialize deployment.

materialize.scraper.mzmon.environmentd #

environmentd metrics are reaching the gateway. If this goes to 0 or disappears, every environmentd-backed panel and alert for the environment is blind — the environment may be perfectly healthy and you simply can’t see it, so rule this out first.
up{
  job="monitoring/mzmon-materialize-environmentd",
  ${mzEnvironmentNamespaceFilter}
} == 1

materialize.scraper.mzmon.clusterd #

clusterd (compute replica) metrics are reaching the gateway. When this drops, per-cluster compute signals — arrangements, peeks, dataflows — go dark even though the replicas may still be serving.
up{
  job="monitoring/mzmon-materialize-clusterd",
  ${mzEnvironmentNamespaceFilter}
} == 1

materialize.scraper.mzmon.orchestratord #

The Materialize operator (orchestratord) is being scraped. Losing it blinds you to cluster and replica lifecycle — creation, resize, and rollout progress — not to the running workloads themselves.
up{
  job="materialize/mzmon-materialize-operator",
  ${mzOperatorNamespaceFilter}
} == 1

materialize.health.clusters.status.percentage #

The share of the environment’s clusters currently reporting ready — the at-a-glance health headline for the whole environment.
count(
  ${mzSqlPrefix}compute_cluster_status{
    ${mzEnvironmentFilter}
  } == 1
) / count(
  ${mzSqlPrefix}compute_cluster_status{
    ${mzEnvironmentFilter}
  }
) * 100

materialize.health.environment.availability.percentage #

An SLO-style snapshot: how much of the selected window the environment’s clusters were ready. Sustained dips are the signal that something restarted or went down while you weren’t watching.
avg by (materialize_cloud_organization_namespace) (
  avg_over_time(
    ${mzSqlPrefix}compute_cluster_status{
      ${mzEnvironmentFilter}
    }${range}
  ) * 100
)

materialize.info.version #

The version of Materialize running in the environment. A single version is the steady state; multiple values appear briefly during a rolling upgrade.
group by (mz_version) (
  ${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}}
)

materialize.info.max_lag #

The worst frontier lag seen anywhere in the environment over the selected window — how far the most-behind collection’s output trailed real time. A top-level freshness pointer.
max(
  max_over_time(
    (
      mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id!="", quantile="1"} < 1e9
    )[1h:1m]
  )
)

materialize-kubernetes#

Kubernetes-side view of a Materialize deployment: capacity, workload readiness, per-pod resource usage, and networking. Adapted from the Overview dashboard’s “Kubernetes Workloads” tab and the k8s-sourced Summary panels.

These read kube-state-metrics and cAdvisor (via the kubelet), NOT Materialize metrics — the same meta-monitoring surface other targets (e.g. Loki health) will draw on. %%{cAdvisorFilter} is the container-scoping filter fragment (namespace + drop empty/pause series); %%{mzNamespaceList} is the raw namespace selector used by the kube_* and container_network_* metrics.

The percent-of-limit panels have an absolute-units sibling for deployments whose metrics source (e.g. GKE’s managed cAdvisor/KSM) doesn’t expose resource limits; the dashboard picks whichever fits the environment.

materialize.kubernetes.cpu.capacity #

Total CPU cores configured across the environment’s containers (sum of cAdvisor CPU limits), excluding the monitoring exporter — i.e. the CPU available to the actual workload.
sum by (container) (
  container_spec_cpu_quota{${cAdvisorFilter}, container!="new-promsql-exporter"}
  / container_spec_cpu_period{${cAdvisorFilter}, container!="new-promsql-exporter"}
)

materialize.kubernetes.memory.capacity #

Total memory configured across the environment’s containers (sum of cAdvisor memory limits), excluding the monitoring exporter. Memory is Materialize’s dominant constraint — in-memory arrangements live in here.
sum by (container) (
  container_spec_memory_limit_bytes{${cAdvisorFilter}, container!="new-promsql-exporter"}
)

materialize.kubernetes.cpu.usage.percent #

Current CPU usage per container type as a fraction of its limit, averaged over the last 5 minutes — shows the worst-loaded container types.
sum by (namespace, container) (
  rate(container_cpu_usage_seconds_total{${cAdvisorFilter}}[5m])
) / sum by (namespace, container) (
  kube_pod_container_resource_limits{resource="cpu", namespace=~"${mzNamespaceList}"}
)

materialize.kubernetes.cpu.usage.absolute #

Current CPU usage per container type in cores (rate over 5 minutes), for deployments whose metrics source doesn’t expose CPU limits — read it against the replica sizes you configured.
sum by (namespace, container) (
  rate(container_cpu_usage_seconds_total{${cAdvisorFilter}}[5m])
)

materialize.kubernetes.memory.usage.percent #

Current memory usage per container type as a fraction of its limit — shows the worst-loaded container types.
sum by (namespace, container) (
  avg by (namespace, pod, container) (
    container_memory_working_set_bytes{${cAdvisorFilter}, container!="new-promsql-exporter"}
  )
) / sum by (namespace, container) (
  avg by (namespace, pod, container) (
    container_spec_memory_limit_bytes{${cAdvisorFilter}, container!="new-promsql-exporter"}
  )
)

materialize.kubernetes.memory.usage.absolute #

Current memory (working set) per container type in bytes, for deployments whose metrics source doesn’t expose memory limits.
sum by (namespace, container) (
  container_memory_working_set_bytes{${cAdvisorFilter}, container!="new-promsql-exporter"}
)

materialize.kubernetes.last_restart #

Seconds since the most recent container restart in the environment.
time()
- topk(1,
    container_start_time_seconds{${cAdvisorFilter}, container!="new-promsql-exporter"}
  )

materialize.kubernetes.pods.readiness #

Pods in the Materialize namespace grouped by phase (Running, Pending, Failed, …).
max by (phase, namespace) (
  sum by (phase, namespace, instance) (
    kube_pod_status_phase{namespace=~"${mzNamespaceList}"}
  )
)

materialize.kubernetes.statefulsets.ready #

StatefulSet replicas reporting Ready. environmentd and the cluster pods are StatefulSets.
max by (namespace) (
  sum by (namespace, instance) (
    kube_statefulset_status_replicas_ready{namespace=~"${mzNamespaceList}"}
  )
)

materialize.kubernetes.deployments.readiness #

Deployment replica health — Ready vs Unavailable. Deployments back stateless services (e.g. the promsql exporter).
max by (namespace) (
  sum by (namespace, instance) (
    kube_deployment_status_replicas_ready{namespace=~"${mzNamespaceList}"}
  )
)

max by (namespace) (
  sum by (namespace, instance) (
    kube_deployment_status_replicas_unavailable{namespace=~"${mzNamespaceList}"}
  )
)

materialize.kubernetes.pods.cpu_usage #

CPU utilization per pod as a fraction of the pod’s limit. Split so the cluster/replica selectors filter the cluster pods while envd/balancer/ exporter stay visible.
sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{${cAdvisorFilter}, pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}${interval})
) / sum by (namespace, pod, container) (
  kube_pod_container_resource_limits{resource="cpu", namespace=~"${mzNamespaceList}", pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}
)

sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{${cAdvisorFilter}, pod!~".*-cluster-.*-replica-.*"}${interval})
) / sum by (namespace, pod, container) (
  kube_pod_container_resource_limits{resource="cpu", namespace=~"${mzNamespaceList}", pod!~".*-cluster-.*-replica-.*"}
)

materialize.kubernetes.pods.memory_usage #

Memory usage per pod as a fraction of the pod’s limit (working-set basis), same cluster/non-cluster split as pod CPU.
avg by (namespace, pod, container) (
  container_memory_working_set_bytes{${cAdvisorFilter}, container!="new-promsql-exporter", pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}
) / avg by (namespace, pod, container) (
  container_spec_memory_limit_bytes{${cAdvisorFilter}, container!="new-promsql-exporter", pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}
)

avg by (namespace, pod, container) (
  container_memory_working_set_bytes{${cAdvisorFilter}, container!="new-promsql-exporter", pod!~".*-cluster-.*-replica-.*"}
) / avg by (namespace, pod, container) (
  container_spec_memory_limit_bytes{${cAdvisorFilter}, container!="new-promsql-exporter", pod!~".*-cluster-.*-replica-.*"}
)

materialize.kubernetes.pods.network_rx #

Network bytes/sec received per pod. For cluster pods, Rx tracks ingest from upstream and inter-pod replication; for envd/balancer it’s client SQL traffic. Surges alongside hydration are normal catchup.
sum by (namespace, pod) (
  rate(container_network_receive_bytes_total{namespace=~"${mzNamespaceList}", pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}${interval})
)

sum by (namespace, pod) (
  rate(container_network_receive_bytes_total{namespace=~"${mzNamespaceList}", pod!~".*-cluster-.*-replica-.*"}${interval})
)

materialize.kubernetes.pods.network_tx #

Network bytes/sec transmitted per pod. For cluster pods Tx covers sink output, inter-pod replication, and query results returning to envd; for envd it’s client query responses.
sum by (namespace, pod) (
  rate(container_network_transmit_bytes_total{namespace=~"${mzNamespaceList}", pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}${interval})
)

sum by (namespace, pod) (
  rate(container_network_transmit_bytes_total{namespace=~"${mzNamespaceList}", pod!~".*-cluster-.*-replica-.*"}${interval})
)

materialize.kubernetes.pods.network_errors #

Network rx + tx errors per pod per second (counted at the NIC/kernel level).
sum by (namespace, pod) (
  rate(container_network_receive_errors_total{namespace=~"${mzNamespaceList}", pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}${interval})
)

sum by (namespace, pod) (
  rate(container_network_receive_errors_total{namespace=~"${mzNamespaceList}", pod!~".*-cluster-.*-replica-.*"}${interval})
)

sum by (namespace, pod) (
  rate(container_network_transmit_errors_total{namespace=~"${mzNamespaceList}", pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}${interval})
)

sum by (namespace, pod) (
  rate(container_network_transmit_errors_total{namespace=~"${mzNamespaceList}", pod!~".*-cluster-.*-replica-.*"}${interval})
)

materialize.kubernetes.pods.network_drops #

Network packets dropped (rx + tx) per pod per second — when kernel buffers fill faster than the app reads (rx) or egress rate-limiting kicks in (tx).
sum by (namespace, pod) (
  rate(container_network_receive_packets_dropped_total{namespace=~"${mzNamespaceList}", pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}${interval})
)

sum by (namespace, pod) (
  rate(container_network_receive_packets_dropped_total{namespace=~"${mzNamespaceList}", pod!~".*-cluster-.*-replica-.*"}${interval})
)

sum by (namespace, pod) (
  rate(container_network_transmit_packets_dropped_total{namespace=~"${mzNamespaceList}", pod=~".*-cluster-${mzClusterList}-replica-${mzReplicaList}-.*"}${interval})
)

sum by (namespace, pod) (
  rate(container_network_transmit_packets_dropped_total{namespace=~"${mzNamespaceList}", pod!~".*-cluster-.*-replica-.*"}${interval})
)

materialize-storage#

Sources and sinks for a Materialize deployment — catalog shape, throughput, lag, and upstream/downstream health. Adapted from the Overview dashboard’s “Sources and Sinks” tab.

The clusterd-side throughput/lag/error metrics (mz_source_* / mz_sink_) carry the long-form cluster_environmentd_materialize_cloud_ id labels. These queries assume one Prometheus job per clusterd endpoint; if the same endpoint is scraped by several jobs, a plain sum-rate reads N× — dedupe the job at the deployment (fix the scrape config, or wrap the inner rate in max without (job)) rather than baking it into the canonical query.

materialize.storage.sources.count #

Active sources in the catalog — each is a continuous ingestion connection from an external system (Kafka, Postgres, MySQL, S3, …), so this is roughly how many upstream feeds the environment maintains. Counts distinct source objects (the hidden per-source _progress subsources are excluded), matching mz_sources.
count(group by (id) (${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type="source"}))

materialize.storage.sinks.count #

Active sinks in the catalog — each emits the results of a materialized view or query to an external system (Kafka, Iceberg, …). Counts distinct sink objects (excluding _progress subsources), matching mz_sinks.
count(group by (id) (${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type="sink"}))

materialize.storage.tables.count #

User-created tables in the catalog. Tables are write-once-read-many; INSERTs feed dataflows downstream. Mostly a catalog-shape signal — for actual ingest activity look at source throughput.
max(sum by (instance) (${mzSqlPrefix}tables_count{${mzEnvironmentFilter}}))

materialize.storage.sources.by_type #

Sources by connector type (kafka / postgres / mysql / …) — what flavors of upstream feed make up the ingest workload. Most environments concentrate on one or two.
count by (object_type) (
  group by (id, object_type) (
    ${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type="source"}
  )
) > 0

materialize.storage.sources.catalog #

A catalog of sources — one row per source (by name) with its connector type, envelope, and the cluster it ingests on. The metric-side “what sources do I have” reference.
group by (id, object_type, connection_type, envelope_type, cluster_id) (
  ${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type="source"}
)

materialize.storage.sources.bytes_received #

Inbound throughput per primary source — bytes/second pulled from upstream. Subsources (e.g. per-table Postgres replication) roll up to their primary, so each line is one logical source.
sum by (parent_source_id) (
  rate(mz_source_bytes_received{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval})
) > 0

materialize.storage.sources.ingestion_by_replica #

Messages ingested per second, split per source AND replica. Replicas read their upstream independently and should track together.
sum by (parent_source_id, cluster_environmentd_materialize_cloud_replica_id) (
  rate(mz_source_messages_received{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval})
)

materialize.storage.sources.upstream_errors #

Per-source upstream health, with two complementary signals — both nominal at 0, so an empty panel is healthy and any series means a source needs attention.
sum by (source_id) (
  rate(mz_source_offset_commit_failures{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval})
) > 0

(
  max by (source_id) (mz_source_offset_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"})
  > bool max by (source_id) (mz_source_offset_known{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"})
) > 0

materialize.storage.sinks.by_type #

Sinks by (type, envelope) — e.g. kafka / upsert, kafka / debezium, iceberg / upsert. The envelope is how Materialize encodes changes: upsert writes the latest value per key, debezium writes change events with old+new values.
count by (object_type, envelope_type) (
  group by (id, object_type, envelope_type) (
    ${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type="sink"}
  )
) > 0

materialize.storage.sinks.throughput #

Outbound throughput per sink — bytes/second successfully committed to the downstream system (Kafka broker, Iceberg catalog, …).
sum by (sink_id) (
  rate(mz_sink_bytes_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval})
) > 0

materialize.storage.sinks.lag #

Bytes staged for a sink but not yet committed downstream — an in-flight queue depth in bytes. Oscillates around a small value in normal operation as commits happen periodically.
clamp_min(
  sum by (sink_id) (mz_sink_bytes_staged{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"})
  - sum by (sink_id) (mz_sink_bytes_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}),
  0
)

materialize.storage.sinks.iceberg.commit_latency #

Iceberg commit-duration percentiles (p50/p90/p99) — how long each COMMIT against the Iceberg catalog takes (write a snapshot manifest, ask the catalog to atomically swap it in).
histogram_quantile(0.50, sum by (le) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval})))

histogram_quantile(0.90, sum by (le) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval})))

histogram_quantile(0.99, sum by (le) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval})))

materialize.storage.sinks.iceberg.commit_failures #

Per-sink rate of failed and conflicting Iceberg commits. Conflicts (concurrent-writer races on the snapshot pointer) are recoverable — Materialize retries — but a high rate means something else is writing the same Iceberg table; failures are commit-side errors (network, auth, schema).
sum by (sink_id) (rate(mz_sink_iceberg_commit_failures{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}))

sum by (sink_id) (rate(mz_sink_iceberg_commit_conflicts{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}))

materialize.storage.sinks.iceberg.file_rate #

Per-sink rate of files and snapshots written to Iceberg. Each commit produces one snapshot with data files (new rows) and delete files (tombstones for upserts). The data:delete ratio reflects your workload — pure-insert sinks produce ~0 deletes; upsert-heavy ones roughly 1:1.
sum by (sink_id) (rate(mz_sink_iceberg_data_files_written{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}))

sum by (sink_id) (rate(mz_sink_iceberg_delete_files_written{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}))

sum by (sink_id) (rate(mz_sink_iceberg_snapshots_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}))

materialize.storage.sinks.kafka.tx_errors #

Per-sink rate of TX errors from the librdkafka client — each is one failed produce-request against the broker.
sum by (sink_id) (rate(mz_sink_rdkafka_txerrs{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}))

materialize.storage.sinks.kafka.output_buffer #

Messages sitting in the librdkafka output buffer, waiting to be sent to the broker. Normal buffer fluctuates briefly as messages flow through.
sum by (sink_id) (mz_sink_rdkafka_outbuf_msg_cnt{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"})

materialize.storage.sinks.kafka.connect_rate #

Connect and disconnect events per sink against the Kafka broker. Healthy connections are persistent — a couple of connects at startup and zero disconnects afterward.
sum by (sink_id) (rate(mz_sink_rdkafka_connects{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}))

sum by (sink_id) (rate(mz_sink_rdkafka_disconnects{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id=~"${mzClusterList}", cluster_environmentd_materialize_cloud_replica_id=~"${mzReplicaList}"}${interval}))
Backward Common Alerts Custom Resource Definitions (CRDs) Forward