Common Queries#

These queries are used in various dashboards and may be a good starting point for a dashboard developer or for someone looking into particulars of a dashboard.

The underlined values in each query are placeholders, and they are editable. Click one, type your environment’s value, and every other occurrence of that placeholder on the page follows — so copying a query gives you something you can paste straight into Prometheus. Press Enter to commit an edit or Escape to discard it.

You can also set placeholders up front by naming them in this page’s URL:

?mzSqlPrefix=v2_mz_&mzNamespaceList=materialize-prod

The two directions are the same mechanism: editing in place rewrites the URL to match, so once the page reads the way you want, the address bar holds a link you can bookmark or share. Only placeholders you have changed appear there, and a customized value is underlined with a solid rather than a dashed line. Reload without the query string to get the defaults back.

infra-logs#

Logs from the platform a Materialize deployment runs on: the monitoring stack itself, the Kubernetes system components, and the nodes underneath both.

Separate from materialize-logs.yaml rather than a widening of it, for two reasons that are structural rather than stylistic.

The selector set differs. These carry component and container filters. component is what tells one Loki or Thanos process from another — loki alone splits into canary, querier, ingester, query-frontend, index-gateway, compactor, distributor and ruler — and container is the only picker that reaches workloads with no app at all, which on a representative install is the whole of kube-system. Neither dimension means anything to a Materialize environment, and adding them to the shared queries would oblige every dashboard using those to define pickers it has no use for.

The node journal is not reachable from a namespace. Journal lines carry unit, component, job, level and service_name and no namespace, app or container, because they come from the node rather than from a pod. Any selector that requires a namespace excludes them by construction, which is why they have their own queries here and their own tab on the dashboard.

The Kubernetes-event queries are not duplicated: materialize.events.cluster.* in materialize-events.yaml is already scoped by the same namespace picker and carries no Materialize-specific filter, so an infrastructure dashboard uses it as it stands.

infra.logs.stream #

The log feed for the selected namespaces, apps, components, containers and levels, newest first.

infra.logs.warnings.stream #

Warning-and-worse lines from the platform, newest first.

infra.logs.rate.by_component #

Log lines per second by application and sub-component — which process of which workload is doing the talking.

infra.logs.rate.by_namespace #

Log lines per second by namespace — where in the cluster the volume is.

infra.logs.warnings.rate #

Warning-and-worse lines per minute across the platform, as one series.

infra.logs.node.stream #

The node journal, newest first — kubelet, containerd, the node problem detector, and the rest of what systemd runs on each node.

infra.logs.node.warnings #

Warning-and-worse lines from the node journal.

infra.logs.node.rate.by_unit #

Node journal lines per second by systemd unit.

infra-nodes#

What kubectl describe node would tell you, for whoever cannot run it.

These answer the questions an operator asks about one machine: what is it, how big is it, how much of it is already promised, is Kubernetes willing to put work on it, and what has it been saying. The measurements of what the machine is actually doing live in node-health.yaml and node-debug.yaml, which read node-exporter; this file is the Kubernetes side, plus the node’s own journal and the events filed against it.

Two identifier conventions meet here, and the difference is the thing to know:

  • kube-state-metrics names a node node="<kubernetes name>". Every query in this file scopes with node="$node" written literally, the same way the node-exporter families write instance=~"$nodeList" literally. A dashboard using either must define the variable; no render parameter supplies it.

  • node-exporter names the same machine instance="<ip>:9100". The join is node_uname_info, whose nodename is the Kubernetes name — which is what the $nodeList variable resolves through. Nothing in this file needs the join, because nothing in this file reads node-exporter.

Loki knows the node a third way: node is structured metadata on journal lines, not a stream label, so it is filtered in the pipeline (| node=...) rather than in the selector. Node events are Kubernetes events whose involved object is the node itself, which is kind="Node" with the node’s name.

Two conventions apply to every query here that aggregates:

  • Deduplicate across kube-state-metrics replicas. instance is the scrape target, so a bare sum or count over an HA deployment adds each object once per replica. Every aggregation keeps instance in the inner step and collapses it with an outer max, which is the shape materialize-kubernetes.yaml established. Queries that only ever max are already safe, since max across identical replicas is idempotent.

  • Terminal pods do not count against the node. A Succeeded or Failed pod has released its CPU and memory, and the scheduler no longer counts it against the pod limit — kube-state-metrics agrees, and stops reporting kube_pod_container_resource_* for it. But kube_pod_info keeps reporting it until garbage collection, so anything counting pods rather than their resources has to subtract them explicitly or it overstates how full the node is. Completed Jobs are the common case.

%%{interval} is the rate window, including its brackets.

infra.nodes.info.kubelet #

The kubelet version this node runs, which is the version Kubernetes itself is on here.
max by (node, kubelet_version) (kube_node_info{node="$node"})

infra.nodes.info.os #

The node’s operating system image.
max by (node, os_image) (kube_node_info{node="$node"})

infra.nodes.info.kernel #

The node’s kernel version.
max by (node, kernel_version) (kube_node_info{node="$node"})

infra.nodes.info.runtime #

The container runtime that starts and stops containers on this node.
max by (node, container_runtime_version) (kube_node_info{node="$node"})

infra.nodes.info.address #

The address the cluster reaches this node on.
max by (node, internal_ip) (kube_node_info{node="$node"})

infra.nodes.created #

Wall-clock time the node joined the cluster.
max by (node) (kube_node_created{node="$node"}) * 1000

infra.nodes.capacity.cpu #

Cores the node reports to Kubernetes.
max by (node) (kube_node_status_capacity{node="$node", resource="cpu"})

infra.nodes.capacity.memory #

Bytes of RAM the node reports to Kubernetes.
max by (node) (kube_node_status_capacity{node="$node", resource="memory"})

infra.nodes.capacity.pods #

The most pods Kubernetes will place on this node.
max by (node) (kube_node_status_capacity{node="$node", resource="pods"})

infra.nodes.capacity.ephemeral_storage #

Bytes of node-local disk available to pods for scratch space.
max by (node) (kube_node_status_capacity{node="$node", resource="ephemeral_storage"})

infra.nodes.allocation.cpu #

Fraction of the node’s schedulable CPU already promised to pods through their requests.
max by (node) (
  sum by (node, instance) (
    kube_pod_container_resource_requests{node="$node", resource="cpu"}
  )
)
/
max by (node) (kube_node_status_allocatable{node="$node", resource="cpu"})

infra.nodes.allocation.memory #

Fraction of the node’s schedulable memory already promised to pods through their requests.
max by (node) (
  sum by (node, instance) (
    kube_pod_container_resource_requests{node="$node", resource="memory"}
  )
)
/
max by (node) (kube_node_status_allocatable{node="$node", resource="memory"})

infra.nodes.allocation.pods #

Fraction of the node’s pod slots in use.
max by (node) (
  count by (node, instance) (
    kube_pod_info{node="$node"}
    unless on (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"} == 1)
  )
)
/
max by (node) (kube_node_status_allocatable{node="$node", resource="pods"})

infra.nodes.pods.by_namespace #

What is actually running on this node, grouped by namespace.
max by (namespace) (
  count by (namespace, instance) (
    kube_pod_info{node="$node"}
    unless on (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"} == 1)
  )
)

infra.nodes.condition.ready #

Whether Kubernetes considers the node healthy enough to run work.
max by (node) (kube_node_status_condition{node="$node", condition="Ready", status="true"})

infra.nodes.conditions #

The node’s pressure and availability conditions — memory, disk, PIDs and network — each 1 when the condition is active.
max by (node, condition) (
  kube_node_status_condition{
    node="$node",
    condition=~"MemoryPressure|DiskPressure|PIDPressure|NetworkUnavailable",
    status="true"
  }
)

infra.nodes.unschedulable #

Whether the node has been cordoned against new work.
max by (node) (kube_node_spec_unschedulable{node="$node"})

infra.nodes.taints #

The taints on this node, which restrict what may be scheduled.
max by (node, key, value, effect) (kube_node_spec_taint{node="$node"})

infra.nodes.pods.by_phase #

Pods on this node, counted by lifecycle phase.
max by (phase) (
  count by (phase, instance) (
    kube_pod_status_phase == 1
    and on (namespace, pod) kube_pod_info{node="$node"}
  )
)

infra.nodes.pods.not_ready #

Pods on this node that are not reporting Ready.
max by (namespace, pod) (
  kube_pod_status_ready{condition="true"}
  and on (namespace, pod) kube_pod_info{node="$node"}
) == 0
unless on (namespace, pod) (kube_pod_status_phase{phase="Succeeded"} == 1)

infra.nodes.pods.restarts #

Container restarts for pods on this node.
max by (namespace, pod) (
  sum by (namespace, pod, instance) (
    kube_pod_container_status_restarts_total
    and on (namespace, pod) kube_pod_info{node="$node"}
  )
)

infra.nodes.pods.budgets #

What each pod on this node reserved and what it is capped at — CPU and memory, requests beside limits.
max by (namespace, pod) (
  sum by (namespace, pod, instance) (
    kube_pod_container_resource_requests{node="$node", resource="cpu"}
  )
)

max by (namespace, pod) (
  sum by (namespace, pod, instance) (
    kube_pod_container_resource_limits{node="$node", resource="cpu"}
  )
)

max by (namespace, pod) (
  sum by (namespace, pod, instance) (
    kube_pod_container_resource_requests{node="$node", resource="memory"}
  )
)

max by (namespace, pod) (
  sum by (namespace, pod, instance) (
    kube_pod_container_resource_limits{node="$node", resource="memory"}
  )
)

infra.nodes.journal.rate.by_unit #

Journal lines per second from this node, split by systemd unit.

infra.nodes.journal.warnings #

Warning-and-worse journal lines from this node.

infra.nodes.journal.stream #

The systemd journal from this node, newest first.

infra.nodes.events.rate.by_reason #

Kubernetes events filed against this node, by reason.

infra.nodes.events.stream #

Kubernetes events filed against this node, newest first.

materialize-clusters#

Inventory and sizing of a Materialize deployment’s clusters and replicas. Adapted from the Overview dashboard’s “Cluster Objects / Replicas” tab.

materialize.clusters.count #

How many clusters exist, split into the Materialize-managed system clusters (mz_catalog_server, mz_system, mz_probe, …) that every environment has and the user clusters you created. The gap between the two is your own footprint.
count(
  group by (compute_cluster_id) (
    mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*"}
  )
)

count(
  group by (compute_cluster_id) (
    mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_cluster_id=~"^s.*"}
  )
)

count_not_null(
  avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}} by {compute_cluster_id}
)

count_not_null(
  avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_cluster_id:s*} by {compute_cluster_id}
)

materialize.clusters.replicas.count #

How many replicas back the selected clusters, and how many of those are redundancy beyond the first. Every cluster needs one replica to run; anything above that is capacity or availability headroom you’ve opted into, so a non-zero “additional” count is the quick check that HA is actually configured where you expect it.
count(
  group by (compute_cluster_id, compute_replica_id) (
    mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*"}
  )
)

count(
  group by (compute_cluster_id, compute_replica_id) (
    mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*", compute_replica_name!="r1"}
  )
)

count_not_null(
  avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}} by {compute_cluster_id,compute_replica_id}
)

default_zero(
  count_not_null(
    avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}, !compute_replica_name:r1} by {compute_cluster_id,compute_replica_id}
  )
)

materialize.clusters.replicas.sizes #

The replica fleet grouped by configured size. Most deployments settle on a handful of sizes; a long tail of one-off sizes usually means an experiment or a half-finished migration. The total agrees with the replica count.
count by (size) (
  mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*"}
)

sum:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}} by {size}

materialize.clusters.info #

A reference row per (cluster, replica): ids, names, size, version, and scheduling metadata. The “what does my fleet actually look like” lookup — most useful for grabbing a cluster or replica id to scope the rest of a dashboard to.
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*"}

avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}}
  by {compute_cluster_id,compute_cluster_name,compute_replica_id,compute_replica_name,size,mz_version}

materialize-compute#

Queries for the compute side of a Materialize deployment — the indexes, materialized views, and subscribes that run as dataflows on cluster replicas, plus their freshness, hydration, and resource footprint.

materialize.compute.materialized_views.count #

Materialized views Materialize is actively maintaining. Each one is a query whose result is kept continuously up to date, so this tracks roughly how much standing compute the environment carries.
max(mz_mzd_views_count{materialize_cloud_organization_name=~".*"})

max:${mzSqlPrefix}mzd_views_count{${mzEnvironmentFilter}}

materialize.compute.indexes.count #

Indexes in the catalog. An index is an in-memory arrangement that makes reads against its relation effectively instant, in exchange for memory — so growth here is a leading indicator of cluster memory growth.
max(sum by (instance) (mz_indexes_count{materialize_cloud_organization_name=~".*"}))

default_zero(sum:${mzSqlPrefix}indexes_count{${mzEnvironmentFilter}} by {instance})

materialize.compute.views.count #

Non-materialized views — query templates evaluated on demand. They cost nothing until something reads them, so this is a catalog-shape signal rather than a load one.
max(mz_views_count{materialize_cloud_organization_name=~".*"})

max:${mzSqlPrefix}views_count{${mzEnvironmentFilter}}

materialize.compute.subscribes.active #

Live SUBSCRIBE sessions — long-running queries that stream updates to a client as data changes. A handful of system subscribes are Materialize’s own internal probes; a persistently climbing user count is a classic leaked-connection signal.
sum by (session_type) (mz_active_subscribes{materialize_cloud_organization_name=~".*"})

sum:mz_active_subscribes{${mzEnvironmentFilter}} by {session_type}

materialize.compute.indexes.by_type #

Indexes split by the kind of relation they sit on. Workloads normally lean heavily on indexes over views (the standard “keep a query’s result hot” pattern); a large share of indexes on base tables is unusual and usually worth a second look.
sum by (relation_type) (mz_indexes_count{materialize_cloud_organization_name=~".*"})

sum:${mzSqlPrefix}indexes_count{${mzEnvironmentFilter}} by {relation_type}

materialize.compute.hydration.currently_hydrating #

Collections still rebuilding their in-memory state — a live hydration-queue proxy. After a restart, replica creation, or some DDL, a dataflow has to rebuild from persisted storage before it can serve, and until it does it produces no results.
count(
  max by (instance_id, collection_id) (
    mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", quantile="1"} > 1e15
  )
)

default_zero(
  count_not_null(
    max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, quantile:1} by {instance_id,collection_id}
  )
)

materialize.compute.hydration.queue_size #

Collections waiting in each replica’s hydration queue. environmentd schedules hydration in batches; a backlog means work is arriving faster than the replica can rebuild it.
sum by (instance_id, replica_id) (
  mz_compute_controller_hydration_queue_size{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*"}
) > 0

sum:mz_compute_controller_hydration_queue_size{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}}
  by {instance_id,replica_id}

materialize.compute.hydration.slowest_collections #

The 15 collections that took longest to finish hydrating. Hydration time scales with the size of the state being rebuilt, so large materialized views and indexes naturally top the list.
topk(15,
  mz_compute_hydration_time_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", hydrated="1"}
)

top(
  max:${mzSqlPrefix}compute_hydration_time_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, hydrated:1} by {instance_id,collection_id},
  15, 'max', 'desc'
)

materialize.compute.freshness.lag_by_cluster #

How far behind real time each cluster’s most-lagged collection is — the worst-case freshness across every index, materialized view, and source on the cluster.
max by (instance_id) (
  mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", quantile="1"} < 1e9
)

max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, quantile:1} by {instance_id}

materialize.compute.freshness.lag_total_by_cluster #

The lag of every collection on each cluster, added together — one number for how far behind the cluster is in total.
sum by (instance_id) (
  max by (instance_id, collection_id) (
    mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", quantile="1"} < 1e9
  )
)

sum:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, quantile:1} by {instance_id}

materialize.compute.freshness.top_collections #

The 15 collections whose results are furthest behind real time — the per-collection breakdown behind the per-cluster freshness lag, labeled by object name.
topk(15,
  max by (instance_id, collection_id) (
    mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", replica_id=~".*", quantile="1"} < 1e9
  )
)

top(
  max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, quantile:1} by {instance_id,collection_id},
  15, 'max', 'desc'
)

materialize.compute.dataflows.count #

Active dataflows on each replica. Every index, materialized view, and live SUBSCRIBE runs as one or more dataflows, so this count rises with DDL and subscribe activity.
max by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id) (
  mz_compute_replica_history_dataflow_count{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}
)

max:mz_compute_replica_history_dataflow_count{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
  by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id}

materialize.compute.dataflows.count_by_worker #

The dataflow count broken out per worker. Workers in a replica run in lockstep and should see exactly the same dataflows, so their series should overlap perfectly.
max by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, worker_id) (
  mz_compute_replica_history_dataflow_count{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}
)

max:mz_compute_replica_history_dataflow_count{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
  by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id,worker_id}

materialize.compute.dataflows.elapsed_rate #

CPU-cores busy inside dataflows, per cluster — the whole of dataflow work: arrangement maintenance, query evaluation, and hydration. Capped by cluster size (a 400cc cluster can’t exceed 400 cores).
sum by (instance_id) (
  max without (job) (rate(
    mz_dataflow_elapsed_seconds_total{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*"}[5m]
  ))
)

sum:${mzSqlPrefix}dataflow_elapsed_seconds_total{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}}
  by {instance_id}.as_rate()

materialize.compute.arrangements.maintenance_rate #

CPU-cores spent maintaining arrangements — the in-memory indexed snapshots behind every index and materialized view — summed across a replica’s workers, so an N-worker replica can reach N.
sum by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id) (
  max without (job) (rate(
    mz_arrangement_maintenance_seconds_total{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]
  ))
)

sum:mz_arrangement_maintenance_seconds_total{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
  by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id}.as_rate()

materialize.compute.arrangements.maintenance_rate_by_worker #

The same maintenance CPU, split per worker — each worker tops out at 1.0.
sum by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, worker_id) (
  max without (job) (rate(
    mz_arrangement_maintenance_seconds_total{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]
  ))
)

sum:mz_arrangement_maintenance_seconds_total{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
  by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id,worker_id}.as_rate()

materialize.compute.arrangements.records.system #

Row counts of arrangements for Materialize’s internal system collections (collection id starts with s). These back the catalog and internal probes, not user data, so they shouldn’t grow with your workload — unexpected growth here can point at a Materialize bug.
max by (collection_id) (
  mz_arrangement_record_count{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", collection_id=~"s.*"}
)

max:${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, collection_id:s*}
  by {collection_id}

materialize.compute.arrangements.records.user #

Row counts of arrangements for your compute objects (collection id starts with u) — the row count of every user index and materialized view, and the primary driver of cluster memory. Growth on a collection tracks the size of its underlying data.
max by (collection_id) (
  mz_arrangement_record_count{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", collection_id=~"u.*"}
)

max:${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, collection_id:u*}
  by {collection_id}

materialize.compute.arrangements.records.transient #

Row counts of transient (collection id t*) and uncategorized (none) arrangements — short-lived intermediates from query optimization and dataflow execution. Normally small and ephemeral.
max by (collection_id) (
  mz_arrangement_record_count{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", collection_id=~"t.*|none"}
)

max:${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, (collection_id:t* OR collection_id:none)}
  by {collection_id}

materialize-connections#

Sessions, query activity, and SQL control-plane (adapter) traffic for a Materialize deployment. Adapted from the Overview dashboard’s “Connections / Activity” tab.

materialize.connections.sessions.active #

Open SQL sessions, split into system (Materialize’s internal probing — a few are always present) and user (client connections).
sum by (session_type) (mz_active_sessions{materialize_cloud_organization_name=~".*"})

sum:mz_active_sessions{${mzEnvironmentFilter}} by {session_type}

materialize.connections.queries.rate #

Queries per second by session type — user tracks your client traffic, system is the steady single-digit baseline of internal health checks. Bursty is normal.
sum by (session_type) (rate(mz_query_total{materialize_cloud_organization_name=~".*"}[5m]))

sum:mz_query_total{${mzEnvironmentFilter}} by {session_type}.as_rate()

materialize.connections.adapter.command_rate #

Commands per second through the adapter — the SQL protocol layer (parse, execute, prepare, fetch). Normally runs higher than the query rate, since one query is several commands.
sum(rate(mz_adapter_commands{materialize_cloud_organization_name=~".*"}[5m]))

sum:mz_adapter_commands{${mzEnvironmentFilter}}.as_rate()

materialize.connections.queries.distribution #

The mix of query kinds over the selected window — a workload-shape signal, not a rate. Heavy set_variable/fetch traffic is normal (that’s how Postgres clients manage session state); heavy insert/update/delete on something you think of as read-mostly is worth a look.
sum by (statement_type) (increase(mz_query_total{materialize_cloud_organization_name=~".*"}[1h])) > 0

sum:mz_query_total{${mzEnvironmentFilter}} by {statement_type}.as_count()

materialize.connections.queries.rate_by_statement #

Query rate broken down by statement type and session type, fully time-resolved — the moving picture behind the distribution donut. A spike in select/user is the thing to line up against peek latency to confirm the system kept pace.
sum by (statement_type, session_type) (rate(mz_query_total{materialize_cloud_organization_name=~".*"}[5m])) > 0

sum:mz_query_total{${mzEnvironmentFilter}} by {statement_type,session_type}.as_rate()

materialize.connections.peek_latency.p50 #

Median read-query latency — the typical time to look up the current state of an arrangement, which is the operation behind every SELECT against an index. Your “what does a normal query feel like” number.
histogram_quantile(0.50,
  sum by (le, instance_id) (
    rate(mz_compute_peek_duration_seconds_bucket{materialize_cloud_organization_name=~".*", instance_id=~".*"}[5m])
  )
)

p50:mz_compute_peek_duration_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}}
  by {instance_id}

materialize.connections.peek_latency.p90 #

90th-percentile read-query latency — how slow the slowest 10% of queries feel. Catches the contention bursts and cold paths that the median hides.
histogram_quantile(0.90,
  sum by (le, instance_id) (
    rate(mz_compute_peek_duration_seconds_bucket{materialize_cloud_organization_name=~".*", instance_id=~".*"}[5m])
  )
)

p90:mz_compute_peek_duration_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}}
  by {instance_id}

materialize.connections.peek_latency.p99 #

Tail read-query latency — the slowest 1% of queries, the ones users complain about.
histogram_quantile(0.99,
  sum by (le, instance_id) (
    rate(mz_compute_peek_duration_seconds_bucket{materialize_cloud_organization_name=~".*", instance_id=~".*"}[5m])
  )
)

p99:mz_compute_peek_duration_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}}
  by {instance_id}

materialize.connections.adapter.commands_by_application #

SQL control-plane command totals per client application_name over the window, so you can see which clients drive the adapter and which are failing. Most clients set application_name in their connection string; those that don’t bucket as unrecognized/unspecified (normal).
sum by (application_name, status) (increase(mz_adapter_commands{materialize_cloud_organization_name=~".*"}[1h]))

sum:mz_adapter_commands{${mzEnvironmentFilter}} by {application_name,status}.as_count()

materialize-events#

Kubernetes events from the namespaces a Materialize deployment occupies: the operator’s own namespace and the environments’ namespace.

Events are logs, not metrics. They arrive through loki.source.kubernetes_events in the monitoring gateway, which reads them from the Kubernetes API and forwards them to Loki, so every query here is LogQL against the logs datasource rather than PromQL. The gateway’s processor lifts reason, name, kind, count, node and reportingcontroller out of each event into structured metadata, which is what these queries filter and group on; the event type becomes the level stream label, Normal as INFO and Warning as WARN.

Two scopes live here. The deployment and operator queries below are rollout-scoped: they answer “is this upgrade going through”, and the env-upgrade dashboard is their consumer. The cluster queries at the end are the general-purpose view, scoped by the same Loki-discovered namespace picker a logs dashboard uses, and are deliberately separate definitions rather than the same ones widened — the rollout queries carry filters (generation, reporting controller) that a general event browser has no business inheriting.

Kubernetes keeps events for about an hour. Loki keeps them for as long as the deployment’s retention says, which is what makes a rollout that finished yesterday still explainable.

An event’s namespace is the involved object’s, not the reporter’s. The operator runs in its own namespace and reconciles resources in the environments’ namespace, and every event it publishes is filed against the resource — so the operator’s own events are found in the environment namespace, where nothing else about them suggests they would be. That is why the queries below scope to both namespaces and pick the operator out by reportingcontroller, which is the reporter’s identity and the only field that actually says an event came from orchestratord.

Only the deployment-wide feeds filter by generation. The operator’s own events are filed against the Materialize, Balancer and Console resources, which carry no generation at all, so %%{mzGenerationEventFilter} could only ever be a no-op on them. The filter itself keeps generation-less objects on purpose — on a representative deployment only 6 of 70 event names carry a generation, and dropping the other 64 would take the whole rollout narrative with them.

materialize.events.deployment.stream #

Every Kubernetes event from the operator and environment namespaces, newest first — the unfiltered record of what the cluster did.

materialize.events.deployment.warnings #

Kubernetes events the reporting component flagged as warnings — a pod that will not schedule, an image that will not pull, a container failing its probes.

materialize.events.deployment.rate.by_reason #

How often each kind of event is being reported, by reason. The shape of a rollout: Pulled, Created and Started rise together as pods are replaced, and fall back to nothing when it finishes.

materialize.events.deployment.warning.rate #

Warning events per interval across both namespaces, as one series — the at-a-glance answer to whether anything is complaining right now.

materialize.events.operator.lifecycle #

Every phase the Materialize resource moved through, as the operator reported it: Applying, ReadyToPromote, WaitingForApproval, Promoting, Applied, and the two that end a rollout badly, RolloutTimeout and FailedDeploy.

materialize.events.operator.lifecycle.rate #

Lifecycle transitions over time, by phase — where a rollout got to, and when.

materialize.events.operator.reconciliation.failures #

Why the operator could not reconcile a resource. The event carries the error’s whole cause chain, which is usually the actionable half — an admission webhook that is down, a secret that does not exist yet, a license key that will not parse.

materialize.events.operator.reconciliation.failures.rate #

Reconciliation failures over time, by the kind of resource that failed.

materialize.events.cluster.stream #

Every Kubernetes event in the selected namespaces, newest first.

materialize.events.cluster.warnings #

Kubernetes events the reporting component flagged as warnings, across the selected namespaces.

materialize.events.cluster.rate.by_reason #

How often each kind of event is being reported, by reason — the shape of what the cluster is doing.

materialize.events.cluster.rate.by_namespace #

Event rate by namespace — where in the cluster things are happening.

materialize-generations#

What each deployment generation of an environment is doing, during and after a blue/green rollout.

A rollout stands a new generation of environmentd and its replicas up beside the old one, lets it rehydrate from persisted storage, and promotes it only once it has caught up. Both generations are live and scraped at the same time, so every ordinary panel sums them together — which is exactly the wrong thing while the question is whether one of them is ready yet.

The generation is not a label. orchestratord records it as a Kubernetes annotation, which neither kube-state-metrics nor cAdvisor surfaces. Where it does reach a query is the object name, in two shapes: …-environmentd-<generation>-<ordinal> and, for a replica, …-gen-<generation>-<ordinal>. %%{mzGenerationFilter} selects on those, and %%{mzGenerationPattern} is the same shape as a capture, for the label_replace that lifts the number into a generation label panels can group by. Both live in the render context, so they cannot drift apart.

materialize.generations.active #

How many deployment generations are currently running — one between rollouts, two while one is in flight.
count(
  count by (generation) (
    label_replace(
      mz_compute_commands_total{
        materialize_cloud_organization_name=~".*", ${mzGenerationFilter}
      },
      "generation", "$1", "pod", "${mzGenerationPattern}"
    )
  )
)

materialize.generations.version #

The Materialize version each generation is running — what the rollout is actually changing.
group by (generation, mz_version) (
  label_replace(
    max_over_time(
      mz_compute_cluster_status{
        materialize_cloud_organization_name=~".*", ${mzGenerationFilter}
      }
      [1h]
    ),
    "generation", "$1", "pod", "${mzGenerationPattern}"
  )
)

materialize.generations.pods #

Pods belonging to each generation — its environmentd and the cluster replicas standing behind it.
count by (generation) (
  group by (pod, generation) (
    label_replace(
      container_memory_working_set_bytes{
        container!="POD", container!="", ${mzGenerationFilter}
      },
      "generation", "$1", "pod", "${mzGenerationPattern}"
    )
  )
)

materialize.generations.hydrating #

Collections still rebuilding their in-memory state, split by generation — the panel that answers whether a new generation is ready to promote.
sum by (generation) (
  max by (generation, instance_id, collection_id) (
    label_replace(
      mz_dataflow_wallclock_lag_seconds{
        materialize_cloud_organization_name=~".*",
        ${mzGenerationFilter},
        instance_id=~".*",
        instance_id!="",
        quantile="1"
      },
      "generation", "$1", "pod", "${mzGenerationPattern}"
    ) > bool 1e15
  )
)

materialize.generations.collections #

Collections each generation is tracking — the denominator for hydration, and the shape of a new generation building out its dataflows.
count by (generation) (
  max by (generation, instance_id, collection_id) (
    label_replace(
      mz_dataflow_wallclock_lag_seconds{
        materialize_cloud_organization_name=~".*",
        ${mzGenerationFilter},
        instance_id=~".*",
        instance_id!="",
        quantile="1"
      },
      "generation", "$1", "pod", "${mzGenerationPattern}"
    )
  )
)

materialize.generations.lag.max #

The worst lag in each generation — how far behind real time its most-lagged collection is.
max by (generation) (
  label_replace(
    mz_dataflow_wallclock_lag_seconds{
      materialize_cloud_organization_name=~".*",
      ${mzGenerationFilter},
      instance_id=~".*",
      instance_id!="",
      quantile="1"
    },
    "generation", "$1", "pod", "${mzGenerationPattern}"
  ) < 1e9
)

materialize.generations.lag.total #

Every hydrated collection’s lag in each generation, added together — how far behind the generation is in total.
sum by (generation) (
  max by (generation, instance_id, collection_id) (
    label_replace(
      mz_dataflow_wallclock_lag_seconds{
        materialize_cloud_organization_name=~".*",
        ${mzGenerationFilter},
        instance_id=~".*",
        instance_id!="",
        quantile="1"
      },
      "generation", "$1", "pod", "${mzGenerationPattern}"
    ) < 1e9
  )
)

materialize.generations.lag.total_by_cluster #

Total lag split by generation and cluster — which cluster in which generation is carrying the lag.
sum by (generation, instance_id) (
  max by (generation, instance_id, collection_id) (
    label_replace(
      mz_dataflow_wallclock_lag_seconds{
        materialize_cloud_organization_name=~".*",
        ${mzGenerationFilter},
        instance_id=~".*",
        instance_id!="",
        quantile="1"
      },
      "generation", "$1", "pod", "${mzGenerationPattern}"
    ) < 1e9
  )
)

materialize.generations.cpu #

CPU used by each generation’s pods — what a rollout costs while both sides are up.
sum by (generation) (
  label_replace(
    rate(
      container_cpu_usage_seconds_total{
        container!="POD", container!="", ${mzGenerationFilter}
      }
      [5m]
    ),
    "generation", "$1", "pod", "${mzGenerationPattern}"
  )
)

materialize.generations.memory #

Memory used by each generation’s pods.
sum by (generation) (
  label_replace(
    container_memory_working_set_bytes{
      container!="POD", container!="", ${mzGenerationFilter}
    },
    "generation", "$1", "pod", "${mzGenerationPattern}"
  )
)

materialize-health#

Common queries for checking the health of a Materialize deployment.

materialize.scraper.mzmon.environmentd #

environmentd metrics are reaching the gateway. If this goes to 0 or disappears, every environmentd-backed panel and alert for the environment is blind — the environment may be perfectly healthy and you simply can’t see it, so rule this out first.
up{
  job="monitoring/mzmon-materialize-environmentd",
  namespace=~"materialize-environment"
} == 1

avg:up{job:monitoring/mzmon-materialize-environmentd, ${mzEnvironmentNamespaceFilter}}

materialize.scraper.mzmon.clusterd #

clusterd (compute replica) metrics are reaching the gateway. When this drops, per-cluster compute signals — arrangements, peeks, dataflows — go dark even though the replicas may still be serving.
up{
  job="monitoring/mzmon-materialize-clusterd",
  namespace=~"materialize-environment"
} == 1

avg:up{job:monitoring/mzmon-materialize-clusterd, ${mzEnvironmentNamespaceFilter}}

materialize.scraper.mzmon.orchestratord #

The Materialize operator (orchestratord) is being scraped. Losing it blinds you to cluster and replica lifecycle — creation, resize, and rollout progress — not to the running workloads themselves.
up{
  job="materialize/mzmon-materialize-operator",
  namespace=~"materialize"
} == 1

avg:up{job:materialize/mzmon-materialize-operator, ${mzOperatorNamespaceFilter}}

materialize.health.clusters.status.percentage #

The share of the environment’s clusters currently reporting ready — the at-a-glance health headline for the whole environment.
count(
  mz_compute_cluster_status{
    materialize_cloud_organization_name=~".*"
  } == 1
) / count(
  mz_compute_cluster_status{
    materialize_cloud_organization_name=~".*"
  }
) * 100

avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}} * 100

materialize.health.environment.availability.percentage #

An SLO-style snapshot: how much of the selected window the environment’s clusters were ready. Sustained dips are the signal that something restarted or went down while you weren’t watching.
avg by (materialize_cloud_organization_namespace) (
  avg_over_time(
    mz_compute_cluster_status{
      materialize_cloud_organization_name=~".*"
    }[1h]
  ) * 100
)

avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}}
  by {materialize_cloud_organization_namespace}.rollup(avg, ${range}) * 100

materialize.info.version #

The version of Materialize running in the environment. A single version is the steady state; multiple values appear briefly during a rolling upgrade.
group by (mz_version) (
  mz_compute_cluster_status{materialize_cloud_organization_name=~".*"}
)

avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}} by {mz_version}

materialize.info.max_lag #

The worst lag seen anywhere in the environment over the selected window — how far the most-behind collection’s output trailed real time. A top-level freshness pointer.
max(
  max_over_time(
    (
      mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id!="", quantile="1"} < 1e9
    )[${rangeWindow}:1m]
  )
)

max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, quantile:1}.rollup(max, 3600)

materialize-kubernetes#

Kubernetes-side view of a Materialize deployment: capacity, workload readiness, per-pod resource usage, and networking. Adapted from the Overview dashboard’s “Kubernetes Workloads” tab and the k8s-sourced Summary panels.

These read kube-state-metrics and cAdvisor (via the kubelet), NOT Materialize metrics — the same meta-monitoring surface other targets (e.g. Loki health) will draw on. %%{cAdvisorFilter} is the container-scoping filter fragment (namespace + drop empty/pause series); %%{mzNamespaceList} is the raw namespace selector used by the kube_* and container_network_* metrics.

The percent-of-limit panels have an absolute-units sibling for deployments whose metrics source (e.g. GKE’s managed cAdvisor/KSM) doesn’t expose resource limits; the dashboard picks whichever fits the environment.

materialize.kubernetes.cpu.capacity #

Total CPU cores configured across the environment’s containers (sum of cAdvisor CPU limits), excluding the monitoring exporter — i.e. the CPU available to the actual workload.
sum by (container) (
  container_spec_cpu_quota{container!="POD", container!="", container!="new-promsql-exporter"}
  / container_spec_cpu_period{container!="POD", container!="", container!="new-promsql-exporter"}
)

sum:container_spec_cpu_quota{${cAdvisorFilter}, !container:new-promsql-exporter} by {container} / 100000

materialize.kubernetes.memory.capacity #

Total memory configured across the environment’s containers (sum of cAdvisor memory limits), excluding the monitoring exporter. Memory is Materialize’s dominant constraint — in-memory arrangements live in here.
sum by (container) (
  container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)

sum:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {container}

materialize.kubernetes.cpu.capacity.all_containers #

Total CPU cores configured across every container in the environment, including the monitoring exporter — the Kubernetes view of what is provisioned rather than what is available to the workload.
sum by (container) (
  container_spec_cpu_quota{container!="POD", container!=""}
  / container_spec_cpu_period{container!="POD", container!=""}
)

sum:container_spec_cpu_quota{${cAdvisorFilter}} by {container} / 100000

materialize.kubernetes.memory.capacity.all_containers #

Total memory configured across every container in the environment, including the monitoring exporter — the Kubernetes view of what is provisioned rather than what is available to the workload.
sum by (container) (
  container_spec_memory_limit_bytes{container!="POD", container!=""}
)

sum:container_spec_memory_limit_bytes{${cAdvisorFilter}} by {container}

materialize.kubernetes.cpu.usage.percent #

Current CPU usage per container type as a fraction of its limit, averaged over the last 5 minutes — shows the worst-loaded container types.
sum by (namespace, container) (
  rate(container_cpu_usage_seconds_total{container!="POD", container!=""}[5m])
) / sum by (namespace, container) (
  kube_pod_container_resource_limits{resource="cpu", namespace=~"materialize-environment"}
)

sum:container_cpu_usage_seconds_total{${cAdvisorFilter}} by {namespace,container}.as_rate()
  / sum:kube_pod_container_resource_limits{resource:cpu, namespace:${mzNamespaceList}} by {namespace,container}

materialize.kubernetes.cpu.usage.absolute #

Current CPU usage per container type in cores (rate over 5 minutes), for deployments whose metrics source doesn’t expose CPU limits — read it against the replica sizes you configured.
sum by (namespace, container) (
  rate(container_cpu_usage_seconds_total{container!="POD", container!=""}[5m])
)

sum:container_cpu_usage_seconds_total{${cAdvisorFilter}} by {namespace,container}.as_rate()

materialize.kubernetes.memory.usage.percent #

Current memory usage per container type as a fraction of its limit — shows the worst-loaded container types.
sum by (namespace, container) (
  avg by (namespace, pod, container) (
    container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
  )
) / sum by (namespace, container) (
  avg by (namespace, pod, container) (
    container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
  )
)

sum:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,container}
  / sum:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,container}

materialize.kubernetes.memory.usage.absolute #

Current memory (working set) per container type in bytes, for deployments whose metrics source doesn’t expose memory limits.
sum by (namespace, container) (
  container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)

sum:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,container}

materialize.kubernetes.last_restart #

Seconds since the most recent container restart in the environment.
time()
- topk(1,
    container_start_time_seconds{container!="POD", container!="", container!="new-promsql-exporter"}
  )

top(
  max:container_start_time_seconds{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,pod,container},
  1, 'max', 'desc'
)

materialize.kubernetes.pods.readiness #

Pods in the Materialize namespace grouped by phase (Running, Pending, Failed, …).
max by (phase, namespace) (
  sum by (phase, namespace, instance) (
    kube_pod_status_phase{namespace=~"materialize-environment"}
  )
)

sum:kube_pod_status_phase{namespace:${mzNamespaceList}} by {phase,namespace}

materialize.kubernetes.statefulsets.ready #

StatefulSet replicas reporting Ready. environmentd and the cluster pods are StatefulSets.
max by (namespace) (
  sum by (namespace, instance) (
    kube_statefulset_status_replicas_ready{namespace=~"materialize-environment"}
  )
)

sum:kube_statefulset_status_replicas_ready{namespace:${mzNamespaceList}} by {namespace}

materialize.kubernetes.deployments.readiness #

Deployment replica health — Ready vs Unavailable. Deployments back stateless services (e.g. the promsql exporter).
max by (namespace) (
  sum by (namespace, instance) (
    kube_deployment_status_replicas_ready{namespace=~"materialize-environment"}
  )
)

max by (namespace) (
  sum by (namespace, instance) (
    kube_deployment_status_replicas_unavailable{namespace=~"materialize-environment"}
  )
)

sum:kube_deployment_status_replicas_ready{namespace:${mzNamespaceList}} by {namespace}

sum:kube_deployment_status_replicas_unavailable{namespace:${mzNamespaceList}} by {namespace}

materialize.kubernetes.pods.cpu_usage #

CPU utilization per pod as a fraction of the pod’s limit. Split so the cluster/replica selectors filter the cluster pods while envd/balancer/ exporter stay visible.
sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{container!="POD", container!="", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
) / sum by (namespace, pod, container) (
  kube_pod_container_resource_limits{resource="cpu", namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}
)

sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{container!="POD", container!="", pod!~".*-cluster-.*-replica-.*"}[5m])
) / sum by (namespace, pod, container) (
  kube_pod_container_resource_limits{resource="cpu", namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}
)

sum:container_cpu_usage_seconds_total{${cAdvisorFilter}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}.as_rate()
  / sum:kube_pod_container_resource_limits{resource:cpu, namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}

sum:container_cpu_usage_seconds_total{${cAdvisorFilter}, !pod:*-cluster-*-replica-*} by {namespace,pod,container}.as_rate()
  / sum:kube_pod_container_resource_limits{resource:cpu, namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod,container}

materialize.kubernetes.pods.memory_usage #

Memory usage per pod as a fraction of the pod’s limit (working-set basis), same cluster/non-cluster split as pod CPU.
avg by (namespace, pod, container) (
  container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}
) / avg by (namespace, pod, container) (
  container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}
)

avg by (namespace, pod, container) (
  container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod!~".*-cluster-.*-replica-.*"}
) / avg by (namespace, pod, container) (
  container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod!~".*-cluster-.*-replica-.*"}
)

avg:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}
  / avg:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}

avg:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, !pod:*-cluster-*-replica-*} by {namespace,pod,container}
  / avg:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, !pod:*-cluster-*-replica-*} by {namespace,pod,container}

materialize.kubernetes.pods.network_rx #

Network bytes/sec received per pod. For cluster pods, Rx tracks ingest from upstream and inter-pod replication; for envd/balancer it’s client SQL traffic. Surges alongside hydration are normal catchup.
sum by (namespace, pod) (
  rate(container_network_receive_bytes_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)

sum by (namespace, pod) (
  rate(container_network_receive_bytes_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)

sum:container_network_receive_bytes_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()

sum:container_network_receive_bytes_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()

materialize.kubernetes.pods.network_tx #

Network bytes/sec transmitted per pod. For cluster pods Tx covers sink output, inter-pod replication, and query results returning to envd; for envd it’s client query responses.
sum by (namespace, pod) (
  rate(container_network_transmit_bytes_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)

sum by (namespace, pod) (
  rate(container_network_transmit_bytes_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)

sum:container_network_transmit_bytes_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()

sum:container_network_transmit_bytes_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()

materialize.kubernetes.pods.network_errors #

Network rx + tx errors per pod per second (counted at the NIC/kernel level).
sum by (namespace, pod) (
  rate(container_network_receive_errors_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)

sum by (namespace, pod) (
  rate(container_network_receive_errors_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)

sum by (namespace, pod) (
  rate(container_network_transmit_errors_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)

sum by (namespace, pod) (
  rate(container_network_transmit_errors_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)

sum:container_network_receive_errors_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()

sum:container_network_receive_errors_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()

sum:container_network_transmit_errors_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()

sum:container_network_transmit_errors_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()

materialize.kubernetes.pods.network_drops #

Network packets dropped (rx + tx) per pod per second — when kernel buffers fill faster than the app reads (rx) or egress rate-limiting kicks in (tx).
sum by (namespace, pod) (
  rate(container_network_receive_packets_dropped_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)

sum by (namespace, pod) (
  rate(container_network_receive_packets_dropped_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)

sum by (namespace, pod) (
  rate(container_network_transmit_packets_dropped_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)

sum by (namespace, pod) (
  rate(container_network_transmit_packets_dropped_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)

sum:container_network_receive_packets_dropped_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()

sum:container_network_receive_packets_dropped_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()

sum:container_network_transmit_packets_dropped_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()

sum:container_network_transmit_packets_dropped_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()

materialize-logs#

Logs, as collected by the monitoring stack and stored in Loki.

Every query here is LogQL, and the scope is Loki-discovered end to end: namespace, app and level come from Loki’s own label values rather than from the metrics pipeline. That is deliberate. Reading logs is frequently how an operator works out why the metrics pipeline is broken, and a logs dashboard that derived its scope from Prometheus would go blind at exactly the moment it is most needed.

The label contract the agent and gateway produce is documented under Logs and Events. What matters here: namespace, app, level and container are stream labels and belong in the selector; pod, node, organization_name and the rest are structured metadata and are filtered after a |. Narrowing the selector before the line filters is the single biggest speedup.

Every selector here carries %%{mzLogJobFilter}, and it is not only a filter. LogQL rejects a stream selector whose every matcher can match the empty string — “queries require at least one regexp or equality matcher that does not have an empty-compatible value” — and a dashboard built from =~ pickers is exactly that shape. The job picker’s “All” is .+ rather than the discovered values, so it always contributes a non-empty matcher and the selector parses whatever the other pickers are set to. Without it, “All” everywhere is a query error rather than a wide result.

The event queries in materialize-events.yaml need no such anchor: they pin job="loki.source.kubernetes_events", which is already a non-empty equality matcher, and a second job matcher would AND with it and zero the panel.

level is normalized by the pipeline where it can be and falls back to UNKNOWN, so the levels present are a property of the workloads running rather than a fixed vocabulary — which is why the dashboard discovers them instead of hard-coding a list.

materialize.logs.stream #

The log feed for the selected namespaces, apps and levels, newest first.

materialize.logs.rate.by_app #

Log lines per second by application — which component is doing the talking.

materialize.logs.rate.by_level #

Log lines per second by severity — the shape of how much of the volume is something going wrong.

materialize.logs.rate.total #

Total log lines per second reaching Loki for the current selection, averaged over each interval.

materialize.logs.warnings.rate #

Warning-and-worse log lines per minute, as one series — the at-a-glance answer to whether anything is complaining.

materialize.logs.warnings.stream #

The warning-and-worse feed, newest first — what the components are actually complaining about.

materialize-operator#

How the Materialize operator’s reconciliation loop is behaving.

orchestratord watches the Materialize, Balancer and Console resources and drives each toward the state its spec asks for. One trip through that work is a pass, and a pass moves through named steps. Both are counted by outcome, and both are timed, which is what lets a stuck rollout say not just that it is stuck but which phase it is stuck in.

These are the operator’s own metrics, scraped from its pods in the operator namespace, so they are scoped by %%{mzOperatorNamespaceFilter} and by nothing else. They carry no organization label, so the environment picker does not narrow them: one operator reconciles every environment in the cluster, and its loop is a single shared thing rather than a per-environment one.

Only the replica holding the leadership lease reconciles. The others export the same metric families sitting at zero, which is why every query here sums across replicas rather than picking one out.

materialize.operator.reconciling.replicas #

How many operator replicas hold the leadership lease and are therefore reconciling. This should be exactly one.
sum(
  orchestratord_is_leader{namespace=~"materialize"}
)

materialize.operator.environments.needing_update #

How many environments in this cluster are still running an outdated pod template — the count an upgrade is working to bring to zero.
sum(
  environmentd_needs_update{namespace=~"materialize"}
)

materialize.operator.reconciliation.rate #

Reconciliation passes per second across every controller — whether the loop is turning at all.
sum(
  rate(
    orchestratord_reconciliations_total{namespace=~"materialize"}
    [5m]
  )
)

materialize.operator.reconciliation.failures.total #

Reconciliation passes that returned an error over the selected time range.
sum(
  increase(
    orchestratord_reconciliations_total{
      namespace=~"materialize", outcome="failed"
    }
    [1h]
  )
)

materialize.operator.reconciliation.outcomes #

What reconciliation passes concluded, by outcome. The shape of a rollout: waiting climbs while the new generation’s pods come up, then gives way to applied when they are ready.
sum by (outcome) (
  rate(
    orchestratord_reconciliations_total{namespace=~"materialize"}
    [5m]
  )
)

materialize.operator.reconciliation.failures.by_controller #

Failing passes broken out by which controller failed and which of its entry points was running — the first question after “something is failing”.
sum by (controller, event_type) (
  rate(
    orchestratord_reconciliations_total{
      namespace=~"materialize", outcome="failed"
    }
    [5m]
  )
)

materialize.operator.reconciliation.duration #

How long one reconciliation pass takes, at the 50th, 90th and 99th percentiles.
histogram_quantile(0.5, sum by (le) (
  rate(
    orchestratord_reconciliation_duration_seconds_bucket{
      namespace=~"materialize"
    }
    [5m]
  )
))

histogram_quantile(0.9, sum by (le) (
  rate(
    orchestratord_reconciliation_duration_seconds_bucket{
      namespace=~"materialize"
    }
    [5m]
  )
))

histogram_quantile(0.99, sum by (le) (
  rate(
    orchestratord_reconciliation_duration_seconds_bucket{
      namespace=~"materialize"
    }
    [5m]
  )
))

materialize.operator.reconciliation.step.duration.p99 #

The slowest phase of a reconciliation pass, at the 99th percentile per step — where the time in a pass actually goes.
histogram_quantile(0.99, sum by (le, step) (
  rate(
    orchestratord_reconciliation_step_duration_seconds_bucket{
      namespace=~"materialize"
    }
    [5m]
  )
))

materialize.operator.reconciliation.steps.rate #

Which phases of reconciliation are running, and how often. A rollout moves through these in order, so the set that is active says where the operator has got to.
sum by (step) (
  rate(
    orchestratord_reconciliation_steps_total{namespace=~"materialize"}
    [5m]
  )
)

materialize.operator.reconciliation.steps.incomplete #

Steps that did not complete, by step and by how they ended — the panel that turns “reconciliation is failing” into “reconciliation is failing here”.
sum by (step, outcome) (
  rate(
    orchestratord_reconciliation_steps_total{
      namespace=~"materialize", outcome=~"failed|abandoned"
    }
    [5m]
  )
)

materialize-storage#

Sources and sinks for a Materialize deployment — catalog shape, throughput, lag, and upstream/downstream health. Adapted from the Overview dashboard’s “Sources and Sinks” tab.

The clusterd-side throughput/lag/error metrics (mz_source_* / mz_sink_) carry the long-form cluster_environmentd_materialize_cloud_ id labels. These queries assume one Prometheus job per clusterd endpoint; if the same endpoint is scraped by several jobs, a plain sum-rate reads N× — dedupe the job at the deployment (fix the scrape config, or wrap the inner rate in max without (job)) rather than baking it into the canonical query.

materialize.storage.sources.count #

Active sources in the catalog — each is a continuous ingestion connection from an external system (Kafka, Postgres, MySQL, S3, …), so this is roughly how many upstream feeds the environment maintains. Counts distinct source objects (the hidden per-source _progress subsources are excluded), matching mz_sources.
count(group by (id) (mz_storage_objects{materialize_cloud_organization_name=~".*", type="source"}))

default_zero(count_not_null(avg:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:source} by {id}))

materialize.storage.sinks.count #

Active sinks in the catalog — each emits the results of a materialized view or query to an external system (Kafka, Iceberg, …). Counts distinct sink objects (excluding _progress subsources), matching mz_sinks.
count(group by (id) (mz_storage_objects{materialize_cloud_organization_name=~".*", type="sink"}))

default_zero(count_not_null(avg:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:sink} by {id}))

materialize.storage.tables.count #

User-created tables in the catalog. Tables are write-once-read-many; INSERTs feed dataflows downstream. Mostly a catalog-shape signal — for actual ingest activity look at source throughput.
max(sum by (instance) (mz_tables_count{materialize_cloud_organization_name=~".*"}))

default_zero(sum:${mzSqlPrefix}tables_count{${mzEnvironmentFilter}} by {instance})

materialize.storage.sources.by_type #

Sources by connector type (kafka / postgres / mysql / …) — what flavors of upstream feed make up the ingest workload. Most environments concentrate on one or two.
count by (object_type) (
  group by (id, object_type) (
    mz_storage_objects{materialize_cloud_organization_name=~".*", type="source"}
  )
) > 0

sum:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:source} by {object_type}

materialize.storage.sources.catalog #

A catalog of sources — one row per source (by name) with its connector type, envelope, and the cluster it ingests on. The metric-side “what sources do I have” reference.
group by (id, object_type, connection_type, envelope_type, cluster_id) (
  mz_storage_objects{materialize_cloud_organization_name=~".*", type="source"}
)

avg:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:source}
  by {id,object_type,connection_type,envelope_type,cluster_id}

materialize.storage.sources.bytes_received #

Inbound throughput per primary source — bytes/second pulled from upstream. Subsources (e.g. per-table Postgres replication) roll up to their primary, so each line is one logical source.
sum by (parent_source_id) (
  max without (job) (rate(mz_source_bytes_received{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
) > 0

sum:mz_source_bytes_received{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {parent_source_id}.as_rate()

materialize.storage.sources.ingestion_by_replica #

Messages ingested per second, split per source AND replica. Replicas read their upstream independently and should track together.
sum by (parent_source_id, cluster_environmentd_materialize_cloud_replica_id) (
  max without (job) (rate(mz_source_messages_received{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
)

sum:mz_source_messages_received{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {parent_source_id,cluster_environmentd_materialize_cloud_replica_id}.as_rate()

materialize.storage.sources.upstream_errors #

Per-source upstream health, with two complementary signals — both nominal at 0, so an empty panel is healthy and any series means a source needs attention.
sum by (source_id) (
  max without (job) (rate(mz_source_offset_commit_failures{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
) > 0

(
  max by (source_id) (mz_source_offset_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"})
  > bool max by (source_id) (mz_source_offset_known{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"})
) > 0

sum:mz_source_offset_commit_failures{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {source_id}.as_rate()

max:mz_source_offset_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {source_id}
  - max:mz_source_offset_known{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {source_id}

materialize.storage.sinks.by_type #

Sinks by (type, envelope) — e.g. kafka / upsert, kafka / debezium, iceberg / upsert. The envelope is how Materialize encodes changes: upsert writes the latest value per key, debezium writes change events with old+new values.
count by (object_type, envelope_type) (
  group by (id, object_type, envelope_type) (
    mz_storage_objects{materialize_cloud_organization_name=~".*", type="sink"}
  )
) > 0

sum:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:sink} by {object_type,envelope_type}

materialize.storage.sinks.throughput #

Outbound throughput per sink — bytes/second successfully committed to the downstream system (Kafka broker, Iceberg catalog, …).
sum by (sink_id) (
  max without (job) (rate(mz_sink_bytes_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
) > 0

sum:mz_sink_bytes_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()

materialize.storage.sinks.lag #

Bytes staged for a sink but not yet committed downstream — an in-flight queue depth in bytes. Oscillates around a small value in normal operation as commits happen periodically.
clamp_min(
  sum by (sink_id) (max without (job) (mz_sink_bytes_staged{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}))
  - sum by (sink_id) (max without (job) (mz_sink_bytes_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"})),
  0
)

clamp_min(
  sum:mz_sink_bytes_staged{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}
    - sum:mz_sink_bytes_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id},
  0
)

materialize.storage.sinks.iceberg.commit_latency #

Iceberg commit-duration percentiles (p50/p90/p99) — how long each COMMIT against the Iceberg catalog takes (write a snapshot manifest, ask the catalog to atomically swap it in).
histogram_quantile(0.50, sum by (le) (max without (job) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))))

histogram_quantile(0.90, sum by (le) (max without (job) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))))

histogram_quantile(0.99, sum by (le) (max without (job) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))))

p50:mz_sink_iceberg_commit_duration_seconds{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}

p90:mz_sink_iceberg_commit_duration_seconds{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}

p99:mz_sink_iceberg_commit_duration_seconds{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}

materialize.storage.sinks.iceberg.commit_failures #

Per-sink rate of failed and conflicting Iceberg commits. Conflicts (concurrent-writer races on the snapshot pointer) are recoverable — Materialize retries — but a high rate means something else is writing the same Iceberg table; failures are commit-side errors (network, auth, schema).
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_commit_failures{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))

sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_commit_conflicts{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))

sum:mz_sink_iceberg_commit_failures{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()

sum:mz_sink_iceberg_commit_conflicts{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()

materialize.storage.sinks.iceberg.file_rate #

Per-sink rate of files and snapshots written to Iceberg. Each commit produces one snapshot with data files (new rows) and delete files (tombstones for upserts). The data:delete ratio reflects your workload — pure-insert sinks produce ~0 deletes; upsert-heavy ones roughly 1:1.
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_data_files_written{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))

sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_delete_files_written{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))

sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_snapshots_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))

sum:mz_sink_iceberg_data_files_written{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()

sum:mz_sink_iceberg_delete_files_written{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()

sum:mz_sink_iceberg_snapshots_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()

materialize.storage.sinks.kafka.tx_errors #

Per-sink rate of TX errors from the librdkafka client — each is one failed produce-request against the broker.
sum by (sink_id) (max without (job) (rate(mz_sink_rdkafka_txerrs{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))

sum:mz_sink_rdkafka_txerrs{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()

materialize.storage.sinks.kafka.output_buffer #

Messages sitting in the librdkafka output buffer, waiting to be sent to the broker. Normal buffer fluctuates briefly as messages flow through.
sum by (sink_id) (max without (job) (mz_sink_rdkafka_outbuf_msg_cnt{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}))

sum:mz_sink_rdkafka_outbuf_msg_cnt{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}

materialize.storage.sinks.kafka.connect_rate #

Connect and disconnect events per sink against the Kafka broker. Healthy connections are persistent — a couple of connects at startup and zero disconnects afterward.
sum by (sink_id) (max without (job) (rate(mz_sink_rdkafka_connects{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))

sum by (sink_id) (max without (job) (rate(mz_sink_rdkafka_disconnects{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))

sum:mz_sink_rdkafka_connects{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()

sum:mz_sink_rdkafka_disconnects{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()

node-debug#

The breakdowns you reach for once node-health.yaml has told you a node is in trouble: which mode the CPU is in, where the memory went, which device is slow, and where the packets are being lost. Adapted from the Node Exporter Full dashboard (https://grafana.com/grafana/dashboards/1860, revision 45).

Split from node-health.yaml on tier rather than on subject. These sit at recommended, so a deployment collecting only the essential tier still gets the health surface and pays nothing for the detail. Nothing here should back an alert — if something here is worth paging on, it belongs in node-health.yaml instead.

The conventions are the same as node-health.yaml: instance=~"$nodeList" rather than =, every query wrapped in max by (instance, ...) (or min where low is the bad direction) so a second scrape job cannot double-count, inner aggregations carrying by (instance, job) so the outer wrapper is what collapses job, and %%{interval} as the rate window.

Only collectors this chart’s allowlist enables are referenced. Notable omissions, because the dashboard has panels for them and they will render empty: node_processes_* (processes collector), node_interrupts_total (interrupts), node_tcp_connection_states (tcpstat), node_systemd_* (systemd), and node_textfile_scrape_error (textfile).

node.debug.cpu.by_mode #

CPU time by mode — system, user, iowait, and the interrupt modes — averaged across cores.
max by (instance) (
  avg by (instance, job) (
    rate(node_cpu_seconds_total{mode="system", instance=~"$nodeList"}[5m])
  )
)

max by (instance) (
  avg by (instance, job) (
    rate(node_cpu_seconds_total{mode="user", instance=~"$nodeList"}[5m])
  )
)

max by (instance) (
  avg by (instance, job) (
    rate(node_cpu_seconds_total{mode="iowait", instance=~"$nodeList"}[5m])
  )
)

max by (instance) (
  avg by (instance, job) (
    sum without (mode) (
      rate(node_cpu_seconds_total{mode=~".*irq", instance=~"$nodeList"}[5m])
    )
  )
)

max by (instance) (
  avg by (instance, job) (
    rate(node_cpu_seconds_total{mode="steal", instance=~"$nodeList"}[5m])
  )
)

avg:node_cpu_seconds_total{instance:$nodeList, mode:system} by {instance}.as_rate()

avg:node_cpu_seconds_total{instance:$nodeList, mode:user} by {instance}.as_rate()

avg:node_cpu_seconds_total{instance:$nodeList, mode:iowait} by {instance}.as_rate()

avg:node_cpu_seconds_total{instance:$nodeList, mode:*irq} by {instance}.as_rate()

avg:node_cpu_seconds_total{instance:$nodeList, mode:steal} by {instance}.as_rate()

node.debug.cpu.per_core #

Non-idle CPU time per core, so a single saturated core is visible.
1 - max by (instance, cpu) (
  rate(node_cpu_seconds_total{mode="idle", instance=~"$nodeList"}[5m])
)

1 - max:node_cpu_seconds_total{instance:$nodeList, mode:idle} by {instance,cpu}.as_rate()

node.debug.schedstat.waiting #

Time tasks spent runnable but not running, per core, from /proc/schedstat.
max by (instance, cpu) (
  rate(node_schedstat_waiting_seconds_total{instance=~"$nodeList"}[5m])
)

max:node_schedstat_waiting_seconds_total{instance:$nodeList} by {instance,cpu}.as_rate()

node.debug.context_switches #

Context switches and hardware interrupts per second.
max by (instance) (
  rate(node_context_switches_total{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_intr_total{instance=~"$nodeList"}[5m])
)

max:node_context_switches_total{instance:$nodeList} by {instance}.as_rate()

max:node_intr_total{instance:$nodeList} by {instance}.as_rate()

node.debug.memory.breakdown #

Where RAM went: total, used by processes, reclaimable cache, and free.
max by (instance) (node_memory_MemTotal_bytes{instance=~"$nodeList"})

max by (instance) (
  node_memory_MemTotal_bytes{instance=~"$nodeList"}
  - node_memory_MemFree_bytes{instance=~"$nodeList"}
  - node_memory_Cached_bytes{instance=~"$nodeList"}
  - node_memory_Buffers_bytes{instance=~"$nodeList"}
  - node_memory_SReclaimable_bytes{instance=~"$nodeList"}
)

max by (instance) (
  node_memory_Cached_bytes{instance=~"$nodeList"}
  + node_memory_Buffers_bytes{instance=~"$nodeList"}
  + node_memory_SReclaimable_bytes{instance=~"$nodeList"}
)

min by (instance) (node_memory_MemFree_bytes{instance=~"$nodeList"})

max:node_memory_MemTotal_bytes{instance:$nodeList} by {instance}

max:node_memory_MemTotal_bytes{instance:$nodeList} by {instance}
  - max:node_memory_MemFree_bytes{instance:$nodeList} by {instance}
  - max:node_memory_Cached_bytes{instance:$nodeList} by {instance}
  - max:node_memory_Buffers_bytes{instance:$nodeList} by {instance}
  - max:node_memory_SReclaimable_bytes{instance:$nodeList} by {instance}

max:node_memory_Cached_bytes{instance:$nodeList} by {instance}
  + max:node_memory_Buffers_bytes{instance:$nodeList} by {instance}
  + max:node_memory_SReclaimable_bytes{instance:$nodeList} by {instance}

min:node_memory_MemFree_bytes{instance:$nodeList} by {instance}

node.debug.memory.kernel #

Kernel-side memory: slab total, reclaimable and unreclaimable slab, and committed address space.
max by (instance) (node_memory_Slab_bytes{instance=~"$nodeList"})

max by (instance) (node_memory_SReclaimable_bytes{instance=~"$nodeList"})

max by (instance) (node_memory_SUnreclaim_bytes{instance=~"$nodeList"})

max by (instance) (node_memory_Committed_AS_bytes{instance=~"$nodeList"})

max:node_memory_Slab_bytes{instance:$nodeList} by {instance}

max:node_memory_SReclaimable_bytes{instance:$nodeList} by {instance}

max:node_memory_SUnreclaim_bytes{instance:$nodeList} by {instance}

max:node_memory_Committed_AS_bytes{instance:$nodeList} by {instance}

node.debug.memory.page_faults #

Total and major page faults per second.
max by (instance) (
  rate(node_vmstat_pgfault{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_vmstat_pgmajfault{instance=~"$nodeList"}[5m])
)

max:node_vmstat_pgfault{instance:$nodeList} by {instance}.as_rate()

max:node_vmstat_pgmajfault{instance:$nodeList} by {instance}.as_rate()

node.debug.memory.reclaim #

Pages scanned and reclaimed per second, split by who did the reclaiming: kswapd (background) or direct (an allocating thread).
max by (instance) (
  rate(node_vmstat_pgscan_kswapd{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_vmstat_pgscan_direct{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_vmstat_pgsteal_kswapd{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_vmstat_pgsteal_direct{instance=~"$nodeList"}[5m])
)

max:node_vmstat_pgscan_kswapd{instance:$nodeList} by {instance}.as_rate()

max:node_vmstat_pgscan_direct{instance:$nodeList} by {instance}.as_rate()

max:node_vmstat_pgsteal_kswapd{instance:$nodeList} by {instance}.as_rate()

max:node_vmstat_pgsteal_direct{instance:$nodeList} by {instance}.as_rate()

node.debug.disk.iops #

Completed reads and writes per second, per device.
max by (instance, device) (
  rate(node_disk_reads_completed_total{instance=~"$nodeList"}[5m])
)

max by (instance, device) (
  rate(node_disk_writes_completed_total{instance=~"$nodeList"}[5m])
)

max:node_disk_reads_completed_total{instance:$nodeList} by {instance,device}.as_rate()

max:node_disk_writes_completed_total{instance:$nodeList} by {instance,device}.as_rate()

node.debug.disk.throughput #

Bytes read and written per second, per device.
max by (instance, device) (
  rate(node_disk_read_bytes_total{instance=~"$nodeList"}[5m])
)

max by (instance, device) (
  rate(node_disk_written_bytes_total{instance=~"$nodeList"}[5m])
)

max:node_disk_read_bytes_total{instance:$nodeList} by {instance,device}.as_rate()

max:node_disk_written_bytes_total{instance:$nodeList} by {instance,device}.as_rate()

node.debug.disk.latency #

Average time per completed read and per completed write, per device.
max by (instance, device) (
  (rate(node_disk_reads_completed_total{instance=~"$nodeList"}[5m]) > bool 0)
  *
  (
    rate(node_disk_read_time_seconds_total{instance=~"$nodeList"}[5m])
    / rate(node_disk_reads_completed_total{instance=~"$nodeList"}[5m])
  )
)

max by (instance, device) (
  (rate(node_disk_writes_completed_total{instance=~"$nodeList"}[5m]) > bool 0)
  *
  (
    rate(node_disk_write_time_seconds_total{instance=~"$nodeList"}[5m])
    / rate(node_disk_writes_completed_total{instance=~"$nodeList"}[5m])
  )
)

max:node_disk_read_time_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
  / max:node_disk_reads_completed_total{instance:$nodeList} by {instance,device}.as_rate()

max:node_disk_write_time_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
  / max:node_disk_writes_completed_total{instance:$nodeList} by {instance,device}.as_rate()

node.debug.disk.queue_depth #

Average I/O queue depth per device.
max by (instance, device) (
  rate(node_disk_io_time_weighted_seconds_total{instance=~"$nodeList"}[5m])
)

max:node_disk_io_time_weighted_seconds_total{instance:$nodeList} by {instance,device}.as_rate()

node.debug.filesystem.inodes.available.ratio #

Fraction of inodes still free, per mountpoint.
min by (instance, mountpoint) (
  node_filesystem_files_free{instance=~"$nodeList", fstype!="rootfs"}
)
/
max by (instance, mountpoint) (
  node_filesystem_files{instance=~"$nodeList", fstype!="rootfs"}
)

min:node_filesystem_files_free{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
  / max:node_filesystem_files{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}

node.debug.network.throughput #

Bytes received and transmitted per second, per interface.
max by (instance, device) (
  rate(node_network_receive_bytes_total{instance=~"$nodeList"}[5m])
)

max by (instance, device) (
  rate(node_network_transmit_bytes_total{instance=~"$nodeList"}[5m])
)

max:node_network_receive_bytes_total{instance:$nodeList} by {instance,device}.as_rate()

max:node_network_transmit_bytes_total{instance:$nodeList} by {instance,device}.as_rate()

node.debug.network.saturation #

Receive and transmit throughput as a fraction of the interface’s reported link speed.
max by (instance, device) (
  (node_network_speed_bytes{instance=~"$nodeList"} > bool 0)
  *
  (
    rate(node_network_receive_bytes_total{instance=~"$nodeList"}[5m])
    / node_network_speed_bytes{instance=~"$nodeList"}
  )
)

max by (instance, device) (
  (node_network_speed_bytes{instance=~"$nodeList"} > bool 0)
  *
  (
    rate(node_network_transmit_bytes_total{instance=~"$nodeList"}[5m])
    / node_network_speed_bytes{instance=~"$nodeList"}
  )
)

max:node_network_receive_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
  / max:node_network_speed_bytes{instance:$nodeList} by {instance,device}

max:node_network_transmit_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
  / max:node_network_speed_bytes{instance:$nodeList} by {instance,device}

node.debug.network.operstate #

Whether each interface is operationally up, and whether it has carrier.
min by (instance, device) (
  node_network_up{instance=~"$nodeList"}
)

min by (instance, device) (
  node_network_carrier{instance=~"$nodeList"}
)

min:node_network_up{instance:$nodeList} by {instance,device}

min:node_network_carrier{instance:$nodeList} by {instance,device}

node.debug.softnet.processed #

Packets processed by the network softirq path, per CPU.
max by (instance, cpu) (
  rate(node_softnet_processed_total{instance=~"$nodeList"}[5m])
)

max:node_softnet_processed_total{instance:$nodeList} by {instance,cpu}.as_rate()

node.debug.softnet.dropped #

Packets dropped in the network softirq path because the backlog queue was full, per CPU.
max by (instance, cpu) (
  rate(node_softnet_dropped_total{instance=~"$nodeList"}[5m])
)

max:node_softnet_dropped_total{instance:$nodeList} by {instance,cpu}.as_rate()

node.debug.softnet.squeezed #

Times the softirq handler exhausted its budget with work still queued, per CPU.
max by (instance, cpu) (
  rate(node_softnet_times_squeezed_total{instance=~"$nodeList"}[5m])
)

max:node_softnet_times_squeezed_total{instance:$nodeList} by {instance,cpu}.as_rate()

node.debug.sockets.tcp #

TCP sockets by state: in use, allocated, orphaned, and TIME_WAIT.
max by (instance) (node_sockstat_TCP_inuse{instance=~"$nodeList"})

max by (instance) (node_sockstat_TCP_alloc{instance=~"$nodeList"})

max by (instance) (node_sockstat_TCP_orphan{instance=~"$nodeList"})

max by (instance) (node_sockstat_TCP_tw{instance=~"$nodeList"})

max:node_sockstat_TCP_inuse{instance:$nodeList} by {instance}

max:node_sockstat_TCP_alloc{instance:$nodeList} by {instance}

max:node_sockstat_TCP_orphan{instance:$nodeList} by {instance}

max:node_sockstat_TCP_tw{instance:$nodeList} by {instance}

node.debug.sockets.memory #

Kernel socket buffer memory held by TCP and UDP.
max by (instance) (node_sockstat_TCP_mem_bytes{instance=~"$nodeList"})

max by (instance) (node_sockstat_UDP_mem_bytes{instance=~"$nodeList"})

max:node_sockstat_TCP_mem_bytes{instance:$nodeList} by {instance}

max:node_sockstat_UDP_mem_bytes{instance:$nodeList} by {instance}

node.debug.tcp.retransmits #

TCP segment retransmits and SYN retransmits per second, against total segments out.
max by (instance) (
  rate(node_netstat_Tcp_RetransSegs{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_netstat_TcpExt_TCPSynRetrans{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_netstat_Tcp_OutSegs{instance=~"$nodeList"}[5m])
)

max:node_netstat_Tcp_RetransSegs{instance:$nodeList} by {instance}.as_rate()

max:node_netstat_TcpExt_TCPSynRetrans{instance:$nodeList} by {instance}.as_rate()

max:node_netstat_Tcp_OutSegs{instance:$nodeList} by {instance}.as_rate()

node.debug.tcp.errors #

TCP listen-queue overflows, listen drops, receive-queue drops, and timeouts per second.
max by (instance) (
  rate(node_netstat_TcpExt_ListenOverflows{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_netstat_TcpExt_ListenDrops{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_netstat_TcpExt_TCPRcvQDrop{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_netstat_TcpExt_TCPTimeouts{instance=~"$nodeList"}[5m])
)

max:node_netstat_TcpExt_ListenOverflows{instance:$nodeList} by {instance}.as_rate()

max:node_netstat_TcpExt_ListenDrops{instance:$nodeList} by {instance}.as_rate()

max:node_netstat_TcpExt_TCPRcvQDrop{instance:$nodeList} by {instance}.as_rate()

max:node_netstat_TcpExt_TCPTimeouts{instance:$nodeList} by {instance}.as_rate()

node.debug.udp.errors #

UDP receive errors, receive-buffer errors, and packets to no listening port, per second.
max by (instance) (
  rate(node_netstat_Udp_InErrors{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_netstat_Udp_RcvbufErrors{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_netstat_Udp_NoPorts{instance=~"$nodeList"}[5m])
)

max:node_netstat_Udp_InErrors{instance:$nodeList} by {instance}.as_rate()

max:node_netstat_Udp_RcvbufErrors{instance:$nodeList} by {instance}.as_rate()

max:node_netstat_Udp_NoPorts{instance:$nodeList} by {instance}.as_rate()

node.debug.udp.queues #

Bytes queued in UDP receive and transmit buffers.
max by (instance) (
  node_udp_queues{ip="v4", queue="rx", instance=~"$nodeList"}
)

max by (instance) (
  node_udp_queues{ip="v4", queue="tx", instance=~"$nodeList"}
)

max:node_udp_queues{instance:$nodeList, ip:v4, queue:rx} by {instance}

max:node_udp_queues{instance:$nodeList, ip:v4, queue:tx} by {instance}

node.debug.arp.entries #

ARP table entries per interface.
max by (instance, device) (
  node_arp_entries{instance=~"$nodeList"}
)

max:node_arp_entries{instance:$nodeList} by {instance,device}

node.debug.time.sync_status #

Whether the kernel clock is synchronized (1) or NTP has given up (0).
min by (instance) (
  node_timex_sync_status{instance=~"$nodeList"}
)

min:node_timex_sync_status{instance:$nodeList} by {instance}

node.debug.time.drift #

Estimated clock offset, maximum error, and estimated error, in seconds.
max by (instance) (node_timex_offset_seconds{instance=~"$nodeList"})

max by (instance) (node_timex_maxerror_seconds{instance=~"$nodeList"})

max by (instance) (node_timex_estimated_error_seconds{instance=~"$nodeList"})

max:node_timex_offset_seconds{instance:$nodeList} by {instance}

max:node_timex_maxerror_seconds{instance:$nodeList} by {instance}

max:node_timex_estimated_error_seconds{instance:$nodeList} by {instance}

node.debug.entropy.available #

Available entropy, against the pool size.
min by (instance) (node_entropy_available_bits{instance=~"$nodeList"})

max by (instance) (node_entropy_pool_size_bits{instance=~"$nodeList"})

min:node_entropy_available_bits{instance:$nodeList} by {instance}

max:node_entropy_pool_size_bits{instance:$nodeList} by {instance}

node.debug.exporter.scrape_duration #

How long each node-exporter collector took on the last scrape.
max by (instance, collector) (
  node_scrape_collector_duration_seconds{instance=~"$nodeList"}
)

max:node_scrape_collector_duration_seconds{instance:$nodeList} by {instance,collector}

node-health#

Node-level health: the at-a-glance answers to “is this machine in trouble”, adapted from the Node Exporter Full dashboard (https://grafana.com/grafana/dashboards/1860, revision 45) and narrowed to what actually alerts.

These read node-exporter, NOT Materialize metrics. Only collectors this chart’s allowlist enables are referenced — see the Node Exporter section of the chart’s values reference for the list and the reasoning. The deeper breakdowns live in node-debug.yaml at the recommended tier.

Three conventions apply to every query here:

  • instance=~"$nodeList", never instance="$nodeList". A regex match makes the selector work unchanged whether the dashboard variable resolves to one node or many, so the same query backs a single-node view and a fleet view.

  • Every query is wrapped in max by (instance, ...), which drops job. A node should only ever be scraped by one job. If a second one appears — a pre-existing node-exporter alongside ours, or a migration with both running — the same series arrives twice under different job labels, and every sum() silently doubles while every binary operation between two metrics loses its match. Aggregating job away makes both failure modes impossible rather than merely unlikely. max is the default; min where low is the bad direction (available memory, free space, collector success), so the aggregate always reports the worst case rather than hiding it behind a healthy duplicate.

  • Where an inner aggregation is needed (averaging across CPUs, summing across devices), it carries by (instance, job) and the outer max/min collapses job afterwards. Aggregating both in one step would blend two jobs' readings into one number instead of picking one.

%%{interval} is the rate window, including its brackets.

node.cpu.utilization #

Fraction of CPU time the node spent doing anything other than idling, averaged across its cores.
1 - max by (instance) (
  avg by (instance, job) (
    rate(node_cpu_seconds_total{mode="idle", instance=~"$nodeList"}[5m])
  )
)

1 - avg:node_cpu_seconds_total{mode:idle, instance:$nodeList} by {instance}.as_rate()

node.load.normalized #

One-minute load average divided by the node’s core count, so it is comparable across instance sizes.
max by (instance) (node_load1{instance=~"$nodeList"})
/
max by (instance) (
  count by (instance, job) (
    count by (instance, job, cpu) (node_cpu_seconds_total{instance=~"$nodeList"})
  )
)

max:node_load1{instance:$nodeList} by {instance}
  / count_not_null(avg:node_cpu_seconds_total{mode:idle, instance:$nodeList} by {cpu})

node.cpu.pressure #

PSI: the fraction of wall time at least one task was stalled waiting for CPU.
max by (instance) (
  rate(node_pressure_cpu_waiting_seconds_total{instance=~"$nodeList"}[5m])
)

max:node_pressure_cpu_waiting_seconds_total{instance:$nodeList} by {instance}.as_rate()

node.memory.available.ratio #

Fraction of RAM the kernel estimates is available for new allocations without swapping, from MemAvailable.
min by (instance) (
  node_memory_MemAvailable_bytes{instance=~"$nodeList"}
)
/
max by (instance) (
  node_memory_MemTotal_bytes{instance=~"$nodeList"}
)

min:node_memory_MemAvailable_bytes{instance:$nodeList} by {instance}
  / max:node_memory_MemTotal_bytes{instance:$nodeList} by {instance}

node.memory.pressure #

PSI: the fraction of wall time at least one task was stalled on memory (waiting), and the fraction where every task was (stalled).
max by (instance) (
  rate(node_pressure_memory_waiting_seconds_total{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_pressure_memory_stalled_seconds_total{instance=~"$nodeList"}[5m])
)

max:node_pressure_memory_waiting_seconds_total{instance:$nodeList} by {instance}.as_rate()

max:node_pressure_memory_stalled_seconds_total{instance:$nodeList} by {instance}.as_rate()

node.swap.used.ratio #

Fraction of configured swap in use. Zero when the node has no swap configured, rather than returning no data.
(
  (
    max by (instance) (node_memory_SwapTotal_bytes{instance=~"$nodeList"})
    -
    min by (instance) (node_memory_SwapFree_bytes{instance=~"$nodeList"})
  )
  /
  max by (instance) (node_memory_SwapTotal_bytes{instance=~"$nodeList"})
)
and
(max by (instance) (node_memory_SwapTotal_bytes{instance=~"$nodeList"}) > 0)

(
  max:node_memory_SwapTotal_bytes{instance:$nodeList} by {instance}
    - min:node_memory_SwapFree_bytes{instance:$nodeList} by {instance}
)
  / max:node_memory_SwapTotal_bytes{instance:$nodeList} by {instance}

node.swap.activity #

Pages swapped in and out per second, from vmstat.
max by (instance) (
  rate(node_vmstat_pswpin{instance=~"$nodeList"}[5m])
)

max by (instance) (
  rate(node_vmstat_pswpout{instance=~"$nodeList"}[5m])
)

max:node_vmstat_pswpin{instance:$nodeList} by {instance}.as_rate()

max:node_vmstat_pswpout{instance:$nodeList} by {instance}.as_rate()

node.memory.oom_kills #

Rate of OOM-killer invocations on the node.
max by (instance) (
  rate(node_vmstat_oom_kill{instance=~"$nodeList"}[5m])
)

max:node_vmstat_oom_kill{instance:$nodeList} by {instance}.as_rate()

node.filesystem.available.ratio #

Fraction of each mounted filesystem still available, per mountpoint.
min by (instance, mountpoint) (
  node_filesystem_avail_bytes{instance=~"$nodeList", fstype!="rootfs"}
)
/
max by (instance, mountpoint) (
  node_filesystem_size_bytes{instance=~"$nodeList", fstype!="rootfs"}
)

min:node_filesystem_avail_bytes{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
  / max:node_filesystem_size_bytes{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}

node.filesystem.readonly #

Whether a filesystem has been remounted read-only, per mountpoint.
max by (instance, mountpoint) (
  node_filesystem_readonly{instance=~"$nodeList", fstype!="rootfs"}
)

max:node_filesystem_readonly{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}

node.disk.io_utilization #

Fraction of wall time each block device had at least one I/O in flight.
max by (instance, device) (
  rate(node_disk_io_time_seconds_total{instance=~"$nodeList"}[5m])
)

max:node_disk_io_time_seconds_total{instance:$nodeList} by {instance,device}.as_rate()

node.filefd.utilization #

Allocated file descriptors as a fraction of the system-wide maximum.
max by (instance) (node_filefd_allocated{instance=~"$nodeList"})
/
max by (instance) (node_filefd_maximum{instance=~"$nodeList"})

max:node_filefd_allocated{instance:$nodeList} by {instance}
  / max:node_filefd_maximum{instance:$nodeList} by {instance}

node.network.errors #

Receive and transmit error rates per interface.
max by (instance, device) (
  rate(node_network_receive_errs_total{instance=~"$nodeList"}[5m])
)

max by (instance, device) (
  rate(node_network_transmit_errs_total{instance=~"$nodeList"}[5m])
)

max:node_network_receive_errs_total{instance:$nodeList} by {instance,device}.as_rate()

max:node_network_transmit_errs_total{instance:$nodeList} by {instance,device}.as_rate()

node.network.drops #

Receive and transmit packet drop rates per interface.
max by (instance, device) (
  rate(node_network_receive_drop_total{instance=~"$nodeList"}[5m])
)

max by (instance, device) (
  rate(node_network_transmit_drop_total{instance=~"$nodeList"}[5m])
)

max:node_network_receive_drop_total{instance:$nodeList} by {instance,device}.as_rate()

max:node_network_transmit_drop_total{instance:$nodeList} by {instance,device}.as_rate()

node.network.rx.total #

Bytes per second received across every interface on the node.
sum by (instance) (
  max by (instance, device) (
    rate(node_network_receive_bytes_total{instance=~"$nodeList"}[5m])
  )
)

node.network.tx.total #

Bytes per second transmitted across every interface on the node.
sum by (instance) (
  max by (instance, device) (
    rate(node_network_transmit_bytes_total{instance=~"$nodeList"}[5m])
  )
)

node.conntrack.utilization #

Netfilter connection-tracking table occupancy as a fraction of its limit.
max by (instance) (node_nf_conntrack_entries{instance=~"$nodeList"})
/
max by (instance) (node_nf_conntrack_entries_limit{instance=~"$nodeList"})

max:node_nf_conntrack_entries{instance:$nodeList} by {instance}
  / max:node_nf_conntrack_entries_limit{instance:$nodeList} by {instance}

node.uptime #

Seconds since the node booted.
min by (instance) (
  node_time_seconds{instance=~"$nodeList"} - node_boot_time_seconds{instance=~"$nodeList"}
)

min:node_time_seconds{instance:$nodeList} by {instance}
  - max:node_boot_time_seconds{instance:$nodeList} by {instance}

node.collector.success #

Whether each enabled node-exporter collector returned data on the last scrape, per collector.
min by (instance, collector) (
  node_scrape_collector_success{instance=~"$nodeList"}
)

min:node_scrape_collector_success{instance:$nodeList} by {instance,collector}
Backward Common Alerts Custom Resource Definitions (CRDs) Forward