Logs from the platform a Materialize deployment runs on: the monitoring stack itself, the Kubernetes system components, and the nodes underneath both.
Separate from materialize-logs.yaml rather than a widening of it, for two
reasons that are structural rather than stylistic.
The selector set differs. These carry component and container filters.
component is what tells one Loki or Thanos process from another — loki
alone splits into canary, querier, ingester, query-frontend,
index-gateway, compactor, distributor and ruler — and container is
the only picker that reaches workloads with no app at all, which on a
representative install is the whole of kube-system. Neither dimension means
anything to a Materialize environment, and adding them to the shared queries
would oblige every dashboard using those to define pickers it has no use for.
The node journal is not reachable from a namespace. Journal lines carry
unit, component, job, level and service_name and no namespace,
app or container, because they come from the node rather than from a pod.
Any selector that requires a namespace excludes them by construction, which is
why they have their own queries here and their own tab on the dashboard.
The Kubernetes-event queries are not duplicated: materialize.events.cluster.*
in materialize-events.yaml is already scoped by the same namespace picker and
carries no Materialize-specific filter, so an infrastructure dashboard uses it
as it stands.
kubelet, containerd, the node
problem detector, and the rest of what systemd runs on each node.
What kubectl describe node would tell you, for whoever cannot run it.
These answer the questions an operator asks about one machine: what is it,
how big is it, how much of it is already promised, is Kubernetes willing to
put work on it, and what has it been saying. The measurements of what the
machine is actually doing live in node-health.yaml and node-debug.yaml,
which read node-exporter; this file is the Kubernetes side, plus the node’s
own journal and the events filed against it.
Two identifier conventions meet here, and the difference is the thing to know:
kube-state-metrics names a node node="<kubernetes name>". Every query
in this file scopes with node="$node" written literally, the same way the
node-exporter families write instance=~"$nodeList" literally. A dashboard
using either must define the variable; no render parameter supplies it.
node-exporter names the same machine instance="<ip>:9100". The join is
node_uname_info, whose nodename is the Kubernetes name — which is what
the $nodeList variable resolves through. Nothing in this file needs the
join, because nothing in this file reads node-exporter.
Loki knows the node a third way: node is structured metadata on journal
lines, not a stream label, so it is filtered in the pipeline (| node=...)
rather than in the selector. Node events are Kubernetes events whose involved
object is the node itself, which is kind="Node" with the node’s name.
Two conventions apply to every query here that aggregates:
Deduplicate across kube-state-metrics replicas. instance is the scrape
target, so a bare sum or count over an HA deployment adds each object
once per replica. Every aggregation keeps instance in the inner step and
collapses it with an outer max, which is the shape
materialize-kubernetes.yaml established. Queries that only ever max are
already safe, since max across identical replicas is idempotent.
Terminal pods do not count against the node. A Succeeded or Failed
pod has released its CPU and memory, and the scheduler no longer counts it
against the pod limit — kube-state-metrics agrees, and stops reporting
kube_pod_container_resource_* for it. But kube_pod_info keeps reporting
it until garbage collection, so anything counting pods rather than their
resources has to subtract them explicitly or it overstates how full the node
is. Completed Jobs are the common case.
%%{interval} is the rate window, including its brackets.
max by (node, kubelet_version) (kube_node_info{node="$node"})
max by (node, os_image) (kube_node_info{node="$node"})
max by (node, kernel_version) (kube_node_info{node="$node"})
max by (node, container_runtime_version) (kube_node_info{node="$node"})
max by (node, internal_ip) (kube_node_info{node="$node"})
max by (node) (kube_node_created{node="$node"}) * 1000
max by (node) (kube_node_status_capacity{node="$node", resource="cpu"})
max by (node) (kube_node_status_capacity{node="$node", resource="memory"})
max by (node) (kube_node_status_capacity{node="$node", resource="pods"})
max by (node) (kube_node_status_capacity{node="$node", resource="ephemeral_storage"})
max by (node) (
sum by (node, instance) (
kube_pod_container_resource_requests{node="$node", resource="cpu"}
)
)
/
max by (node) (kube_node_status_allocatable{node="$node", resource="cpu"})
max by (node) (
sum by (node, instance) (
kube_pod_container_resource_requests{node="$node", resource="memory"}
)
)
/
max by (node) (kube_node_status_allocatable{node="$node", resource="memory"})
max by (node) (
count by (node, instance) (
kube_pod_info{node="$node"}
unless on (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"} == 1)
)
)
/
max by (node) (kube_node_status_allocatable{node="$node", resource="pods"})
max by (namespace) (
count by (namespace, instance) (
kube_pod_info{node="$node"}
unless on (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"} == 1)
)
)
max by (node) (kube_node_status_condition{node="$node", condition="Ready", status="true"})
max by (node, condition) (
kube_node_status_condition{
node="$node",
condition=~"MemoryPressure|DiskPressure|PIDPressure|NetworkUnavailable",
status="true"
}
)
max by (node) (kube_node_spec_unschedulable{node="$node"})
max by (node, key, value, effect) (kube_node_spec_taint{node="$node"})
max by (phase) (
count by (phase, instance) (
kube_pod_status_phase == 1
and on (namespace, pod) kube_pod_info{node="$node"}
)
)
max by (namespace, pod) (
kube_pod_status_ready{condition="true"}
and on (namespace, pod) kube_pod_info{node="$node"}
) == 0
unless on (namespace, pod) (kube_pod_status_phase{phase="Succeeded"} == 1)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_status_restarts_total
and on (namespace, pod) kube_pod_info{node="$node"}
)
)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_resource_requests{node="$node", resource="cpu"}
)
)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_resource_limits{node="$node", resource="cpu"}
)
)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_resource_requests{node="$node", resource="memory"}
)
)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_resource_limits{node="$node", resource="memory"}
)
)
count(
group by (compute_cluster_id) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*"}
)
)
count(
group by (compute_cluster_id) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_cluster_id=~"^s.*"}
)
)
count_not_null(
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}} by {compute_cluster_id}
)
count_not_null(
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_cluster_id:s*} by {compute_cluster_id}
)
count(
group by (compute_cluster_id, compute_replica_id) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*"}
)
)
count(
group by (compute_cluster_id, compute_replica_id) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*", compute_replica_name!="r1"}
)
)
count_not_null(
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}} by {compute_cluster_id,compute_replica_id}
)
default_zero(
count_not_null(
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}, !compute_replica_name:r1} by {compute_cluster_id,compute_replica_id}
)
)
count by (size) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*"}
)
sum:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}} by {size}
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*"}
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}}
by {compute_cluster_id,compute_cluster_name,compute_replica_id,compute_replica_name,size,mz_version}
max(mz_mzd_views_count{materialize_cloud_organization_name=~".*"})
max:${mzSqlPrefix}mzd_views_count{${mzEnvironmentFilter}}
max(sum by (instance) (mz_indexes_count{materialize_cloud_organization_name=~".*"}))
default_zero(sum:${mzSqlPrefix}indexes_count{${mzEnvironmentFilter}} by {instance})
max(mz_views_count{materialize_cloud_organization_name=~".*"})
max:${mzSqlPrefix}views_count{${mzEnvironmentFilter}}
system subscribes are
Materialize’s own internal probes; a persistently climbing user count
is a classic leaked-connection signal.
sum by (session_type) (mz_active_subscribes{materialize_cloud_organization_name=~".*"})
sum:mz_active_subscribes{${mzEnvironmentFilter}} by {session_type}
sum by (relation_type) (mz_indexes_count{materialize_cloud_organization_name=~".*"})
sum:${mzSqlPrefix}indexes_count{${mzEnvironmentFilter}} by {relation_type}
count(
max by (instance_id, collection_id) (
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", quantile="1"} > 1e15
)
)
default_zero(
count_not_null(
max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, quantile:1} by {instance_id,collection_id}
)
)
sum by (instance_id, replica_id) (
mz_compute_controller_hydration_queue_size{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*"}
) > 0
sum:mz_compute_controller_hydration_queue_size{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}}
by {instance_id,replica_id}
topk(15,
mz_compute_hydration_time_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", hydrated="1"}
)
top(
max:${mzSqlPrefix}compute_hydration_time_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, hydrated:1} by {instance_id,collection_id},
15, 'max', 'desc'
)
max by (instance_id) (
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", quantile="1"} < 1e9
)
max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, quantile:1} by {instance_id}
sum by (instance_id) (
max by (instance_id, collection_id) (
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", quantile="1"} < 1e9
)
)
sum:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, quantile:1} by {instance_id}
topk(15,
max by (instance_id, collection_id) (
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", replica_id=~".*", quantile="1"} < 1e9
)
)
top(
max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, quantile:1} by {instance_id,collection_id},
15, 'max', 'desc'
)
max by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id) (
mz_compute_replica_history_dataflow_count{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}
)
max:mz_compute_replica_history_dataflow_count{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id}
max by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, worker_id) (
mz_compute_replica_history_dataflow_count{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}
)
max:mz_compute_replica_history_dataflow_count{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id,worker_id}
sum by (instance_id) (
max without (job) (rate(
mz_dataflow_elapsed_seconds_total{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*"}[5m]
))
)
sum:${mzSqlPrefix}dataflow_elapsed_seconds_total{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}}
by {instance_id}.as_rate()
sum by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id) (
max without (job) (rate(
mz_arrangement_maintenance_seconds_total{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]
))
)
sum:mz_arrangement_maintenance_seconds_total{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id}.as_rate()
sum by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, worker_id) (
max without (job) (rate(
mz_arrangement_maintenance_seconds_total{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]
))
)
sum:mz_arrangement_maintenance_seconds_total{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id,worker_id}.as_rate()
s). These back the catalog and internal
probes, not user data, so they shouldn’t grow with your workload —
unexpected growth here can point at a Materialize bug.
max by (collection_id) (
mz_arrangement_record_count{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", collection_id=~"s.*"}
)
max:${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, collection_id:s*}
by {collection_id}
u) — the row count of every user index and materialized view, and
the primary driver of cluster memory. Growth on a collection tracks the
size of its underlying data.
max by (collection_id) (
mz_arrangement_record_count{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", collection_id=~"u.*"}
)
max:${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, collection_id:u*}
by {collection_id}
t*) and uncategorized (none)
arrangements — short-lived intermediates from query optimization and
dataflow execution. Normally small and ephemeral.
max by (collection_id) (
mz_arrangement_record_count{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", collection_id=~"t.*|none"}
)
max:${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, (collection_id:t* OR collection_id:none)}
by {collection_id}
system (Materialize’s internal probing —
a few are always present) and user (client connections).
sum by (session_type) (mz_active_sessions{materialize_cloud_organization_name=~".*"})
sum:mz_active_sessions{${mzEnvironmentFilter}} by {session_type}
user tracks your client traffic,
system is the steady single-digit baseline of internal health checks.
Bursty is normal.
sum by (session_type) (rate(mz_query_total{materialize_cloud_organization_name=~".*"}[5m]))
sum:mz_query_total{${mzEnvironmentFilter}} by {session_type}.as_rate()
sum(rate(mz_adapter_commands{materialize_cloud_organization_name=~".*"}[5m]))
sum:mz_adapter_commands{${mzEnvironmentFilter}}.as_rate()
set_variable/fetch traffic is normal
(that’s how Postgres clients manage session state); heavy
insert/update/delete on something you think of as read-mostly is
worth a look.
sum by (statement_type) (increase(mz_query_total{materialize_cloud_organization_name=~".*"}[1h])) > 0
sum:mz_query_total{${mzEnvironmentFilter}} by {statement_type}.as_count()
select/user is the thing to line up against peek latency
to confirm the system kept pace.
sum by (statement_type, session_type) (rate(mz_query_total{materialize_cloud_organization_name=~".*"}[5m])) > 0
sum:mz_query_total{${mzEnvironmentFilter}} by {statement_type,session_type}.as_rate()
SELECT
against an index. Your “what does a normal query feel like” number.
histogram_quantile(0.50,
sum by (le, instance_id) (
rate(mz_compute_peek_duration_seconds_bucket{materialize_cloud_organization_name=~".*", instance_id=~".*"}[5m])
)
)
p50:mz_compute_peek_duration_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}}
by {instance_id}
histogram_quantile(0.90,
sum by (le, instance_id) (
rate(mz_compute_peek_duration_seconds_bucket{materialize_cloud_organization_name=~".*", instance_id=~".*"}[5m])
)
)
p90:mz_compute_peek_duration_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}}
by {instance_id}
histogram_quantile(0.99,
sum by (le, instance_id) (
rate(mz_compute_peek_duration_seconds_bucket{materialize_cloud_organization_name=~".*", instance_id=~".*"}[5m])
)
)
p99:mz_compute_peek_duration_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}}
by {instance_id}
application_name over the
window, so you can see which clients drive the adapter and which are
failing. Most clients set application_name in their connection string;
those that don’t bucket as unrecognized/unspecified (normal).
sum by (application_name, status) (increase(mz_adapter_commands{materialize_cloud_organization_name=~".*"}[1h]))
sum:mz_adapter_commands{${mzEnvironmentFilter}} by {application_name,status}.as_count()
Kubernetes events from the namespaces a Materialize deployment occupies: the operator’s own namespace and the environments’ namespace.
Events are logs, not metrics. They arrive through
loki.source.kubernetes_events in the monitoring gateway, which reads them
from the Kubernetes API and forwards them to Loki, so every query here is
LogQL against the logs datasource rather than PromQL. The gateway’s processor
lifts reason, name, kind, count, node and reportingcontroller out
of each event into structured metadata, which is what these queries filter and
group on; the event type becomes the level stream label, Normal as
INFO and Warning as WARN.
Two scopes live here. The deployment and operator queries below are
rollout-scoped: they answer “is this upgrade going through”, and the
env-upgrade dashboard is their consumer. The cluster queries at the end are
the general-purpose view, scoped by the same Loki-discovered namespace picker a
logs dashboard uses, and are deliberately separate definitions rather than the
same ones widened — the rollout queries carry filters (generation, reporting
controller) that a general event browser has no business inheriting.
Kubernetes keeps events for about an hour. Loki keeps them for as long as the deployment’s retention says, which is what makes a rollout that finished yesterday still explainable.
An event’s namespace is the involved object’s, not the reporter’s. The
operator runs in its own namespace and reconciles resources in the
environments’ namespace, and every event it publishes is filed against the
resource — so the operator’s own events are found in the environment
namespace, where nothing else about them suggests they would be. That is why
the queries below scope to both namespaces and pick the operator out by
reportingcontroller, which is the reporter’s identity and the only field
that actually says an event came from orchestratord.
Only the deployment-wide feeds filter by generation. The operator’s own
events are filed against the Materialize, Balancer and Console resources,
which carry no generation at all, so %%{mzGenerationEventFilter} could only
ever be a no-op on them. The filter itself keeps generation-less objects on
purpose — on a representative deployment only 6 of 70 event names carry a
generation, and dropping the other 64 would take the whole rollout narrative
with them.
Pulled, Created and Started rise together as pods are
replaced, and fall back to nothing when it finishes.
Applying, ReadyToPromote, WaitingForApproval,
Promoting, Applied, and the two that end a rollout badly,
RolloutTimeout and FailedDeploy.
What each deployment generation of an environment is doing, during and after a blue/green rollout.
A rollout stands a new generation of environmentd and its replicas up beside
the old one, lets it rehydrate from persisted storage, and promotes it only once
it has caught up. Both generations are live and scraped at the same time, so
every ordinary panel sums them together — which is exactly the wrong thing while
the question is whether one of them is ready yet.
The generation is not a label. orchestratord records it as a Kubernetes
annotation, which neither kube-state-metrics nor cAdvisor surfaces. Where it
does reach a query is the object name, in two shapes:
…-environmentd-<generation>-<ordinal> and, for a replica,
…-gen-<generation>-<ordinal>. %%{mzGenerationFilter} selects on those, and
%%{mzGenerationPattern} is the same shape as a capture, for the
label_replace that lifts the number into a generation label panels can group
by. Both live in the render context, so they cannot drift apart.
count(
count by (generation) (
label_replace(
mz_compute_commands_total{
materialize_cloud_organization_name=~".*", ${mzGenerationFilter}
},
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
)
group by (generation, mz_version) (
label_replace(
max_over_time(
mz_compute_cluster_status{
materialize_cloud_organization_name=~".*", ${mzGenerationFilter}
}
[1h]
),
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
count by (generation) (
group by (pod, generation) (
label_replace(
container_memory_working_set_bytes{
container!="POD", container!="", ${mzGenerationFilter}
},
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
)
sum by (generation) (
max by (generation, instance_id, collection_id) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
) > bool 1e15
)
)
count by (generation) (
max by (generation, instance_id, collection_id) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
)
max by (generation) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
) < 1e9
)
sum by (generation) (
max by (generation, instance_id, collection_id) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
) < 1e9
)
)
sum by (generation, instance_id) (
max by (generation, instance_id, collection_id) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
) < 1e9
)
)
sum by (generation) (
label_replace(
rate(
container_cpu_usage_seconds_total{
container!="POD", container!="", ${mzGenerationFilter}
}
[5m]
),
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
sum by (generation) (
label_replace(
container_memory_working_set_bytes{
container!="POD", container!="", ${mzGenerationFilter}
},
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
up{
job="monitoring/mzmon-materialize-environmentd",
namespace=~"materialize-environment"
} == 1
avg:up{job:monitoring/mzmon-materialize-environmentd, ${mzEnvironmentNamespaceFilter}}
up{
job="monitoring/mzmon-materialize-clusterd",
namespace=~"materialize-environment"
} == 1
avg:up{job:monitoring/mzmon-materialize-clusterd, ${mzEnvironmentNamespaceFilter}}
up{
job="materialize/mzmon-materialize-operator",
namespace=~"materialize"
} == 1
avg:up{job:materialize/mzmon-materialize-operator, ${mzOperatorNamespaceFilter}}
count(
mz_compute_cluster_status{
materialize_cloud_organization_name=~".*"
} == 1
) / count(
mz_compute_cluster_status{
materialize_cloud_organization_name=~".*"
}
) * 100
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}} * 100
avg by (materialize_cloud_organization_namespace) (
avg_over_time(
mz_compute_cluster_status{
materialize_cloud_organization_name=~".*"
}[1h]
) * 100
)
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}}
by {materialize_cloud_organization_namespace}.rollup(avg, ${range}) * 100
group by (mz_version) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*"}
)
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}} by {mz_version}
max(
max_over_time(
(
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id!="", quantile="1"} < 1e9
)[${rangeWindow}:1m]
)
)
max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, quantile:1}.rollup(max, 3600)
Kubernetes-side view of a Materialize deployment: capacity, workload readiness, per-pod resource usage, and networking. Adapted from the Overview dashboard’s “Kubernetes Workloads” tab and the k8s-sourced Summary panels.
These read kube-state-metrics and cAdvisor (via the kubelet), NOT Materialize
metrics — the same meta-monitoring surface other targets (e.g. Loki health)
will draw on. %%{cAdvisorFilter} is the container-scoping filter fragment
(namespace + drop empty/pause series); %%{mzNamespaceList} is the raw
namespace selector used by the kube_* and container_network_* metrics.
The percent-of-limit panels have an absolute-units sibling for deployments whose metrics source (e.g. GKE’s managed cAdvisor/KSM) doesn’t expose resource limits; the dashboard picks whichever fits the environment.
sum by (container) (
container_spec_cpu_quota{container!="POD", container!="", container!="new-promsql-exporter"}
/ container_spec_cpu_period{container!="POD", container!="", container!="new-promsql-exporter"}
)
sum:container_spec_cpu_quota{${cAdvisorFilter}, !container:new-promsql-exporter} by {container} / 100000
sum by (container) (
container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)
sum:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {container}
sum by (container) (
container_spec_cpu_quota{container!="POD", container!=""}
/ container_spec_cpu_period{container!="POD", container!=""}
)
sum:container_spec_cpu_quota{${cAdvisorFilter}} by {container} / 100000
sum by (container) (
container_spec_memory_limit_bytes{container!="POD", container!=""}
)
sum:container_spec_memory_limit_bytes{${cAdvisorFilter}} by {container}
sum by (namespace, container) (
rate(container_cpu_usage_seconds_total{container!="POD", container!=""}[5m])
) / sum by (namespace, container) (
kube_pod_container_resource_limits{resource="cpu", namespace=~"materialize-environment"}
)
sum:container_cpu_usage_seconds_total{${cAdvisorFilter}} by {namespace,container}.as_rate()
/ sum:kube_pod_container_resource_limits{resource:cpu, namespace:${mzNamespaceList}} by {namespace,container}
sum by (namespace, container) (
rate(container_cpu_usage_seconds_total{container!="POD", container!=""}[5m])
)
sum:container_cpu_usage_seconds_total{${cAdvisorFilter}} by {namespace,container}.as_rate()
sum by (namespace, container) (
avg by (namespace, pod, container) (
container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)
) / sum by (namespace, container) (
avg by (namespace, pod, container) (
container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)
)
sum:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,container}
/ sum:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,container}
sum by (namespace, container) (
container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)
sum:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,container}
time()
- topk(1,
container_start_time_seconds{container!="POD", container!="", container!="new-promsql-exporter"}
)
top(
max:container_start_time_seconds{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,pod,container},
1, 'max', 'desc'
)
max by (phase, namespace) (
sum by (phase, namespace, instance) (
kube_pod_status_phase{namespace=~"materialize-environment"}
)
)
sum:kube_pod_status_phase{namespace:${mzNamespaceList}} by {phase,namespace}
max by (namespace) (
sum by (namespace, instance) (
kube_statefulset_status_replicas_ready{namespace=~"materialize-environment"}
)
)
sum:kube_statefulset_status_replicas_ready{namespace:${mzNamespaceList}} by {namespace}
max by (namespace) (
sum by (namespace, instance) (
kube_deployment_status_replicas_ready{namespace=~"materialize-environment"}
)
)
max by (namespace) (
sum by (namespace, instance) (
kube_deployment_status_replicas_unavailable{namespace=~"materialize-environment"}
)
)
sum:kube_deployment_status_replicas_ready{namespace:${mzNamespaceList}} by {namespace}
sum:kube_deployment_status_replicas_unavailable{namespace:${mzNamespaceList}} by {namespace}
sum by (namespace, pod, container) (
rate(container_cpu_usage_seconds_total{container!="POD", container!="", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
) / sum by (namespace, pod, container) (
kube_pod_container_resource_limits{resource="cpu", namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}
)
sum by (namespace, pod, container) (
rate(container_cpu_usage_seconds_total{container!="POD", container!="", pod!~".*-cluster-.*-replica-.*"}[5m])
) / sum by (namespace, pod, container) (
kube_pod_container_resource_limits{resource="cpu", namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}
)
sum:container_cpu_usage_seconds_total{${cAdvisorFilter}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}.as_rate()
/ sum:kube_pod_container_resource_limits{resource:cpu, namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}
sum:container_cpu_usage_seconds_total{${cAdvisorFilter}, !pod:*-cluster-*-replica-*} by {namespace,pod,container}.as_rate()
/ sum:kube_pod_container_resource_limits{resource:cpu, namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod,container}
avg by (namespace, pod, container) (
container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}
) / avg by (namespace, pod, container) (
container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}
)
avg by (namespace, pod, container) (
container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod!~".*-cluster-.*-replica-.*"}
) / avg by (namespace, pod, container) (
container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod!~".*-cluster-.*-replica-.*"}
)
avg:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}
/ avg:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}
avg:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, !pod:*-cluster-*-replica-*} by {namespace,pod,container}
/ avg:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, !pod:*-cluster-*-replica-*} by {namespace,pod,container}
sum by (namespace, pod) (
rate(container_network_receive_bytes_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_receive_bytes_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum:container_network_receive_bytes_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_receive_bytes_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum by (namespace, pod) (
rate(container_network_transmit_bytes_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_bytes_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum:container_network_transmit_bytes_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_bytes_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum by (namespace, pod) (
rate(container_network_receive_errors_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_receive_errors_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_errors_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_errors_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum:container_network_receive_errors_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_receive_errors_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_errors_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_errors_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum by (namespace, pod) (
rate(container_network_receive_packets_dropped_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_receive_packets_dropped_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_packets_dropped_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_packets_dropped_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum:container_network_receive_packets_dropped_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_receive_packets_dropped_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_packets_dropped_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_packets_dropped_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
Logs, as collected by the monitoring stack and stored in Loki.
Every query here is LogQL, and the scope is Loki-discovered end to end: namespace, app and level come from Loki’s own label values rather than from the metrics pipeline. That is deliberate. Reading logs is frequently how an operator works out why the metrics pipeline is broken, and a logs dashboard that derived its scope from Prometheus would go blind at exactly the moment it is most needed.
The label contract the agent and gateway produce is documented under
Logs and Events. What matters here:
namespace, app, level and container are stream labels and belong in the
selector; pod, node, organization_name and the rest are structured
metadata and are filtered after a |. Narrowing the selector before the line
filters is the single biggest speedup.
Every selector here carries %%{mzLogJobFilter}, and it is not only a filter.
LogQL rejects a stream selector whose every matcher can match the empty string —
“queries require at least one regexp or equality matcher that does not have an
empty-compatible value” — and a dashboard built from =~ pickers is exactly
that shape. The job picker’s “All” is .+ rather than the discovered values, so
it always contributes a non-empty matcher and the selector parses whatever the
other pickers are set to. Without it, “All” everywhere is a query error rather
than a wide result.
The event queries in materialize-events.yaml need no such anchor: they pin
job="loki.source.kubernetes_events", which is already a non-empty equality
matcher, and a second job matcher would AND with it and zero the panel.
level is normalized by the pipeline where it can be and falls back to
UNKNOWN, so the levels present are a property of the workloads running rather
than a fixed vocabulary — which is why the dashboard discovers them instead of
hard-coding a list.
How the Materialize operator’s reconciliation loop is behaving.
orchestratord watches the Materialize, Balancer and Console resources
and drives each toward the state its spec asks for. One trip through that
work is a pass, and a pass moves through named steps. Both are counted by
outcome, and both are timed, which is what lets a stuck rollout say not just
that it is stuck but which phase it is stuck in.
These are the operator’s own metrics, scraped from its pods in the operator
namespace, so they are scoped by %%{mzOperatorNamespaceFilter} and by
nothing else. They carry no organization label, so the environment picker
does not narrow them: one operator reconciles every environment in the
cluster, and its loop is a single shared thing rather than a per-environment
one.
Only the replica holding the leadership lease reconciles. The others export the same metric families sitting at zero, which is why every query here sums across replicas rather than picking one out.
sum(
orchestratord_is_leader{namespace=~"materialize"}
)
sum(
environmentd_needs_update{namespace=~"materialize"}
)
sum(
rate(
orchestratord_reconciliations_total{namespace=~"materialize"}
[5m]
)
)
sum(
increase(
orchestratord_reconciliations_total{
namespace=~"materialize", outcome="failed"
}
[1h]
)
)
waiting climbs while the new generation’s pods come up, then
gives way to applied when they are ready.
sum by (outcome) (
rate(
orchestratord_reconciliations_total{namespace=~"materialize"}
[5m]
)
)
sum by (controller, event_type) (
rate(
orchestratord_reconciliations_total{
namespace=~"materialize", outcome="failed"
}
[5m]
)
)
histogram_quantile(0.5, sum by (le) (
rate(
orchestratord_reconciliation_duration_seconds_bucket{
namespace=~"materialize"
}
[5m]
)
))
histogram_quantile(0.9, sum by (le) (
rate(
orchestratord_reconciliation_duration_seconds_bucket{
namespace=~"materialize"
}
[5m]
)
))
histogram_quantile(0.99, sum by (le) (
rate(
orchestratord_reconciliation_duration_seconds_bucket{
namespace=~"materialize"
}
[5m]
)
))
histogram_quantile(0.99, sum by (le, step) (
rate(
orchestratord_reconciliation_step_duration_seconds_bucket{
namespace=~"materialize"
}
[5m]
)
))
sum by (step) (
rate(
orchestratord_reconciliation_steps_total{namespace=~"materialize"}
[5m]
)
)
sum by (step, outcome) (
rate(
orchestratord_reconciliation_steps_total{
namespace=~"materialize", outcome=~"failed|abandoned"
}
[5m]
)
)
Sources and sinks for a Materialize deployment — catalog shape, throughput, lag, and upstream/downstream health. Adapted from the Overview dashboard’s “Sources and Sinks” tab.
The clusterd-side throughput/lag/error metrics (mz_source_* / mz_sink_) carry
the long-form cluster_environmentd_materialize_cloud_ id labels. These queries
assume one Prometheus job per clusterd endpoint; if the same endpoint is
scraped by several jobs, a plain sum-rate reads N× — dedupe the job at the
deployment (fix the scrape config, or wrap the inner rate in max without (job)) rather than baking it into the canonical query.
_progress
subsources are excluded), matching mz_sources.
count(group by (id) (mz_storage_objects{materialize_cloud_organization_name=~".*", type="source"}))
default_zero(count_not_null(avg:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:source} by {id}))
_progress subsources), matching mz_sinks.
count(group by (id) (mz_storage_objects{materialize_cloud_organization_name=~".*", type="sink"}))
default_zero(count_not_null(avg:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:sink} by {id}))
INSERTs feed dataflows downstream. Mostly a catalog-shape signal — for
actual ingest activity look at source throughput.
max(sum by (instance) (mz_tables_count{materialize_cloud_organization_name=~".*"}))
default_zero(sum:${mzSqlPrefix}tables_count{${mzEnvironmentFilter}} by {instance})
count by (object_type) (
group by (id, object_type) (
mz_storage_objects{materialize_cloud_organization_name=~".*", type="source"}
)
) > 0
sum:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:source} by {object_type}
group by (id, object_type, connection_type, envelope_type, cluster_id) (
mz_storage_objects{materialize_cloud_organization_name=~".*", type="source"}
)
avg:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:source}
by {id,object_type,connection_type,envelope_type,cluster_id}
sum by (parent_source_id) (
max without (job) (rate(mz_source_bytes_received{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
) > 0
sum:mz_source_bytes_received{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {parent_source_id}.as_rate()
sum by (parent_source_id, cluster_environmentd_materialize_cloud_replica_id) (
max without (job) (rate(mz_source_messages_received{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
)
sum:mz_source_messages_received{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {parent_source_id,cluster_environmentd_materialize_cloud_replica_id}.as_rate()
sum by (source_id) (
max without (job) (rate(mz_source_offset_commit_failures{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
) > 0
(
max by (source_id) (mz_source_offset_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"})
> bool max by (source_id) (mz_source_offset_known{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"})
) > 0
sum:mz_source_offset_commit_failures{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {source_id}.as_rate()
max:mz_source_offset_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {source_id}
- max:mz_source_offset_known{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {source_id}
kafka / upsert, kafka / debezium,
iceberg / upsert. The envelope is how Materialize encodes changes:
upsert writes the latest value per key, debezium writes change
events with old+new values.
count by (object_type, envelope_type) (
group by (id, object_type, envelope_type) (
mz_storage_objects{materialize_cloud_organization_name=~".*", type="sink"}
)
) > 0
sum:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:sink} by {object_type,envelope_type}
sum by (sink_id) (
max without (job) (rate(mz_sink_bytes_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
) > 0
sum:mz_sink_bytes_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
clamp_min(
sum by (sink_id) (max without (job) (mz_sink_bytes_staged{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}))
- sum by (sink_id) (max without (job) (mz_sink_bytes_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"})),
0
)
clamp_min(
sum:mz_sink_bytes_staged{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}
- sum:mz_sink_bytes_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id},
0
)
COMMIT against the Iceberg catalog takes (write a snapshot manifest,
ask the catalog to atomically swap it in).
histogram_quantile(0.50, sum by (le) (max without (job) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))))
histogram_quantile(0.90, sum by (le) (max without (job) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))))
histogram_quantile(0.99, sum by (le) (max without (job) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))))
p50:mz_sink_iceberg_commit_duration_seconds{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
p90:mz_sink_iceberg_commit_duration_seconds{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
p99:mz_sink_iceberg_commit_duration_seconds{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_commit_failures{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_commit_conflicts{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum:mz_sink_iceberg_commit_failures{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum:mz_sink_iceberg_commit_conflicts{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_data_files_written{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_delete_files_written{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_snapshots_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum:mz_sink_iceberg_data_files_written{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum:mz_sink_iceberg_delete_files_written{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum:mz_sink_iceberg_snapshots_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum by (sink_id) (max without (job) (rate(mz_sink_rdkafka_txerrs{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum:mz_sink_rdkafka_txerrs{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum by (sink_id) (max without (job) (mz_sink_rdkafka_outbuf_msg_cnt{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}))
sum:mz_sink_rdkafka_outbuf_msg_cnt{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}
sum by (sink_id) (max without (job) (rate(mz_sink_rdkafka_connects{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum by (sink_id) (max without (job) (rate(mz_sink_rdkafka_disconnects{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum:mz_sink_rdkafka_connects{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum:mz_sink_rdkafka_disconnects{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
The breakdowns you reach for once node-health.yaml has told you a node is in
trouble: which mode the CPU is in, where the memory went, which device is
slow, and where the packets are being lost. Adapted from the Node Exporter
Full dashboard (https://grafana.com/grafana/dashboards/1860, revision 45).
Split from node-health.yaml on tier rather than on subject. These sit at
recommended, so a deployment collecting only the essential tier still gets
the health surface and pays nothing for the detail. Nothing here should back
an alert — if something here is worth paging on, it belongs in
node-health.yaml instead.
The conventions are the same as node-health.yaml: instance=~"$nodeList" rather
than =, every query wrapped in max by (instance, ...) (or min where low
is the bad direction) so a second scrape job cannot double-count, inner
aggregations carrying by (instance, job) so the outer wrapper is what
collapses job, and %%{interval} as the rate window.
Only collectors this chart’s allowlist enables are referenced. Notable
omissions, because the dashboard has panels for them and they will render
empty: node_processes_* (processes collector), node_interrupts_total
(interrupts), node_tcp_connection_states (tcpstat), node_systemd_*
(systemd), and node_textfile_scrape_error (textfile).
max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="system", instance=~"$nodeList"}[5m])
)
)
max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="user", instance=~"$nodeList"}[5m])
)
)
max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="iowait", instance=~"$nodeList"}[5m])
)
)
max by (instance) (
avg by (instance, job) (
sum without (mode) (
rate(node_cpu_seconds_total{mode=~".*irq", instance=~"$nodeList"}[5m])
)
)
)
max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="steal", instance=~"$nodeList"}[5m])
)
)
avg:node_cpu_seconds_total{instance:$nodeList, mode:system} by {instance}.as_rate()
avg:node_cpu_seconds_total{instance:$nodeList, mode:user} by {instance}.as_rate()
avg:node_cpu_seconds_total{instance:$nodeList, mode:iowait} by {instance}.as_rate()
avg:node_cpu_seconds_total{instance:$nodeList, mode:*irq} by {instance}.as_rate()
avg:node_cpu_seconds_total{instance:$nodeList, mode:steal} by {instance}.as_rate()
1 - max by (instance, cpu) (
rate(node_cpu_seconds_total{mode="idle", instance=~"$nodeList"}[5m])
)
1 - max:node_cpu_seconds_total{instance:$nodeList, mode:idle} by {instance,cpu}.as_rate()
/proc/schedstat.
max by (instance, cpu) (
rate(node_schedstat_waiting_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_schedstat_waiting_seconds_total{instance:$nodeList} by {instance,cpu}.as_rate()
max by (instance) (
rate(node_context_switches_total{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_intr_total{instance=~"$nodeList"}[5m])
)
max:node_context_switches_total{instance:$nodeList} by {instance}.as_rate()
max:node_intr_total{instance:$nodeList} by {instance}.as_rate()
max by (instance) (node_memory_MemTotal_bytes{instance=~"$nodeList"})
max by (instance) (
node_memory_MemTotal_bytes{instance=~"$nodeList"}
- node_memory_MemFree_bytes{instance=~"$nodeList"}
- node_memory_Cached_bytes{instance=~"$nodeList"}
- node_memory_Buffers_bytes{instance=~"$nodeList"}
- node_memory_SReclaimable_bytes{instance=~"$nodeList"}
)
max by (instance) (
node_memory_Cached_bytes{instance=~"$nodeList"}
+ node_memory_Buffers_bytes{instance=~"$nodeList"}
+ node_memory_SReclaimable_bytes{instance=~"$nodeList"}
)
min by (instance) (node_memory_MemFree_bytes{instance=~"$nodeList"})
max:node_memory_MemTotal_bytes{instance:$nodeList} by {instance}
max:node_memory_MemTotal_bytes{instance:$nodeList} by {instance}
- max:node_memory_MemFree_bytes{instance:$nodeList} by {instance}
- max:node_memory_Cached_bytes{instance:$nodeList} by {instance}
- max:node_memory_Buffers_bytes{instance:$nodeList} by {instance}
- max:node_memory_SReclaimable_bytes{instance:$nodeList} by {instance}
max:node_memory_Cached_bytes{instance:$nodeList} by {instance}
+ max:node_memory_Buffers_bytes{instance:$nodeList} by {instance}
+ max:node_memory_SReclaimable_bytes{instance:$nodeList} by {instance}
min:node_memory_MemFree_bytes{instance:$nodeList} by {instance}
max by (instance) (node_memory_Slab_bytes{instance=~"$nodeList"})
max by (instance) (node_memory_SReclaimable_bytes{instance=~"$nodeList"})
max by (instance) (node_memory_SUnreclaim_bytes{instance=~"$nodeList"})
max by (instance) (node_memory_Committed_AS_bytes{instance=~"$nodeList"})
max:node_memory_Slab_bytes{instance:$nodeList} by {instance}
max:node_memory_SReclaimable_bytes{instance:$nodeList} by {instance}
max:node_memory_SUnreclaim_bytes{instance:$nodeList} by {instance}
max:node_memory_Committed_AS_bytes{instance:$nodeList} by {instance}
max by (instance) (
rate(node_vmstat_pgfault{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pgmajfault{instance=~"$nodeList"}[5m])
)
max:node_vmstat_pgfault{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pgmajfault{instance:$nodeList} by {instance}.as_rate()
kswapd (background) or direct (an allocating thread).
max by (instance) (
rate(node_vmstat_pgscan_kswapd{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pgscan_direct{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pgsteal_kswapd{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pgsteal_direct{instance=~"$nodeList"}[5m])
)
max:node_vmstat_pgscan_kswapd{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pgscan_direct{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pgsteal_kswapd{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pgsteal_direct{instance:$nodeList} by {instance}.as_rate()
max by (instance, device) (
rate(node_disk_reads_completed_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_disk_writes_completed_total{instance=~"$nodeList"}[5m])
)
max:node_disk_reads_completed_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_disk_writes_completed_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
rate(node_disk_read_bytes_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_disk_written_bytes_total{instance=~"$nodeList"}[5m])
)
max:node_disk_read_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_disk_written_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
(rate(node_disk_reads_completed_total{instance=~"$nodeList"}[5m]) > bool 0)
*
(
rate(node_disk_read_time_seconds_total{instance=~"$nodeList"}[5m])
/ rate(node_disk_reads_completed_total{instance=~"$nodeList"}[5m])
)
)
max by (instance, device) (
(rate(node_disk_writes_completed_total{instance=~"$nodeList"}[5m]) > bool 0)
*
(
rate(node_disk_write_time_seconds_total{instance=~"$nodeList"}[5m])
/ rate(node_disk_writes_completed_total{instance=~"$nodeList"}[5m])
)
)
max:node_disk_read_time_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
/ max:node_disk_reads_completed_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_disk_write_time_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
/ max:node_disk_writes_completed_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
rate(node_disk_io_time_weighted_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_disk_io_time_weighted_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
min by (instance, mountpoint) (
node_filesystem_files_free{instance=~"$nodeList", fstype!="rootfs"}
)
/
max by (instance, mountpoint) (
node_filesystem_files{instance=~"$nodeList", fstype!="rootfs"}
)
min:node_filesystem_files_free{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
/ max:node_filesystem_files{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
max by (instance, device) (
rate(node_network_receive_bytes_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_network_transmit_bytes_total{instance=~"$nodeList"}[5m])
)
max:node_network_receive_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_network_transmit_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
(node_network_speed_bytes{instance=~"$nodeList"} > bool 0)
*
(
rate(node_network_receive_bytes_total{instance=~"$nodeList"}[5m])
/ node_network_speed_bytes{instance=~"$nodeList"}
)
)
max by (instance, device) (
(node_network_speed_bytes{instance=~"$nodeList"} > bool 0)
*
(
rate(node_network_transmit_bytes_total{instance=~"$nodeList"}[5m])
/ node_network_speed_bytes{instance=~"$nodeList"}
)
)
max:node_network_receive_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
/ max:node_network_speed_bytes{instance:$nodeList} by {instance,device}
max:node_network_transmit_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
/ max:node_network_speed_bytes{instance:$nodeList} by {instance,device}
min by (instance, device) (
node_network_up{instance=~"$nodeList"}
)
min by (instance, device) (
node_network_carrier{instance=~"$nodeList"}
)
min:node_network_up{instance:$nodeList} by {instance,device}
min:node_network_carrier{instance:$nodeList} by {instance,device}
max by (instance, cpu) (
rate(node_softnet_processed_total{instance=~"$nodeList"}[5m])
)
max:node_softnet_processed_total{instance:$nodeList} by {instance,cpu}.as_rate()
max by (instance, cpu) (
rate(node_softnet_dropped_total{instance=~"$nodeList"}[5m])
)
max:node_softnet_dropped_total{instance:$nodeList} by {instance,cpu}.as_rate()
max by (instance, cpu) (
rate(node_softnet_times_squeezed_total{instance=~"$nodeList"}[5m])
)
max:node_softnet_times_squeezed_total{instance:$nodeList} by {instance,cpu}.as_rate()
max by (instance) (node_sockstat_TCP_inuse{instance=~"$nodeList"})
max by (instance) (node_sockstat_TCP_alloc{instance=~"$nodeList"})
max by (instance) (node_sockstat_TCP_orphan{instance=~"$nodeList"})
max by (instance) (node_sockstat_TCP_tw{instance=~"$nodeList"})
max:node_sockstat_TCP_inuse{instance:$nodeList} by {instance}
max:node_sockstat_TCP_alloc{instance:$nodeList} by {instance}
max:node_sockstat_TCP_orphan{instance:$nodeList} by {instance}
max:node_sockstat_TCP_tw{instance:$nodeList} by {instance}
max by (instance) (node_sockstat_TCP_mem_bytes{instance=~"$nodeList"})
max by (instance) (node_sockstat_UDP_mem_bytes{instance=~"$nodeList"})
max:node_sockstat_TCP_mem_bytes{instance:$nodeList} by {instance}
max:node_sockstat_UDP_mem_bytes{instance:$nodeList} by {instance}
max by (instance) (
rate(node_netstat_Tcp_RetransSegs{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_TcpExt_TCPSynRetrans{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_Tcp_OutSegs{instance=~"$nodeList"}[5m])
)
max:node_netstat_Tcp_RetransSegs{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_TcpExt_TCPSynRetrans{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_Tcp_OutSegs{instance:$nodeList} by {instance}.as_rate()
max by (instance) (
rate(node_netstat_TcpExt_ListenOverflows{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_TcpExt_ListenDrops{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_TcpExt_TCPRcvQDrop{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_TcpExt_TCPTimeouts{instance=~"$nodeList"}[5m])
)
max:node_netstat_TcpExt_ListenOverflows{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_TcpExt_ListenDrops{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_TcpExt_TCPRcvQDrop{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_TcpExt_TCPTimeouts{instance:$nodeList} by {instance}.as_rate()
max by (instance) (
rate(node_netstat_Udp_InErrors{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_Udp_RcvbufErrors{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_Udp_NoPorts{instance=~"$nodeList"}[5m])
)
max:node_netstat_Udp_InErrors{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_Udp_RcvbufErrors{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_Udp_NoPorts{instance:$nodeList} by {instance}.as_rate()
max by (instance) (
node_udp_queues{ip="v4", queue="rx", instance=~"$nodeList"}
)
max by (instance) (
node_udp_queues{ip="v4", queue="tx", instance=~"$nodeList"}
)
max:node_udp_queues{instance:$nodeList, ip:v4, queue:rx} by {instance}
max:node_udp_queues{instance:$nodeList, ip:v4, queue:tx} by {instance}
max by (instance, device) (
node_arp_entries{instance=~"$nodeList"}
)
max:node_arp_entries{instance:$nodeList} by {instance,device}
min by (instance) (
node_timex_sync_status{instance=~"$nodeList"}
)
min:node_timex_sync_status{instance:$nodeList} by {instance}
max by (instance) (node_timex_offset_seconds{instance=~"$nodeList"})
max by (instance) (node_timex_maxerror_seconds{instance=~"$nodeList"})
max by (instance) (node_timex_estimated_error_seconds{instance=~"$nodeList"})
max:node_timex_offset_seconds{instance:$nodeList} by {instance}
max:node_timex_maxerror_seconds{instance:$nodeList} by {instance}
max:node_timex_estimated_error_seconds{instance:$nodeList} by {instance}
min by (instance) (node_entropy_available_bits{instance=~"$nodeList"})
max by (instance) (node_entropy_pool_size_bits{instance=~"$nodeList"})
min:node_entropy_available_bits{instance:$nodeList} by {instance}
max:node_entropy_pool_size_bits{instance:$nodeList} by {instance}
max by (instance, collector) (
node_scrape_collector_duration_seconds{instance=~"$nodeList"}
)
max:node_scrape_collector_duration_seconds{instance:$nodeList} by {instance,collector}
Node-level health: the at-a-glance answers to “is this machine in trouble”, adapted from the Node Exporter Full dashboard (https://grafana.com/grafana/dashboards/1860, revision 45) and narrowed to what actually alerts.
These read node-exporter, NOT Materialize metrics. Only collectors this
chart’s allowlist enables are referenced — see the Node Exporter section of
the chart’s values reference for the list and the reasoning. The deeper
breakdowns live in node-debug.yaml at the recommended tier.
Three conventions apply to every query here:
instance=~"$nodeList", never instance="$nodeList". A regex match makes the
selector work unchanged whether the dashboard variable resolves to one node
or many, so the same query backs a single-node view and a fleet view.
Every query is wrapped in max by (instance, ...), which drops job.
A node should only ever be scraped by one job. If a second one appears —
a pre-existing node-exporter alongside ours, or a migration with both
running — the same series arrives twice under different job labels, and
every sum() silently doubles while every binary operation between two
metrics loses its match. Aggregating job away makes both failure modes
impossible rather than merely unlikely. max is the default; min where
low is the bad direction (available memory, free space, collector
success), so the aggregate always reports the worst case rather than hiding
it behind a healthy duplicate.
Where an inner aggregation is needed (averaging across CPUs, summing across
devices), it carries by (instance, job) and the outer max/min collapses
job afterwards. Aggregating both in one step would blend two jobs'
readings into one number instead of picking one.
%%{interval} is the rate window, including its brackets.
1 - max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="idle", instance=~"$nodeList"}[5m])
)
)
1 - avg:node_cpu_seconds_total{mode:idle, instance:$nodeList} by {instance}.as_rate()
max by (instance) (node_load1{instance=~"$nodeList"})
/
max by (instance) (
count by (instance, job) (
count by (instance, job, cpu) (node_cpu_seconds_total{instance=~"$nodeList"})
)
)
max:node_load1{instance:$nodeList} by {instance}
/ count_not_null(avg:node_cpu_seconds_total{mode:idle, instance:$nodeList} by {cpu})
max by (instance) (
rate(node_pressure_cpu_waiting_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_pressure_cpu_waiting_seconds_total{instance:$nodeList} by {instance}.as_rate()
MemAvailable.
min by (instance) (
node_memory_MemAvailable_bytes{instance=~"$nodeList"}
)
/
max by (instance) (
node_memory_MemTotal_bytes{instance=~"$nodeList"}
)
min:node_memory_MemAvailable_bytes{instance:$nodeList} by {instance}
/ max:node_memory_MemTotal_bytes{instance:$nodeList} by {instance}
waiting), and the fraction where every task was (stalled).
max by (instance) (
rate(node_pressure_memory_waiting_seconds_total{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_pressure_memory_stalled_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_pressure_memory_waiting_seconds_total{instance:$nodeList} by {instance}.as_rate()
max:node_pressure_memory_stalled_seconds_total{instance:$nodeList} by {instance}.as_rate()
(
(
max by (instance) (node_memory_SwapTotal_bytes{instance=~"$nodeList"})
-
min by (instance) (node_memory_SwapFree_bytes{instance=~"$nodeList"})
)
/
max by (instance) (node_memory_SwapTotal_bytes{instance=~"$nodeList"})
)
and
(max by (instance) (node_memory_SwapTotal_bytes{instance=~"$nodeList"}) > 0)
(
max:node_memory_SwapTotal_bytes{instance:$nodeList} by {instance}
- min:node_memory_SwapFree_bytes{instance:$nodeList} by {instance}
)
/ max:node_memory_SwapTotal_bytes{instance:$nodeList} by {instance}
vmstat.
max by (instance) (
rate(node_vmstat_pswpin{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pswpout{instance=~"$nodeList"}[5m])
)
max:node_vmstat_pswpin{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pswpout{instance:$nodeList} by {instance}.as_rate()
max by (instance) (
rate(node_vmstat_oom_kill{instance=~"$nodeList"}[5m])
)
max:node_vmstat_oom_kill{instance:$nodeList} by {instance}.as_rate()
min by (instance, mountpoint) (
node_filesystem_avail_bytes{instance=~"$nodeList", fstype!="rootfs"}
)
/
max by (instance, mountpoint) (
node_filesystem_size_bytes{instance=~"$nodeList", fstype!="rootfs"}
)
min:node_filesystem_avail_bytes{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
/ max:node_filesystem_size_bytes{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
max by (instance, mountpoint) (
node_filesystem_readonly{instance=~"$nodeList", fstype!="rootfs"}
)
max:node_filesystem_readonly{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
max by (instance, device) (
rate(node_disk_io_time_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_disk_io_time_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance) (node_filefd_allocated{instance=~"$nodeList"})
/
max by (instance) (node_filefd_maximum{instance=~"$nodeList"})
max:node_filefd_allocated{instance:$nodeList} by {instance}
/ max:node_filefd_maximum{instance:$nodeList} by {instance}
max by (instance, device) (
rate(node_network_receive_errs_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_network_transmit_errs_total{instance=~"$nodeList"}[5m])
)
max:node_network_receive_errs_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_network_transmit_errs_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
rate(node_network_receive_drop_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_network_transmit_drop_total{instance=~"$nodeList"}[5m])
)
max:node_network_receive_drop_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_network_transmit_drop_total{instance:$nodeList} by {instance,device}.as_rate()
sum by (instance) (
max by (instance, device) (
rate(node_network_receive_bytes_total{instance=~"$nodeList"}[5m])
)
)
sum by (instance) (
max by (instance, device) (
rate(node_network_transmit_bytes_total{instance=~"$nodeList"}[5m])
)
)
max by (instance) (node_nf_conntrack_entries{instance=~"$nodeList"})
/
max by (instance) (node_nf_conntrack_entries_limit{instance=~"$nodeList"})
max:node_nf_conntrack_entries{instance:$nodeList} by {instance}
/ max:node_nf_conntrack_entries_limit{instance:$nodeList} by {instance}
min by (instance) (
node_time_seconds{instance=~"$nodeList"} - node_boot_time_seconds{instance=~"$nodeList"}
)
min:node_time_seconds{instance:$nodeList} by {instance}
- max:node_boot_time_seconds{instance:$nodeList} by {instance}
min by (instance, collector) (
node_scrape_collector_success{instance=~"$nodeList"}
)
min:node_scrape_collector_success{instance:$nodeList} by {instance,collector}