Is telemetry collection healthy, and if not, which stage of it broke.
Alloy runs in two roles in this stack. The agent is a DaemonSet with one pod per node. It reads container log files and the node journal, and pushes both to the gateway. The gateway is a Deployment. It receives the agents' logs, collects Kubernetes events, scrapes every metrics target in the cluster, and delivers all of it to the stores the dashboards read.
Every signal in this file travels through the gateway before it can be read. The gateway scrapes Alloy’s own metrics, and Alloy’s own logs reach Loki through the gateway. An empty panel here can therefore mean the gateway is down rather than that nothing happened. When every panel is empty at once, the first check is whether any gateway pod is running.
Three variables, written literally on the same precedent as infra-loki.yaml.
$alloyNamespace selects where the collectors run.$alloyRole selects alloy-agent, alloy-gateway, or both, matched
against app. Metrics and logs both take app from the pod’s
app.kubernetes.io/name label, so one picker serves both engines. Its
“All” value is the literal pair rather than .+, which makes it the anchor
that keeps every selector to this chart’s collectors. thanos-ruler
publishes the same prometheus_remote_storage_* family, and an unanchored
selector would add its remote-write queue to the gateway’s.$alloyPod selects one collector, most often the agent on one node.kube-state-metrics and cAdvisor series carry no app label. Those queries
scope with and on (namespace, pod) up{…} against the collectors’ own scrape
targets instead, which applies all three pickers without assuming anything
about pod names.
Queries over up and scrape_* for the targets the gateway scrapes carry
no Alloy scope. Those series describe the target, not the collector, and
nothing on them records which gateway replica produced them. The same holds
for the remote-write senders’ own queues, which are found by their url.
Kubernetes events carry the involved object’s name rather than an app
label. The event queries match name against the role pattern, as
${alloyRole:regex} since it is embedded in a longer pattern, and against the
pod picker. They also match the pre-install validation Jobs, whose names
follow the release rather than the role, as .+-validate-(agent|gateway).
Several counts here take the form sum(A) or 0 * sum(B), where B is the
population A is drawn from. Alloy creates an unhealthy-component series only
once a component is unhealthy, so sum(A) alone is empty on a healthy
install. The or arm turns that into a zero, and only when B exists. A
panel reading zero has heard from the collectors; a panel reading nothing
has not.
loki_process_dropped_lines_total counts two different things under one
name. The debug tap in both pipelines forks a copy of the stream, keeps about
1% of it, and drops the rest with reasons sampleDebug, skip self and
skip loki. The gateway also drops Alloy’s own loki.echo output with reason
alloy loki.echo, which stops the tap re-collecting itself. None of those
lines were bound for a store. Every other reason is a guard on the main path,
and a line dropped there is lost. infra.alloy.log_pipeline.guard_drops
excludes the four deliberate reasons by name, so a reason added to a pipeline
later shows up as a loss until it is classified here.
clamp_min(
sum(
kube_daemonset_status_desired_number_scheduled{
namespace=~"$alloyNamespace",
daemonset="alloy-agent"
}
)
- (
sum(
up{
app="alloy-agent",
namespace=~"$alloyNamespace"
}
)
or vector(0)
),
0
)
sum(
alloy_component_controller_running_components{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod",
health_type!="healthy"
}
)
or
0 * sum(
alloy_component_controller_running_components{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
count(
alloy_config_last_load_successful{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
} == 0
)
or
0 * count(
alloy_config_last_load_successful{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum(
rate(
loki_write_dropped_entries_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
(
(
sum(
rate(
prometheus_remote_storage_samples_failed_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
or vector(0)
)
+ (
sum(
rate(
otelcol_exporter_send_failed_metric_points_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
or vector(0)
)
)
and on ()
(
sum(
rate(
prometheus_remote_storage_samples_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
or
sum(
rate(
otelcol_exporter_sent_metric_points_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
)
(
(
sum(
rate(
loki_source_api_request_duration_seconds_count{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod",
route!="other",
status_code!~"2.."
}
[5m]
)
)
or vector(0)
)
+ (
sum(
rate(
prometheus_receive_http_request_duration_seconds_count{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod",
route!="other",
status_code!~"2.."
}
[5m]
)
)
or vector(0)
)
)
and on ()
(
sum(
rate(
loki_source_api_request_duration_seconds_count{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
or
sum(
rate(
prometheus_receive_http_request_duration_seconds_count{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
)
max by (app, pod) (
up{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (namespace, pod) (
increase(
kube_pod_container_status_restarts_total{
namespace=~"$alloyNamespace",
container="alloy"
}
[1h]
)
) > 0
and on (namespace, pod)
max_over_time(
up{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[1h]
)
count by (app, version) (
alloy_build_info{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (app, host) (
rate(
loki_write_sent_entries_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (component_id) (
rate(
prometheus_remote_storage_samples_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (component_id) (
rate(
otelcol_exporter_sent_metric_points_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
loki_source_file_read_lines_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
+ (
sum(
rate(
loki_source_journal_target_lines_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
or vector(0)
)
sum(
rate(
loki_write_sent_entries_total{
app=~"$alloyRole",
app="alloy-gateway",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
histogram_quantile(
0.99,
sum by (le, app) (
rate(
loki_write_entry_propagation_latency_seconds_bucket{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
)
sum by (pod) (
loki_source_file_files_active_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (pod) (
rate(
loki_source_file_read_lines_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (pod) (
rate(
loki_source_journal_target_lines_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (host, status_code) (
rate(
loki_write_request_duration_seconds_count{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (app, host) (
rate(
loki_write_batch_retries_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (app, reason) (
rate(
loki_write_dropped_entries_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
) > 0
or on (app)
0 * sum by (app) (
rate(
loki_write_sent_entries_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
histogram_quantile(
0.99,
sum by (le, host) (
rate(
loki_write_request_duration_seconds_bucket{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
)
sum by (app, reason) (
rate(
loki_process_dropped_lines_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod",
reason!~"sampleDebug|skip self|skip loki|alloy loki.echo"
}
[5m]
)
) > 0
or on (app)
0 * sum by (app) (
rate(
loki_process_dropped_lines_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (pod) (
rate(
loki_process_cri_lines_truncated_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
prometheus_target_scrape_pool_targets{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
count(up == 0)
or
0 * count(up)
sum(
rate(
prometheus_remote_storage_samples_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
clamp_min(
max by (pod, component_id) (
prometheus_remote_storage_highest_timestamp_in_seconds{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
- ignoring (url, remote_name) group_right (pod)
prometheus_remote_storage_queue_highest_sent_timestamp_seconds{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
),
0
) < 31536000
max(
clamp_min(
prometheus_remote_storage_highest_timestamp_in_seconds{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
- ignoring (url, remote_name) group_right (pod)
prometheus_remote_storage_queue_highest_sent_timestamp_seconds{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
},
0
) < 31536000
)
max by (job, namespace, pod, instance) (
up == 0
)
sum by (scrape_job) (
prometheus_target_scrape_pool_targets{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum(
rate(
prometheus_target_scrapes_exceeded_sample_limit_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
prometheus_target_scrapes_exceeded_body_size_limit_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
prometheus_target_scrape_pool_exceeded_target_limit_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
prometheus_target_scrape_pool_exceeded_label_limits_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
prometheus_target_scrapes_sample_out_of_order_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
prometheus_target_scrapes_sample_duplicate_timestamp_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
prometheus_target_scrapes_sample_out_of_bounds_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
topk(
10,
max by (job) (
scrape_duration_seconds
)
)
topk(
10,
sum by (job) (
scrape_samples_post_metric_relabeling
)
)
sum by (component_id) (
rate(
prometheus_forwarded_samples_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod",
component_id=~"prometheus\\.(scrape|operator)\\..*"
}
[5m]
)
)
sum by (component_id) (
rate(
prometheus_remote_storage_samples_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (component_id) (
rate(
prometheus_remote_storage_samples_failed_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (component_id) (
rate(
prometheus_remote_storage_samples_retried_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
max by (component_id) (
prometheus_remote_storage_shards{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
max by (component_id) (
prometheus_remote_storage_shards_desired{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
max by (component_id) (
prometheus_remote_storage_shards_max{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
histogram_quantile(
0.99,
sum by (le, component_id) (
rate(
prometheus_remote_storage_sent_batch_duration_seconds_bucket{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
)
sum by (pod) (
prometheus_remote_write_wal_storage_active_series{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (pod) (
prometheus_tsdb_wal_storage_size_bytes{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum(
rate(
prometheus_remote_write_wal_corruptions_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
prometheus_tsdb_wal_writes_failed_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
prometheus_remote_write_wal_checkpoint_creations_failed_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
prometheus_tsdb_wal_truncations_failed_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (component_id) (
rate(
otelcol_exporter_sent_metric_points_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (component_id) (
rate(
otelcol_exporter_sent_log_records_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (component_id) (
rate(
otelcol_exporter_send_failed_metric_points_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (component_id) (
rate(
otelcol_exporter_send_failed_log_records_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
max by (component_id, data_type) (
otelcol_exporter_queue_size{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
/ otelcol_exporter_queue_capacity{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (component_id) (
rate(
otelcol_processor_filter_datapoints_filtered_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
loki_source_api_entries_written{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (route, status_code) (
rate(
loki_source_api_request_duration_seconds_count{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
histogram_quantile(
0.99,
sum by (le) (
rate(
loki_source_api_request_duration_seconds_bucket{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
)
sum(
rate(
prometheus_forwarded_samples_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod",
component_id=~"prometheus\\.receive_http\\..*"
}
[5m]
)
)
sum(
rate(
prometheus_forwarded_samples_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod",
component_id=~"prometheus\\.receive_http\\..*"
}
[5m]
)
)
sum(
rate(
prometheus_api_remote_write_invalid_labels_samples_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (route, status_code) (
rate(
prometheus_receive_http_request_duration_seconds_count{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
histogram_quantile(
0.99,
sum by (le) (
rate(
prometheus_receive_http_request_duration_seconds_bucket{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
)
sum by (job) (
rate(
prometheus_remote_storage_samples_total{
url=~".*alloy-gateway.*"
}
[5m]
)
)
sum by (job) (
rate(
prometheus_remote_storage_samples_failed_total{
url=~".*alloy-gateway.*"
}
[5m]
)
)
sum by (job) (
rate(
prometheus_remote_storage_samples_retried_total{
url=~".*alloy-gateway.*"
}
[5m]
)
)
sum(
rate(
otelcol_receiver_accepted_metric_points_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
otelcol_receiver_refused_metric_points_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
otelcol_receiver_accepted_log_records_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum(
rate(
otelcol_receiver_refused_log_records_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
sum by (pod, health_type) (
alloy_component_controller_running_components{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod",
health_type!="healthy"
}
)
or on (pod)
0 * sum by (pod) (
alloy_component_controller_running_components{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (pod) (
alloy_component_controller_running_components{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
max by (app, pod, sha256) (
alloy_config_hash{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
* on (namespace, pod) group_left ()
alloy_config_last_load_successful{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
max by (app, pod) (
time()
- alloy_resources_process_start_time_seconds{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
count by (app) (
count by (app, sha256) (
alloy_config_hash{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
)
sum by (app) (
rate(
alloy_component_evaluation_seconds_count{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
histogram_quantile(
0.99,
sum by (le, app) (
rate(
alloy_component_evaluation_seconds_bucket{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
)
sum by (app, component_id) (
rate(
alloy_component_evaluation_slow_seconds{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
or on (app)
0 * sum by (app) (
rate(
alloy_component_evaluation_seconds_count{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
max by (pod) (
alloy_component_evaluation_queue_size{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (pod) (
cluster_node_peers{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
count(
cluster_node_info{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
max by (pod) (
cluster_node_gossip_health_score{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (pod) (
prometheus_target_scrape_pool_targets{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (pod) (
rate(
cluster_transport_tx_packets_failed_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
+ sum by (pod) (
rate(
cluster_transport_stream_tx_packets_failed_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
min by (pod, state) (
cluster_ready_for_traffic{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
* on (namespace, pod) group_left (state)
cluster_node_info{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
max by (namespace, pod) (
rate(
container_cpu_usage_seconds_total{
namespace=~"$alloyNamespace",
container="alloy"
}
[5m]
)
)
/ max by (namespace, pod) (
kube_pod_container_resource_limits{
namespace=~"$alloyNamespace",
container="alloy",
resource="cpu"
}
)
and on (namespace, pod)
up{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
sum by (namespace, pod) (
rate(
container_cpu_cfs_throttled_periods_total{
namespace=~"$alloyNamespace",
container="alloy"
}
[5m]
)
)
/ sum by (namespace, pod) (
rate(
container_cpu_cfs_periods_total{
namespace=~"$alloyNamespace",
container="alloy"
}
[5m]
)
)
and on (namespace, pod)
up{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
max by (namespace, pod) (
container_memory_working_set_bytes{
namespace=~"$alloyNamespace",
container="alloy"
}
)
/ max by (namespace, pod) (
kube_pod_container_resource_limits{
namespace=~"$alloyNamespace",
container="alloy",
resource="memory"
}
)
and on (namespace, pod)
up{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
GOMEMLIMIT sets for it.
max by (pod) (
(
go_memstats_sys_bytes{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
- go_memstats_heap_released_bytes{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
/ go_gc_gomemlimit_bytes{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
sum by (pod) (
rate(
otelcol_processor_memory_limiter_refused_metric_points_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
or
0 * sum by (pod) (
rate(
otelcol_processor_memory_limiter_accepted_metric_points_total{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
[5m]
)
)
max by (pod) (
go_goroutines{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
max by (pod) (
process_open_fds{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
/ process_max_fds{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
)
max by (namespace, pod, reason) (
kube_pod_container_status_last_terminated_reason{
namespace=~"$alloyNamespace",
container="alloy"
}
) == 1
and on (namespace, pod)
up{
app=~"$alloyRole",
namespace=~"$alloyNamespace",
pod=~"$alloyPod"
}
How the cluster’s nodes follow its pods, whatever does the following.
Three things add nodes to a Materialize deployment, depending on the cloud: Karpenter on EKS, and the managed cluster autoscaler on GKE and AKS. They expose nothing in common, so everything here is read from what every cluster has instead:
Nominated and Launched, the cluster autoscaler’s TriggeredScaleUp
and NotTriggerScaleUp, the scheduler’s FailedScheduling, and the node
lifecycle’s RegisteredNode and DeletingNode.Karpenter’s own metrics, which say far more on EKS, are the Karpenter
dashboard’s (infra-karpenter.yaml).
%%{nodePools} gives every node its pool, instance type and zone, from
kube_node_labels. The pool is whichever label the node’s provisioner
sets: a Karpenter NodePool, an EKS managed node group, a GKE node pool or an
AKS agent pool. kube-state-metrics publishes those labels only because the
chart’s metricLabelsAllowlist names them, and publishes no
kube_node_labels at all without it.
Every kube-state-metrics family here is read through max by its own
identity first, because each extra kube-state-metrics replica reports every
object again. The same shape infra-nodes.yaml uses.
Provider samples arrive with the pull, every five minutes by default, so
they are read through last_over_time(...[15m]) and never rated over
$__rate_interval; infra-cloud.yaml explains why.
sum(max by (namespace, pod) (kube_pod_status_phase{phase="Pending"}))
sum(max by (namespace, pod) (kube_pod_status_scheduled{condition="false"}))
count(max by (node) (kube_node_info))
sum(1 - max by (node) (kube_node_status_condition{condition="Ready", status="true"}))
count(
max by (node) (kube_node_info)
unless max by (node) (kube_node_info offset ${rangeWindow})
)
or 0 * count(max by (node) (kube_node_info))
count(
max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_current_replicas)
>= max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas)
)
or 0 * count(max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas))
sum by (namespace) (max by (namespace, pod) (kube_pod_status_scheduled{condition="false"})) > 0
sum by (namespace) (max by (namespace, pod) (kube_pod_status_phase{phase="Pending"})) > 0
count by (pool) (${nodePools})
count by (instance_type) (${nodePools})
count by (zone) (${nodePools})
sum(
max by (namespace, pod, container, node) (kube_pod_container_resource_requests{resource="cpu", node!=""})
unless on (namespace, pod) (max by (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"}) == 1)
)
/
sum(max by (node) (kube_node_status_allocatable{resource="cpu"}))
sum(
max by (namespace, pod, container, node) (kube_pod_container_resource_requests{resource="memory", node!=""})
unless on (namespace, pod) (max by (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"}) == 1)
)
/
sum(max by (node) (kube_node_status_allocatable{resource="memory"}))
sum by (pool) (
sum by (node) (
max by (namespace, pod, container, node) (kube_pod_container_resource_requests{resource="cpu", node!=""})
unless on (namespace, pod) (max by (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"}) == 1)
)
* on (node) group_left (pool) ${nodePools}
)
/
sum by (pool) (
max by (node) (kube_node_status_allocatable{resource="cpu"})
* on (node) group_left (pool) ${nodePools}
)
sum by (pool) (
sum by (node) (
max by (namespace, pod, container, node) (kube_pod_container_resource_requests{resource="memory", node!=""})
unless on (namespace, pod) (max by (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"}) == 1)
)
* on (node) group_left (pool) ${nodePools}
)
/
sum by (pool) (
max by (node) (kube_node_status_allocatable{resource="memory"})
* on (node) group_left (pool) ${nodePools}
)
sum by (pool) (
(1 - max by (node) (kube_node_status_condition{condition="Ready", status="true"}))
* on (node) group_left (pool) ${nodePools}
)
sum by (pool) (
max by (node) (kube_node_spec_unschedulable)
* on (node) group_left (pool) ${nodePools}
)
(time() - max by (node) (kube_node_created))
* on (node) group_left (pool, instance_type, zone) ${nodePools}
(time() - max by (namespace, pod) (kube_pod_created))
and on (namespace, pod) (max by (namespace, pod) (kube_pod_status_scheduled{condition="false"}) == 1)
max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_current_replicas)
max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_desired_replicas)
max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_min_replicas)
max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas)
max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_current_replicas)
/
max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas)
sum by (condition) (
max by (namespace, horizontalpodautoscaler, condition) (
kube_horizontalpodautoscaler_status_condition{condition=~"AbleToScale|ScalingActive", status="false"}
)
)
sum(last_over_time(aws_ec2_status_check_failed_system_maximum[15m]))
sum(last_over_time(aws_ec2_status_check_failed_instance_maximum[15m]))
sum(last_over_time(aws_ec2_status_check_failed_attached_ebs_maximum[15m]))
max by (tag_eks_nodegroup_name) (last_over_time(aws_autoscaling_group_desired_capacity_average[15m]))
max by (tag_eks_nodegroup_name) (last_over_time(aws_autoscaling_group_in_service_instances_average[15m]))
max by (tag_eks_nodegroup_name) (last_over_time(aws_autoscaling_group_max_size_average[15m]))
max(last_over_time(aws_usage_resource_count_maximum{dimension_Resource="vCPU"}[15m]))
max by (vm_family, location) (
last_over_time(stackdriver_compute_googleapis_com_location_compute_googleapis_com_quota_cpus_per_vm_family_usage{limit_name=~".*per-project-region"}[15m])
)
/
max by (vm_family, location) (
last_over_time(stackdriver_compute_googleapis_com_location_compute_googleapis_com_quota_cpus_per_vm_family_limit{limit_name=~".*per-project-region"}[15m])
)
max by (vm_family, location) (
last_over_time(stackdriver_compute_googleapis_com_location_compute_googleapis_com_quota_local_ssd_total_storage_per_vm_family_usage{limit_name=~".*per-project-region"}[15m])
)
/
max by (vm_family, location) (
last_over_time(stackdriver_compute_googleapis_com_location_compute_googleapis_com_quota_local_ssd_total_storage_per_vm_family_limit{limit_name=~".*per-project-region"}[15m])
)
sum by (vm_family, location) (
increase({__name__=~"stackdriver_compute_googleapis_com_location_compute_googleapis_com_quota_cpus_per_vm_family_exceeded(_total)?"}[1h])
)
sum by (vm_family, location) (
increase({__name__=~"stackdriver_compute_googleapis_com_location_compute_googleapis_com_quota_local_ssd_total_storage_per_vm_family_exceeded(_total)?"}[1h])
)
max by (resourceName) (
last_over_time(azure_microsoft_containerservice_managedclusters_cluster_autoscaler_unschedulable_pods_count_maximum_count[15m])
)
min by (resourceName) (
last_over_time(azure_microsoft_containerservice_managedclusters_cluster_autoscaler_cluster_safe_to_autoscale_minimum_count[15m])
)
max by (resourceName) (
last_over_time(azure_microsoft_containerservice_managedclusters_cluster_autoscaler_scale_down_in_cooldown_maximum_count[15m])
)
max by (resourceName) (
last_over_time(azure_microsoft_containerservice_managedclusters_cluster_autoscaler_unneeded_nodes_count_maximum_count[15m])
)
sum by (resourceName) (
1 - min by (resourceName, dimensionVmname) (
last_over_time(azure_microsoft_compute_virtualmachinescalesets_vmavailabilitymetric_minimum_count[15m])
)
)
What the cloud provider publishes about the managed services a Materialize deployment depends on: the metadata database and the object storage buckets.
These are the families the gateway pulls from CloudWatch, Cloud Monitoring and
Azure Monitor when pipeline.metrics.provider.* is enabled — see
packages/alloy-pipelines/gateway-provider.yaml, whose metric sets are the
ones named here. Each family belongs to exactly one provider, so every query
below that covers all three is a list of one expression per provider, and on
any given install two of them return nothing. That is deliberate: it is
cheaper and more honest than a normalization layer, and it keeps each
provider’s own units and resource names visible.
The client’s view of the same dependencies — what Materialize experienced —
is materialize-consensus.yaml and materialize-persist.yaml. The external
dependency design doc records why the two are separate dashboards.
Each provider speaks its own: CloudWatch and Azure Monitor publish percent
as 0–100, Cloud Monitoring publishes the same quantities as 0–1 (its unit
label reads 10^2.%). Everything here is converted to 0–1 so a panel can
pin its axis. Transaction-ID consumption is converted to a fraction of the
2^31 wraparound limit, which is what Cloud SQL’s own ratio is measured
against.
Provider samples are sparse and old. The pull runs every five minutes by
default; CloudWatch and Azure Monitor samples are stamped at the pull,
Cloud SQL’s arrive about three minutes old and GCS’s over ten. A default
five-minute lookback would therefore draw gaps, so every expression reads
its series through last_over_time: 15 minutes for the scrape-stamped
providers and Cloud SQL, 30 for GCS. The cost is that a panel goes on
showing a stopped pull’s last value for that long. rate() is never used
on a provider counter for the same reason: $__rate_interval is sized
from the datasource’s scrape interval, which is far shorter than a pull.
| Provider | Database | Bucket |
|---|---|---|
| CloudWatch | dimension_DBInstanceIdentifier |
dimension_BucketName |
| Cloud Monitoring | database_id (project:instance) |
bucket_name |
| Azure Monitor | resourceName |
resourceName (the storage account) |
Azure metric names are the exporter’s azure_{type}_{metric}_{aggregation}_{unit},
lowercased, where the type is the metric namespace when one is set — hence
…_storageaccounts_blobservices_… for the blob families.
max by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_cpuutilization_average[15m])
) / 100
max by (database_id) (
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_cpu_utilization[15m])
)
max by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_cpu_percent_average_percent[15m])
) / 100
max by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_database_connections_maximum[15m])
)
sum by (database_id) (
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_postgresql_num_backends[15m])
)
max by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_active_connections_maximum_count[15m])
)
sum(mz_persist_postgres_connpool_size) + (sum(mz_ts_oracle_postgres_connpool_size) or vector(0))
max by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_maximum_used_transaction_ids_maximum[15m])
) / 2147483648
max by (database_id) (
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_postgresql_transaction_id_utilization[15m])
)
max by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_maximum_used_transactionids_maximum_count[15m])
) / 2147483648
min by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_free_storage_space_minimum[15m])
)
min by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_freeable_memory_minimum[15m])
)
db.t*) instances.
min by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_cpucredit_balance_minimum[15m])
)
max by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_read_latency_average[15m])
)
max by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_write_latency_average[15m])
)
max by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_disk_queue_depth_average[15m])
)
min by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_burst_balance_minimum[15m])
) / 100
min by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_ebsiobalance_percent_minimum[15m])
) / 100
min by (dimension_DBInstanceIdentifier) (
last_over_time(aws_rds_ebsbyte_balance_percent_minimum[15m])
) / 100
max by (database_id) (
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_disk_utilization[15m])
)
max by (database_id) (
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_memory_utilization[15m])
)
min by (database_id) (
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_up[15m])
)
sum by (database_id, database) (
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_postgresql_num_backends[15m])
)
sum by (database_id, state) (
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_postgresql_num_backends_by_state[15m])
)
max by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_storage_percent_maximum_percent[15m])
) / 100
max by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_memory_percent_maximum_percent[15m])
) / 100
min by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_cpu_credits_remaining_minimum_count[15m])
)
min by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_is_db_alive_minimum_count[15m])
)
max by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_disk_iops_consumed_percentage_maximum_percent[15m])
) / 100
max by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_disk_bandwidth_consumed_percentage_maximum_percent[15m])
) / 100
max by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_disk_queue_depth_average_count[15m])
)
max by (resourceName) (
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_connections_failed_total_count[15m])
)
max by (dimension_BucketName) (
last_over_time(aws_s3_bucket_size_bytes_average[15m])
)
sum by (bucket_name) (
last_over_time(stackdriver_gcs_bucket_storage_googleapis_com_storage_v_2_total_bytes[30m])
)
max by (resourceName) (
last_over_time(azure_microsoft_storage_storageaccounts_blobservices_blobcapacity_average_bytes[15m])
)
sum(max by (shard) (mz_persist_shard_usage_current_state_batches_bytes))
+ sum(max by (shard) (mz_persist_shard_usage_current_state_rollups_bytes))
+ sum(max by (shard) (mz_persist_shard_usage_referenced_not_current_state_bytes))
+ sum(max by (shard) (mz_persist_shard_usage_not_leaked_not_referenced_bytes))
+ sum(max by (shard) (mz_persist_shard_usage_leaked_bytes))
max by (dimension_BucketName) (
last_over_time(aws_s3_number_of_objects_average[15m])
)
sum by (bucket_name) (
last_over_time(stackdriver_gcs_bucket_storage_googleapis_com_storage_v_2_total_count[30m])
)
max by (resourceName) (
last_over_time(azure_microsoft_storage_storageaccounts_blobservices_blobcount_average_count[15m])
)
sum by (bucket_name, type) (
last_over_time(stackdriver_gcs_bucket_storage_googleapis_com_storage_v_2_total_bytes[30m])
)
sum by (bucket_name, type) (
last_over_time(stackdriver_gcs_bucket_storage_googleapis_com_storage_v_2_total_count[30m])
)
min by (resourceName) (
last_over_time(azure_microsoft_storage_storageaccounts_blobservices_availability_average_percent[15m])
) / 100
max by (resourceName) (
last_over_time(azure_microsoft_storage_storageaccounts_blobservices_successserverlatency_average_milliseconds[15m])
) / 1000
max by (resourceName) (
last_over_time(azure_microsoft_storage_storageaccounts_blobservices_successe2elatency_average_milliseconds[15m])
) / 1000
max by (job, instance) (up{job=~"integrations/(cloudwatch|gcp|azure)"})
count(max by (dimension_DBInstanceIdentifier) (last_over_time(aws_rds_cpuutilization_average[15m])))
count(max by (dimension_BucketName) (last_over_time(aws_s3_bucket_size_bytes_average[15m])))
count(max by (database_id) (last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_cpu_utilization[15m])))
count(max by (bucket_name) (last_over_time(stackdriver_gcs_bucket_storage_googleapis_com_storage_v_2_total_bytes[30m])))
count(max by (resourceName) (last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_cpu_percent_average_percent[15m])))
count(max by (resourceName) (last_over_time(azure_microsoft_storage_storageaccounts_blobservices_blobcapacity_average_bytes[15m])))
max by (instance) (
last_over_time(stackdriver_monitoring_last_scrape_error[15m])
)
sum by (instance) (
rate(stackdriver_monitoring_api_calls_total[30m])
) * 60
sum by (statusCode) (
rate(azurerm_api_request_count{statusCode!~"2.."}[30m])
) * 60
label_replace(
label_replace(
sum by (resourceProvider, method) (
rate(azurerm_api_request_count[30m])
) * 60,
"resourceProvider", "resource graph", "method", "post"
),
"resourceProvider", "other", "resourceProvider", ""
)
Karpenter’s own account of provisioning and disruption, from the metrics its controller publishes and the events it records.
Karpenter adds nodes on EKS in the self-managed Terraform: it watches for pods the scheduler cannot place, launches an EC2 instance that fits them, and later removes or replaces nodes that are empty, underused, drifted from their NodePool’s spec, or being interrupted by AWS. Everything it does to a node goes through a NodeClaim, one per instance.
The controller runs two replicas and only the leader does any work, so most
families here come from one pod. Counters restart on a leader change; every
query reads them through rate or increase, which tolerates that.
The chart’s ServiceMonitor is enabled by the Terraform’s karpenter module.
It drops the offering price estimates and per-instance-type CPU and memory,
which describe every instance type in the region rather than the cluster,
and keeps offering availability only for the node pools’ instance types.
Karpenter’s generic families (controller_runtime_*, workqueue_*,
client_go_*, aws_sdk_go_*) carry no karpenter_ prefix, so they are
scoped by app="karpenter", which the gateway sets from the pod’s name label.
_count#karpenter_scheduler_unschedulable_pods_count and the
operator_*_status_condition_count families are gauges, not the counts of a
histogram.
NodeClaims are created and removed a few times a day, so their duration
histograms are read over an hour rather than $__rate_interval, which would
leave the panel mostly empty.
count(
max by (name) (operator_nodepool_status_condition_count{type="Ready", status!="True"}) == 1
)
or 0 * count(max by (name) (operator_nodepool_status_condition_count{type="Ready"}))
count(
max by (node_name) (karpenter_nodes_allocatable{resource_type="cpu", nodepool!="", nodepool=~"$karpenterNodePool"})
)
max(karpenter_scheduler_unschedulable_pods_count{controller="provisioner"})
sum(increase(karpenter_cloudprovider_errors_total{method="Create"}[1h]))
or 0 * max(karpenter_build_info)
sum(increase(karpenter_nodeclaims_disrupted_total{nodepool=~"$karpenterNodePool"}[1h]))
or 0 * max(karpenter_build_info)
max(karpenter_cluster_state_synced)
max by (version) (karpenter_build_info)
count by (nodepool) (
max by (node_name, nodepool) (karpenter_nodes_allocatable{resource_type="cpu", nodepool!="", nodepool=~"$karpenterNodePool"})
)
max by (nodepool) (karpenter_nodepools_usage{resource_type="cpu", nodepool=~"$karpenterNodePool"})
max by (nodepool) (karpenter_nodepools_limit{resource_type="cpu", nodepool=~"$karpenterNodePool"})
max by (nodepool) (karpenter_nodepools_usage{resource_type="memory", nodepool=~"$karpenterNodePool"})
max by (nodepool) (karpenter_nodepools_limit{resource_type="memory", nodepool=~"$karpenterNodePool"})
max by (resource_type) (karpenter_cluster_utilization_percent{resource_type=~"cpu|memory|pods"}) / 100
max by (kind, name, type, status, reason) (
operator_status_condition_count{kind=~"NodePool|EC2NodeClass"}
) == 1
sum by (nodepool, reason) (
rate(karpenter_nodeclaims_created_total{nodepool=~"$karpenterNodePool"}[5m])
) * 3600
sum by (error, method) (
rate(karpenter_cloudprovider_errors_total[5m])
) * 3600
sum by (instance_type, capacity_type) (
max by (instance_type, capacity_type, zone) (karpenter_cloudprovider_instance_type_offering_available{capacity_type!="reserved"})
)
max(karpenter_scheduler_unschedulable_pods_count{controller="provisioner"})
max by (controller) (karpenter_scheduler_queue_depth)
histogram_quantile(0.5, sum by (le) (increase(karpenter_pods_provisioning_startup_duration_seconds_bucket[1h])))
histogram_quantile(0.99, sum by (le) (increase(karpenter_pods_provisioning_startup_duration_seconds_bucket[1h])))
histogram_quantile(0.99,
sum by (type, le) (
increase(operator_nodeclaim_status_condition_transition_seconds_bucket{type=~"Launched|Registered|Initialized", karpenter_sh_nodepool=~"$karpenterNodePool"}[1h])
)
)
histogram_quantile(0.99,
sum by (controller, le) (rate(karpenter_scheduler_scheduling_duration_seconds_bucket[5m]))
)
histogram_quantile(0.99,
sum by (method, le) (rate(karpenter_cloudprovider_duration_seconds_bucket[5m]))
)
sum by (nodepool, reason) (
rate(karpenter_nodeclaims_disrupted_total{nodepool=~"$karpenterNodePool"}[5m])
) * 3600
sum by (reason, decision) (
rate(karpenter_voluntary_disruption_decisions_total[5m])
) * 3600
max by (reason) (karpenter_voluntary_disruption_eligible_nodes)
min by (nodepool, reason) (karpenter_nodepools_allowed_disruptions{nodepool=~"$karpenterNodePool"})
DisruptionBlocked
and Unconsolidatable events per five minutes.
sum by (code) (rate(karpenter_nodes_eviction_requests_total[5m])) * 60
histogram_quantile(0.99,
sum by (le) (increase(operator_nodeclaim_termination_duration_seconds_bucket{karpenter_sh_nodepool=~"$karpenterNodePool"}[1h]))
)
sum by (consolidation_type) (
rate(karpenter_voluntary_disruption_consolidation_timeouts_total[5m])
) * 3600
max by (node_name, nodepool, instance_type, zone) (
karpenter_nodes_current_lifetime_seconds{nodepool!="", nodepool=~"$karpenterNodePool"}
)
sum by (message_type) (
rate(karpenter_interruption_received_messages_total[5m])
) * 3600
histogram_quantile(0.99,
sum by (le) (increase(karpenter_interruption_message_queue_duration_seconds_bucket[1h]))
)
sum by (controller) (
rate(controller_runtime_reconcile_errors_total{app="karpenter"}[5m])
) * 60 > 0
max by (name) (workqueue_depth{app="karpenter"}) > 0
sum by (exported_service, action, code) (
rate(aws_sdk_go_request_total{app="karpenter", code!~"2..|412"}[5m])
) * 60 > 0
sum by (method, code) (
rate(client_go_request_total{app="karpenter", code!~"2.."}[5m])
) * 60 > 0
max by (pod) (leader_election_master_status{app="karpenter"})
sum by (pod) (
rate(container_cpu_usage_seconds_total{container="controller", pod=~"karpenter-.+"}[5m])
)
max by (pod) (container_memory_working_set_bytes{container="controller", pod=~"karpenter-.+"})
max by (pod) (kube_pod_container_resource_limits{container="controller", pod=~"karpenter-.+", resource="memory"})
Logs from the platform a Materialize deployment runs on: the monitoring stack itself, the Kubernetes system components, and the nodes underneath both.
Separate from materialize-logs.yaml rather than a widening of it, for two
reasons that are structural rather than stylistic.
The selector set differs. These carry component and container filters.
component is what tells one Loki or Thanos process from another — loki
alone splits into canary, querier, ingester, query-frontend,
index-gateway, compactor, distributor and ruler — and container is
the only picker that reaches workloads with no app at all, which on a
representative install is the whole of kube-system. Neither dimension means
anything to a Materialize environment, and adding them to the shared queries
would oblige every dashboard using those to define pickers it has no use for.
The node journal is not reachable from a namespace. Journal lines carry
unit, component, job, level and service_name and no namespace,
app or container, because they come from the node rather than from a pod.
Any selector that requires a namespace excludes them by construction, which is
why they have their own queries here and their own tab on the dashboard.
The Kubernetes-event queries are not duplicated: materialize.events.cluster.*
in materialize-events.yaml is already scoped by the same namespace picker and
carries no Materialize-specific filter, so an infrastructure dashboard uses it
as it stands.
kubelet, containerd, the node
problem detector, and the rest of what systemd runs on each node.
Is the log store healthy — and if not, which half of it broke.
Every other logs query in this repository reads Loki. These watch it. That makes this file the one place where the monitoring stack is the subject rather than the instrument, and it changes what a reading means: a blank panel here is itself a finding, because the thing that would have reported the problem is the thing that is down.
The audience is whoever operates the monitoring stack. Loki’s vocabulary is not assumed — a stream is one label-set’s worth of lines, a chunk is a stream’s lines batched up for storage, and the write path turns the first into the second before handing it to object storage. Where a panel needs one of those to be legible, its description says so.
loki_ metric families#The prefix is shared and the producers are not. Reading one for the other is the single easiest mistake to make here, and nothing about the metric name warns you.
job=~"loki/.*", and everything this file’s panels draw.
391 metric names on a reference install.loki.* components. job=~"alloy-.*", 43 names, all under
loki_write_*, loki_source_{file,api,journal}_*, loki_process_* and
loki_relabel_*. These describe the collector — what Alloy read off disk
and shipped — not the store that received it.Only loki_experimental_features_in_use_total is emitted by both.
The two are kept apart by app_instance="loki", which both of this chart’s
Loki ServiceMonitors stamp on every target they collect and nothing else sets.
A job=~"loki/.*" matcher would work too, but it hardcodes the subchart’s
jobPrefix, which is a value an operator may change.
The Alloy-side family belongs to infra-alloy.yaml, which watches the
collectors and is where those metrics are drawn.
Two variables, both written literally rather than reaching through a render
parameter, on the same precedent as $nodeList in node-health.yaml and
$namespaceList in infra-networking.yaml: a meta-monitoring dashboard’s
scope is the monitoring stack’s own, which no Materialize-shaped parameter
describes.
$lokiNamespace — where Loki runs. Every metric query here carries it,
so two materialize-monitoring releases sharing a cluster do not read as
one store.$lokiComponent — which Loki process. Discovered from the metrics
side, off the container label, and applied to the log queries as
component=~"$lokiComponent" as well.One picker across both engines is a deliberate call, because the two label sets
agree by construction: container on a metric and component on a log line
are both the Kubernetes container name. The one value that exists on the
metrics side and not the logs side is exporter, the memcached sidecar, which
produces no log lines at all — so selecting it empties the log feeds, which is
the honest answer rather than a bug.
Three of Loki’s targets never serve TLS, whatever Loki itself is configured to
do: the canary’s own /metrics, and the two memcached exporters. They are
collected by a separate ServiceMonitor for that reason
(templates/scrapers/monitor-loki-plaintext.yaml), and infra.loki.health.up
is the panel that says whether that split is working. It was not, before this
file existed: under profiles/mtls all three were scraped over HTTPS, failed,
and vanished — taking the end-to-end canary with them.
sum(
rate(
loki_canary_missing_entries_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum(
rate(
loki_canary_entries_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
histogram_quantile(
0.50,
sum by (le) (
rate(
loki_canary_response_latency_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
histogram_quantile(
0.99,
sum by (le) (
rate(
loki_canary_response_latency_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
sum(
rate(
loki_canary_missing_entries_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum(
rate(
loki_canary_spot_check_missing_entries_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
max by (container, service) (
up{
app_instance="loki",
namespace=~"$lokiNamespace",
container=~"$lokiComponent"
}
)
sum(
rate(
loki_request_duration_seconds_count{
app_instance="loki",
namespace=~"$lokiNamespace",
container=~"$lokiComponent",
status_code=~"5.."
}
[5m]
)
)
/
sum(
rate(
loki_request_duration_seconds_count{
app_instance="loki",
namespace=~"$lokiNamespace",
container=~"$lokiComponent"
}
[5m]
)
)
sum(
rate(
loki_request_duration_seconds_count{
app_instance="loki",
namespace=~"$lokiNamespace",
container=~"$lokiComponent",
status_code=~"4.."
}
[5m]
)
)
/
sum(
rate(
loki_request_duration_seconds_count{
app_instance="loki",
namespace=~"$lokiNamespace",
container=~"$lokiComponent"
}
[5m]
)
)
sum by (container) (
rate(
loki_request_duration_seconds_count{
app_instance="loki",
namespace=~"$lokiNamespace",
container=~"$lokiComponent",
status_code="error"
}
[5m]
)
)
histogram_quantile(
0.99,
sum by (le, route) (
rate(
loki_request_duration_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace",
container=~"$lokiComponent",
route!~"(?i).*tail.*|/schedulerpb.SchedulerForQuerier/QuerierLoop"
}
[5m]
)
)
)
sum by (pod) (
increase(
kube_pod_container_status_restarts_total{
namespace=~"$lokiNamespace",
pod=~"loki-.*"
}
[1h]
)
) > 0
sum(
rate(
loki_distributor_lines_received_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum(
rate(
loki_distributor_bytes_received_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum by (reason) (
rate(
loki_discarded_samples_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum by (reason) (
rate(
loki_discarded_bytes_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum by (pod) (
loki_ingester_memory_streams{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
sum by (pod) (
loki_ingester_memory_chunks{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
sum(
rate(
loki_ingester_chunks_flushed_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
histogram_quantile(
0.50,
sum by (le) (
rate(
loki_ingester_chunk_utilization_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
histogram_quantile(
0.10,
sum by (le) (
rate(
loki_ingester_chunk_utilization_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
histogram_quantile(
0.50,
sum by (le) (
rate(
loki_ingester_chunk_age_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
histogram_quantile(
0.99,
sum by (le) (
rate(
loki_ingester_chunk_age_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
max by (pod) (
loki_ingester_wal_disk_usage_percent{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
sum by (pod) (
increase(
loki_ingester_wal_disk_full_failures_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[1h]
)
)
max by (pod) (
loki_ingester_wal_replay_active{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
histogram_quantile(
0.50,
sum by (le) (
rate(
loki_request_duration_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace",
route=~"loki_api_v1_push|api_prom_push|/logproto.Pusher/Push"
}
[5m]
)
)
)
histogram_quantile(
0.99,
sum by (le) (
rate(
loki_request_duration_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace",
route=~"loki_api_v1_push|api_prom_push|/logproto.Pusher/Push"
}
[5m]
)
)
)
sum by (route) (
rate(
loki_request_duration_seconds_count{
app_instance="loki",
namespace=~"$lokiNamespace",
container="query-frontend",
route=~"loki_api_v1_.*"
}
[5m]
)
)
histogram_quantile(
0.50,
sum by (le) (
rate(
loki_request_duration_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace",
container="query-frontend",
route=~"loki_api_v1_.*",
route!~"(?i).*tail.*"
}
[5m]
)
)
)
histogram_quantile(
0.90,
sum by (le) (
rate(
loki_request_duration_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace",
container="query-frontend",
route=~"loki_api_v1_.*",
route!~"(?i).*tail.*"
}
[5m]
)
)
)
histogram_quantile(
0.99,
sum by (le) (
rate(
loki_request_duration_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace",
container="query-frontend",
route=~"loki_api_v1_.*",
route!~"(?i).*tail.*"
}
[5m]
)
)
)
histogram_quantile(
0.50,
sum by (le) (
rate(
loki_query_scheduler_queue_duration_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
histogram_quantile(
0.99,
sum by (le) (
rate(
loki_query_scheduler_queue_duration_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
sum(
loki_query_frontend_queries_in_progress{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
sum(
rate(
loki_logql_querystats_bytes_processed_per_seconds_sum{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum by (name) (
rate(
loki_cache_fetched_keys{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum by (name) (
rate(
loki_cache_hits{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
/
sum by (name) (
rate(
loki_cache_fetched_keys{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum by (service) (
memcached_current_bytes{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
/
sum by (service) (
memcached_limit_bytes{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
sum by (operation) (
rate(
loki_objstore_bucket_operations_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum by (operation) (
rate(
loki_objstore_bucket_operation_failures_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
histogram_quantile(
0.99,
sum by (le, operation) (
rate(
loki_objstore_bucket_operation_duration_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
sum(
rate(
loki_tsdb_shipper_tables_sync_operation_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
sum(
rate(
loki_tsdb_shipper_tables_upload_operation_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
histogram_quantile(
0.99,
sum by (le) (
rate(
loki_tsdb_shipper_query_wait_time_seconds_bucket{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[5m]
)
)
)
time()
-
max(
loki_compactor_apply_retention_last_successful_run_timestamp_seconds{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
time()
-
max(
loki_boltdb_shipper_compact_tables_operation_last_successful_run_timestamp_seconds{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
sum by (status) (
increase(
loki_boltdb_shipper_compact_tables_operation_total{
app_instance="loki",
namespace=~"$lokiNamespace"
}
[1h]
)
)
sum(
loki_compactor_pending_delete_requests_count{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
max(
loki_compactor_oldest_pending_delete_request_age_seconds{
app_instance="loki",
namespace=~"$lokiNamespace"
}
)
How traffic moves through the cluster, and what stops it.
Where infra-nodes asks what one machine is doing, this asks what the
network is doing across all of them: what pods are sending, what Kubernetes
is routing, what the CNI is doing underneath, and which packets a policy
dropped.
Three families meet here, and knowing which is which is most of what makes this file readable:
cAdvisor (container_network_*) is the only per-pod view. It is always
present, because the gateway scrapes every kubelet. Note that its series
split in two by whether namespace is set: with a namespace they describe a
pod, and without one they carry id="/" and describe the node’s root
cgroup — every interface on the machine, including the veth of every pod on
it. Panels must pick one and say which.
Host-network pods are excluded from every rollup here, and must be.
cAdvisor reads a container’s counters from its network namespace, and a
host-network pod’s namespace is the node’s — so such a pod reports all 31 of
a reference node’s interfaces rather than its own one, and its “pod traffic”
is the whole machine’s, counting every other pod’s veth. Several DaemonSets
do this on every node. Left in, a cluster-wide sum read 27,831 KiB/s against
a true 1,700, and the ten busiest pods were all host-network DaemonSets
reporting their node. %%{excludeHostNetworkPods} is the unless clause
that drops them; it appends to a sum by (namespace, pod) and cannot be a
label matcher, because being host-network is a property of a pod’s whole
series set rather than of any one series.
kube-state-metrics (kube_service_*, kube_endpointslice_*,
kube_networkpolicy_*) is the declared intent: what Services exist, what is
behind them, and which policies were written. It says nothing about whether
any of it works.
The CNI is the only family that can say what the dataplane actually did — which packet a policy dropped, how many addresses are left, whether the datapath is erroring. It is also the only family that is not guaranteed to be there, which is why every query reading it is on a row the dashboard shows only when that vendor was detected. See “Dataplane detection” below.
node_* is a fourth family and deliberately not redefined here:
node-health.yaml and node-debug.yaml already carry it, every one of their
expressions is scoped by instance=~"$nodeList", and infra-net defines that
variable as a multi-select across the fleet rather than the single address
infra-nodes resolves it to. The same expressions therefore answer for one
node or for all of them depending only on the dashboard that asks.
A cluster’s CNI is not something this repository can know: EKS defaults to the
AWS VPC CNI, GKE to Dataplane V2, AKS to Azure CNI powered by Cilium, and a
bring-your-own cluster to anything at all. Rather than ask the operator, the
CNI monitors in packages/prometheus-scrapers/ stamp every series they
collect with a network_component label, and the dashboard discovers the
answer with label_values(up{network_component=~".+"}, network_component).
That puts the vendor knowledge in the scrape config, where it already had to live, and leaves the queries below reading ordinary vendor metric names.
Two conventions apply throughout:
Scope literally, not by parameter. namespace=~"$namespaceList" and
instance=~"$nodeList" are written out, the same way infra-nodes.yaml
writes node="$node". A dashboard using these must define the variables; no
render parameter supplies them.
$nodeList holds node-exporter addresses, inherited from the node query
families. Nothing here scopes on a node label, which every other family
spells as the Kubernetes name – the two cannot be filtered by one picker,
and the node-exporter form is the one that buys the 87 vetted expressions in
node-health.yaml and node-debug.yaml.
Deduplicate across scrape replicas. instance is the scrape target, so
a bare sum over an HA kube-state-metrics adds each object once per
replica. Aggregations keep instance in the inner step and collapse it with
an outer max.
%%{interval} is the rate window, including its brackets.
max by (network_component) (up{network_component=~".+"})
sum (
sum by (namespace, pod) (
rate(container_network_receive_bytes_total{namespace=~"$namespaceList", interface!="lo"}[5m])
) ${excludeHostNetworkPods}
)
sum (
sum by (namespace, pod) (
rate(container_network_transmit_bytes_total{namespace=~"$namespaceList", interface!="lo"}[5m])
) ${excludeHostNetworkPods}
)
sum (
sum by (namespace, pod) (
rate(container_network_receive_errors_total{namespace=~"$namespaceList"}[5m])
+ rate(container_network_transmit_errors_total{namespace=~"$namespaceList"}[5m])
) ${excludeHostNetworkPods}
)
sum (
sum by (namespace, pod) (
rate(container_network_receive_packets_dropped_total{namespace=~"$namespaceList"}[5m])
+ rate(container_network_transmit_packets_dropped_total{namespace=~"$namespaceList"}[5m])
) ${excludeHostNetworkPods}
)
topk(10,
sum by (namespace, pod) (
rate(container_network_receive_bytes_total{namespace=~"$namespaceList", interface!="lo"}[5m])
+ rate(container_network_transmit_bytes_total{namespace=~"$namespaceList", interface!="lo"}[5m])
) ${excludeHostNetworkPods}
)
sum by (namespace) (
sum by (namespace, pod) (
rate(container_network_receive_bytes_total{namespace=~"$namespaceList", interface!="lo"}[5m])
) ${excludeHostNetworkPods}
)
sum by (namespace) (
sum by (namespace, pod) (
rate(container_network_transmit_bytes_total{namespace=~"$namespaceList", interface!="lo"}[5m])
) ${excludeHostNetworkPods}
)
sum by (namespace) (
sum by (namespace, pod) (
rate(container_network_receive_errors_total{namespace=~"$namespaceList"}[5m])
+ rate(container_network_transmit_errors_total{namespace=~"$namespaceList"}[5m])
+ rate(container_network_receive_packets_dropped_total{namespace=~"$namespaceList"}[5m])
+ rate(container_network_transmit_packets_dropped_total{namespace=~"$namespaceList"}[5m])
) ${excludeHostNetworkPods}
)
max by (type) (
sum by (instance, type) (kube_service_spec_type)
)
max by (namespace) (
sum by (instance, namespace) (kube_endpointslice_endpoints{namespace=~"$namespaceList"})
)
histogram_quantile(0.99,
sum by (le) (
rate(kubeproxy_sync_proxy_rules_duration_seconds_bucket[5m])
)
)
histogram_quantile(0.99,
sum by (le) (
rate(kubeproxy_network_programming_duration_seconds_bucket[5m])
)
)
max by (node) (awscni_assigned_ip_addresses)
/
max by (node) (awscni_ip_max)
max by (node) (awscni_assigned_ip_addresses)max by (node) (awscni_total_ip_addresses)max by (node) (awscni_ip_max)max by (node) (awscni_eni_allocated)max by (node) (awscni_eni_max)sum (max by (node) (awscni_no_available_ip_addresses))
sum by (api) (rate(awscni_aws_api_latency_ms_sum[5m]))
/
sum by (api) (rate(awscni_aws_api_latency_ms_count[5m]))
sum by (node, endpoint_state) (cilium_endpoint_state)
sum by (reason) (rate(cilium_drop_count_total[5m]))
sum by (action) (rate(cilium_policy_verdict_total[5m]))
max by (node, map_name) (cilium_bpf_map_pressure)
histogram_quantile(0.99,
sum by (le, scope) (
rate(cilium_endpoint_regeneration_time_stats_seconds_bucket[5m])
)
)
sum (max by (node) (cilium_unreachable_nodes))sum (max by (node) (cilium_unreachable_health_endpoints))sum by (verdict) (rate(hubble_flows_processed_total[5m]))
max by (namespace) (
count by (instance, namespace) (kube_networkpolicy_spec_ingress_rules{namespace=~"$namespaceList"})
)
count by (namespace) (kube_pod_info{namespace=~"$namespaceList"})
unless on (namespace)
count by (namespace) (kube_networkpolicy_spec_ingress_rules)
max by (namespace, networkpolicy) (kube_networkpolicy_spec_ingress_rules{namespace=~"$namespaceList"})
max by (namespace, networkpolicy) (kube_networkpolicy_spec_egress_rules{namespace=~"$namespaceList"})
sum by (node) (rate(network_policy_drop_count_total[5m]))
sum by (node) (rate(cilium_drop_count_total{reason=~"Policy denied.*"}[5m]))
max by (namespace, service, ip, hostname) (kube_service_status_load_balancer_ingress)
What kubectl describe node would tell you, for whoever cannot run it.
These answer the questions an operator asks about one machine: what is it,
how big is it, how much of it is already promised, is Kubernetes willing to
put work on it, and what has it been saying. The measurements of what the
machine is actually doing live in node-health.yaml and node-debug.yaml,
which read node-exporter; this file is the Kubernetes side, plus the node’s
own journal and the events filed against it.
Two identifier conventions meet here, and the difference is the thing to know:
kube-state-metrics names a node node="<kubernetes name>". Every query
in this file scopes with node="$node" written literally, the same way the
node-exporter families write instance=~"$nodeList" literally. A dashboard
using either must define the variable; no render parameter supplies it.
node-exporter names the same machine instance="<ip>:9100". The join is
node_uname_info, whose nodename is the Kubernetes name — which is what
the $nodeList variable resolves through. Nothing in this file needs the
join, because nothing in this file reads node-exporter.
Loki knows the node a third way: node is structured metadata on journal
lines, not a stream label, so it is filtered in the pipeline (| node=...)
rather than in the selector. Node events are Kubernetes events whose involved
object is the node itself, which is kind="Node" with the node’s name.
Two conventions apply to every query here that aggregates:
Deduplicate across kube-state-metrics replicas. instance is the scrape
target, so a bare sum or count over an HA deployment adds each object
once per replica. Every aggregation keeps instance in the inner step and
collapses it with an outer max, which is the shape
materialize-kubernetes.yaml established. Queries that only ever max are
already safe, since max across identical replicas is idempotent.
Terminal pods do not count against the node. A Succeeded or Failed
pod has released its CPU and memory, and the scheduler no longer counts it
against the pod limit — kube-state-metrics agrees, and stops reporting
kube_pod_container_resource_* for it. But kube_pod_info keeps reporting
it until garbage collection, so anything counting pods rather than their
resources has to subtract them explicitly or it overstates how full the node
is. Completed Jobs are the common case.
%%{interval} is the rate window, including its brackets.
max by (node, kubelet_version) (kube_node_info{node="$node"})
max by (node, os_image) (kube_node_info{node="$node"})
max by (node, kernel_version) (kube_node_info{node="$node"})
max by (node, container_runtime_version) (kube_node_info{node="$node"})
max by (node, internal_ip) (kube_node_info{node="$node"})
max by (node) (kube_node_created{node="$node"}) * 1000
max by (node) (kube_node_status_capacity{node="$node", resource="cpu"})
max by (node) (kube_node_status_capacity{node="$node", resource="memory"})
max by (node) (kube_node_status_capacity{node="$node", resource="pods"})
max by (node) (kube_node_status_capacity{node="$node", resource="ephemeral_storage"})
max by (node) (
sum by (node, instance) (
kube_pod_container_resource_requests{node="$node", resource="cpu"}
)
)
/
max by (node) (kube_node_status_allocatable{node="$node", resource="cpu"})
max by (node) (
sum by (node, instance) (
kube_pod_container_resource_requests{node="$node", resource="memory"}
)
)
/
max by (node) (kube_node_status_allocatable{node="$node", resource="memory"})
max by (node) (
count by (node, instance) (
kube_pod_info{node="$node"}
unless on (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"} == 1)
)
)
/
max by (node) (kube_node_status_allocatable{node="$node", resource="pods"})
max by (namespace) (
count by (namespace, instance) (
kube_pod_info{node="$node"}
unless on (namespace, pod) (kube_pod_status_phase{phase=~"Succeeded|Failed"} == 1)
)
)
max by (node) (kube_node_status_condition{node="$node", condition="Ready", status="true"})
max by (node, condition) (
kube_node_status_condition{
node="$node",
condition=~"MemoryPressure|DiskPressure|PIDPressure|NetworkUnavailable",
status="true"
}
)
max by (node) (kube_node_spec_unschedulable{node="$node"})
max by (node, key, value, effect) (kube_node_spec_taint{node="$node"})
max by (phase) (
count by (phase, instance) (
kube_pod_status_phase == 1
and on (namespace, pod) kube_pod_info{node="$node"}
)
)
max by (namespace, pod) (
kube_pod_status_ready{condition="true"}
and on (namespace, pod) kube_pod_info{node="$node"}
) == 0
unless on (namespace, pod) (kube_pod_status_phase{phase="Succeeded"} == 1)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_status_restarts_total
and on (namespace, pod) kube_pod_info{node="$node"}
)
)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_resource_requests{node="$node", resource="cpu"}
)
)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_resource_limits{node="$node", resource="cpu"}
)
)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_resource_requests{node="$node", resource="memory"}
)
)
max by (namespace, pod) (
sum by (namespace, pod, instance) (
kube_pod_container_resource_limits{node="$node", resource="memory"}
)
)
count(
group by (compute_cluster_id) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*"}
)
)
count(
group by (compute_cluster_id) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_cluster_id=~"^s.*"}
)
)
count_not_null(
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}} by {compute_cluster_id}
)
count_not_null(
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_cluster_id:s*} by {compute_cluster_id}
)
count(
group by (compute_cluster_id, compute_replica_id) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*"}
)
)
count(
group by (compute_cluster_id, compute_replica_id) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*", compute_replica_name!="r1"}
)
)
count_not_null(
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}} by {compute_cluster_id,compute_replica_id}
)
default_zero(
count_not_null(
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}, !compute_replica_name:r1} by {compute_cluster_id,compute_replica_id}
)
)
count by (size) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*"}
)
sum:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}} by {size}
mz_compute_cluster_status{materialize_cloud_organization_name=~".*", compute_cluster_id=~".*", compute_replica_id=~".*"}
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}, compute_cluster_id:${mzClusterList}, compute_replica_id:${mzReplicaList}}
by {compute_cluster_id,compute_cluster_name,compute_replica_id,compute_replica_name,size,mz_version}
max(mz_mzd_views_count{materialize_cloud_organization_name=~".*"})
max:${mzSqlPrefix}mzd_views_count{${mzEnvironmentFilter}}
max(sum by (instance) (mz_indexes_count{materialize_cloud_organization_name=~".*"}))
default_zero(sum:${mzSqlPrefix}indexes_count{${mzEnvironmentFilter}} by {instance})
max(mz_views_count{materialize_cloud_organization_name=~".*"})
max:${mzSqlPrefix}views_count{${mzEnvironmentFilter}}
system subscribes are
Materialize’s own internal probes; a persistently climbing user count
usually means a client is opening subscribes and never closing them.
sum by (session_type) (mz_active_subscribes{materialize_cloud_organization_name=~".*"})
sum:mz_active_subscribes{${mzEnvironmentFilter}} by {session_type}
sum by (relation_type) (mz_indexes_count{materialize_cloud_organization_name=~".*"})
sum:${mzSqlPrefix}indexes_count{${mzEnvironmentFilter}} by {relation_type}
count(
max by (instance_id, collection_id) (
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", quantile="1"} > 1e15
)
)
default_zero(
count_not_null(
max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, quantile:1} by {instance_id,collection_id}
)
)
sum by (instance_id, replica_id) (
mz_compute_controller_hydration_queue_size{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*"}
) > 0
sum:mz_compute_controller_hydration_queue_size{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}}
by {instance_id,replica_id}
topk(15,
mz_compute_hydration_time_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", hydrated="1"}
)
top(
max:${mzSqlPrefix}compute_hydration_time_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, hydrated:1} by {instance_id,collection_id},
15, 'max', 'desc'
)
max by (instance_id) (
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", quantile="1"} < 1e9
)
max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, quantile:1} by {instance_id}
sum by (instance_id) (
max by (instance_id, collection_id) (
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", quantile="1"} < 1e9
)
)
sum:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, quantile:1} by {instance_id}
topk(15,
max by (instance_id, collection_id) (
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id=~".*", instance_id!="", replica_id=~".*", quantile="1"} < 1e9
)
)
top(
max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, quantile:1} by {instance_id,collection_id},
15, 'max', 'desc'
)
max by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id) (
mz_compute_replica_history_dataflow_count{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}
)
max:mz_compute_replica_history_dataflow_count{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id}
max by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, worker_id) (
mz_compute_replica_history_dataflow_count{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}
)
max:mz_compute_replica_history_dataflow_count{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id,worker_id}
sum by (instance_id) (
max without (job) (rate(
mz_dataflow_elapsed_seconds_total{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*"}[5m]
))
)
sum:${mzSqlPrefix}dataflow_elapsed_seconds_total{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}}
by {instance_id}.as_rate()
sum by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id) (
max without (job) (rate(
mz_arrangement_maintenance_seconds_total{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]
))
)
sum:mz_arrangement_maintenance_seconds_total{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id}.as_rate()
sum by (cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, worker_id) (
max without (job) (rate(
mz_arrangement_maintenance_seconds_total{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]
))
)
sum:mz_arrangement_maintenance_seconds_total{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id,worker_id}.as_rate()
s). These back the catalog and internal
probes, not user data, so they shouldn’t grow with your workload —
unexpected growth here can point at a Materialize bug.
max by (collection_id) (
mz_arrangement_record_count{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", collection_id=~"s.*"}
)
max:${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, collection_id:s*}
by {collection_id}
u) — the row count of every user index and materialized view, and
the primary driver of cluster memory. Growth on an object tracks the
size of its underlying data.
max by (collection_id) (
mz_arrangement_record_count{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", collection_id=~"u.*"}
)
max:${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, collection_id:u*}
by {collection_id}
t*) and uncategorized (none)
arrangements — short-lived intermediates from query optimization and
dataflow execution. Normally small and ephemeral.
max by (collection_id) (
mz_arrangement_record_count{materialize_cloud_organization_name=~".*", instance_id=~".*", replica_id=~".*", collection_id=~"t.*|none"}
)
max:${mzSqlPrefix}arrangement_record_count{${mzEnvironmentFilter}, instance_id:${mzClusterList}, replica_id:${mzReplicaList}, (collection_id:t* OR collection_id:none)}
by {collection_id}
system (Materialize’s internal probing —
a few are always present) and user (client connections).
sum by (session_type) (mz_active_sessions{materialize_cloud_organization_name=~".*"})
sum:mz_active_sessions{${mzEnvironmentFilter}} by {session_type}
user tracks your client traffic,
system is the steady single-digit baseline of internal health checks.
Bursty is normal.
sum by (session_type) (rate(mz_query_total{materialize_cloud_organization_name=~".*"}[5m]))
sum:mz_query_total{${mzEnvironmentFilter}} by {session_type}.as_rate()
sum(rate(mz_adapter_commands{materialize_cloud_organization_name=~".*"}[5m]))
sum:mz_adapter_commands{${mzEnvironmentFilter}}.as_rate()
set_variable/fetch traffic is normal
(that’s how Postgres clients manage session state); heavy
insert/update/delete on something you think of as read-mostly is
worth a look.
sum by (statement_type) (increase(mz_query_total{materialize_cloud_organization_name=~".*"}[1h])) > 0
sum:mz_query_total{${mzEnvironmentFilter}} by {statement_type}.as_count()
select/user is the thing to line up against peek latency
to confirm the system kept pace.
sum by (statement_type, session_type) (rate(mz_query_total{materialize_cloud_organization_name=~".*"}[5m])) > 0
sum:mz_query_total{${mzEnvironmentFilter}} by {statement_type,session_type}.as_rate()
SELECT
against an index. Your “what does a normal query feel like” number.
histogram_quantile(0.50,
sum by (le, instance_id) (
rate(mz_compute_peek_duration_seconds_bucket{materialize_cloud_organization_name=~".*", instance_id=~".*"}[5m])
)
)
p50:mz_compute_peek_duration_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}}
by {instance_id}
histogram_quantile(0.90,
sum by (le, instance_id) (
rate(mz_compute_peek_duration_seconds_bucket{materialize_cloud_organization_name=~".*", instance_id=~".*"}[5m])
)
)
p90:mz_compute_peek_duration_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}}
by {instance_id}
histogram_quantile(0.99,
sum by (le, instance_id) (
rate(mz_compute_peek_duration_seconds_bucket{materialize_cloud_organization_name=~".*", instance_id=~".*"}[5m])
)
)
p99:mz_compute_peek_duration_seconds{${mzEnvironmentFilter}, instance_id:${mzClusterList}}
by {instance_id}
application_name over the
window, so you can see which clients drive the adapter and which are
failing. Most clients set application_name in their connection string;
those that don’t bucket as unrecognized/unspecified (normal).
sum by (application_name, status) (increase(mz_adapter_commands{materialize_cloud_organization_name=~".*"}[1h]))
sum:mz_adapter_commands{${mzEnvironmentFilter}} by {application_name,status}.as_count()
The metadata (consensus) database, from the vantage point of the Materialize processes that use it.
Persist is Materialize’s storage layer. Every durable collection — a table, a source, a materialized view, the catalog itself — is a persist shard: data files in object storage, plus a small record of the shard’s current state in the metadata database. Every change to a shard commits a new version of that record with a compare-and-set, so the database sees a steady stream of small conditional writes (about 80 a second on an idle environment) and very little else. The timestamp oracle is the second client: environmentd keeps the read and write timestamps of each timeline in the same database.
Everything here is the client’s measurement, which is the SLI: it is what
a Materialize process experienced, including the network, DNS, TLS and its
own connection pool. It needs no cloud credentials and is identical on every
database flavor. What the database was doing at the time — CPU, storage,
connection ceilings — is the provider’s view, on infra-cloud. The external
dependency design doc records why the two are separate.
op on mz_persist_external_* names the operation: consensus_cas (the
commit), consensus_head, consensus_scan, consensus_truncate and
consensus_list_keys. The blob_* values of the same label are object
storage and belong to materialize-persist.yaml. The latency histogram
mz_persist_external_op_latency covers consensus_cas only on this side;
the other operations have a mean from _seconds / _started_count.
op on mz_persist_retry_* is a retry loop, named after the work rather
than the call. The five listed in materialize.consensus.retries.by_operation
are the ones that wrap a metadata-database call; next_listen_batch and
snapshot also retry constantly, but they are polling for new data rather
than recovering from a failure and must never be drawn as retries. The same
names, as they appear in the logs, back materialize.consensus.logs.retries.
The process breakdown is app (environmentd or clusterd) plus the
long-form cluster id, which only clusterd carries. mzClusterName turns
the id into the cluster’s name and leaves environmentd’s blank.
mz_persist_postgres_connpool_available goes negative by the number of
callers waiting for a connection (deadpool’s Status.available), and is
sampled at each acquire. clamp_min(-available, 0) is the queue.max_connections. The pool’s configured maximum is
not published.Every mean here is a ratio of two rates wrapped in (…) >= 0. An operation
nobody called in the window divides zero by zero, and the resulting NaN
would otherwise be drawn — and would put the panel’s empty-state text into
its legend, which Grafana applies per field.
sum(rate(mz_persist_external_failed_count{materialize_cloud_organization_name=~".*", op=~"consensus_.*"}[5m]))
sum(rate(mz_persist_postgres_connpool_connection_errors{materialize_cloud_organization_name=~".*"}[5m]))
+ (sum(rate(mz_ts_oracle_postgres_connpool_connection_errors{materialize_cloud_organization_name=~".*"}[5m])) or vector(0))
histogram_quantile(0.99,
sum by (le) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="consensus_cas"}[5m])
)
)
sum(clamp_min(-mz_persist_postgres_connpool_available{materialize_cloud_organization_name=~".*"}, 0))
+ (sum(clamp_min(-mz_ts_oracle_postgres_connpool_available{materialize_cloud_organization_name=~".*"}, 0)) or vector(0))
sum(mz_persist_postgres_connpool_size{materialize_cloud_organization_name=~".*"})
+ (sum(mz_ts_oracle_postgres_connpool_size{materialize_cloud_organization_name=~".*"}) or vector(0))
sum(rate(mz_persist_external_started_count{materialize_cloud_organization_name=~".*", op="consensus_cas"}[5m]))
histogram_quantile(0.50,
sum by (le) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="consensus_cas"}[5m])
)
)
histogram_quantile(0.99,
sum by (le) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="consensus_cas"}[5m])
)
)
max by (app, cluster_environmentd_materialize_cloud_cluster_id) (
mz_persist_external_rtt_latency{materialize_cloud_organization_name=~".*", external="consensus"}
)
sum by (op) (
rate(mz_persist_external_failed_count{materialize_cloud_organization_name=~".*", op=~"consensus_.*"}[5m])
)
sum by (op) (
rate(mz_ts_oracle_failed_count{materialize_cloud_organization_name=~".*"}[5m])
)
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
rate(mz_persist_postgres_connpool_connection_errors{materialize_cloud_organization_name=~".*"}[5m])
)
sum(rate(mz_ts_oracle_postgres_connpool_connection_errors{materialize_cloud_organization_name=~".*"}[5m]))
sum by (op) (
rate(mz_persist_external_started_count{materialize_cloud_organization_name=~".*", op=~"consensus_.*"}[5m])
)
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
rate(mz_persist_external_started_count{materialize_cloud_organization_name=~".*", op="consensus_cas"}[5m])
)
sum by (op) (
rate(mz_persist_external_bytes_count{materialize_cloud_organization_name=~".*", op=~"consensus_.*"}[5m])
)
count(group by (shard) (mz_persist_shard_usage_current_state_batches_bytes{materialize_cloud_organization_name=~".*"}))
(
sum by (op) (
rate(mz_persist_external_seconds{materialize_cloud_organization_name=~".*", op=~"consensus_.*"}[5m])
)
/
sum by (op) (
rate(mz_persist_external_started_count{materialize_cloud_organization_name=~".*", op=~"consensus_.*"}[5m])
)
) >= 0
histogram_quantile(0.99,
sum by (le, app, cluster_environmentd_materialize_cloud_cluster_id) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="consensus_cas"}[5m])
)
)
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
rate(mz_persist_cmd_cas_mismatch_count{materialize_cloud_organization_name=~".*"}[5m])
)
sum by (op) (
rate(mz_persist_retry_retries_count{materialize_cloud_organization_name=~".*", op=~"fetch_state::scan|apply_unbatched_cmd::cas|gc::truncate|maybe_init::cas|consensus::open"}[5m])
)
sum by (op) (
rate(mz_ts_oracle_retry_retries_count{materialize_cloud_organization_name=~".*"}[5m])
)
sum by (op) (
rate(mz_ts_oracle_started_count{materialize_cloud_organization_name=~".*"}[5m])
)
(
sum by (op) (
rate(mz_ts_oracle_seconds{materialize_cloud_organization_name=~".*"}[5m])
)
/
sum by (op) (
rate(mz_ts_oracle_started_count{materialize_cloud_organization_name=~".*"}[5m])
)
) >= 0
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
mz_persist_postgres_connpool_size{materialize_cloud_organization_name=~".*"}
)
sum(mz_ts_oracle_postgres_connpool_size{materialize_cloud_organization_name=~".*"})
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
mz_persist_postgres_connpool_size{materialize_cloud_organization_name=~".*"}
- clamp_min(mz_persist_postgres_connpool_available{materialize_cloud_organization_name=~".*"}, 0)
)
sum(
mz_ts_oracle_postgres_connpool_size{materialize_cloud_organization_name=~".*"}
- clamp_min(mz_ts_oracle_postgres_connpool_available{materialize_cloud_organization_name=~".*"}, 0)
)
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
clamp_min(-mz_persist_postgres_connpool_available{materialize_cloud_organization_name=~".*"}, 0)
)
sum(clamp_min(-mz_ts_oracle_postgres_connpool_available{materialize_cloud_organization_name=~".*"}, 0))
(
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
rate(mz_persist_postgres_connpool_acquire_seconds{materialize_cloud_organization_name=~".*"}[5m])
)
/
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
rate(mz_persist_postgres_connpool_acquires{materialize_cloud_organization_name=~".*"}[5m])
)
) >= 0
(
sum(rate(mz_ts_oracle_postgres_connpool_acquire_seconds{materialize_cloud_organization_name=~".*"}[5m]))
/
sum(rate(mz_ts_oracle_postgres_connpool_acquires{materialize_cloud_organization_name=~".*"}[5m]))
) >= 0
sum(rate(mz_persist_postgres_connpool_connections_created{materialize_cloud_organization_name=~".*"}[5m]))
+ (sum(rate(mz_ts_oracle_postgres_connpool_connections_created{materialize_cloud_organization_name=~".*"}[5m])) or vector(0))
sum(rate(mz_persist_postgres_connpool_ttl_reconnections{materialize_cloud_organization_name=~".*"}[5m]))
+ (sum(rate(mz_ts_oracle_postgres_connpool_ttl_reconnections{materialize_cloud_organization_name=~".*"}[5m])) or vector(0))
sum(rate(mz_persist_external_succeeded_count{materialize_cloud_organization_name=~".*", op="consensus_cas"}[5m]))
sum(rate(mz_persist_external_consensus_truncated_count{materialize_cloud_organization_name=~".*"}[5m]))
sum(max by (shard) (mz_persist_shard_gc_live_diffs{materialize_cloud_organization_name=~".*"}))
topk(10, max by (shard) (mz_persist_shard_seqnos_held{materialize_cloud_organization_name=~".*"}))
sum(rate(mz_persist_gc_started{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_gc_finished{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_gc_noop{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_lease_timeout_read{materialize_cloud_organization_name=~".*"}[5m]))
Kubernetes events from the namespaces a Materialize deployment occupies: the operator’s own namespace and the environments’ namespace.
Events are logs, not metrics. They arrive through
loki.source.kubernetes_events in the monitoring gateway, which reads them
from the Kubernetes API and forwards them to Loki, so every query here is
LogQL against the logs datasource rather than PromQL. The gateway’s processor
lifts reason, name, kind, count, node and reportingcontroller out
of each event into structured metadata, which is what these queries filter and
group on; the event type becomes the level stream label, Normal as
INFO and Warning as WARN.
Two scopes live here. The deployment and operator queries below are
rollout-scoped: they answer “is this upgrade going through”, and the
env-upgrade dashboard is their consumer. The cluster queries at the end are
the general-purpose view, scoped by the same Loki-discovered namespace picker a
logs dashboard uses, and are deliberately separate definitions rather than the
same ones widened — the rollout queries carry filters (generation, reporting
controller) that a general event browser has no business inheriting.
Kubernetes keeps events for about an hour. Loki keeps them for as long as the deployment’s retention says, which is what makes a rollout that finished yesterday still explainable.
An event’s namespace is the involved object’s, not the reporter’s. The
operator runs in its own namespace and reconciles resources in the
environments’ namespace, and every event it publishes is filed against the
resource — so the operator’s own events are found in the environment
namespace, where nothing else about them suggests they would be. That is why
the queries below scope to both namespaces and pick the operator out by
reportingcontroller, which is the reporter’s identity and the only field
that actually says an event came from orchestratord.
Only the deployment-wide feeds filter by generation. The operator’s own
events are filed against the Materialize, Balancer and Console resources,
which carry no generation at all, so %%{mzGenerationEventFilter} could only
ever be a no-op on them. The filter itself keeps generation-less objects on
purpose — on a representative deployment only 6 of 70 event names carry a
generation, and dropping the other 64 would take the whole rollout narrative
with them.
Pulled, Created and Started rise together as pods are
replaced, and fall back to nothing when it finishes.
Applying, ReadyToPromote, WaitingForApproval,
Promoting, Applied, and the two that end a rollout badly,
RolloutTimeout and FailedDeploy.
What each deployment generation of an environment is doing, during and after a blue/green rollout.
A rollout stands a new generation of environmentd and its replicas up beside
the old one, lets it rehydrate from persisted storage, and promotes it only once
it has caught up. Both generations are live and scraped at the same time, so
every ordinary panel sums them together — which is exactly the wrong thing while
the question is whether one of them is ready yet.
The generation is not a label. orchestratord records it as a Kubernetes
annotation, which neither kube-state-metrics nor cAdvisor surfaces. Where it
does reach a query is the object name, in two shapes:
…-environmentd-<generation>-<ordinal> and, for a replica,
…-gen-<generation>-<ordinal>. %%{mzGenerationFilter} selects on those, and
%%{mzGenerationPattern} is the same shape as a capture, for the
label_replace that lifts the number into a generation label panels can group
by. Both live in the render context, so they cannot drift apart.
count(
count by (generation) (
label_replace(
mz_compute_commands_total{
materialize_cloud_organization_name=~".*", ${mzGenerationFilter}
},
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
)
group by (generation, mz_version) (
label_replace(
max_over_time(
mz_compute_cluster_status{
materialize_cloud_organization_name=~".*", ${mzGenerationFilter}
}
[1h]
),
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
count by (generation) (
group by (pod, generation) (
label_replace(
container_memory_working_set_bytes{
container!="POD", container!="", ${mzGenerationFilter}
},
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
)
sum by (generation) (
max by (generation, instance_id, collection_id) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
) > bool 1e15
)
)
count by (generation) (
max by (generation, instance_id, collection_id) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
)
max by (generation) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
) < 1e9
)
sum by (generation) (
max by (generation, instance_id, collection_id) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
) < 1e9
)
)
sum by (generation, instance_id) (
max by (generation, instance_id, collection_id) (
label_replace(
mz_dataflow_wallclock_lag_seconds{
materialize_cloud_organization_name=~".*",
${mzGenerationFilter},
instance_id=~".*",
instance_id!="",
quantile="1"
},
"generation", "$1", "pod", "${mzGenerationPattern}"
) < 1e9
)
)
sum by (generation) (
label_replace(
rate(
container_cpu_usage_seconds_total{
container!="POD", container!="", ${mzGenerationFilter}
}
[5m]
),
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
sum by (generation) (
label_replace(
container_memory_working_set_bytes{
container!="POD", container!="", ${mzGenerationFilter}
},
"generation", "$1", "pod", "${mzGenerationPattern}"
)
)
up{
job="monitoring/mzmon-materialize-environmentd",
namespace=~"materialize-environment"
} == 1
avg:up{job:monitoring/mzmon-materialize-environmentd, ${mzEnvironmentNamespaceFilter}}
up{
job="monitoring/mzmon-materialize-clusterd",
namespace=~"materialize-environment"
} == 1
avg:up{job:monitoring/mzmon-materialize-clusterd, ${mzEnvironmentNamespaceFilter}}
up{
job="materialize/mzmon-materialize-operator",
namespace=~"materialize"
} == 1
avg:up{job:materialize/mzmon-materialize-operator, ${mzOperatorNamespaceFilter}}
count(
mz_compute_cluster_status{
materialize_cloud_organization_name=~".*"
} == 1
) / count(
mz_compute_cluster_status{
materialize_cloud_organization_name=~".*"
}
) * 100
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}} * 100
avg by (materialize_cloud_organization_namespace) (
avg_over_time(
mz_compute_cluster_status{
materialize_cloud_organization_name=~".*"
}[1h]
) * 100
)
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}}
by {materialize_cloud_organization_namespace}.rollup(avg, ${range}) * 100
group by (mz_version) (
mz_compute_cluster_status{materialize_cloud_organization_name=~".*"}
)
avg:${mzSqlPrefix}compute_cluster_status{${mzEnvironmentFilter}} by {mz_version}
max(
max_over_time(
(
mz_dataflow_wallclock_lag_seconds{materialize_cloud_organization_name=~".*", instance_id!="", quantile="1"} < 1e9
)[${rangeWindow}:1m]
)
)
max:mz_dataflow_wallclock_lag_seconds{${mzEnvironmentFilter}, quantile:1}.rollup(max, 3600)
Kubernetes-side view of a Materialize deployment: capacity, workload readiness, per-pod resource usage, and networking. Adapted from the Overview dashboard’s “Kubernetes Workloads” tab and the k8s-sourced Summary panels.
These read kube-state-metrics and cAdvisor (via the kubelet), NOT Materialize
metrics — the same meta-monitoring surface other targets (e.g. Loki health)
will draw on. %%{cAdvisorFilter} is the container-scoping filter fragment
(namespace + drop empty/pause series); %%{mzNamespaceList} is the raw
namespace selector used by the kube_* and container_network_* metrics.
The percent-of-limit panels have an absolute-units sibling for deployments whose metrics source (e.g. GKE’s managed cAdvisor/KSM) doesn’t expose resource limits; the dashboard picks whichever fits the environment.
sum by (container) (
container_spec_cpu_quota{container!="POD", container!="", container!="new-promsql-exporter"}
/ container_spec_cpu_period{container!="POD", container!="", container!="new-promsql-exporter"}
)
sum:container_spec_cpu_quota{${cAdvisorFilter}, !container:new-promsql-exporter} by {container} / 100000
sum by (container) (
container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)
sum:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {container}
sum by (container) (
container_spec_cpu_quota{container!="POD", container!=""}
/ container_spec_cpu_period{container!="POD", container!=""}
)
sum:container_spec_cpu_quota{${cAdvisorFilter}} by {container} / 100000
sum by (container) (
container_spec_memory_limit_bytes{container!="POD", container!=""}
)
sum:container_spec_memory_limit_bytes{${cAdvisorFilter}} by {container}
sum by (namespace, container) (
rate(container_cpu_usage_seconds_total{container!="POD", container!=""}[5m])
) / sum by (namespace, container) (
kube_pod_container_resource_limits{resource="cpu", namespace=~"materialize-environment"}
)
sum:container_cpu_usage_seconds_total{${cAdvisorFilter}} by {namespace,container}.as_rate()
/ sum:kube_pod_container_resource_limits{resource:cpu, namespace:${mzNamespaceList}} by {namespace,container}
sum by (namespace, container) (
rate(container_cpu_usage_seconds_total{container!="POD", container!=""}[5m])
)
sum:container_cpu_usage_seconds_total{${cAdvisorFilter}} by {namespace,container}.as_rate()
sum by (namespace, container) (
avg by (namespace, pod, container) (
container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)
) / sum by (namespace, container) (
avg by (namespace, pod, container) (
container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)
)
sum:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,container}
/ sum:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,container}
sum by (namespace, container) (
container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter"}
)
sum:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,container}
time()
- topk(1,
container_start_time_seconds{container!="POD", container!="", container!="new-promsql-exporter"}
)
top(
max:container_start_time_seconds{${cAdvisorFilter}, !container:new-promsql-exporter} by {namespace,pod,container},
1, 'max', 'desc'
)
max by (phase, namespace) (
sum by (phase, namespace, instance) (
kube_pod_status_phase{namespace=~"materialize-environment"}
)
)
sum:kube_pod_status_phase{namespace:${mzNamespaceList}} by {phase,namespace}
max by (namespace) (
sum by (namespace, instance) (
kube_statefulset_status_replicas_ready{namespace=~"materialize-environment"}
)
)
sum:kube_statefulset_status_replicas_ready{namespace:${mzNamespaceList}} by {namespace}
max by (namespace) (
sum by (namespace, instance) (
kube_deployment_status_replicas_ready{namespace=~"materialize-environment"}
)
)
max by (namespace) (
sum by (namespace, instance) (
kube_deployment_status_replicas_unavailable{namespace=~"materialize-environment"}
)
)
sum:kube_deployment_status_replicas_ready{namespace:${mzNamespaceList}} by {namespace}
sum:kube_deployment_status_replicas_unavailable{namespace:${mzNamespaceList}} by {namespace}
sum by (namespace, pod, container) (
rate(container_cpu_usage_seconds_total{container!="POD", container!="", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
) / sum by (namespace, pod, container) (
kube_pod_container_resource_limits{resource="cpu", namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}
)
sum by (namespace, pod, container) (
rate(container_cpu_usage_seconds_total{container!="POD", container!="", pod!~".*-cluster-.*-replica-.*"}[5m])
) / sum by (namespace, pod, container) (
kube_pod_container_resource_limits{resource="cpu", namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}
)
sum:container_cpu_usage_seconds_total{${cAdvisorFilter}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}.as_rate()
/ sum:kube_pod_container_resource_limits{resource:cpu, namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}
sum:container_cpu_usage_seconds_total{${cAdvisorFilter}, !pod:*-cluster-*-replica-*} by {namespace,pod,container}.as_rate()
/ sum:kube_pod_container_resource_limits{resource:cpu, namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod,container}
avg by (namespace, pod, container) (
container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}
) / avg by (namespace, pod, container) (
container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}
)
avg by (namespace, pod, container) (
container_memory_working_set_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod!~".*-cluster-.*-replica-.*"}
) / avg by (namespace, pod, container) (
container_spec_memory_limit_bytes{container!="POD", container!="", container!="new-promsql-exporter", pod!~".*-cluster-.*-replica-.*"}
)
avg:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}
/ avg:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod,container}
avg:container_memory_working_set_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, !pod:*-cluster-*-replica-*} by {namespace,pod,container}
/ avg:container_spec_memory_limit_bytes{${cAdvisorFilter}, !container:new-promsql-exporter, !pod:*-cluster-*-replica-*} by {namespace,pod,container}
sum by (namespace, pod) (
rate(container_network_receive_bytes_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_receive_bytes_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum:container_network_receive_bytes_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_receive_bytes_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum by (namespace, pod) (
rate(container_network_transmit_bytes_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_bytes_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum:container_network_transmit_bytes_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_bytes_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum by (namespace, pod) (
rate(container_network_receive_errors_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_receive_errors_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_errors_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_errors_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum:container_network_receive_errors_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_receive_errors_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_errors_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_errors_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum by (namespace, pod) (
rate(container_network_receive_packets_dropped_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_receive_packets_dropped_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_packets_dropped_total{namespace=~"materialize-environment", pod=~".*-cluster-${mzClusterListRegex}-replica-${mzReplicaListRegex}-.*"}[5m])
)
sum by (namespace, pod) (
rate(container_network_transmit_packets_dropped_total{namespace=~"materialize-environment", pod!~".*-cluster-.*-replica-.*"}[5m])
)
sum:container_network_receive_packets_dropped_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_receive_packets_dropped_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_packets_dropped_total{namespace:${mzNamespaceList}, pod:*-cluster-${mzClusterList}-replica-${mzReplicaList}-*} by {namespace,pod}.as_rate()
sum:container_network_transmit_packets_dropped_total{namespace:${mzNamespaceList}, !pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate()
Logs, as collected by the monitoring stack and stored in Loki.
Every query here is LogQL, and the scope is Loki-discovered end to end: namespace, app and level come from Loki’s own label values rather than from the metrics pipeline. That is deliberate. Reading logs is frequently how an operator works out why the metrics pipeline is broken, and a logs dashboard that derived its scope from Prometheus would go blind at exactly the moment it is most needed.
The label contract the agent and gateway produce is documented under
Logs and Events. What matters here:
namespace, app, level and container are stream labels and belong in the
selector; pod, node, organization_name and the rest are structured
metadata and are filtered after a |. Narrowing the selector before the line
filters is the single biggest speedup.
Every selector here carries %%{mzLogJobFilter}, and it is not only a filter.
LogQL rejects a stream selector whose every matcher can match the empty string —
“queries require at least one regexp or equality matcher that does not have an
empty-compatible value” — and a dashboard built from =~ pickers is exactly
that shape. The job picker’s “All” is .+ rather than the discovered values, so
it always contributes a non-empty matcher and the selector parses whatever the
other pickers are set to. Without it, “All” everywhere is a query error rather
than a wide result.
The event queries in materialize-events.yaml need no such anchor: they pin
job="loki.source.kubernetes_events", which is already a non-empty equality
matcher, and a second job matcher would AND with it and zero the panel.
level is normalized by the pipeline where it can be and falls back to
UNKNOWN, so the levels present are a property of the workloads running rather
than a fixed vocabulary — which is why the dashboard discovers them instead of
hard-coding a list.
How the Materialize operator’s reconciliation loop is behaving.
orchestratord watches the Materialize, Balancer and Console resources
and drives each toward the state its spec asks for. One trip through that
work is a pass, and a pass moves through named steps. Both are counted by
outcome, and both are timed, which is what lets a stuck rollout say not just
that it is stuck but which phase it is stuck in.
These are the operator’s own metrics, scraped from its pods in the operator
namespace, so they are scoped by %%{mzOperatorNamespaceFilter} and by
nothing else. They carry no organization label, so the environment picker
does not narrow them: one operator reconciles every environment in the
cluster, and its loop is a single shared thing rather than a per-environment
one.
Only the replica holding the leadership lease reconciles. The others export the same metric families sitting at zero, which is why every query here sums across replicas rather than picking one out.
sum(
orchestratord_is_leader{namespace=~"materialize"}
)
sum(
environmentd_needs_update{namespace=~"materialize"}
)
sum(
rate(
orchestratord_reconciliations_total{namespace=~"materialize"}
[5m]
)
)
sum(
increase(
orchestratord_reconciliations_total{
namespace=~"materialize", outcome="failed"
}
[1h]
)
)
waiting climbs while the new generation’s pods come up, then
gives way to applied when they are ready.
sum by (outcome) (
rate(
orchestratord_reconciliations_total{namespace=~"materialize"}
[5m]
)
)
sum by (controller, event_type) (
rate(
orchestratord_reconciliations_total{
namespace=~"materialize", outcome="failed"
}
[5m]
)
)
histogram_quantile(0.5, sum by (le) (
rate(
orchestratord_reconciliation_duration_seconds_bucket{
namespace=~"materialize"
}
[5m]
)
))
histogram_quantile(0.9, sum by (le) (
rate(
orchestratord_reconciliation_duration_seconds_bucket{
namespace=~"materialize"
}
[5m]
)
))
histogram_quantile(0.99, sum by (le) (
rate(
orchestratord_reconciliation_duration_seconds_bucket{
namespace=~"materialize"
}
[5m]
)
))
histogram_quantile(0.99, sum by (le, step) (
rate(
orchestratord_reconciliation_step_duration_seconds_bucket{
namespace=~"materialize"
}
[5m]
)
))
sum by (step) (
rate(
orchestratord_reconciliation_steps_total{namespace=~"materialize"}
[5m]
)
)
sum by (step, outcome) (
rate(
orchestratord_reconciliation_steps_total{
namespace=~"materialize", outcome=~"failed|abandoned"
}
[5m]
)
)
Object storage (the persist bucket), from the vantage point of the Materialize processes that use it.
Persist is Materialize’s storage layer. Every durable collection — a table, a
source, a materialized view, the catalog itself — is a persist shard: data
files in object storage, plus a small record of the shard’s current state in
the metadata database. That second half is materialize-consensus.yaml; this
file is the first. Writers upload new data as immutable parts, compaction
merges small parts into larger ones and deletes what it replaced, and readers
fetch parts — through an in-memory cache — to hydrate and to serve queries.
Everything here is the client’s measurement: what a Materialize process
experienced, including the network between it and the store. It is identical
on S3, GCS (which Materialize reaches through GCS’s S3-compatible API), Azure
Blob and any S3-compatible on-premise store, and needs no cloud credentials.
What the bucket itself reports — its size, its object count, reclaimable
waste — is on infra-cloud.
op on mz_persist_external_* is the operation: blob_get, blob_set,
blob_delete, blob_list_keys and restore. The consensus_* values of
the same label are the metadata database. mz_persist_external_op_latency is
a histogram for blob_get and blob_set only; the rest have a mean from
_seconds / _started_count.
mz_persist_s3_* is the S3 client beneath persist, used on S3, GCS and
S3-compatible stores but not on Azure Blob, which Materialize reaches with
Azure’s own client. The counters are registered on every install, though, so
on Azure they exist and read zero; queries on them gate on the client making
calls. It is the only family carrying the store’s error code (created on
first error, so absent until then), and its op values are raw API calls
(get_part, set_single, delete_object, …) rather than persist
operations.
op on mz_persist_retry_* is a retry loop named after the work. The ones
in materialize.persist.retries.by_operation wrap object-storage calls; the
consensus file owns the rest, and next_listen_batch and snapshot are
polling loops that must never be drawn as retries.
The mz_persist_shard_usage_* families are per shard, and each process
reports every shard it has open, so any total over them takes a max by (shard) first. environmentd has every shard open, so the deduplicated total
is the environment’s.
Every mean and share here is a ratio of two rates wrapped in (…) >= 0, for
the reason materialize-consensus.yaml gives: a window with no calls divides
zero by zero.
sum(rate(mz_persist_external_failed_count{materialize_cloud_organization_name=~".*", op=~"blob_.*"}[5m]))
histogram_quantile(0.99,
sum by (le) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="blob_get"}[5m])
)
)
histogram_quantile(0.99,
sum by (le) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="blob_set"}[5m])
)
)
sum(rate(mz_persist_user_write_stall_count{materialize_cloud_organization_name=~".*"}[5m]))
+ sum(rate(mz_persist_compaction_write_stall_count{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_compaction_failed{materialize_cloud_organization_name=~".*"}[5m]))
+ sum(rate(mz_persist_compaction_timed_out{materialize_cloud_organization_name=~".*"}[5m]))
sum(max by (shard) (mz_persist_shard_usage_current_state_batches_bytes{materialize_cloud_organization_name=~".*"}))
+ sum(max by (shard) (mz_persist_shard_usage_current_state_rollups_bytes{materialize_cloud_organization_name=~".*"}))
histogram_quantile(0.50,
sum by (le) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="blob_get"}[5m])
)
)
histogram_quantile(0.99,
sum by (le) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="blob_get"}[5m])
)
)
histogram_quantile(0.50,
sum by (le) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="blob_set"}[5m])
)
)
histogram_quantile(0.99,
sum by (le) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="blob_set"}[5m])
)
)
max by (app, cluster_environmentd_materialize_cloud_cluster_id) (
mz_persist_external_rtt_latency{materialize_cloud_organization_name=~".*", external="blob"}
)
sum by (op) (
rate(mz_persist_external_failed_count{materialize_cloud_organization_name=~".*", op=~"blob_.*"}[5m])
)
sum by (op, code) (
rate(mz_persist_s3_errors{materialize_cloud_organization_name=~".*"}[5m])
)
sum(rate(mz_persist_s3_connect_timeouts{materialize_cloud_organization_name=~".*"}[5m]))
and on () (sum(rate(mz_persist_s3_operations{materialize_cloud_organization_name=~".*"}[5m])) > 0)
sum(rate(mz_persist_s3_read_timeouts{materialize_cloud_organization_name=~".*"}[5m]))
and on () (sum(rate(mz_persist_s3_operations{materialize_cloud_organization_name=~".*"}[5m])) > 0)
sum(rate(mz_persist_s3_operation_attempt_timeouts{materialize_cloud_organization_name=~".*"}[5m]))
and on () (sum(rate(mz_persist_s3_operations{materialize_cloud_organization_name=~".*"}[5m])) > 0)
sum(rate(mz_persist_s3_operation_timeouts{materialize_cloud_organization_name=~".*"}[5m]))
and on () (sum(rate(mz_persist_s3_operations{materialize_cloud_organization_name=~".*"}[5m])) > 0)
sum by (op) (
rate(mz_persist_external_started_count{materialize_cloud_organization_name=~".*", op=~"blob_.*"}[5m])
)
sum by (op) (
rate(mz_persist_external_bytes_count{materialize_cloud_organization_name=~".*", op=~"blob_.*"}[5m])
)
sum by (op) (
rate(mz_persist_s3_operations{materialize_cloud_organization_name=~".*"}[5m])
)
and on () (sum(rate(mz_persist_s3_operations{materialize_cloud_organization_name=~".*"}[5m])) > 0)
(
sum by (op) (
rate(mz_persist_external_seconds{materialize_cloud_organization_name=~".*", op=~"blob_.*"}[5m])
)
/
sum by (op) (
rate(mz_persist_external_started_count{materialize_cloud_organization_name=~".*", op=~"blob_.*"}[5m])
)
) >= 0
histogram_quantile(0.99,
sum by (le, app, cluster_environmentd_materialize_cloud_cluster_id) (
rate(mz_persist_external_op_latency_bucket{materialize_cloud_organization_name=~".*", op="blob_get"}[5m])
)
)
sum by (op) (
rate(mz_persist_retry_retries_count{materialize_cloud_organization_name=~".*", op=~"batch::set|batch::delete|fetch_batch::get|rollup::get|rollup::set|rollup::delete|hollow_run::get|hollow_run::set|blob::open|compaction_noop::delete|storage_usage::shard_size"}[5m])
)
(
sum(rate(mz_persist_blob_cache_hits_bytes{materialize_cloud_organization_name=~".*"}[5m]))
/
(
sum(rate(mz_persist_blob_cache_hits_bytes{materialize_cloud_organization_name=~".*"}[5m]))
+ sum(rate(mz_persist_external_bytes_count{materialize_cloud_organization_name=~".*", op="blob_get"}[5m]))
)
) >= 0
sum(rate(mz_persist_blob_hedges_fired{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_blob_hedges_won{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_compaction_requested{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_compaction_started{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_compaction_applied{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_compaction_failed{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_compaction_timed_out{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_compaction_dropped{materialize_cloud_organization_name=~".*"}[5m]))
(
sum(rate(mz_persist_compaction_queued_seconds{materialize_cloud_organization_name=~".*"}[5m]))
/
sum(rate(mz_persist_compaction_started{materialize_cloud_organization_name=~".*"}[5m]))
) >= 0
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
rate(mz_persist_compaction_seconds{materialize_cloud_organization_name=~".*"}[5m])
)
sum(rate(mz_persist_user_bytes{materialize_cloud_organization_name=~".*"}[5m]))
sum(rate(mz_persist_compaction_bytes{materialize_cloud_organization_name=~".*"}[5m]))
sum by (app, cluster_environmentd_materialize_cloud_cluster_id) (
rate(mz_persist_user_write_stall_count{materialize_cloud_organization_name=~".*"}[5m])
+ rate(mz_persist_compaction_write_stall_count{materialize_cloud_organization_name=~".*"}[5m])
)
sum(max by (shard) (mz_persist_shard_usage_current_state_batches_bytes{materialize_cloud_organization_name=~".*"}))
sum(max by (shard) (mz_persist_shard_usage_current_state_rollups_bytes{materialize_cloud_organization_name=~".*"}))
sum(max by (shard) (mz_persist_shard_usage_referenced_not_current_state_bytes{materialize_cloud_organization_name=~".*"}))
sum(max by (shard) (mz_persist_shard_usage_not_leaked_not_referenced_bytes{materialize_cloud_organization_name=~".*"}))
sum(max by (shard) (mz_persist_shard_usage_leaked_bytes{materialize_cloud_organization_name=~".*"}))
sum(max by (shard) (mz_persist_shard_usage_leaked_bytes{materialize_cloud_organization_name=~".*"}))
topk(10,
max by (shard) (mz_persist_shard_usage_current_state_batches_bytes{materialize_cloud_organization_name=~".*"})
+ max by (shard) (mz_persist_shard_usage_current_state_rollups_bytes{materialize_cloud_organization_name=~".*"})
)
topk(10, max by (shard) (mz_persist_shard_usage_referenced_not_current_state_bytes{materialize_cloud_organization_name=~".*"}))
Sources and sinks for a Materialize deployment — catalog shape, throughput, lag, and upstream/downstream health. Adapted from the Overview dashboard’s “Sources and Sinks” tab.
The clusterd-side throughput/lag/error metrics (mz_source_* / mz_sink_) carry
the long-form cluster_environmentd_materialize_cloud_ id labels. These queries
assume one Prometheus job per clusterd endpoint; if the same endpoint is
scraped by several jobs, a plain sum-rate reads N× — dedupe the job at the
deployment (fix the scrape config, or wrap the inner rate in max without (job)) rather than baking it into the canonical query.
_progress
subsources are excluded), matching mz_sources.
count(group by (id) (mz_storage_objects{materialize_cloud_organization_name=~".*", type="source"}))
default_zero(count_not_null(avg:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:source} by {id}))
_progress subsources), matching mz_sinks.
count(group by (id) (mz_storage_objects{materialize_cloud_organization_name=~".*", type="sink"}))
default_zero(count_not_null(avg:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:sink} by {id}))
INSERTs feed dataflows downstream. Mostly a catalog-shape signal — for
actual ingest activity look at source throughput.
max(sum by (instance) (mz_tables_count{materialize_cloud_organization_name=~".*"}))
default_zero(sum:${mzSqlPrefix}tables_count{${mzEnvironmentFilter}} by {instance})
count by (object_type) (
group by (id, object_type) (
mz_storage_objects{materialize_cloud_organization_name=~".*", type="source"}
)
) > 0
sum:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:source} by {object_type}
group by (id, object_type, connection_type, envelope_type, cluster_id) (
mz_storage_objects{materialize_cloud_organization_name=~".*", type="source"}
)
avg:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:source}
by {id,object_type,connection_type,envelope_type,cluster_id}
sum by (parent_source_id) (
max without (job) (rate(mz_source_bytes_received{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
) > 0
sum:mz_source_bytes_received{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {parent_source_id}.as_rate()
sum by (parent_source_id, cluster_environmentd_materialize_cloud_replica_id) (
max without (job) (rate(mz_source_messages_received{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
)
sum:mz_source_messages_received{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {parent_source_id,cluster_environmentd_materialize_cloud_replica_id}.as_rate()
sum by (source_id) (
max without (job) (rate(mz_source_offset_commit_failures{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
) > 0
(
max by (source_id) (mz_source_offset_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"})
> bool max by (source_id) (mz_source_offset_known{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"})
) > 0
sum:mz_source_offset_commit_failures{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {source_id}.as_rate()
max:mz_source_offset_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {source_id}
- max:mz_source_offset_known{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {source_id}
kafka / upsert, kafka / debezium,
iceberg / upsert. The envelope is how Materialize encodes changes:
upsert writes the latest value per key, debezium writes change
events with old+new values.
count by (object_type, envelope_type) (
group by (id, object_type, envelope_type) (
mz_storage_objects{materialize_cloud_organization_name=~".*", type="sink"}
)
) > 0
sum:${mzSqlPrefix}storage_objects{${mzEnvironmentFilter}, type:sink} by {object_type,envelope_type}
sum by (sink_id) (
max without (job) (rate(mz_sink_bytes_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))
) > 0
sum:mz_sink_bytes_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
clamp_min(
sum by (sink_id) (max without (job) (mz_sink_bytes_staged{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}))
- sum by (sink_id) (max without (job) (mz_sink_bytes_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"})),
0
)
clamp_min(
sum:mz_sink_bytes_staged{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}
- sum:mz_sink_bytes_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id},
0
)
COMMIT against the Iceberg catalog takes (write a snapshot manifest,
ask the catalog to atomically swap it in).
histogram_quantile(0.50, sum by (le) (max without (job) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))))
histogram_quantile(0.90, sum by (le) (max without (job) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))))
histogram_quantile(0.99, sum by (le) (max without (job) (rate(mz_sink_iceberg_commit_duration_seconds_bucket{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m]))))
p50:mz_sink_iceberg_commit_duration_seconds{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
p90:mz_sink_iceberg_commit_duration_seconds{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
p99:mz_sink_iceberg_commit_duration_seconds{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}}
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_commit_failures{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_commit_conflicts{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum:mz_sink_iceberg_commit_failures{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum:mz_sink_iceberg_commit_conflicts{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_data_files_written{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_delete_files_written{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum by (sink_id) (max without (job) (rate(mz_sink_iceberg_snapshots_committed{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum:mz_sink_iceberg_data_files_written{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum:mz_sink_iceberg_delete_files_written{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum:mz_sink_iceberg_snapshots_committed{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum by (sink_id) (max without (job) (rate(mz_sink_rdkafka_txerrs{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum:mz_sink_rdkafka_txerrs{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum by (sink_id) (max without (job) (mz_sink_rdkafka_outbuf_msg_cnt{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}))
sum:mz_sink_rdkafka_outbuf_msg_cnt{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}
sum by (sink_id) (max without (job) (rate(mz_sink_rdkafka_connects{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum by (sink_id) (max without (job) (rate(mz_sink_rdkafka_disconnects{materialize_cloud_organization_name=~".*", cluster_environmentd_materialize_cloud_cluster_id=~".*", cluster_environmentd_materialize_cloud_replica_id=~".*"}[5m])))
sum:mz_sink_rdkafka_connects{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
sum:mz_sink_rdkafka_disconnects{${mzEnvironmentFilter}, cluster_environmentd_materialize_cloud_cluster_id:${mzClusterList}, cluster_environmentd_materialize_cloud_replica_id:${mzReplicaList}} by {sink_id}.as_rate()
Alerting rules for the workloads running on Materialize.
Every alert here carries audience: workload, so a route can send them to
the people who own the clusters rather than to whoever runs the deployment.
Each is scoped to user clusters (u*); the same condition on a system
cluster is a platform alert.
group by (namespace, cluster_id, name) (mz_cluster_info{materialize_cloud_organization_name=~".*"})
max:mz_cluster_info{${mzEnvironmentFilter}} by {namespace,cluster_id,name}
The breakdowns you reach for once node-health.yaml has told you a node is in
trouble: which mode the CPU is in, where the memory went, which device is
slow, and where the packets are being lost. Adapted from the Node Exporter
Full dashboard (https://grafana.com/grafana/dashboards/1860, revision 45).
Split from node-health.yaml on tier rather than on subject. These sit at
recommended, so a deployment collecting only the essential tier still gets
the health surface and pays nothing for the detail. Nothing here should back
an alert — if something here is worth paging on, it belongs in
node-health.yaml instead.
The conventions are the same as node-health.yaml: instance=~"$nodeList" rather
than =, every query wrapped in max by (instance, ...) (or min where low
is the bad direction) so a second scrape job cannot double-count, inner
aggregations carrying by (instance, job) so the outer wrapper is what
collapses job, and %%{interval} as the rate window.
Only collectors this chart’s allowlist enables are referenced. Notable
omissions, because the dashboard has panels for them and they will render
empty: node_processes_* (processes collector), node_interrupts_total
(interrupts), node_tcp_connection_states (tcpstat), node_systemd_*
(systemd), and node_textfile_scrape_error (textfile).
max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="system", instance=~"$nodeList"}[5m])
)
)
max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="user", instance=~"$nodeList"}[5m])
)
)
max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="iowait", instance=~"$nodeList"}[5m])
)
)
max by (instance) (
avg by (instance, job) (
sum without (mode) (
rate(node_cpu_seconds_total{mode=~".*irq", instance=~"$nodeList"}[5m])
)
)
)
max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="steal", instance=~"$nodeList"}[5m])
)
)
avg:node_cpu_seconds_total{instance:$nodeList, mode:system} by {instance}.as_rate()
avg:node_cpu_seconds_total{instance:$nodeList, mode:user} by {instance}.as_rate()
avg:node_cpu_seconds_total{instance:$nodeList, mode:iowait} by {instance}.as_rate()
avg:node_cpu_seconds_total{instance:$nodeList, mode:*irq} by {instance}.as_rate()
avg:node_cpu_seconds_total{instance:$nodeList, mode:steal} by {instance}.as_rate()
1 - max by (instance, cpu) (
rate(node_cpu_seconds_total{mode="idle", instance=~"$nodeList"}[5m])
)
1 - max:node_cpu_seconds_total{instance:$nodeList, mode:idle} by {instance,cpu}.as_rate()
/proc/schedstat.
max by (instance, cpu) (
rate(node_schedstat_waiting_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_schedstat_waiting_seconds_total{instance:$nodeList} by {instance,cpu}.as_rate()
max by (instance) (
rate(node_context_switches_total{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_intr_total{instance=~"$nodeList"}[5m])
)
max:node_context_switches_total{instance:$nodeList} by {instance}.as_rate()
max:node_intr_total{instance:$nodeList} by {instance}.as_rate()
max by (instance) (node_memory_MemTotal_bytes{instance=~"$nodeList"})
max by (instance) (
node_memory_MemTotal_bytes{instance=~"$nodeList"}
- node_memory_MemFree_bytes{instance=~"$nodeList"}
- node_memory_Cached_bytes{instance=~"$nodeList"}
- node_memory_Buffers_bytes{instance=~"$nodeList"}
- node_memory_SReclaimable_bytes{instance=~"$nodeList"}
)
max by (instance) (
node_memory_Cached_bytes{instance=~"$nodeList"}
+ node_memory_Buffers_bytes{instance=~"$nodeList"}
+ node_memory_SReclaimable_bytes{instance=~"$nodeList"}
)
min by (instance) (node_memory_MemFree_bytes{instance=~"$nodeList"})
max:node_memory_MemTotal_bytes{instance:$nodeList} by {instance}
max:node_memory_MemTotal_bytes{instance:$nodeList} by {instance}
- max:node_memory_MemFree_bytes{instance:$nodeList} by {instance}
- max:node_memory_Cached_bytes{instance:$nodeList} by {instance}
- max:node_memory_Buffers_bytes{instance:$nodeList} by {instance}
- max:node_memory_SReclaimable_bytes{instance:$nodeList} by {instance}
max:node_memory_Cached_bytes{instance:$nodeList} by {instance}
+ max:node_memory_Buffers_bytes{instance:$nodeList} by {instance}
+ max:node_memory_SReclaimable_bytes{instance:$nodeList} by {instance}
min:node_memory_MemFree_bytes{instance:$nodeList} by {instance}
max by (instance) (node_memory_Slab_bytes{instance=~"$nodeList"})
max by (instance) (node_memory_SReclaimable_bytes{instance=~"$nodeList"})
max by (instance) (node_memory_SUnreclaim_bytes{instance=~"$nodeList"})
max by (instance) (node_memory_Committed_AS_bytes{instance=~"$nodeList"})
max:node_memory_Slab_bytes{instance:$nodeList} by {instance}
max:node_memory_SReclaimable_bytes{instance:$nodeList} by {instance}
max:node_memory_SUnreclaim_bytes{instance:$nodeList} by {instance}
max:node_memory_Committed_AS_bytes{instance:$nodeList} by {instance}
max by (instance) (
rate(node_vmstat_pgfault{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pgmajfault{instance=~"$nodeList"}[5m])
)
max:node_vmstat_pgfault{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pgmajfault{instance:$nodeList} by {instance}.as_rate()
kswapd (background) or direct (an allocating thread).
max by (instance) (
rate(node_vmstat_pgscan_kswapd{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pgscan_direct{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pgsteal_kswapd{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pgsteal_direct{instance=~"$nodeList"}[5m])
)
max:node_vmstat_pgscan_kswapd{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pgscan_direct{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pgsteal_kswapd{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pgsteal_direct{instance:$nodeList} by {instance}.as_rate()
max by (instance, device) (
rate(node_disk_reads_completed_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_disk_writes_completed_total{instance=~"$nodeList"}[5m])
)
max:node_disk_reads_completed_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_disk_writes_completed_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
rate(node_disk_read_bytes_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_disk_written_bytes_total{instance=~"$nodeList"}[5m])
)
max:node_disk_read_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_disk_written_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
(rate(node_disk_reads_completed_total{instance=~"$nodeList"}[5m]) > bool 0)
*
(
rate(node_disk_read_time_seconds_total{instance=~"$nodeList"}[5m])
/ rate(node_disk_reads_completed_total{instance=~"$nodeList"}[5m])
)
)
max by (instance, device) (
(rate(node_disk_writes_completed_total{instance=~"$nodeList"}[5m]) > bool 0)
*
(
rate(node_disk_write_time_seconds_total{instance=~"$nodeList"}[5m])
/ rate(node_disk_writes_completed_total{instance=~"$nodeList"}[5m])
)
)
max:node_disk_read_time_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
/ max:node_disk_reads_completed_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_disk_write_time_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
/ max:node_disk_writes_completed_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
rate(node_disk_io_time_weighted_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_disk_io_time_weighted_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
min by (instance, mountpoint) (
node_filesystem_files_free{instance=~"$nodeList", fstype!="rootfs"}
)
/
max by (instance, mountpoint) (
node_filesystem_files{instance=~"$nodeList", fstype!="rootfs"}
)
min:node_filesystem_files_free{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
/ max:node_filesystem_files{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
max by (instance, device) (
rate(node_network_receive_bytes_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_network_transmit_bytes_total{instance=~"$nodeList"}[5m])
)
max:node_network_receive_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_network_transmit_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
(node_network_speed_bytes{instance=~"$nodeList"} > bool 0)
*
(
rate(node_network_receive_bytes_total{instance=~"$nodeList"}[5m])
/ node_network_speed_bytes{instance=~"$nodeList"}
)
)
max by (instance, device) (
(node_network_speed_bytes{instance=~"$nodeList"} > bool 0)
*
(
rate(node_network_transmit_bytes_total{instance=~"$nodeList"}[5m])
/ node_network_speed_bytes{instance=~"$nodeList"}
)
)
max:node_network_receive_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
/ max:node_network_speed_bytes{instance:$nodeList} by {instance,device}
max:node_network_transmit_bytes_total{instance:$nodeList} by {instance,device}.as_rate()
/ max:node_network_speed_bytes{instance:$nodeList} by {instance,device}
min by (instance, device) (
node_network_up{instance=~"$nodeList"}
)
min by (instance, device) (
node_network_carrier{instance=~"$nodeList"}
)
min:node_network_up{instance:$nodeList} by {instance,device}
min:node_network_carrier{instance:$nodeList} by {instance,device}
max by (instance, cpu) (
rate(node_softnet_processed_total{instance=~"$nodeList"}[5m])
)
max:node_softnet_processed_total{instance:$nodeList} by {instance,cpu}.as_rate()
max by (instance, cpu) (
rate(node_softnet_dropped_total{instance=~"$nodeList"}[5m])
)
max:node_softnet_dropped_total{instance:$nodeList} by {instance,cpu}.as_rate()
max by (instance, cpu) (
rate(node_softnet_times_squeezed_total{instance=~"$nodeList"}[5m])
)
max:node_softnet_times_squeezed_total{instance:$nodeList} by {instance,cpu}.as_rate()
max by (instance) (node_sockstat_TCP_inuse{instance=~"$nodeList"})
max by (instance) (node_sockstat_TCP_alloc{instance=~"$nodeList"})
max by (instance) (node_sockstat_TCP_orphan{instance=~"$nodeList"})
max by (instance) (node_sockstat_TCP_tw{instance=~"$nodeList"})
max:node_sockstat_TCP_inuse{instance:$nodeList} by {instance}
max:node_sockstat_TCP_alloc{instance:$nodeList} by {instance}
max:node_sockstat_TCP_orphan{instance:$nodeList} by {instance}
max:node_sockstat_TCP_tw{instance:$nodeList} by {instance}
max by (instance) (node_sockstat_TCP_mem_bytes{instance=~"$nodeList"})
max by (instance) (node_sockstat_UDP_mem_bytes{instance=~"$nodeList"})
max:node_sockstat_TCP_mem_bytes{instance:$nodeList} by {instance}
max:node_sockstat_UDP_mem_bytes{instance:$nodeList} by {instance}
max by (instance) (
rate(node_netstat_Tcp_RetransSegs{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_TcpExt_TCPSynRetrans{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_Tcp_OutSegs{instance=~"$nodeList"}[5m])
)
max:node_netstat_Tcp_RetransSegs{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_TcpExt_TCPSynRetrans{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_Tcp_OutSegs{instance:$nodeList} by {instance}.as_rate()
max by (instance) (
rate(node_netstat_TcpExt_ListenOverflows{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_TcpExt_ListenDrops{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_TcpExt_TCPRcvQDrop{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_TcpExt_TCPTimeouts{instance=~"$nodeList"}[5m])
)
max:node_netstat_TcpExt_ListenOverflows{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_TcpExt_ListenDrops{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_TcpExt_TCPRcvQDrop{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_TcpExt_TCPTimeouts{instance:$nodeList} by {instance}.as_rate()
max by (instance) (
rate(node_netstat_Udp_InErrors{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_Udp_RcvbufErrors{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_netstat_Udp_NoPorts{instance=~"$nodeList"}[5m])
)
max:node_netstat_Udp_InErrors{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_Udp_RcvbufErrors{instance:$nodeList} by {instance}.as_rate()
max:node_netstat_Udp_NoPorts{instance:$nodeList} by {instance}.as_rate()
max by (instance) (
node_udp_queues{ip="v4", queue="rx", instance=~"$nodeList"}
)
max by (instance) (
node_udp_queues{ip="v4", queue="tx", instance=~"$nodeList"}
)
max:node_udp_queues{instance:$nodeList, ip:v4, queue:rx} by {instance}
max:node_udp_queues{instance:$nodeList, ip:v4, queue:tx} by {instance}
max by (instance, device) (
node_arp_entries{instance=~"$nodeList"}
)
max:node_arp_entries{instance:$nodeList} by {instance,device}
min by (instance) (
node_timex_sync_status{instance=~"$nodeList"}
)
min:node_timex_sync_status{instance:$nodeList} by {instance}
max by (instance) (node_timex_offset_seconds{instance=~"$nodeList"})
max by (instance) (node_timex_maxerror_seconds{instance=~"$nodeList"})
max by (instance) (node_timex_estimated_error_seconds{instance=~"$nodeList"})
max:node_timex_offset_seconds{instance:$nodeList} by {instance}
max:node_timex_maxerror_seconds{instance:$nodeList} by {instance}
max:node_timex_estimated_error_seconds{instance:$nodeList} by {instance}
min by (instance) (node_entropy_available_bits{instance=~"$nodeList"})
max by (instance) (node_entropy_pool_size_bits{instance=~"$nodeList"})
min:node_entropy_available_bits{instance:$nodeList} by {instance}
max:node_entropy_pool_size_bits{instance:$nodeList} by {instance}
max by (instance, collector) (
node_scrape_collector_duration_seconds{instance=~"$nodeList"}
)
max:node_scrape_collector_duration_seconds{instance:$nodeList} by {instance,collector}
Node-level health: the at-a-glance answers to “is this machine in trouble”, adapted from the Node Exporter Full dashboard (https://grafana.com/grafana/dashboards/1860, revision 45) and narrowed to what actually alerts.
These read node-exporter, NOT Materialize metrics. Only collectors this
chart’s allowlist enables are referenced — see the Node Exporter section of
the chart’s values reference for the list and the reasoning. The deeper
breakdowns live in node-debug.yaml at the recommended tier.
Three conventions apply to every query here:
instance=~"$nodeList", never instance="$nodeList". A regex match makes the
selector work unchanged whether the dashboard variable resolves to one node
or many, so the same query backs a single-node view and a fleet view.
Every query is wrapped in max by (instance, ...), which drops job.
A node should only ever be scraped by one job. If a second one appears —
a pre-existing node-exporter alongside ours, or a migration with both
running — the same series arrives twice under different job labels, and
every sum() silently doubles while every binary operation between two
metrics loses its match. Aggregating job away makes both failure modes
impossible rather than merely unlikely. max is the default; min where
low is the bad direction (available memory, free space, collector
success), so the aggregate always reports the worst case rather than hiding
it behind a healthy duplicate.
Where an inner aggregation is needed (averaging across CPUs, summing across
devices), it carries by (instance, job) and the outer max/min collapses
job afterwards. Aggregating both in one step would blend two jobs'
readings into one number instead of picking one.
%%{interval} is the rate window, including its brackets.
1 - max by (instance) (
avg by (instance, job) (
rate(node_cpu_seconds_total{mode="idle", instance=~"$nodeList"}[5m])
)
)
1 - avg:node_cpu_seconds_total{mode:idle, instance:$nodeList} by {instance}.as_rate()
max by (instance) (node_load1{instance=~"$nodeList"})
/
max by (instance) (
count by (instance, job) (
count by (instance, job, cpu) (node_cpu_seconds_total{instance=~"$nodeList"})
)
)
max:node_load1{instance:$nodeList} by {instance}
/ count_not_null(avg:node_cpu_seconds_total{mode:idle, instance:$nodeList} by {cpu})
max by (instance) (
rate(node_pressure_cpu_waiting_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_pressure_cpu_waiting_seconds_total{instance:$nodeList} by {instance}.as_rate()
MemAvailable.
min by (instance) (
node_memory_MemAvailable_bytes{instance=~"$nodeList"}
)
/
max by (instance) (
node_memory_MemTotal_bytes{instance=~"$nodeList"}
)
min:node_memory_MemAvailable_bytes{instance:$nodeList} by {instance}
/ max:node_memory_MemTotal_bytes{instance:$nodeList} by {instance}
waiting), and the fraction where every task was (stalled).
max by (instance) (
rate(node_pressure_memory_waiting_seconds_total{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_pressure_memory_stalled_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_pressure_memory_waiting_seconds_total{instance:$nodeList} by {instance}.as_rate()
max:node_pressure_memory_stalled_seconds_total{instance:$nodeList} by {instance}.as_rate()
(
(
max by (instance) (node_memory_SwapTotal_bytes{instance=~"$nodeList"})
-
min by (instance) (node_memory_SwapFree_bytes{instance=~"$nodeList"})
)
/
max by (instance) (node_memory_SwapTotal_bytes{instance=~"$nodeList"})
)
and
(max by (instance) (node_memory_SwapTotal_bytes{instance=~"$nodeList"}) > 0)
(
max:node_memory_SwapTotal_bytes{instance:$nodeList} by {instance}
- min:node_memory_SwapFree_bytes{instance:$nodeList} by {instance}
)
/ max:node_memory_SwapTotal_bytes{instance:$nodeList} by {instance}
vmstat.
max by (instance) (
rate(node_vmstat_pswpin{instance=~"$nodeList"}[5m])
)
max by (instance) (
rate(node_vmstat_pswpout{instance=~"$nodeList"}[5m])
)
max:node_vmstat_pswpin{instance:$nodeList} by {instance}.as_rate()
max:node_vmstat_pswpout{instance:$nodeList} by {instance}.as_rate()
max by (instance) (
rate(node_vmstat_oom_kill{instance=~"$nodeList"}[5m])
)
max:node_vmstat_oom_kill{instance:$nodeList} by {instance}.as_rate()
min by (instance, mountpoint) (
node_filesystem_avail_bytes{instance=~"$nodeList", fstype!="rootfs"}
)
/
max by (instance, mountpoint) (
node_filesystem_size_bytes{instance=~"$nodeList", fstype!="rootfs"}
)
min:node_filesystem_avail_bytes{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
/ max:node_filesystem_size_bytes{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
max by (instance, mountpoint) (
node_filesystem_readonly{instance=~"$nodeList", fstype!="rootfs"}
)
max:node_filesystem_readonly{instance:$nodeList, !fstype:rootfs} by {instance,mountpoint}
max by (instance, device) (
rate(node_disk_io_time_seconds_total{instance=~"$nodeList"}[5m])
)
max:node_disk_io_time_seconds_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance) (node_filefd_allocated{instance=~"$nodeList"})
/
max by (instance) (node_filefd_maximum{instance=~"$nodeList"})
max:node_filefd_allocated{instance:$nodeList} by {instance}
/ max:node_filefd_maximum{instance:$nodeList} by {instance}
max by (instance, device) (
rate(node_network_receive_errs_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_network_transmit_errs_total{instance=~"$nodeList"}[5m])
)
max:node_network_receive_errs_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_network_transmit_errs_total{instance:$nodeList} by {instance,device}.as_rate()
max by (instance, device) (
rate(node_network_receive_drop_total{instance=~"$nodeList"}[5m])
)
max by (instance, device) (
rate(node_network_transmit_drop_total{instance=~"$nodeList"}[5m])
)
max:node_network_receive_drop_total{instance:$nodeList} by {instance,device}.as_rate()
max:node_network_transmit_drop_total{instance:$nodeList} by {instance,device}.as_rate()
sum by (instance) (
max by (instance, device) (
rate(node_network_receive_bytes_total{instance=~"$nodeList"}[5m])
)
)
sum by (instance) (
max by (instance, device) (
rate(node_network_transmit_bytes_total{instance=~"$nodeList"}[5m])
)
)
max by (instance) (node_nf_conntrack_entries{instance=~"$nodeList"})
/
max by (instance) (node_nf_conntrack_entries_limit{instance=~"$nodeList"})
max:node_nf_conntrack_entries{instance:$nodeList} by {instance}
/ max:node_nf_conntrack_entries_limit{instance:$nodeList} by {instance}
min by (instance) (
node_time_seconds{instance=~"$nodeList"} - node_boot_time_seconds{instance=~"$nodeList"}
)
min:node_time_seconds{instance:$nodeList} by {instance}
- max:node_boot_time_seconds{instance:$nodeList} by {instance}
min by (instance, collector) (
node_scrape_collector_success{instance=~"$nodeList"}
)
min:node_scrape_collector_success{instance:$nodeList} by {instance,collector}