Common Alerts#

Many of these alerts are not suited for all deployments. Do not use the entire set as is!

infra-alerts#

Alerting rules for the infrastructure Materialize depends on.

Each alert carries its firing expression as an inline query, is graded by a severity label (critical > warning > notice), and is tagged with a component label naming the subsystem it watches.

crdb-disk-usage-critical #

CockroachDB disk usage is above 90% and likely to impact Materialize. Labels:

component: cockroachdb
severity: critical

max(
  crdb_dedicated_capacity_used / crdb_dedicated_capacity
) * 100 > 90

crdb-disk-usage-high #

CockroachDB disk usage is above 70% and may need a capacity increase. Labels:

component: cockroachdb
severity: warning

max(
  crdb_dedicated_capacity_used / crdb_dedicated_capacity
) * 100 > 70

crdb-disk-usage-elevated #

CockroachDB disk usage is above 30% and worth keeping an eye on. Labels:

component: cockroachdb
severity: notice

max(
  crdb_dedicated_capacity_used / crdb_dedicated_capacity
) * 100 > 30

crdb-cpu-usage-critical #

CockroachDB CPU usage is critically high (>89%) and likely to impact Materialize. Labels:

component: cockroachdb
severity: warning

max(
  avg_over_time(crdb_dedicated_sys_cpu_combined_percent_normalized[30m]) * 100
) > 89

crdb-cpu-usage-high #

CockroachDB CPU usage is above 85% over 2h and may impact Materialize. Labels:

component: cockroachdb
severity: warning

max(
  avg_over_time(crdb_dedicated_sys_cpu_combined_percent_normalized[2h]) * 100
) > 85

crdb-query-latency #

CockroachDB p95 SQL service latency has been above 250ms for an extended period. Labels:

component: cockroachdb
severity: warning

histogram_quantile(0.95,
  sum by (le, node) (
    rate(crdb_dedicated_sql_service_latency_bucket[20m])
  )
) > 250 * 1000 * 1000

crdb-lsm-read-amplification-critical #

CockroachDB LSM read amplification is critically high (>150), indicating severe I/O overload. Labels:

component: cockroachdb
severity: warning

max(crdb_dedicated_rocksdb_read_amplification) > 150

crdb-lsm-read-amplification-high #

CockroachDB LSM read amplification is elevated (>50), a sign writes may be outpacing compaction. Labels:

component: cockroachdb
severity: notice

max(crdb_dedicated_rocksdb_read_amplification) > 50

crdb-ranges-unavailable #

CockroachDB has unavailable ranges, which may indicate node failures or replication issues. Labels:

component: cockroachdb
severity: warning

max(crdb_dedicated_ranges_unavailable) > 0

crdb-ranges-underreplicated #

CockroachDB has under-replicated ranges, which may indicate node failures or replication issues. Labels:

component: cockroachdb
severity: warning

max(crdb_dedicated_ranges_underreplicated) > 0

crdb-sql-memory-rapid-growth #

CockroachDB SQL memory is growing faster than 8 MB/s, a sign of a runaway query heading toward OOM. Labels:

component: cockroachdb
severity: warning

max by (node_id) (
  deriv(crdb_dedicated_sql_mem_distsql_current[5m])
) > 8 * 1000 * 1000

crdb-sql-memory-pressure-high #

CockroachDB distsql memory exceeds 18% of node RAM, indicating high user-query memory pressure. Labels:

component: cockroachdb
severity: notice

max by (node_id) (
  crdb_dedicated_sql_mem_distsql_current
)
/ on (node_id) max by (node_id) (
  crdb_dedicated_sys_totalmem
)
> 0.18

crdb-write-intent-accumulation-critical #

CockroachDB write-intent count has exceeded 10M for 10m, indicating large transactions holding locks. Labels:

component: cockroachdb
severity: warning

max(crdb_dedicated_intentcount) > 10 * 1000 * 1000

crdb-write-intent-accumulation-high #

CockroachDB write-intent count has exceeded 10M, a sign of large transactions accumulating locks. Labels:

component: cockroachdb
severity: notice

max(crdb_dedicated_intentcount) > 10 * 1000 * 1000

crdb-backup-missing #

A CockroachDB backup may be missing — the last completed backup is more than 80 minutes old. Labels:

component: cockroachdb
severity: notice

(
  time() - max(max_over_time(crdb_dedicated_schedules_backup_last_completed_time[60m]))
) / 60 > 80

cilium-bpf-map-pressure #

A Cilium BPF map is filling up, which will cause networking issues if left unaddressed. Labels:

component: cilium
severity: warning

max by (map_name, instance) (
  cilium_bpf_map_pressure
) > 0.7

cilium-drop-rate-elevated #

Cilium is dropping more packets than expected, which can indicate networking issues. Labels:

component: cilium
severity: notice

max by (direction, reason, pod) (
  rate(cilium_drop_count_total[5m:1m])
) > 10

coredns-slow-queries #

CoreDNS p99 request latency is above 500ms, which can cause or accompany broader outages. Labels:

component: coredns
severity: notice

histogram_quantile(0.99,
  sum by (le, service) (
    rate(coredns_dns_request_duration_seconds_bucket{zone="."}[5m])
  )
) > 0.5

node-unreachable #

A Kubernetes node is marked unreachable, which may indicate a node or network problem. Labels:

component: kubernetes
severity: warning

kube_node_spec_taint{key="node.kubernetes.io/unreachable"}

k8s-horizontalpodautoscaler-replicas-critical #

A HorizontalPodAutoscaler is above 90% of its maximum replicas — nearly out of headroom to scale. Labels:

component: kubernetes
severity: critical

(
  kube_horizontalpodautoscaler_status_current_replicas{namespace!~"alloy|loki"}
  / kube_horizontalpodautoscaler_spec_max_replicas{namespace!~"alloy|loki"}
) > 0.9

k8s-horizontalpodautoscaler-replicas-high #

A HorizontalPodAutoscaler is above 70% of its maximum replicas. Labels:

component: kubernetes
severity: warning

(
  kube_horizontalpodautoscaler_status_current_replicas
  / kube_horizontalpodautoscaler_spec_max_replicas
) > 0.7

k8s-container-disk-usage #

A container filesystem is above 70% usage and should be cleaned up or resized. Labels:

component: kubernetes
severity: warning

100 * (
  max by (kubernetes_io_hostname) (container_fs_usage_bytes)
  / max by (kubernetes_io_hostname) (container_fs_limit_bytes)
) > 70

k8s-node-disk-pressure #

A Kubernetes node is under disk pressure, which can cause pods to fail scheduling or be evicted. Labels:

component: kubernetes
severity: warning

max by (node) (
  kube_node_status_condition{condition="DiskPressure", status="true"}
) > 0

k8s-volume-usage #

A Kubernetes persistent volume is above 70% usage and should be cleaned up or resized. Labels:

component: kubernetes
severity: warning

100 * (
  max by (namespace, persistentvolumeclaim) (
    kubelet_volume_stats_used_bytes{persistentvolumeclaim!~".*cluster-s2-.*|.*cluster-u.*"}
  )
  / max by (namespace, persistentvolumeclaim) (
    kubelet_volume_stats_capacity_bytes{persistentvolumeclaim!~".*cluster-s2-.*|.*cluster-u.*"}
  )
) > 70

alloy-log-drops #

Alloy is dropping log entries — logs are being lost right now. Labels:

component: loki
severity: warning

sum by (namespace, job, reason) (
  rate(loki_write_dropped_entries_total[2m])
) > 0

loki-push-err-high #

Loki’s push endpoint is returning 10%+ write errors, so logs are being rejected. Labels:

component: loki
severity: warning

100 * (
  sum by (namespace, app_kubernetes_io_component) (
    rate(loki_request_duration_seconds_count{status_code=~"4.*|5.*", route=~".*push.*"}[2m])
  )
  / sum by (namespace, app_kubernetes_io_component) (
    rate(loki_request_duration_seconds_count{route=~".*push.*"}[2m])
  )
) > 10

loki-panics #

A Loki component has panicked. Labels:

component: loki
severity: notice

sum by (namespace, job, app_kubernetes_io_component) (
  increase(loki_panic_total[10m])
) > 0

loki-req-duration-high #

Loki p95 request duration is above 1s (excluding tail/long-poll routes). Labels:

component: loki
severity: notice

histogram_quantile(0.95,
  sum by (le, app_kubernetes_io_component) (
    rate(loki_request_duration_seconds_bucket{route!~"(?i).*tail.*|/schedulerpb.SchedulerForQuerier/QuerierLoop"}[5m])
  )
) > 1

loki-req-err-high #

Loki read requests are returning 10%+ 5xx errors. Labels:

component: loki
severity: notice

100 * (
  sum by (namespace, app_kubernetes_io_component, route) (
    rate(loki_request_duration_seconds_count{status_code=~"5.*"}[2m])
  )
  / sum by (namespace, app_kubernetes_io_component, route) (
    rate(loki_request_duration_seconds_count[2m])
  )
) > 10

loki-writer-err-high #

Loki write requests are erroring for 10%+ of requests. Labels:

component: loki
severity: notice

100 * (
  sum by (namespace, app_kubernetes_io_component, route) (
    rate(loki_request_duration_seconds_count{status_code="error"}[2m])
  )
  / sum by (namespace, app_kubernetes_io_component, route) (
    rate(loki_request_duration_seconds_count[2m])
  )
) > 10

clusterd-metrics-missing #

All metrics for a Materialize pod have been missing for 60m — the scrape target is likely down. Labels:

component: monitoring
severity: warning

max by (pod, namespace) (
  up{job=~"materialize", cluster_environmentd_materialize_cloud_cluster_id!="s5"}
) == 0

critical-metrics-missing #

More than 30% of a critical scrape job’s targets have been down for 30m. Labels:

component: monitoring
severity: warning

(
  count by (job, app) (
    up{job=~"cilium|crdb|environment_controller|kubernetes-nodes.*|kubernetes-pods|lvm-exporter|node-exporter|new-promsql-exporter|region_controller"} == 0
  )
  / count by (job, app) (
    up{job=~"cilium|crdb|environment_controller|kubernetes-nodes.*|kubernetes-pods|lvm-exporter|node-exporter|new-promsql-exporter|region_controller"}
  )
) * 100 > 30

logging-collection-down #

Loki has received no logs for 15m — log collection is down. Labels:

component: monitoring
severity: warning

sum(rate(loki_distributor_bytes_received_total{namespace="loki"}[5m])) == 0

k8s-deployment-unavailable-critical #

A core control-plane deployment is unavailable. Labels:

component: kubernetes
severity: critical

group by (deployment, namespace) (
  kube_deployment_status_condition{condition=~"Available", status=~"true", deployment=~"admission-webhook|aws-load-balancer-controller|balancer.*|cilium-agent|cilium-operator|controller|coredns|csi-attacher|csi-node-driver-registrar|csi-provisioner|csi-resizer|csi-snapshotter|daemonset-taint-remover|environment-controller|external-dns|.*internal-api.*|.*region-api.*|.*sync-server|karpenter|node-cache|node-driver-registrar|openebs-lvm.*|region-controller|.*scheduler|snapshot-controller", ${excludeEnvironmentFilter}} == 0
)

k8s-deployment-unavailable-warning #

An important supporting deployment is unavailable. Labels:

component: kubernetes
severity: warning

group by (deployment, namespace) (
  kube_deployment_status_condition{condition=~"Available", status=~"true", deployment=~"cert-manager.*|ebs-plugin|ebs-csi-controller|eip-operator|hubble-relay|kube-state-metrics|liveness-probe|metrics-server|new-promsql-exporter|node-driver-registrar|node-exporter|prometheus-adapter|sidecar-aws-sigv4-proxy", ${excludeEnvironmentFilter}} == 0
)

k8s-deployment-unavailable-notice #

A non-essential deployment is unavailable. Labels:

component: kubernetes
severity: notice

group by (deployment, namespace) (
  kube_deployment_status_condition{condition=~"Available", status=~"true", deployment=~"backend|egress-check.*|egress-noop.*|exporter|frontend|hubble-ui|loki.*|memcached|metrics-proxy|nginx|overprovisioning-placeholder|pause|polarsignals-scraper|tbot|teleport|cert-manager-csi-driver|lvm-exporter|parca-agent|vector", ${excludeEnvironmentFilter}} == 0
)

k8s-daemonset-saturating-cpu #

Daemonsets are requesting nearly all the CPU reserved for them, squeezing clusterd headroom. Labels:

component: kubernetes
severity: warning

(
  1.77 - sum(
    max by (container) (
      kube_pod_container_resource_requests{container=~"cilium-agent|csi-node-driver-registrar|ebs-plugin|eip-operator|lvm-exporter|node-driver-registrar|node-exporter|openebs-lvm-plugin|parca-agent|vector", unit="core", resource="cpu"}
    )
  )
) < 0.01

k8s-daemonset-saturating-mem #

Daemonsets are requesting nearly all the memory reserved for them, squeezing clusterd headroom. Labels:

component: kubernetes
severity: warning

(
  10730942464 - sum(
    max by (container) (
      kube_pod_container_resource_requests{container=~"cilium-agent|csi-node-driver-registrar|ebs-plugin|eip-operator|lvm-exporter|node-driver-registrar|node-exporter|openebs-lvm-plugin|parca-agent|vector", unit="byte", resource="memory"}
    )
  )
) < 5243000

k8s-daemonset-high-cpu #

Daemonsets are approaching the CPU budget reserved for them. Labels:

component: kubernetes
severity: notice

(
  1.77 - sum(
    max by (container) (
      kube_pod_container_resource_requests{container=~"cilium-agent|csi-node-driver-registrar|ebs-plugin|eip-operator|lvm-exporter|node-driver-registrar|node-exporter|openebs-lvm-plugin|parca-agent|vector", unit="core", resource="cpu"}
    )
  )
) < 0.1

container-file-descriptors-critical #

A core container’s open file descriptors are above 70% of its limit and it may be killed. Labels:

component: kubernetes
severity: critical

100 * (
  sum by (pod, namespace, container) (
    container_file_descriptors{container=~"admission-webhook|aws-load-balancer-controller|balancer.*|cilium-agent|cilium-operator|controller|coredns|csi-attacher|csi-node-driver-registrar|csi-provisioner|csi-resizer|csi-snapshotter|daemonset-taint-remover|environment-controller|external-dns|.*internal-api.*|.*region-api.*|.*sync-server|karpenter|node-cache|node-driver-registrar|openebs-lvm.*|region-controller|.*scheduler|snapshot-controller"}
  )
  / min by (pod, namespace, container) (
    container_ulimits_soft{ulimit="max_open_files"}
  )
) > 70

container-file-descriptors-warning #

A non-core container’s open file descriptors are above 70% of its limit and it may be killed. Labels:

component: kubernetes
severity: warning

100 * (
  sum by (pod, namespace, container) (
    container_file_descriptors{container!~"admission-webhook|aws-load-balancer-controller|balancer.*|cilium-agent|cilium-operator|controller|coredns|csi-attacher|csi-node-driver-registrar|csi-provisioner|csi-resizer|csi-snapshotter|daemonset-taint-remover|environment-controller|external-dns|.*internal-api.*|.*region-api.*|.*sync-server|karpenter|node-cache|node-driver-registrar|openebs-lvm.*|region-controller|.*scheduler|snapshot-controller"}
  )
  / min by (pod, namespace, container) (
    container_ulimits_soft{ulimit="max_open_files"}
  )
) > 70

container-file-descriptors-elevated #

A container’s open file descriptors are above 20% of its limit. Labels:

component: kubernetes
severity: notice

100 * (
  sum by (pod, namespace, container) (container_file_descriptors)
  / min by (pod, namespace, container) (
    container_ulimits_soft{ulimit="max_open_files"}
  )
) > 20

infra-pod-high-cpu-ratio #

A vector pod has used more than its full CPU request for 6h and may be throttled. Labels:

component: kubernetes
severity: notice

(
  sum by (pod, namespace, container) (
    rate(container_cpu_usage_seconds_total{container!="", pod=~"vector-.*"}[5m])
  )
  / max by (pod, namespace, container) (
    kube_pod_container_resource_requests{resource="cpu", container!="", pod=~"vector-.*"}
  )
) > 1

infra-pods-high-cpu-ratio #

An infra pod is using more than its full CPU request and may be throttled. Labels:

component: kubernetes
severity: notice

(
  sum by (pod, namespace, container) (
    rate(container_cpu_usage_seconds_total{namespace!~"environment.*", container!="", pod!~"parca-agent-.*|vector-.*|cilium-egress-.*"}[5m])
  )
  / max by (pod, namespace, container) (
    kube_pod_container_resource_requests{namespace!~"environment.*", resource="cpu", container!="", pod!~"parca-agent-.*|vector-.*|cilium-egress-.*"}
  )
) > 1

infra-memory-high #

An important infra container is above 80% memory usage. Labels:

component: kubernetes
severity: warning

100 * (
  sum by (namespace, pod, container) (
    container_memory_working_set_bytes{container=~"admission-webhook|aws-load-balancer-controller|balancer.*|cilium-agent|cilium-operator|controller|coredns|csi-attacher|csi-node-driver-registrar|csi-provisioner|csi-resizer|csi-snapshotter|daemonset-taint-remover|environment-controller|external-dns|.*internal-api.*|.*region-api.*|.*sync-server|karpenter|node-cache|node-driver-registrar|openebs-lvm.*|region-controller|.*scheduler|snapshot-controller"}
  )
  / sum by (namespace, pod, container) (
    container_spec_memory_limit_bytes{container=~"admission-webhook|aws-load-balancer-controller|balancer.*|cilium-agent|cilium-operator|controller|coredns|csi-attacher|csi-node-driver-registrar|csi-provisioner|csi-resizer|csi-snapshotter|daemonset-taint-remover|environment-controller|external-dns|.*internal-api.*|.*region-api.*|.*sync-server|karpenter|node-cache|node-driver-registrar|openebs-lvm.*|region-controller|.*scheduler|snapshot-controller"}
  )
) > 80

infra-memory-elevated #

A supporting infra container is above 80% memory usage. Labels:

component: kubernetes
severity: notice

100 * (
  sum by (namespace, pod, container) (
    container_memory_working_set_bytes{container=~"cert-manager.*|ebs-plugin|ebs-csi-controller|eip-operator|hubble-relay|kube-state-metrics|liveness-probe|metrics-server|new-promsql-exporter|node-driver-registrar|node-exporter|prometheus-adapter|sidecar-aws-sigv4-proxy"}
  )
  / sum by (namespace, pod, container) (
    container_spec_memory_limit_bytes{container=~"cert-manager.*|ebs-plugin|ebs-csi-controller|eip-operator|hubble-relay|kube-state-metrics|liveness-probe|metrics-server|new-promsql-exporter|node-driver-registrar|node-exporter|prometheus-adapter|sidecar-aws-sigv4-proxy"}
  )
) > 80

infra-oomkill-core-systems #

A core infra container has been OOMKilled. Labels:

component: kubernetes
severity: warning

sum by (container, namespace, pod) (
  kube_pod_container_status_restarts_total{container=~"admission-webhook|aws-load-balancer-controller|balancer.*|cilium-agent|cilium-operator|controller|coredns|csi-attacher|csi-node-driver-registrar|csi-provisioner|csi-resizer|csi-snapshotter|daemonset-taint-remover|environment-controller|external-dns|.*internal-api.*|.*region-api.*|.*sync-server|karpenter|node-cache|node-driver-registrar|openebs-lvm.*|region-controller|.*scheduler|snapshot-controller"}
  - kube_pod_container_status_restarts_total{container=~"admission-webhook|aws-load-balancer-controller|balancer.*|cilium-agent|cilium-operator|controller|coredns|csi-attacher|csi-node-driver-registrar|csi-provisioner|csi-resizer|csi-snapshotter|daemonset-taint-remover|environment-controller|external-dns|.*internal-api.*|.*region-api.*|.*sync-server|karpenter|node-cache|node-driver-registrar|openebs-lvm.*|region-controller|.*scheduler|snapshot-controller"} offset 3h > 1
  and ignoring(reason)
  kube_pod_container_status_last_terminated_reason{reason="OOMKilled", container=~"admission-webhook|aws-load-balancer-controller|balancer.*|cilium-agent|cilium-operator|controller|coredns|csi-attacher|csi-node-driver-registrar|csi-provisioner|csi-resizer|csi-snapshotter|daemonset-taint-remover|environment-controller|external-dns|.*internal-api.*|.*region-api.*|.*sync-server|karpenter|node-cache|node-driver-registrar|openebs-lvm.*|region-controller|.*scheduler|snapshot-controller"}
) > 0

infra-oomkill-important-systems #

An important infra container has been OOMKilled. Labels:

component: kubernetes
severity: warning

sum by (container, namespace, pod) (
  kube_pod_container_status_restarts_total{container=~"cert-manager.*|ebs-plugin|ebs-csi-controller|eip-operator|hubble-relay|kube-state-metrics|liveness-probe|metrics-server|new-promsql-exporter|node-driver-registrar|node-exporter|prometheus-adapter|sidecar-aws-sigv4-proxy"}
  - kube_pod_container_status_restarts_total{container=~"cert-manager.*|ebs-plugin|ebs-csi-controller|eip-operator|hubble-relay|kube-state-metrics|liveness-probe|metrics-server|new-promsql-exporter|node-driver-registrar|node-exporter|prometheus-adapter|sidecar-aws-sigv4-proxy"} offset 3h > 1
  and ignoring(reason)
  kube_pod_container_status_last_terminated_reason{reason="OOMKilled", container=~"cert-manager.*|ebs-plugin|ebs-csi-controller|eip-operator|hubble-relay|kube-state-metrics|liveness-probe|metrics-server|new-promsql-exporter|node-driver-registrar|node-exporter|prometheus-adapter|sidecar-aws-sigv4-proxy"}
) > 0

infra-oomkill-nonessential-systems #

A non-essential infra container has been OOMKilled. Labels:

component: kubernetes
severity: notice

sum by (container, namespace, pod) (
  kube_pod_container_status_restarts_total{container=~"awslimitchecker|backend|egress-check.*|egress-noop.*|exporter|external-uptime-checker|frontend|hubble-ui|loki.*|memcached|metrics-proxy|nginx|overprovisioning-placeholder|pause|polarsignals-scraper|tbot|teleport|cert-manager-csi-driver|lvm-exporter|parca-agent|vector"}
  - kube_pod_container_status_restarts_total{container=~"awslimitchecker|backend|egress-check.*|egress-noop.*|exporter|external-uptime-checker|frontend|hubble-ui|loki.*|memcached|metrics-proxy|nginx|overprovisioning-placeholder|pause|polarsignals-scraper|tbot|teleport|cert-manager-csi-driver|lvm-exporter|parca-agent|vector"} offset 3h > 1
  and ignoring(reason)
  kube_pod_container_status_last_terminated_reason{reason="OOMKilled", container=~"awslimitchecker|backend|egress-check.*|egress-noop.*|exporter|external-uptime-checker|frontend|hubble-ui|loki.*|memcached|metrics-proxy|nginx|overprovisioning-placeholder|pause|polarsignals-scraper|tbot|teleport|cert-manager-csi-driver|lvm-exporter|parca-agent|vector"}
) > 0

k8s-infra-pod-pending-too-long #

An infra pod has been Pending for 15m — the cluster may be unhealthy or out of capacity. Labels:

component: kubernetes
severity: warning

max by (namespace, pod) (
  kube_pod_status_phase{namespace!~"environment.*", phase="Pending"}
) > 0

pod-restart-rate-high #

An important infra container is restarting frequently, which may indicate a crash loop. Labels:

component: kubernetes
severity: warning

avg by (container, namespace) (
  rate(kube_pod_container_status_restarts_total{namespace!~"environment.*", container!~"awslimitchecker|backend|egress-check.*|egress-noop.*|exporter|external-uptime-checker|frontend|hubble-ui|loki.*|memcached|metrics-proxy|nginx|overprovisioning-placeholder|pause|polarsignals-scraper|tbot|teleport|cert-manager-csi-driver|lvm-exporter|parca-agent|vector"}[10m])
) * 100 > 0

pod-restart-rate-high-nonessential #

A non-essential infra container is restarting frequently, which may indicate a crash loop. Labels:

component: kubernetes
severity: notice

avg by (container, namespace) (
  rate(kube_pod_container_status_restarts_total{container=~"awslimitchecker|backend|egress-check.*|egress-noop.*|exporter|external-uptime-checker|frontend|hubble-ui|loki.*|memcached|metrics-proxy|nginx|overprovisioning-placeholder|pause|polarsignals-scraper|tbot|teleport|cert-manager-csi-driver|lvm-exporter|parca-agent|vector"}[10m])
) * 100 > 0

pods-stuck-in-waiting #

A pod has been stuck in Waiting for over 10m, which can indicate a scheduling or resource problem. Labels:

component: kubernetes
severity: notice

sum by (namespace, pod) (
  kube_pod_container_status_waiting == 1
  and time() - kube_pod_start_time > 10 * 60
) > 0

egress-traffic-missing-metrics #

Egress-gateway node throughput metrics are missing, which may indicate an egress-gateway problem. Labels:

component: egress-gateway
severity: warning

absent(
  sum by (node) (
    rate(node_network_receive_bytes_total{device="eth0"}[2m])
  )
  * on (node) group_left (workload) sum by (node, workload) (
    label_replace(container_network_receive_bytes_total{workload="materialize-egress"}, "node", "$1", "instance", "(.+)")
  ) ^ 0
)

excessive-egress-traffic #

An egress-gateway node has very high traffic, which may indicate traffic is not routing through the internet gateway. Labels:

component: egress-gateway
severity: warning

sum by (node) (
  rate(node_network_receive_bytes_total{device="eth0"}[2m])
)
* on (node) group_left (workload) sum by (node, workload) (
  label_replace(container_network_receive_bytes_total{workload="materialize-egress"}, "node", "$1", "instance", "(.+)")
) ^ 0
> 5.5 * 1000 * 1000 * 1000 / 8

high-egress-traffic #

An egress-gateway node is above 90% of its allowed traffic and may soon exceed capacity. Labels:

component: egress-gateway
severity: notice

sum by (node) (
  rate(node_network_receive_bytes_total{device="eth0"}[2m])
)
* on (node) group_left (workload) sum by (node, workload) (
  label_replace(container_network_receive_bytes_total{workload="materialize-egress"}, "node", "$1", "instance", "(.+)")
) ^ 0
> 4.5 * 1000 * 1000 * 1000 / 8

low-egress-traffic #

An egress-gateway node has unusually low traffic, which may indicate the gateway is unhealthy. Labels:

component: egress-gateway
severity: warning

sum by (node) (
  rate(node_network_receive_bytes_total{device="eth0"}[2m])
)
* on (node) group_left (workload) sum by (node, workload) (
  label_replace(container_network_receive_bytes_total{workload="materialize-egress"}, "node", "$1", "instance", "(.+)")
) ^ 0
< 100

no-egress-traffic #

An egress-gateway node has had no traffic for 5m, which may indicate external traffic is blocked. Labels:

component: egress-gateway
severity: warning

(
  min by (node) (
    rate(node_network_receive_bytes_total{device="eth0"}[5m:1m])
    or rate(node_network_transmit_bytes_total{device="eth0"}[5m:1m])
  )
  and on (node) kubelet_node_name{workload="materialize-egress"}
) == 0

materialize-alerts#

Alerting rules for Materialize.

Each alert carries its firing expression as an inline query, is graded by a severity label (critical > warning > notice), and is tagged with a component label. Per-environment alerts group abstractly and attach the environment name via the mzEnvironmentName template function; the optional %%{excludeEnvironmentFilter} fragment lets a deployment exclude environments (e.g. internal test environments). Alerts that only apply to Materialize Cloud carry deploymentMode: cloud-only.

env-uptime-sla #

environmentd is not accepting basic connections and may be unreachable. Labels:

component: environmentd
severity: critical

avg by (namespace, pod) (
  v2_mz_can_connect{${excludeEnvironmentFilter}}
) <= 0.1
and on (namespace, pod) (
  time() - kube_pod_start_time > 5 * 60
)

env-uptime-slo #

environmentd is not accepting basic connections and may be unreachable. Labels:

component: environmentd
severity: warning

avg by (namespace, pod) (
  v2_mz_can_connect{${excludeEnvironmentFilter}}
) <= 0.1
and on (namespace, pod) (
  time() - kube_pod_start_time > 5 * 60
)

envd-simplest-query #

environmentd is not responding to a SELECT 1 and may be unhealthy. Labels:

component: environmentd
severity: critical

avg by (namespace) (
  v2_mz_envd_up{${excludeEnvironmentFilter}}
) <= 0.1

env-query-views-critical #

environmentd is not answering a simple query that reads from object storage. Labels:

component: environmentd
severity: critical

avg by (namespace) (
  v2_mz_views_query_successful{${excludeEnvironmentFilter}}
) <= 0.1

env-query-views-warning #

environmentd is not answering a simple query that reads from object storage. Labels:

component: environmentd
severity: warning

avg by (namespace) (
  v2_mz_views_query_successful{${excludeEnvironmentFilter}}
) <= 0.1

clusterd-not-receiving-commands #

A clusterd has not received commands from environmentd for 5m — the replica may be stalled or disconnected. Labels:

component: clusterd
severity: warning

max by (namespace, cluster_environmentd_materialize_cloud_cluster_id, cluster_environmentd_materialize_cloud_replica_id, pod) (
  rate(mz_cluster_server_last_command_received{server_name="compute", pod=~".*0", ${excludeEnvironmentFilter}}[5m:1m])
) == 0
and on (pod) (
  time() - kube_pod_start_time > 5 * 60
)

storage-collection-finalization-stuck #

Storage shards have been stuck finalizing for 60m. Labels:

component: storage
severity: notice

max by (namespace) (
  mz_shard_finalization_outstanding
) > 0

auth-errors #

An elevated rate of unexpected authentication errors. Labels:

component: auth
severity: warning

avg by (namespace, status) (
  rate(mz_auth_request_count{status!~"(2|401).*"}[30m])
) > 0.01

auth-refresh-failures #

An elevated rate of failed auth-token refreshes. Labels:

component: auth
severity: warning

avg(
  rate(mz_auth_request_count{status=~"401.*", path="refresh_token"}[15m])
) > 0.002

console-errors #

The web console has returned an error for 2% or more of commands for 30m, across more than one environment. Labels:

component: console
severity: warning

(
  sum by (application_name) (
    rate(mz_adapter_commands{application_name="web_console", status="error", ${excludeEnvironmentFilter}}[5m])
  )
  / sum by (application_name) (
    rate(mz_adapter_commands{application_name="web_console", ${excludeEnvironmentFilter}}[5m])
  )
) * 100 > 2
and
count by (application_name) (
  (
    sum by (application_name, namespace) (
      rate(mz_adapter_commands{application_name="web_console", status="error", ${excludeEnvironmentFilter}}[5m])
    )
    / sum by (application_name, namespace) (
      rate(mz_adapter_commands{application_name="web_console", ${excludeEnvironmentFilter}}[5m])
    )
  ) * 100 > 2
) > 1

persist-failures #

Failures in Persist that should be rare are happening frequently. Labels:

component: persist
severity: notice

sum by (metric) (rate(label_replace(mz_persist_blob_failures, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_consensus_failures, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_state_update_state_slow_path, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_lease_timeout_read, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 3
or sum by (metric) (rate(label_replace(mz_persist_compaction_noop, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 3
or sum by (metric) (rate(label_replace(mz_persist_compaction_failed, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_external_blob_delete_noop_count, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_external_failed_count, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_cmd_failed_count, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_compaction_dropped, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_pushdown_parts_mismatched_stats_count, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_columnar_validation_count{result="invalid"}, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_txn_placeholder_schema_apply, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_columnar_op_count{op="validation", result="invalid"}, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_schema_cache_fetch_state_count, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1
or sum by (metric) (rate(label_replace(mz_persist_shard_unconsolidated_snapshot, "metric", "$1", "__name__", "(.*)")[1m:15s])) > 1

envd-terminated #

An environmentd was unexpectedly terminated — this should not happen. Labels:

component: environmentd
severity: warning

kube_pod_container_status_last_terminated_exitcode{container="environmentd"} != 166

environmentd-high-cpu #

An environmentd has been above 80% CPU usage for 90m. Labels:

component: environmentd
severity: warning

100 * sum by (namespace, pod) (
  rate(container_cpu_usage_seconds_total{pod=~".*environmentd.+", container!=""}[30m])
)
/ sum by (namespace, pod) (
  container_spec_cpu_quota{pod=~".*environmentd.+", container!=""}
  / container_spec_cpu_period{pod=~".*environmentd.+", container!=""}
) > 80

environmentd-high-memory #

An environmentd has been above 80% memory usage for 30m. Labels:

component: environmentd
severity: warning

100 * (
  sum by (namespace, pod, container) (
    container_memory_working_set_bytes{pod=~".*environmentd.+", container!=""}
  )
  / sum by (namespace, pod, container) (
    container_spec_memory_limit_bytes{pod=~".*environmentd.+", container!=""} != 0
  )
) > 80

environmentd-cpu-throttled #

An environmentd container is being CPU throttled more than 50% of the time. Labels:

component: environmentd
severity: notice

100 * avg by (instance, container, namespace) (
  rate(container_cpu_cfs_throttled_periods_total{container=~"environmentd"}[30m])
  / rate(container_cpu_cfs_periods_total{container=~"environmentd"}[30m])
) > 50

environmentd-memory-elevated #

An environmentd is above 80% memory usage (lower-severity companion to environmentd-high-memory). Labels:

component: environmentd
severity: notice

100 * (
  sum by (namespace, pod, container) (
    container_memory_working_set_bytes{container!="POD", container!="", container=~"environmentd"}
  )
  / sum by (namespace, pod, container) (
    container_spec_memory_limit_bytes{container!="POD", container!="", container=~"environmentd"}
  )
) > 80

clusterd-error-kill #

A clusterd was terminated with an unexpected exit code — this should not happen. Labels:

component: clusterd
severity: warning

max by (namespace, pod) (
  increase(kube_pod_container_status_restarts_total{namespace=~"environment.+", container="clusterd"}[1h])
) > 0
and max by (namespace, pod) (kube_pod_container_status_last_terminated_exitcode) != 137
and max by (namespace, pod) (kube_pod_container_status_last_terminated_exitcode) != 135
and max by (namespace, pod) (kube_pod_container_status_last_terminated_exitcode) != 166
and max by (namespace, pod) (kube_pod_container_status_last_terminated_exitcode) != 167

system-cluster-terminated #

A system cluster was unexpectedly terminated — this should not happen. Labels:

component: clusterd
severity: warning

kube_pod_container_status_last_terminated_exitcode{pod=~".*cluster-s.*"} != 166

system-cluster-high-memory #

A system cluster is above 80% memory usage — this should not happen for system clusters. Labels:

component: clusterd
severity: warning

100 * (
  sum by (namespace, pod) (
    mz_memory_limiter_memory_usage_bytes{pod=~".*cluster-s.+"}
  )
  / sum by (namespace, pod) (
    mz_memory_limiter_memory_limit_bytes{pod=~".*cluster-s.+"} != 0
  )
) > 80

clusterd-expiration-7d #

A cluster replica will expire in less than a week. Labels:

component: clusterd
severity: warning

mz_dataflow_replica_expiration_remaining_seconds{pod=~".*cluster.+", ${excludeEnvironmentFilter}} > 0
< 60 * 60 * 24 * 7

swap-cluster-oom #

A swap-enabled cluster was OOMKilled while below 80% swap usage. Labels:

component: clusterd
severity: notice

increase(
  (
    max by (namespace, pod) (
      kube_pod_container_status_last_terminated_reason{reason="OOMKilled", namespace=~"environment.*"}
    )
    or on (namespace, pod) count by (namespace, pod) (
      container_last_seen{materialize_cloud_swap="true", ${excludeEnvironmentFilter}}
    ) * 0
  )[10m:1m]
) > 0
and on (namespace, pod) (
  max_over_time(
    (
      max by (namespace, pod) (container_memory_swap)
      / max by (namespace, pod) (container_spec_memory_swap_limit_bytes)
    )[10m:1m]
  ) < 0.8
)
and on (namespace, pod) (
  increase(
    max by (namespace, pod) (kube_pod_container_status_restarts_total)[10m:1m]
  ) > 0
)

environment-pod-pending-critical #

An environment pod has been Pending for 15m — the cluster may be unhealthy or out of capacity. Labels:

component: environmentd
severity: critical

min by (namespace, pod_base) (
  label_replace(
    kube_pod_status_phase{namespace=~"environment.*", phase="Pending", ${excludeEnvironmentFilter}},
    "pod_base", "$1", "pod", "(.*?)(?:-gen-[0-9]+-[0-9]+|([0-9]+)-[0-9]+)?"
  )
) > 0

environment-pod-pending #

An environment pod has been Pending for 15m — the cluster may be unhealthy or out of capacity. Labels:

component: environmentd
severity: warning

max by (namespace, pod) (
  kube_pod_status_phase{namespace=~"environment.*", phase="Pending", ${excludeEnvironmentFilter}}
) > 0

certificate-not-ready #

A certificate has not become ready in 20m, which can prevent an environment from coming up. Labels:

component: environmentd
severity: notice

max by (name, namespace, condition) (
  certmanager_certificate_ready_status{condition!="True", ${excludeEnvironmentFilter}} == 1
)

console-query-latency #

Web-console query p95 latency has exceeded 10s for 15m. Labels:

component: console
severity: warning

sum by (namespace) (
  histogram_quantile(0.95,
    sum by (le, namespace) (
      rate(mz_time_to_first_row_seconds_bucket{instance_id=~"s2", application_name="web_console", ${excludeEnvironmentFilter}}[2m])
    )
  )
) > 10

new-clusterd-restarts #

A previously healthy clusterd is restarting during a release. Labels:

component: clusterd
severity: warning

label_replace(
  max by (namespace, pod) (kube_pod_container_status_restarts_total{container="clusterd"}) > 0
  and on (namespace, pod) (
    min by (namespace, pod) (time() - kube_pod_created)
  ) < 60 * 60 * 24,
  "pod_base", "$1", "pod", ".*-(cluster-.*)-gen-([0-9]+)-[0-9]+$"
)
and on (namespace, pod_base) label_replace(
  max by (namespace, pod) (increase(kube_pod_container_status_restarts_total{container="clusterd"}[12h:1m])) == 0
  and on (namespace, pod) (
    min by (namespace, pod) (time() - kube_pod_created)
  ) > 60 * 60 * 24,
  "pod_base", "$1", "pod", ".*-(cluster-.*)-gen-([0-9]+)-[0-9]+$"
)

external-env-uptime #

environmentd is unreachable from outside the network by the external uptime checker. Labels:

component: external-uptime
deploymentMode: cloud-only
severity: critical

avg by (namespace) (
  mz_external_envd_up
) < 1

external-env-uptime-failed #

New external connections to environmentd are failing, per the external uptime checker. Labels:

component: external-uptime
deploymentMode: cloud-only
severity: warning

sum by (connection_type, namespace) (
  rate(mz_external_calls_count{status="failed"}[2m])
) > 0

external-uptime-checker-not-calling #

The external uptime checker has stopped making calls — the checker itself may be down. Labels:

component: external-uptime
deploymentMode: cloud-only
severity: warning

sum by (connection_type, namespace) (
  increase(mz_external_calls_count{status="attempted"}[2m])
) == 0

launchdarkly-stale-sse #

The last LaunchDarkly server-side event is more than 40 minutes old — flag updates may not be reaching environmentd. Labels:

component: launchdarkly
deploymentMode: cloud-only
severity: warning

sum by (namespace) (
  timestamp(mz_parameter_frontend_last_sse_time_seconds{${excludeEnvironmentFilter}})
  - mz_parameter_frontend_last_sse_time_seconds{${excludeEnvironmentFilter}}
) >= 40 * 60

launchdarkly-stale-cse #

The last LaunchDarkly client-side event is more than 40 minutes old — client analytics may be stale. Labels:

component: launchdarkly
deploymentMode: cloud-only
severity: notice

sum by (namespace) (
  timestamp(mz_parameter_frontend_last_cse_time_seconds{${excludeEnvironmentFilter}})
  - mz_parameter_frontend_last_cse_time_seconds{${excludeEnvironmentFilter}}
) >= 40 * 60