{ "description": "Overview of a Materialize environment, ported from the Grafana dashboard of the same name.", "layout_type": "ordered", "reflow_type": "auto", "tabs": [ { "id": "7ec89514-ad10-431b-9b76-f9010ec73e4c", "name": "Summary", "widget_ids": [ 100, 101, 102 ] }, { "id": "25683934-ccbb-4720-8229-323cadb3b117", "name": "Kubernetes Workloads", "widget_ids": [ 201, 202, 203, 204 ] }, { "id": "5dcbe88a-d3f2-4540-abc8-d9e0a9eee795", "name": "Connections / Activity", "widget_ids": [ 301, 302, 303 ] }, { "id": "863c1f92-b7b6-4a16-b3c1-4f54ac3465b3", "name": "Cluster Objects / Replicas", "widget_ids": [ 401, 402, 403 ] }, { "id": "86fdb3c2-1576-4559-b05b-dca90a44cd40", "name": "Compute Objects", "widget_ids": [ 501, 502, 503, 504, 505 ] }, { "id": "2f6125dd-a5b2-4540-bf86-f9615f8dd4a9", "name": "Sources and Sinks", "widget_ids": [ 601, 602, 603, 604, 605 ] } ], "template_variables": [ { "defaults": [ "*" ], "name": "environment", "prefix": "materialize_cloud_organization_name" }, { "defaults": [ "*" ], "name": "namespace", "prefix": "namespace" }, { "defaults": [ "*" ], "name": "cluster", "prefix": "cluster_id" }, { "defaults": [ "*" ], "name": "replica", "prefix": "replica_id" } ], "title": "Materialize Environment Overview", "widgets": [ { "definition": { "background_color": "yellow", "content": "**Pick one environment, and the namespace it runs in.** Environments reuse cluster and replica ids, so the panels merge them when several are selected. The Kubernetes panels read the namespace picker, because pod metrics carry no environment tag. Pick one value or `*` in each picker. The peek and Iceberg commit latency panels need percentiles enabled for their metrics in Datadog.", "type": "note" }, "id": 100 }, { "definition": { "background_color": "gray", "layout_type": "ordered", "show_title": true, "title": "Environment Health", "type": "group", "widgets": [ { "definition": { "autoscale": true, "description": "**The share of the environment's clusters currently reporting ready — the at-a-glance health headline for the whole environment.**\n\n**Nominal:** Every cluster is ready.\n\n**Degraded:** Some clusters aren't ready, but most are. Usually a rollout in flight — replicas cycling during an upgrade or resize — rather than a real outage. Find which clusters are affected before reacting; a brief dip during a known change is expected.\n\n**Unhealthy:** A large fraction of clusters are down and not recovering on their own. Check pod restart history and narrow to the specific clusters and replicas involved; this is an outage, not a rollout.\n\nThe healthy/degraded/unhealthy bands (100% / 80-100% / <80%) are set by the consuming dashboard, not by this query.", "precision": 0, "requests": [ { "formulas": [ { "conditional_formats": [ { "comparator": ">=", "palette": "white_on_green", "value": 100 }, { "comparator": ">=", "palette": "white_on_yellow", "value": 80 }, { "comparator": "<", "palette": "white_on_red", "value": 80 } ], "formula": "query1 * 100", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "percent" } } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_compute_cluster_status{$environment}" } ], "response_format": "scalar" } ], "title": "Environment Status", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**An SLO-style snapshot: how much of the selected window the environment's clusters were ready. Sustained dips are the signal that something restarted or went down while you weren't watching.**\n\n**Nominal:** At or near full availability. Watch the decimals — 99.999% is five-nines, a very different story from 100%.\n\n**Degraded:** Availability slipped below target over the window. Correlate the dip with cluster restarts and outages; widen the range to pin down when it happened, then work backwards from there.\n\nFour-nines is about the ceiling this can measure: at a 30s scrape resolution over a week, a single missed scrape already costs a nine, so don't chase phantom sub-nine dips that are really scrape granularity.", "precision": 4, "requests": [ { "formulas": [ { "conditional_formats": [ { "comparator": ">=", "palette": "white_on_green", "value": 99 }, { "comparator": ">=", "palette": "white_on_yellow", "value": 95 }, { "comparator": "<", "palette": "white_on_red", "value": 95 } ], "formula": "query1 * 100", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "percent" } } } ], "queries": [ { "aggregator": "avg", "data_source": "metrics", "name": "query1", "query": "avg:mz_compute_cluster_status{$environment} by {materialize_cloud_organization_namespace}" } ], "response_format": "scalar" } ], "title": "Environment Availability (Select Time Range)", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**Objects still rebuilding their in-memory state — a live hydration-queue proxy. After a restart, replica creation, or some DDL, a dataflow has to rebuild from persisted storage before it can serve, and until it does it produces no results.**\n\n**Nominal:** 0, with brief spikes right after a replica restart that drain back to 0 as dataflows catch up. Those spikes are healthy re-hydration, not a problem.\n\n**Unhealthy:** A count that stays elevated means something can't finish hydrating — a source whose CREATE never completed, or a wedged dataflow. Confirm what with `SELECT * FROM mz_internal.mz_hydration_statuses WHERE NOT hydrated`.\n\nStopgap, not a real signal: self-managed has no hydration-state metric, so this counts objects whose freshness reads the u64::MAX sentinel (> 1e15) as a proxy. Deprecate this query once an actual hydration-state metric exists.", "precision": 0, "requests": [ { "formulas": [ { "formula": "default_zero(count_not_null(cutoff_min(query1, 1e15)))" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "max:mz_dataflow_wallclock_lag_seconds{$environment AND instance_id:$cluster.value AND quantile:1} by {instance_id,collection_id}" } ], "response_format": "scalar" } ], "title": "Currently Hydrating", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**The worst freshness seen anywhere in the environment over the selected window — how far the most-behind object's output trailed real time. A top-level freshness pointer.**\n\n**Degraded:** Climbing toward an hour means some object is falling behind; drill into per-cluster and per-object freshness to find which, and into hydration / source health for why.\n\nNot-yet-hydrated objects are excluded (they report the sentinel and surface as \"currently hydrating\"). This is the peak over the window, so it reflects the worst moment rather than the current value.", "precision": 0, "requests": [ { "formulas": [ { "conditional_formats": [ { "comparator": ">=", "palette": "white_on_red", "value": 3600 }, { "comparator": ">=", "palette": "white_on_yellow", "value": 60 }, { "comparator": "<", "palette": "white_on_green", "value": 60 } ], "formula": "reduce(max(cutoff_max(query1, 1e9), {*}), 'max')", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "max:mz_dataflow_wallclock_lag_seconds{$environment AND instance_id:* AND quantile:1} by {namespace,pod,instance_id,replica_id,collection_id}.rollup(max)" } ], "response_format": "scalar" } ], "title": "Worst Freshness (Select Time Range)", "type": "query_value" } }, { "definition": { "description": "**Current CPU usage per container type as a fraction of its limit, averaged over the last 5 minutes — shows the worst-loaded container types.**\n\n**Nominal:** Well below 1.0.\n\n**Degraded:** Sustained near 1.0 means a container type is CPU-bound.\n\nNeeds CPU limits from kube-state-metrics; use the `.absolute` sibling where limits aren't exposed. Requires cAdvisor + kube-state-metrics.", "requests": [ { "formulas": [ { "formula": "(query1 / query2) * 100", "limit": { "count": 10, "order": "desc" }, "number_format": { "unit": { "type": "canonical_unit", "unit_name": "percent" } } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:container_cpu_usage_seconds_total{namespace:$namespace.value AND container:* AND NOT container:POD} by {namespace,container}.as_rate().rollup(sum, 60)" }, { "aggregator": "last", "data_source": "metrics", "name": "query2", "query": "sum:kube_pod_container_resource_limits{resource:cpu AND namespace:$namespace.value} by {namespace,container}" } ], "response_format": "scalar" } ], "title": "Current CPU Usage (5 min)", "type": "toplist" } }, { "definition": { "description": "**Current memory usage per container type as a fraction of its limit — shows the worst-loaded container types.**\n\n**Unhealthy:** Sustained near 1.0 is dangerous: hitting the limit triggers an OOM-kill, which on a compute replica forces a hydration cycle (in-memory state rebuilt from persisted storage, minutes-to-hours). The offending workload usually shows in the arrangement record counts.\n\nNeeds memory limits from kube-state-metrics; use the `.absolute` sibling where limits aren't exposed. Requires cAdvisor + kube-state-metrics.", "requests": [ { "formulas": [ { "formula": "(query1 / query2) * 100", "limit": { "count": 10, "order": "desc" }, "number_format": { "unit": { "type": "canonical_unit", "unit_name": "percent" } } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:container_memory_working_set_bytes{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT container:new-promsql-exporter} by {namespace,container}" }, { "aggregator": "last", "data_source": "metrics", "name": "query2", "query": "sum:container_spec_memory_limit_bytes{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT container:new-promsql-exporter} by {namespace,container}" } ], "response_format": "scalar" } ], "title": "Current Memory Usage", "type": "toplist" } } ] }, "id": 101 }, { "definition": { "background_color": "gray", "layout_type": "ordered", "show_title": true, "title": "Environment Info", "type": "group", "widgets": [ { "definition": { "description": "**The version of Materialize running in the environment. A single version is the steady state; multiple values appear briefly during a rolling upgrade.**\n\nThe presentation layer extracts the commit hash from the `mz_version` label (e.g. to link the commit). If the version isn't what you expect, check the environment's manifest in the deployment pipeline.", "requests": [ { "formulas": [ { "formula": "query1", "limit": { "count": 10, "order": "desc" } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_compute_cluster_status{$environment} by {mz_version}" } ], "response_format": "scalar" } ], "title": "Materialize Version", "type": "toplist" } }, { "definition": { "description": "**Total CPU cores configured across the environment's containers (sum of cAdvisor CPU limits), excluding the monitoring exporter — i.e. the CPU available to the actual workload.**\n\nSteps track `ALTER CLUSTER REPLICA SIZE`, replica create/drop, and pod restarts. Requires cAdvisor. For the same total *including* the monitoring exporter see `.all_containers`.", "requests": [ { "formulas": [ { "formula": "reduce(sum(query1 / query2, {container}), 'last')", "limit": { "count": 10, "order": "desc" }, "number_format": { "unit": { "type": "canonical_unit", "unit_name": "core" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:container_spec_cpu_quota{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT container:new-promsql-exporter} by {container,pod}" }, { "data_source": "metrics", "name": "query2", "query": "sum:container_spec_cpu_period{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT container:new-promsql-exporter} by {container,pod}" } ], "response_format": "scalar" } ], "title": "Total CPU Capacity", "type": "toplist" } }, { "definition": { "description": "**Total memory configured across the environment's containers (sum of cAdvisor memory limits), excluding the monitoring exporter. Memory is Materialize's dominant constraint — in-memory arrangements live in here.**\n\nSteps track replica sizing and pod restarts. Requires cAdvisor.", "requests": [ { "formulas": [ { "formula": "query1", "limit": { "count": 10, "order": "desc" }, "number_format": { "unit": { "type": "canonical_unit", "unit_name": "byte" } } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:container_spec_memory_limit_bytes{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT container:new-promsql-exporter} by {container}" } ], "response_format": "scalar" } ], "title": "Total Memory", "type": "toplist" } } ] }, "id": 102 }, { "definition": { "background_color": "blue", "layout_type": "ordered", "show_title": false, "title": "Resources Summary", "type": "group", "widgets": [ { "definition": { "description": "**Total CPU cores configured across every container in the environment, including the monitoring exporter — the Kubernetes view of what is provisioned rather than what is available to the workload.**\n\nDiffers from `materialize.kubernetes.cpu.capacity` only by counting the monitoring exporter. Use that one to read workload capacity; use this one when reconciling against what Kubernetes reports. Requires cAdvisor.", "requests": [ { "formulas": [ { "formula": "reduce(sum(query1 / query2, {container}), 'last')", "limit": { "count": 10, "order": "desc" }, "number_format": { "unit": { "type": "canonical_unit", "unit_name": "core" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:container_spec_cpu_quota{namespace:$namespace.value AND container:* AND NOT container:POD} by {container,pod}" }, { "data_source": "metrics", "name": "query2", "query": "sum:container_spec_cpu_period{namespace:$namespace.value AND container:* AND NOT container:POD} by {container,pod}" } ], "response_format": "scalar" } ], "title": "Total CPU Capacity", "type": "toplist" } }, { "definition": { "description": "**Total memory configured across every container in the environment, including the monitoring exporter — the Kubernetes view of what is provisioned rather than what is available to the workload.**\n\nDiffers from `materialize.kubernetes.memory.capacity` only by counting the monitoring exporter. Memory is the dominant constraint on Materialize: in-memory arrangements live inside this total. Requires cAdvisor.", "requests": [ { "formulas": [ { "formula": "query1", "limit": { "count": 10, "order": "desc" }, "number_format": { "unit": { "type": "canonical_unit", "unit_name": "byte" } } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:container_spec_memory_limit_bytes{namespace:$namespace.value AND container:* AND NOT container:POD} by {container}" } ], "response_format": "scalar" } ], "title": "Total Memory", "type": "toplist" } } ] }, "id": 201 }, { "definition": { "background_color": "blue", "layout_type": "ordered", "show_title": false, "title": "Workload Readiness", "type": "group", "widgets": [ { "definition": { "description": "**Pods in the Materialize namespace grouped by phase (Running, Pending, Failed, …).**\n\n**Nominal:** Nearly all Running.\n\n**Degraded:** Pending usually means Kubernetes can't schedule them (capacity, taints, AZ constraints); Failed means a container exited and won't be restarted.\n\nRequires kube-state-metrics.\n\nCanonical because the `kube_*` families it reads are asserted against a running stack by `kube_state::labels_are_honored` in the e2e suite. That check exists for one specific failure: when the scrape does not set `honorLabels`, kube-state-metrics' own `namespace` and `pod` labels are renamed `exported_*` and replaced by the exporter pod's own identity, so this query matches nothing while `up` and `scrape_samples_scraped` stay healthy. It shipped that way once and was found by eye.", "legend": { "type": "table" }, "requests": [ { "formulas": [ { "formula": "reduce(max(query1, {phase,namespace}), 'last')" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:kube_pod_status_phase{namespace:$namespace.value} by {phase,namespace,service.instance.id}" } ], "response_format": "scalar" } ], "title": "Pod Readiness", "type": "sunburst" } }, { "definition": { "description": "**StatefulSet replicas reporting Ready. environmentd and the cluster pods are StatefulSets.**\n\n**Nominal:** Matches the configured replica count.\n\n**Degraded:** A drop indicates a pod stuck in initialization or hydration.\n\nRequires kube-state-metrics.\n\nCanonical because the `kube_*` families it reads are asserted against a running stack by `kube_state::labels_are_honored` in the e2e suite. That check exists for one specific failure: when the scrape does not set `honorLabels`, kube-state-metrics' own `namespace` and `pod` labels are renamed `exported_*` and replaced by the exporter pod's own identity, so this query matches nothing while `up` and `scrape_samples_scraped` stay healthy. It shipped that way once and was found by eye.", "legend": { "type": "table" }, "requests": [ { "formulas": [ { "formula": "query1" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:kube_statefulset_status_replicas_ready{namespace:$namespace.value} by {namespace}" } ], "response_format": "scalar" } ], "title": "StatefulSet Readiness", "type": "sunburst" } }, { "definition": { "description": "**Deployment replica health — Ready vs Unavailable. Deployments back stateless services (e.g. the promsql exporter).**\n\n**Nominal:** All replicas Ready, zero Unavailable.\n\n**Degraded:** Unavailable counts indicate failed rollouts or crashing pods.\n\nRequires kube-state-metrics.\n\nCanonical because the `kube_*` families it reads are asserted against a running stack by `kube_state::labels_are_honored` in the e2e suite. That check exists for one specific failure: when the scrape does not set `honorLabels`, kube-state-metrics' own `namespace` and `pod` labels are renamed `exported_*` and replaced by the exporter pod's own identity, so this query matches nothing while `up` and `scrape_samples_scraped` stay healthy. It shipped that way once and was found by eye.", "requests": [ { "formulas": [ { "alias": "Ready", "formula": "query1", "limit": { "count": 500, "order": "desc" } }, { "alias": "Unavailable", "formula": "query2" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:kube_deployment_status_replicas_ready{namespace:$namespace.value} by {namespace}" }, { "aggregator": "last", "data_source": "metrics", "name": "query2", "query": "sum:kube_deployment_status_replicas_unavailable{namespace:$namespace.value} by {namespace}" } ], "response_format": "scalar" } ], "title": "Deployment Readiness", "type": "query_table" } } ] }, "id": 202 }, { "definition": { "background_color": "blue", "layout_type": "ordered", "show_title": true, "title": "Pod Metrics", "type": "group", "widgets": [ { "definition": { "description": "**CPU utilization per pod as a fraction of the pod's limit. Split so the cluster/replica selectors filter the cluster pods while envd/balancer/ exporter stay visible.**\n\n**Degraded:** Sustained near 1.0 for a pod means it's CPU-bound; for the Materialize-level cause see the dataflow elapsed / arrangement rates.\n\nRequires cAdvisor + kube-state-metrics.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "Cluster replicas", "formula": "(query1 / query2) * 100", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "percent" } } }, { "alias": "Other pods", "formula": "(query3 / query4) * 100", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "percent" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:container_cpu_usage_seconds_total{namespace:$namespace.value AND container:* AND NOT container:POD AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod,container}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query2", "query": "sum:kube_pod_container_resource_limits{resource:cpu AND namespace:$namespace.value AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod,container}" }, { "data_source": "metrics", "name": "query3", "query": "sum:container_cpu_usage_seconds_total{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT pod:*-cluster-*-replica-*} by {namespace,pod,container}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query4", "query": "sum:kube_pod_container_resource_limits{resource:cpu AND namespace:$namespace.value AND NOT pod:*-cluster-*-replica-*} by {namespace,pod,container}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Pod CPU Usage", "type": "timeseries" } }, { "definition": { "description": "**Memory usage per pod as a fraction of the pod's limit (working-set basis), same cluster/non-cluster split as pod CPU.**\n\n**Unhealthy:** A sustained climb toward 1.0 is dangerous — the pod gets OOM-killed, and on a compute replica that triggers a hydration cycle. If a cluster pod is the offender, the arrangement record counts show which arrangements hold the memory.\n\nRequires cAdvisor.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "Cluster replicas", "formula": "(query1 / query2) * 100", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "percent" } } }, { "alias": "Other pods", "formula": "(query3 / query4) * 100", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "percent" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "avg:container_memory_working_set_bytes{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT container:new-promsql-exporter AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod,container}" }, { "data_source": "metrics", "name": "query2", "query": "avg:container_spec_memory_limit_bytes{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT container:new-promsql-exporter AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod,container}" }, { "data_source": "metrics", "name": "query3", "query": "avg:container_memory_working_set_bytes{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT container:new-promsql-exporter AND NOT pod:*-cluster-*-replica-*} by {namespace,pod,container}" }, { "data_source": "metrics", "name": "query4", "query": "avg:container_spec_memory_limit_bytes{namespace:$namespace.value AND container:* AND NOT container:POD AND NOT container:new-promsql-exporter AND NOT pod:*-cluster-*-replica-*} by {namespace,pod,container}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Pod Memory Usage", "type": "timeseries" } } ] }, "id": 203 }, { "definition": { "background_color": "blue", "layout_type": "ordered", "show_title": true, "title": "Pod Networking", "type": "group", "widgets": [ { "definition": { "description": "**Network bytes/sec received per pod. For cluster pods, Rx tracks ingest from upstream and inter-pod replication; for envd/balancer it's client SQL traffic. Surges alongside hydration are normal catchup.**\n\nRequires cAdvisor.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "Cluster replicas", "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "byte" } } }, { "alias": "Other pods", "formula": "query2", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "byte" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:container_network_receive_bytes_total{namespace:$namespace.value AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query2", "query": "sum:container_network_receive_bytes_total{namespace:$namespace.value AND NOT pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Pod Network Rx", "type": "timeseries" } }, { "definition": { "description": "**Network bytes/sec transmitted per pod. For cluster pods Tx covers sink output, inter-pod replication, and query results returning to envd; for envd it's client query responses.**\n\nRequires cAdvisor.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "Cluster replicas", "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "byte" } } }, { "alias": "Other pods", "formula": "query2", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "byte" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:container_network_transmit_bytes_total{namespace:$namespace.value AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query2", "query": "sum:container_network_transmit_bytes_total{namespace:$namespace.value AND NOT pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Pod Network Tx", "type": "timeseries" } }, { "definition": { "description": "**Network rx + tx errors per pod per second (counted at the NIC/kernel level).**\n\n**Nominal:** 0 — no NIC/kernel-level errors.\n\n**Unhealthy:** Non-zero is unusual and points at infrastructure (faulty NIC, kernel network stack, container runtime) — not a Materialize-level problem; file an infra ticket.\n\nRequires cAdvisor.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "Cluster replicas rx", "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "error" } } }, { "alias": "Other pods rx", "formula": "query2", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "error" } } }, { "alias": "Cluster replicas tx", "formula": "query3", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "error" } } }, { "alias": "Other pods tx", "formula": "query4", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "error" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:container_network_receive_errors_total{namespace:$namespace.value AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query2", "query": "sum:container_network_receive_errors_total{namespace:$namespace.value AND NOT pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query3", "query": "sum:container_network_transmit_errors_total{namespace:$namespace.value AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query4", "query": "sum:container_network_transmit_errors_total{namespace:$namespace.value AND NOT pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Pod Network Errors", "type": "timeseries" } }, { "definition": { "description": "**Network packets dropped (rx + tx) per pod per second — when kernel buffers fill faster than the app reads (rx) or egress rate-limiting kicks in (tx).**\n\n**Nominal:** 0 (single-digit pps drops are usually harmless background noise).\n\n**Degraded:** Sustained higher rates mean the pod is overwhelmed at the network layer — often paired with elevated pod CPU.\n\nRequires cAdvisor.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "Cluster replicas rx", "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "packet" } } }, { "alias": "Other pods rx", "formula": "query2", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "packet" } } }, { "alias": "Cluster replicas tx", "formula": "query3", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "packet" } } }, { "alias": "Other pods tx", "formula": "query4", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "packet" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:container_network_receive_packets_dropped_total{namespace:$namespace.value AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query2", "query": "sum:container_network_receive_packets_dropped_total{namespace:$namespace.value AND NOT pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query3", "query": "sum:container_network_transmit_packets_dropped_total{namespace:$namespace.value AND pod:*-cluster-$cluster.value-replica-$replica.value-*} by {namespace,pod}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query4", "query": "sum:container_network_transmit_packets_dropped_total{namespace:$namespace.value AND NOT pod:*-cluster-*-replica-*} by {namespace,pod}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Pod Network Packet Drops", "type": "timeseries" } } ] }, "id": 204 }, { "definition": { "background_color": "purple", "layout_type": "ordered", "show_title": false, "title": "Connection Summary", "type": "group", "widgets": [ { "definition": { "description": "**Open SQL sessions, split into `system` (Materialize's internal probing — a few are always present) and `user` (client connections).**\n\n**Degraded:** A `user` count that climbs and never settles usually means a client is opening connections and never closing them; cross-check whether query activity rose to match, or whether connections are just accumulating idle.\n\nEnvironment-scoped — the cluster/replica selectors don't narrow it.", "requests": [ { "formulas": [ { "formula": "query1", "limit": { "count": 10, "order": "desc" } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:mz_active_sessions{$environment} by {session_type}" } ], "response_format": "scalar" } ], "title": "Active Sessions", "type": "toplist" } }, { "definition": { "description": "**Queries per second by session type — `user` tracks your client traffic, `system` is the steady single-digit baseline of internal health checks. Bursty is normal.**\n\nEnvironment-scoped.", "requests": [ { "formulas": [ { "formula": "query1", "limit": { "count": 10, "order": "desc" }, "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "query" } } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:mz_query_total{$environment} by {session_type}.as_rate().rollup(sum, 60)" } ], "response_format": "scalar" } ], "title": "Active Queries", "type": "toplist" } }, { "definition": { "autoscale": true, "description": "**Commands per second through the adapter — the SQL protocol layer (parse, execute, prepare, fetch). Normally runs higher than the query rate, since one query is several commands.**\n\n**Degraded:** A flat-line on an environment that's usually busy is unusual and can indicate the adapter is in trouble; check per-application command activity and its error column.\n\nEnvironment-scoped.", "precision": 2, "requests": [ { "formulas": [ { "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "command" } } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:mz_adapter_commands{$environment}.as_rate().rollup(sum, 60)" } ], "response_format": "scalar" } ], "title": "SQL Control Plane Command Rate", "type": "query_value" } } ] }, "id": 301 }, { "definition": { "background_color": "purple", "layout_type": "ordered", "show_title": true, "title": "Queries", "type": "group", "widgets": [ { "definition": { "description": "**The mix of query kinds over the selected window — a workload-shape signal, not a rate. Heavy `set_variable`/`fetch` traffic is normal (that's how Postgres clients manage session state); heavy `insert`/`update`/`delete` on something you think of as read-mostly is worth a look.**\n\nIdle statement types are filtered out. Environment-scoped.", "legend": { "type": "table" }, "requests": [ { "formulas": [ { "formula": "cutoff_min(query1, 1e-9)" } ], "queries": [ { "aggregator": "sum", "data_source": "metrics", "name": "query1", "query": "sum:mz_query_total{$environment} by {statement_type}.as_count()" } ], "response_format": "scalar" } ], "title": "Query Distribution (by statement_type)", "type": "sunburst" } }, { "definition": { "description": "**Query rate broken down by statement type and session type, fully time-resolved — the moving picture behind the distribution donut. A spike in `select`/`user` is the thing to line up against peek latency to confirm the system kept pace.**\n\nIdle (statement, session) tuples are filtered out. Environment-scoped.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "cutoff_min(query1, 1e-9)", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "query" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_query_total{$environment} by {statement_type,session_type}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Query Rate (by statement_type / session_type)", "type": "timeseries", "yaxis": { "min": "0" } } }, { "definition": { "description": "**Median read-query latency — the typical time to look up the current state of an arrangement, which is the operation behind every `SELECT` against an index. Your \"what does a normal query feel like\" number.**\n\n**Nominal:** A few milliseconds on a healthy cluster.\n\n**Degraded:** Sustained multi-second p50 means the cluster is overwhelmed — it can't keep up with the read load on it.\n\nPer-cluster: the self-managed metric carries the cluster (instance_id) but not the replica, so the replica selector doesn't split it further.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "p50:mz_compute_peek_duration_seconds{$environment AND instance_id:$cluster.value} by {instance_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Peek Latency (p50)", "type": "timeseries", "yaxis": { "scale": "log" } } }, { "definition": { "description": "**90th-percentile read-query latency — how slow the slowest 10% of queries feel. Catches the contention bursts and cold paths that the median hides.**\n\n**Nominal:** A small multiple of p50, roughly 2–5×.\n\n**Degraded:** p90 running 10–100× p50 means a bimodal distribution — usually cold-cache effects on infrequently-queried indexes, or contention.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "p90:mz_compute_peek_duration_seconds{$environment AND instance_id:$cluster.value} by {instance_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Peek Latency (p90)", "type": "timeseries", "yaxis": { "scale": "log" } } }, { "definition": { "description": "**Tail read-query latency — the slowest 1% of queries, the ones users complain about.**\n\n**Nominal:** A small multiple of p50 (2–10×), with occasional spikes during query plan recompilation or hydration.\n\n**Degraded:** Sustained p99 in the seconds range — especially when p50/p90 are NOT elevated — points at a single bad query or a genuinely tail-sensitive use case. Confirm the latency is on real traffic (against the query rate), not idle scrapes.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "p99:mz_compute_peek_duration_seconds{$environment AND instance_id:$cluster.value} by {instance_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Peek Latency (p99)", "type": "timeseries", "yaxis": { "scale": "log" } } } ] }, "id": 302 }, { "definition": { "background_color": "purple", "layout_type": "ordered", "show_title": true, "title": "SQL Control Plane Commands", "type": "group", "widgets": [ { "definition": { "description": "**SQL control-plane command totals per client `application_name` over the window, so you can see which clients drive the adapter and which are failing. Most clients set `application_name` in their connection string; those that don't bucket as `unrecognized`/`unspecified` (normal).**\n\n**Unhealthy:** Sustained errors against a real application mean that app is consistently failing its commands — correlate with the app's own logs and the `mz_internal` activity-log views for the specific failures.\n\nThe presentation layer pivots the `status` label into Success/Errors columns. Environment-scoped.", "requests": [ { "formulas": [ { "alias": "Commands", "formula": "query1", "limit": { "count": 500, "order": "desc" } } ], "queries": [ { "aggregator": "sum", "data_source": "metrics", "name": "query1", "query": "sum:mz_adapter_commands{$environment} by {application_name,status}.as_count()" } ], "response_format": "scalar" } ], "title": "SQL Control Plane Commands by Application", "type": "query_table" } } ] }, "id": 303 }, { "definition": { "background_color": "green", "layout_type": "ordered", "show_title": false, "title": "Cluster Summary", "type": "group", "widgets": [ { "definition": { "autoscale": true, "description": "**How many clusters exist, split into the Materialize-managed system clusters (mz_catalog_server, mz_system, mz_probe, …) that every environment has and the user clusters you created. The gap between the two is your own footprint.**\n\nSteady in normal operation; steps up and down on CREATE CLUSTER / DROP CLUSTER. Sustained drift with no DDL you recognize points at automation creating clusters.", "precision": 0, "requests": [ { "formulas": [ { "formula": "count_not_null(query1)" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_compute_cluster_status{$environment AND compute_cluster_id:$cluster.value} by {compute_cluster_id}" } ], "response_format": "scalar" } ], "title": "Total Clusters", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**How many clusters exist, split into the Materialize-managed system clusters (mz_catalog_server, mz_system, mz_probe, …) that every environment has and the user clusters you created. The gap between the two is your own footprint.**\n\nSteady in normal operation; steps up and down on CREATE CLUSTER / DROP CLUSTER. Sustained drift with no DDL you recognize points at automation creating clusters.", "precision": 0, "requests": [ { "formulas": [ { "formula": "count_not_null(query1)" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_compute_cluster_status{$environment AND compute_cluster_id:$cluster.value AND compute_cluster_id:s*} by {compute_cluster_id}" } ], "response_format": "scalar" } ], "title": "System Clusters", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**How many replicas back the selected clusters, and how many of those are redundancy beyond the first. Every cluster needs one replica to run; anything above that is capacity or availability headroom you've opted into, so a non-zero \"additional\" count is the quick check that HA is actually configured where you expect it.**\n\nSteps on CREATE/DROP CLUSTER REPLICA. The \"additional\" line is hardened to read 0 rather than \"no data\" on clusters with a single replica.", "precision": 0, "requests": [ { "formulas": [ { "formula": "count_not_null(query1)" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_compute_cluster_status{$environment AND compute_cluster_id:$cluster.value AND compute_replica_id:$replica.value} by {compute_cluster_id,compute_replica_id}" } ], "response_format": "scalar" } ], "title": "Total Replicas", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**How many replicas back the selected clusters, and how many of those are redundancy beyond the first. Every cluster needs one replica to run; anything above that is capacity or availability headroom you've opted into, so a non-zero \"additional\" count is the quick check that HA is actually configured where you expect it.**\n\nSteps on CREATE/DROP CLUSTER REPLICA. The \"additional\" line is hardened to read 0 rather than \"no data\" on clusters with a single replica.", "precision": 0, "requests": [ { "formulas": [ { "formula": "default_zero(count_not_null(query1))" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_compute_cluster_status{$environment AND compute_cluster_id:$cluster.value AND compute_replica_id:$replica.value AND NOT compute_replica_name:r1} by {compute_cluster_id,compute_replica_id}" } ], "response_format": "scalar" } ], "title": "Redundant Replicas", "type": "query_value" } } ] }, "id": 401 }, { "definition": { "background_color": "green", "layout_type": "ordered", "show_title": true, "title": "Replication / Availability", "type": "group", "widgets": [ { "definition": { "description": "**The replica fleet grouped by configured size. Most deployments settle on a handful of sizes; a long tail of one-off sizes usually means an experiment or a half-finished migration. The total agrees with the replica count.**", "legend": { "type": "table" }, "requests": [ { "formulas": [ { "formula": "reduce(count(query1, {size}), 'last')" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "max:mz_compute_cluster_status{$environment AND compute_cluster_id:$cluster.value AND compute_replica_id:$replica.value} by {compute_cluster_id,compute_replica_id,size}" } ], "response_format": "scalar" } ], "title": "Replica Sizes", "type": "sunburst" } } ] }, "id": 402 }, { "definition": { "background_color": "green", "layout_type": "ordered", "show_title": true, "title": "Cluster Information", "type": "group", "widgets": [ { "definition": { "description": "**A reference row per (cluster, replica): ids, names, size, version, and scheduling metadata. The \"what does my fleet actually look like\" lookup — most useful for grabbing a cluster or replica id to scope the rest of a dashboard to.**\n\nA catalog snapshot, not a health signal. The presentation layer projects the metric's labels (compute_cluster_name, size, mz_version, region, …) into table columns; this query just selects the underlying series.", "requests": [ { "formulas": [ { "alias": "Status", "formula": "query1", "limit": { "count": 500, "order": "desc" } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_compute_cluster_status{$environment AND compute_cluster_id:$cluster.value AND compute_replica_id:$replica.value} by {compute_cluster_id,compute_cluster_name,compute_replica_id,compute_replica_name,size,mz_version}" } ], "response_format": "scalar" } ], "title": "Cluster Information", "type": "query_table" } } ] }, "id": 403 }, { "definition": { "background_color": "orange", "layout_type": "ordered", "show_title": false, "title": "Compute Objects Summary", "type": "group", "widgets": [ { "definition": { "autoscale": true, "description": "**Materialized views Materialize is actively maintaining. Each one is a query whose result is kept continuously up to date, so this tracks roughly how much standing compute the environment carries.**\n\nEnvironment-scoped — the cluster/replica selectors don't narrow it. Steps on CREATE/DROP MATERIALIZED VIEW; steady otherwise.", "precision": 0, "requests": [ { "formulas": [ { "formula": "query1" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "max:mz_mzd_views_count{$environment}" } ], "response_format": "scalar" } ], "title": "Active Materialized Views", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**Indexes in the catalog. An index is an in-memory arrangement that makes reads against its relation effectively instant, in exchange for memory — so growth here is a leading indicator of cluster memory growth.**\n\nEnvironment-scoped. Hardened to read 0 on a fresh environment (the metric is absent until the first index exists).", "precision": 0, "requests": [ { "formulas": [ { "formula": "default_zero(query1)" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:mz_indexes_count{$environment} by {service.instance.id}" } ], "response_format": "scalar" } ], "title": "Active Indexes", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**Non-materialized views — query templates evaluated on demand. They cost nothing until something reads them, so this is a catalog-shape signal rather than a load one.**\n\nEnvironment-scoped.", "precision": 0, "requests": [ { "formulas": [ { "formula": "query1" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "max:mz_views_count{$environment}" } ], "response_format": "scalar" } ], "title": "Active Views", "type": "query_value" } }, { "definition": { "description": "**Live SUBSCRIBE sessions — long-running queries that stream updates to a client as data changes. A handful of `system` subscribes are Materialize's own internal probes; a persistently climbing `user` count usually means a client is opening subscribes and never closing them.**\n\nEnvironment-scoped.", "legend": { "type": "table" }, "requests": [ { "formulas": [ { "formula": "query1" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:mz_active_subscribes{$environment} by {session_type}" } ], "response_format": "scalar" } ], "title": "Active Subscribes", "type": "sunburst" } }, { "definition": { "description": "**Indexes split by the kind of relation they sit on. Workloads normally lean heavily on indexes over views (the standard \"keep a query's result hot\" pattern); a large share of indexes on base tables is unusual and usually worth a second look.**\n\nEnvironment-scoped.", "legend": { "type": "table" }, "requests": [ { "formulas": [ { "formula": "query1" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:mz_indexes_count{$environment} by {relation_type}" } ], "response_format": "scalar" } ], "title": "Index Relationship Types", "type": "sunburst" } } ] }, "id": 501 }, { "definition": { "background_color": "orange", "layout_type": "ordered", "show_title": true, "title": "Freshness", "type": "group", "widgets": [ { "definition": { "description": "**How far behind real time each cluster's least fresh object is — the worst-case freshness across every index, materialized view, and source on the cluster.**\n\n**Nominal:** Low and flat, sub-second to a few seconds, for a cluster keeping up.\n\n**Degraded:** A sustained climb means an object can't keep pace with its input — an under-provisioned cluster or an expensive dataflow. A sharp spike that decays on its own is normal re-hydration after a restart or DDL, not a problem.\n\nThe catalog cluster (s2) sits a couple seconds back as its baseline. Objects that have not produced results yet report a sentinel and are excluded here. For which object is responsible, see _Least Fresh Objects_ (in this row).", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "max(cutoff_max(query1, 1e9), {instance_id})", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "max:mz_dataflow_wallclock_lag_seconds{$environment AND instance_id:$cluster.value AND quantile:1} by {namespace,pod,instance_id,replica_id,collection_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Worst Freshness by Cluster", "type": "timeseries", "yaxis": { "scale": "log" } } }, { "definition": { "description": "**The freshness of every object on each cluster, added together — one number for how far behind the cluster is in total.**\n\n**Nominal:** Low and flat. The absolute value is not comparable between clusters, since a cluster with more objects has more to add up; the *shape* is what this panel is for.\n\n**Degraded:** A climb that does not come back down. A rise that decays steadily is a cluster catching up after a restart, a rollout, or DDL, and the decay is the thing worth watching.\n\n**Read this beside the worst-case panel rather than instead of it.** The max says whether anything is behind; this says how much is behind in total, and the two answer different questions. When a hundred objects rehydrate and fifty of them catch up, the max barely moves — it is still reporting whichever one is furthest behind — while this halves. That descent is the clearest picture of hydration progress the metrics can give, and it is invisible in the max.\n\n**Deduplicated across replicas.** An object served by several replicas reports once per replica, so the inner `max` picks the worst reading per object before the outer `sum` adds objects together. Without it a two-replica cluster would report double, which is exactly the shape a blue/green rollout has.\n\nObjects that have not produced results yet report the `u64::MAX` sentinel and are excluded (`< 1e9`) — including them would swamp the total by nineteen orders of magnitude. They are counted by the hydration panels instead, so the two are complementary: this totals the objects that are *producing results*, those count the ones that are not.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "sum(max(cutoff_max(query1, 1e9), {instance_id,collection_id}), {instance_id})", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "max:mz_dataflow_wallclock_lag_seconds{$environment AND instance_id:$cluster.value AND quantile:1} by {namespace,pod,instance_id,replica_id,collection_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Total Freshness by Cluster", "type": "timeseries", "yaxis": { "min": "0" } } }, { "definition": { "description": "**The 15 objects whose results are furthest behind real time — the per-object breakdown behind _Worst Freshness by Cluster_, labeled by object name.**\n\nIdle or still-hydrating objects report a sentinel and are filtered out; check hydration state directly in SQL if an object you expect is missing. Catalog (s2) objects commonly appear with a small, expected baseline freshness of a few seconds.", "requests": [ { "formulas": [ { "formula": "reduce(top(max(cutoff_max(query1, 1e9), {instance_id,collection_id}), 15, 'last', 'desc'), 'last')", "limit": { "count": 15, "order": "desc" }, "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "max:mz_dataflow_wallclock_lag_seconds{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND quantile:1} by {namespace,pod,instance_id,replica_id,collection_id}" } ], "response_format": "scalar" } ], "time": { "live_span": "5m" }, "title": "Least Fresh Objects", "type": "toplist" } } ] }, "id": 502 }, { "definition": { "background_color": "orange", "layout_type": "ordered", "show_title": true, "title": "Hydration", "type": "group", "widgets": [ { "definition": { "autoscale": true, "description": "**Objects still rebuilding their in-memory state — a live hydration-queue proxy. After a restart, replica creation, or some DDL, a dataflow has to rebuild from persisted storage before it can serve, and until it does it produces no results.**\n\n**Nominal:** 0, with brief spikes right after a replica restart that drain back to 0 as dataflows catch up. Those spikes are healthy re-hydration, not a problem.\n\n**Unhealthy:** A count that stays elevated means something can't finish hydrating — a source whose CREATE never completed, or a wedged dataflow. Confirm what with `SELECT * FROM mz_internal.mz_hydration_statuses WHERE NOT hydrated`.\n\nStopgap, not a real signal: self-managed has no hydration-state metric, so this counts objects whose freshness reads the u64::MAX sentinel (> 1e15) as a proxy. Deprecate this query once an actual hydration-state metric exists.", "precision": 0, "requests": [ { "formulas": [ { "formula": "default_zero(count_not_null(cutoff_min(query1, 1e15)))" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "max:mz_dataflow_wallclock_lag_seconds{$environment AND instance_id:$cluster.value AND quantile:1} by {instance_id,collection_id}" } ], "response_format": "scalar" } ], "title": "Currently Hydrating", "type": "query_value" } }, { "definition": { "description": "**Objects waiting in each replica's hydration queue. environmentd schedules hydration in batches; a backlog means work is arriving faster than the replica can rebuild it.**\n\n**Nominal:** 0 — nothing is waiting to hydrate.\n\n**Unhealthy:** Sustained non-zero means the replica is undersized for its workload, or one slow-hydrating object is holding up the line behind it.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "cutoff_min(query1, 1e-9)" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_compute_controller_hydration_queue_size{$environment AND instance_id:$cluster.value AND replica_id:$replica.value} by {instance_id,replica_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Hydration Queue Size", "type": "timeseries", "yaxis": { "min": "0" } } }, { "definition": { "description": "**The 15 objects that took longest to finish hydrating. Hydration time scales with the size of the state being rebuilt, so large materialized views and indexes naturally top the list.**\n\nThe per-object hydration-time metric is cloud-only, so this is blank on self-managed. Get it from SQL instead: `SELECT object_id, time_ns/1e9 AS seconds FROM mz_internal.mz_compute_hydration_times ORDER BY time_ns DESC`. The live metric-side proxy for what's behind is _Freshness -> Least Fresh Objects_.", "requests": [ { "formulas": [ { "formula": "top(query1, 15, 'last', 'desc')", "limit": { "count": 15, "order": "desc" }, "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "max:mz_compute_hydration_time_seconds{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND hydrated:1} by {instance_id,collection_id}" } ], "response_format": "scalar" } ], "time": { "live_span": "5m" }, "title": "Slowest Hydrating Objects", "type": "toplist" } } ] }, "id": 503 }, { "definition": { "background_color": "orange", "layout_type": "ordered", "show_title": true, "title": "Dataflows", "type": "group", "widgets": [ { "definition": { "description": "**Active dataflows on each replica. Every index, materialized view, and live SUBSCRIBE runs as one or more dataflows, so this count rises with DDL and subscribe activity.**\n\n**Nominal:** Stable for a steady workload.\n\n**Degraded:** A sharp drop with no matching DDL usually means a replica restarted; a sharp rise usually means bulk object creation.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "max:mz_compute_replica_history_dataflow_count{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Dataflow Count", "type": "timeseries", "yaxis": { "min": "0" } } }, { "definition": { "description": "**The dataflow count broken out per worker. Workers in a replica run in lockstep and should see exactly the same dataflows, so their series should overlap perfectly.**\n\n**Unhealthy:** Visible divergence between worker series within one (cluster, replica) is a bug-class signal — the workers disagree on what they're running — and worth filing.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "max:mz_compute_replica_history_dataflow_count{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id,worker_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Dataflow Count (per worker)", "type": "timeseries", "yaxis": { "min": "0" } } }, { "definition": { "description": "**CPU-cores busy inside dataflows, per cluster — the whole of dataflow work: arrangement maintenance, query evaluation, and hydration. Capped by cluster size (a 400cc cluster can't exceed 400 cores).**\n\n**Nominal:** Well below the cluster's size.\n\n**Degraded:** Sustained near the size cap means the cluster is CPU-bound and a candidate for upsizing. The catalog cluster (s2) is typically the busiest in any environment.\n\nReads high if the same clusterd endpoint is scraped by more than one Prometheus job (a known multi-job scrape-config pitfall); fix the scrape config, or collapse duplicate `job` series, at the deployment.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "per_second(query1)" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_dataflow_elapsed_seconds_total{$environment AND instance_id:$cluster.value AND replica_id:$replica.value} by {instance_id}.rollup(max, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Dataflow Elapsed Rate", "type": "timeseries", "yaxis": { "scale": "log" } } } ] }, "id": 504 }, { "definition": { "background_color": "orange", "layout_type": "ordered", "show_title": true, "title": "Arrangements", "type": "group", "widgets": [ { "definition": { "description": "**CPU-cores spent maintaining arrangements — the in-memory indexed snapshots behind every index and materialized view — summed across a replica's workers, so an N-worker replica can reach N.**\n\n**Nominal:** Well below the worker count.\n\n**Degraded:** Sustained near the worker count means the replica is bottlenecked on maintenance, usually a high upstream change rate or many heavy indexes. The catalog cluster's baseline runs higher than typical user clusters.\n\nUses the long-form cluster_environmentd_* id labels. Reads high under the same multi-job scrape pitfall as the dataflow elapsed rate.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_arrangement_maintenance_seconds_total{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Arrangement Maintenance Rate", "type": "timeseries" } }, { "definition": { "description": "**The same maintenance CPU, split per worker — each worker tops out at 1.0.**\n\n**Degraded:** Watch for skew: one worker sitting near 1.0 while its siblings idle near 0 means the cluster is bottlenecked on a single worker (a hot key or hot arrangement) — the most common reason an 8-worker cluster performs like a 1-worker one. `EXPLAIN PHYSICAL PLAN` on the heavy object is the usual next step.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "(query1) * 100", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "percent" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_arrangement_maintenance_seconds_total{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {cluster_environmentd_materialize_cloud_cluster_id,cluster_environmentd_materialize_cloud_replica_id,worker_id}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Arrangement Maintenance Rate (per worker)", "type": "timeseries" } }, { "definition": { "description": "**Row counts of arrangements for Materialize's internal system objects (object id starts with `s`). These back the catalog and internal probes, not user data, so they shouldn't grow with your workload — unexpected growth here can point at a Materialize bug.**", "requests": [ { "formulas": [ { "alias": "Min", "formula": "query1", "limit": { "count": 500, "order": "desc" } }, { "alias": "Max", "formula": "query2" }, { "alias": "Last", "formula": "query3" } ], "queries": [ { "aggregator": "min", "data_source": "metrics", "name": "query1", "query": "max:mz_arrangement_record_count{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND collection_id:s*} by {collection_id}" }, { "aggregator": "max", "data_source": "metrics", "name": "query2", "query": "max:mz_arrangement_record_count{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND collection_id:s*} by {collection_id}" }, { "aggregator": "last", "data_source": "metrics", "name": "query3", "query": "max:mz_arrangement_record_count{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND collection_id:s*} by {collection_id}" } ], "response_format": "scalar" } ], "title": "System Objects — Record Counts", "type": "query_table" } }, { "definition": { "description": "**Row counts of arrangements for your compute objects (object id starts with `u`) — the row count of every user index and materialized view, and the primary driver of cluster memory. Growth on an object tracks the size of its underlying data.**", "requests": [ { "formulas": [ { "alias": "Min", "formula": "query1", "limit": { "count": 500, "order": "desc" } }, { "alias": "Max", "formula": "query2" }, { "alias": "Last", "formula": "query3" } ], "queries": [ { "aggregator": "min", "data_source": "metrics", "name": "query1", "query": "max:mz_arrangement_record_count{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND collection_id:u*} by {collection_id}" }, { "aggregator": "max", "data_source": "metrics", "name": "query2", "query": "max:mz_arrangement_record_count{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND collection_id:u*} by {collection_id}" }, { "aggregator": "last", "data_source": "metrics", "name": "query3", "query": "max:mz_arrangement_record_count{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND collection_id:u*} by {collection_id}" } ], "response_format": "scalar" } ], "title": "User Objects — Record Counts", "type": "query_table" } }, { "definition": { "description": "**Row counts of transient (object id `t*`) and uncategorized (`none`) arrangements — short-lived intermediates from query optimization and dataflow execution. Normally small and ephemeral.**\n\n**Unhealthy:** Sustained non-trivial entries here can mean a stuck or orphaned dataflow holding intermediate state it should have released.", "requests": [ { "formulas": [ { "alias": "Min", "formula": "query1", "limit": { "count": 500, "order": "desc" } }, { "alias": "Max", "formula": "query2" }, { "alias": "Last", "formula": "query3" } ], "queries": [ { "aggregator": "min", "data_source": "metrics", "name": "query1", "query": "max:mz_arrangement_record_count{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND (collection_id:t* OR collection_id:none)} by {collection_id}" }, { "aggregator": "max", "data_source": "metrics", "name": "query2", "query": "max:mz_arrangement_record_count{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND (collection_id:t* OR collection_id:none)} by {collection_id}" }, { "aggregator": "last", "data_source": "metrics", "name": "query3", "query": "max:mz_arrangement_record_count{$environment AND instance_id:$cluster.value AND replica_id:$replica.value AND (collection_id:t* OR collection_id:none)} by {collection_id}" } ], "response_format": "scalar" } ], "title": "Transient / Uncategorized — Record Counts", "type": "query_table" } } ] }, "id": 505 }, { "definition": { "background_color": "yellow", "layout_type": "ordered", "show_title": false, "title": "Storage Objects Summary", "type": "group", "widgets": [ { "definition": { "autoscale": true, "description": "**Active sources in the catalog — each is a continuous ingestion connection from an external system (Kafka, Postgres, MySQL, S3, …), so this is roughly how many upstream feeds the environment maintains. Counts distinct source objects (the hidden per-source `_progress` subsources are excluded), matching `mz_sources`.**\n\nEnvironment-scoped — the cluster/replica selectors don't narrow it.", "precision": 0, "requests": [ { "formulas": [ { "formula": "default_zero(count_not_null(query1))" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_storage_objects{$environment AND type:source} by {id}" } ], "response_format": "scalar" } ], "title": "Active Sources", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**Active sinks in the catalog — each emits the results of a materialized view or query to an external system (Kafka, Iceberg, …). Counts distinct sink objects (excluding `_progress` subsources), matching `mz_sinks`.**\n\nEnvironment-scoped.", "precision": 0, "requests": [ { "formulas": [ { "formula": "default_zero(count_not_null(query1))" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_storage_objects{$environment AND type:sink} by {id}" } ], "response_format": "scalar" } ], "title": "Active Sinks", "type": "query_value" } }, { "definition": { "autoscale": true, "description": "**User-created tables in the catalog. Tables are write-once-read-many; `INSERT`s feed dataflows downstream. Mostly a catalog-shape signal — for actual ingest activity look at source throughput.**\n\nEnvironment-scoped.", "precision": 0, "requests": [ { "formulas": [ { "formula": "default_zero(query1)" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:mz_tables_count{$environment} by {service.instance.id}" } ], "response_format": "scalar" } ], "title": "Active Tables", "type": "query_value" } } ] }, "id": 601 }, { "definition": { "background_color": "yellow", "layout_type": "ordered", "show_title": true, "title": "Sources", "type": "group", "widgets": [ { "definition": { "description": "**Sources by connector type (kafka / postgres / mysql / …) — what flavors of upstream feed make up the ingest workload. Most environments concentrate on one or two.**\n\nEnvironment-scoped.", "legend": { "type": "table" }, "requests": [ { "formulas": [ { "formula": "reduce(count(query1, {object_type}), 'last')" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "max:mz_storage_objects{$environment AND type:source} by {id,object_type}" } ], "response_format": "scalar" } ], "title": "Source Types", "type": "sunburst" } }, { "definition": { "description": "**A catalog of sources — one row per source (by name) with its connector type, envelope, and the cluster it ingests on. The metric-side \"what sources do I have\" reference.**\n\nSelf-managed exposes no source *status* metric, so running/stalled isn't here — check `SELECT name, type, status FROM mz_internal.mz_source_statuses`, and use source throughput to confirm a source is actively ingesting. `_progress` subsources are excluded, so the row count matches the source count. Environment-scoped.", "requests": [ { "formulas": [ { "alias": "Objects", "formula": "query1", "limit": { "count": 500, "order": "desc" } } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "avg:mz_storage_objects{$environment AND type:source} by {id,object_type,connection_type,envelope_type,cluster_id}" } ], "response_format": "scalar" } ], "title": "Sources", "type": "query_table" } }, { "definition": { "description": "**Inbound throughput per primary source — bytes/second pulled from upstream. Subsources (e.g. per-table Postgres replication) roll up to their primary, so each line is one logical source.**\n\nIdle sources are filtered out. Scoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "byte" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_source_bytes_received{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {parent_source_id}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Source Bytes Received (rate)", "type": "timeseries", "yaxis": { "scale": "log" } } }, { "definition": { "description": "**Messages ingested per second, split per source AND replica. Replicas read their upstream independently and should track together.**\n\n**Unhealthy:** A replica flat at 0 while its siblings keep ingesting has lost its upstream connection (e.g. restarted and couldn't resume pulling from Kafka). The source still shows `Running` overall and the aggregate throughput hides it, so this per-replica split is where it surfaces — the source will be falling behind real time too; restarting that replica usually clears the stale connection.\n\nScoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "message" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_source_messages_received{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {parent_source_id,cluster_environmentd_materialize_cloud_replica_id}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Source Ingestion by Replica", "type": "timeseries", "yaxis": { "min": "0" } } }, { "definition": { "description": "**Per-source upstream health, with two complementary signals — both nominal at 0, so an empty panel is healthy and any series means a source needs attention.**\n\n**Unhealthy:** Commit failures fire when the upstream is reachable but rejects the offset/replication-slot commit (auth/ACL, broker rejecting). The disconnected indicator flips to 1 when the source has lost sight of its upstream (`offset_known` fell below `offset_committed`) — the broker/DB-unreachable case (BrokerTransportFailure, severed security group, DNS) that commit-failures can't catch, because the source never reaches the commit step. When disconnected is 1, throughput flat-lines and the source falls further behind real time. Read the exact error via `SELECT name, status, error FROM mz_internal.mz_source_statuses WHERE status != 'running'`.\n\nData-decode errors are separate (`mz_source_error_inserts`). Scoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "commit failures", "formula": "cutoff_min(query1, 1e-9)" }, { "alias": "disconnected", "formula": "cutoff_min(is_greater(query2 - query3, 0), 1)" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_source_offset_commit_failures{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {source_id}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query2", "query": "max:mz_source_offset_committed{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {source_id}" }, { "data_source": "metrics", "name": "query3", "query": "max:mz_source_offset_known{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {source_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Source Upstream Errors", "type": "timeseries", "yaxis": { "min": "0" } } } ] }, "id": 602 }, { "definition": { "background_color": "yellow", "layout_type": "ordered", "show_title": true, "title": "Sinks", "type": "group", "widgets": [ { "definition": { "description": "**Sinks by (type, envelope) — e.g. `kafka / upsert`, `kafka / debezium`, `iceberg / upsert`. The envelope is how Materialize encodes changes: `upsert` writes the latest value per key, `debezium` writes change events with old+new values.**\n\nEnvironment-scoped.", "legend": { "type": "table" }, "requests": [ { "formulas": [ { "formula": "query1" } ], "queries": [ { "aggregator": "last", "data_source": "metrics", "name": "query1", "query": "sum:mz_storage_objects{$environment AND type:sink} by {object_type,envelope_type}" } ], "response_format": "scalar" } ], "title": "Sink Types", "type": "sunburst" } }, { "definition": { "description": "**Outbound throughput per sink — bytes/second successfully committed to the downstream system (Kafka broker, Iceberg catalog, …).**\n\nIdle sinks are filtered out. Scoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "byte" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_sink_bytes_committed{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Sink Throughput (committed)", "type": "timeseries", "yaxis": { "scale": "log" } } }, { "definition": { "description": "**Bytes staged for a sink but not yet committed downstream — an in-flight queue depth in bytes. Oscillates around a small value in normal operation as commits happen periodically.**\n\n**Unhealthy:** Sustained growth means the sink can't keep up — usually downstream back-pressure (broker overloaded, Iceberg catalog slow) or repeated commit failures (see the Iceberg/Kafka sink error panels).\n\nClamped at 0 (brief negatives come from scrape skew between the staged and committed counters). Scoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "clamp_min(query1 - query2, 0)", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "byte" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_sink_bytes_staged{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}" }, { "data_source": "metrics", "name": "query2", "query": "sum:mz_sink_bytes_committed{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Sink Lag (staged minus committed)", "type": "timeseries", "yaxis": { "min": "0" } } } ] }, "id": 603 }, { "definition": { "background_color": "yellow", "layout_type": "ordered", "show_title": true, "title": "Iceberg Sinks", "type": "group", "widgets": [ { "definition": { "description": "**Iceberg commit-duration percentiles (p50/p90/p99) — how long each `COMMIT` against the Iceberg catalog takes (write a snapshot manifest, ask the catalog to atomically swap it in).**\n\n**Nominal:** p50 sub-second to low seconds; p99 a few seconds even when healthy.\n\n**Degraded:** Sustained p99 in the tens of seconds points at a slow Iceberg catalog (REST catalog under load, Glue API throttling); sink lag will be growing at the same time.\n\nScoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "p50", "formula": "query1", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } }, { "alias": "p90", "formula": "query2", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } }, { "alias": "p99", "formula": "query3", "number_format": { "unit": { "type": "canonical_unit", "unit_name": "second" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "p50:mz_sink_iceberg_commit_duration_seconds{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value}" }, { "data_source": "metrics", "name": "query2", "query": "p90:mz_sink_iceberg_commit_duration_seconds{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value}" }, { "data_source": "metrics", "name": "query3", "query": "p99:mz_sink_iceberg_commit_duration_seconds{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Iceberg Commit Latency (p50 / p90 / p99)", "type": "timeseries", "yaxis": { "scale": "log" } } }, { "definition": { "description": "**Per-sink rate of failed and conflicting Iceberg commits. Conflicts (concurrent-writer races on the snapshot pointer) are recoverable — Materialize retries — but a high rate means something else is writing the same Iceberg table; failures are commit-side errors (network, auth, schema).**\n\n**Unhealthy:** Non-zero in either dimension is worth investigating; climbing failures drag sink lag up with them.\n\nScoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "failures", "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "error" } } }, { "alias": "conflicts", "formula": "query2", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "error" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_sink_iceberg_commit_failures{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query2", "query": "sum:mz_sink_iceberg_commit_conflicts{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Iceberg Commit Failures & Conflicts", "type": "timeseries", "yaxis": { "min": "0" } } }, { "definition": { "description": "**Per-sink rate of files and snapshots written to Iceberg. Each commit produces one snapshot with data files (new rows) and delete files (tombstones for upserts). The data:delete ratio reflects your workload — pure-insert sinks produce ~0 deletes; upsert-heavy ones roughly 1:1.**\n\nSustained delete-file rate with no data files means the sink is mostly deleting (data evaporating upstream). Scoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "data", "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "file" } } }, { "alias": "deletes", "formula": "query2", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "file" } } }, { "alias": "snapshots", "formula": "query3", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "file" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_sink_iceberg_data_files_written{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query2", "query": "sum:mz_sink_iceberg_delete_files_written{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query3", "query": "sum:mz_sink_iceberg_snapshots_committed{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Iceberg File & Snapshot Rate", "type": "timeseries", "yaxis": { "min": "0" } } } ] }, "id": 604 }, { "definition": { "background_color": "yellow", "layout_type": "ordered", "show_title": true, "title": "Kafka Sinks", "type": "group", "widgets": [ { "definition": { "description": "**Per-sink rate of TX errors from the librdkafka client — each is one failed produce-request against the broker.**\n\n**Unhealthy:** Non-zero is a problem: broker outages, ACL changes, topic deletion/recreation, or partition rebalancing. Sustained errors grow sink lag and can fill the output buffer.\n\nScoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "error" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_sink_rdkafka_txerrs{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Kafka TX Error Rate", "type": "timeseries", "yaxis": { "min": "0" } } }, { "definition": { "description": "**Messages sitting in the librdkafka output buffer, waiting to be sent to the broker. Normal buffer fluctuates briefly as messages flow through.**\n\n**Unhealthy:** Sustained high values mean Materialize is producing faster than the broker accepts (often paired with TX errors); if the buffer hits its bound the sink stalls and lag climbs.\n\nScoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "formula": "query1" } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_sink_rdkafka_outbuf_msg_cnt{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Kafka Output Buffer (messages)", "type": "timeseries", "yaxis": { "min": "0" } } }, { "definition": { "description": "**Connect and disconnect events per sink against the Kafka broker. Healthy connections are persistent — a couple of connects at startup and zero disconnects afterward.**\n\n**Unhealthy:** A sustained non-zero disconnect rate signals unhealthy connectivity (network flakiness, broker restarting, auth tokens expiring); pairs with TX errors when the issue is broker-side rather than purely network.\n\nScoped to the selected clusters/replicas.", "legend_layout": "auto", "requests": [ { "display_type": "line", "formulas": [ { "alias": "connects", "formula": "query1", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "connection" } } }, { "alias": "disconnects", "formula": "query2", "number_format": { "unit": { "per_unit_name": "second", "type": "canonical_unit", "unit_name": "connection" } } } ], "queries": [ { "data_source": "metrics", "name": "query1", "query": "sum:mz_sink_rdkafka_connects{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}.as_rate().rollup(sum, 60)" }, { "data_source": "metrics", "name": "query2", "query": "sum:mz_sink_rdkafka_disconnects{$environment AND cluster_environmentd_materialize_cloud_cluster_id:$cluster.value AND cluster_environmentd_materialize_cloud_replica_id:$replica.value} by {sink_id}.as_rate().rollup(sum, 60)" } ], "response_format": "timeseries" } ], "show_legend": true, "title": "Kafka Connect / Disconnect Rate", "type": "timeseries", "yaxis": { "min": "0" } } } ] }, "id": 605 } ] }