{ "apiVersion": "dashboard.grafana.app/v2", "kind": "Dashboard", "metadata": { "annotations": { "grafana.app/folder": "materialize", "monitoring.materialize.cloud/min-mz-version": "v26.41.0", "monitoring.materialize.cloud/rec-mz-version": "v26.41.0", "monitoring.materialize.cloud/sql-metric-prefix": "mz_", "monitoring.materialize.cloud/target-export": "generic" }, "name": "mz-mon-env-upgrade" }, "spec": { "annotations": [ { "kind": "AnnotationQuery", "spec": { "builtIn": true, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "query": { "datasource": { "name": "-- Grafana --" }, "group": "grafana", "kind": "DataQuery", "spec": {}, "version": "v0" } } } ], "cursorSync": "Crosshair", "description": "What happened during a Materialize upgrade.\n\nThe operator's own account of a rollout, alongside\nwhat the cluster did underneath it.", "editable": true, "elements": { "active-generations": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count(\n count by (generation) (\n label_replace(\n mz_compute_commands_total{\n materialize_cloud_organization_name=~\"$environmentNameList\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n )\n)\n", "legendFormat": "generations", "qryType": 1, "query": "count(\n count by (generation) (\n label_replace(\n mz_compute_commands_total{\n materialize_cloud_organization_name=~\"$environmentNameList\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How many deployment generations are currently running — one between rollouts, two while one is in flight.**\n\n**Nominal:** One. A settled environment runs a single generation.\n\n**Degraded:** Two, for as long as a rollout takes. Both generations hold read holds and both cost compute, so this is the state you want to leave promptly rather than linger in.\n\n**Unhealthy:** Two for far longer than a rollout should take. Either the new generation never became ready, or it is ready and waiting for a manual promotion — _Events -> Rollout_ tells you which, and a `WaitingForApproval` there means the rollout is doing what it was configured to do.\n\nCounted from the environmentd pods that are reporting, so a generation appears here as soon as its environmentd starts serving metrics and disappears once it is torn down.", "id": 1010, "links": [], "title": "Active Generations", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE7733", "mode": "shades" }, "decimals": 0, "min": 0, "noValue": "No generations match the current filters" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "all-event-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\"}\n | name=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\" or name!~\".*-(environmentd|gen)-([0-9]+)-[0-9]+\"\n | line_format \"{{.reason}} {{.kind}}/{{.name}} — {{.msg}}\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Every Kubernetes event from the operator and environment namespaces, newest first — the unfiltered record of what the cluster did.**\n\nThe reading of last resort. Prefer the warning and lifecycle feeds first; come here when the cause is something neither of them names, or to see an ordinary event in the context of the ones around it.\n\nLines are reformatted to `Reason Kind/name — message`. The raw event carries a dozen more fields, most of them resource versions and forwarding addresses, and expanding a line shows all of them.", "id": 1009, "links": [], "title": "All Events", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No events in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "collections-by-generation": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count by (generation) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n )\n)\n", "legendFormat": "gen {{generation}}", "qryType": 1, "query": "count by (generation) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Collections each generation is tracking — the denominator for hydration, and the shape of a new generation building out its dataflows.**\n\n**Nominal:** Both generations converging on the same count. The new one is meant to end up serving what the old one served.\n\n**Unhealthy:** A new generation whose count settles well below the old one's. It did not build every dataflow, so promoting it would lose what is missing.\n\nCounts every collection reporting a wallclock lag, hydrated or not, which is what makes it the denominator for _Hydration -> Hydrating Collections by Generation_.", "id": 1016, "links": [], "title": "Collections by Generation", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No generations match the current filters" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "cpu-by-generation": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (generation) (\n label_replace(\n rate(\n container_cpu_usage_seconds_total{\n namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n }\n [$__rate_interval]\n ),\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n)\n", "legendFormat": "gen {{generation}}", "qryType": 1, "query": "sum by (generation) (\n label_replace(\n rate(\n container_cpu_usage_seconds_total{\n namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n }\n [$__rate_interval]\n ),\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**CPU used by each generation's pods — what a rollout costs while both sides are up.**\n\n**Nominal:** The new generation drawing heavily while it hydrates, then settling to something like the old one's draw.\n\n**Unhealthy:** A new generation pinned at its limit long after hydration should have finished, or an old one still drawing after promotion, which means it was never torn down.\n\nCovers environmentd and the cluster replicas of each generation together. This is the panel that says whether the cluster can actually afford to run two generations at once — the answer is roughly double, for as long as the rollout lasts.", "id": 1020, "links": [], "title": "CPU by Generation", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: cadvisor metrics are required (via kubelet)", "unit": "cores" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "environments-needing-update": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n environmentd_needs_update{namespace=~\"$operatorNamespace\"}\n)\n", "legendFormat": "outdated", "qryType": 1, "query": "sum(\n environmentd_needs_update{namespace=~\"$operatorNamespace\"}\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How many environments in this cluster are still running an outdated pod template — the count an upgrade is working to bring to zero.**\n\n**Nominal:** Zero between upgrades. Non-zero during one, falling as environments are rolled.\n\n**Unhealthy:** A count that stops falling. The rollout has stalled rather than finished; read _Reconciliation Passes -> Pass Outcomes_ for whether the operator is still trying, and the Events tab for what it said about why.\n\nDeliberately not alarm-coloured. A non-zero reading is what an upgrade in progress looks like, and colouring it red would fire on every routine one. What matters is the *direction*, which is why this is a sparkline rather than a bare number.\n\nOnly the leader observes this, and it resets to zero when a replica loses the lease, so summing across replicas does not double-count an environment across a leadership handoff.", "id": 1023, "links": [], "title": "Environments Needing Update", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#009988", "mode": "shades" }, "decimals": 0, "min": 0, "noValue": "No metrics: the Materialize operator is not being scraped" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "event-rate-by-reason": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (reason) (\n count_over_time(\n {job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\"}\n | name=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\" or name!~\".*-(environmentd|gen)-([0-9]+)-[0-9]+\"\n [$__auto]\n )\n)\n", "legendFormat": "{{reason}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How often each kind of event is being reported, by reason. The shape of a rollout: `Pulled`, `Created` and `Started` rise together as pods are replaced, and fall back to nothing when it finishes.**\n\n**Nominal:** Quiet, apart from bursts that coincide with a deliberate change.\n\n**Unhealthy:** A reason that keeps firing at a steady rate is a loop rather than a transition. `FailedScheduling` and `BackOff` are the two that most often look like this, and both mean the rollout is not going to finish on its own.\n\nCounts events as Loki received them. Kubernetes aggregates a repeated event into one object with a rising `count` rather than sending it again, so a tight loop reports fewer lines than it has occurrences — the `count` on each line is the number this one stands for.", "id": 1007, "links": [], "title": "Event Rate by Reason", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No events in this time range" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "failed-passes": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n increase(\n orchestratord_reconciliations_total{\n namespace=~\"$operatorNamespace\", outcome=\"failed\"\n }\n [$__range]\n )\n)\n", "legendFormat": "failed", "qryType": 1, "query": "sum(\n increase(\n orchestratord_reconciliations_total{\n namespace=~\"$operatorNamespace\", outcome=\"failed\"\n }\n [$__range]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Reconciliation passes that returned an error over the selected time range.**\n\n**Nominal:** Zero.\n\n**Degraded:** A handful that stopped. The operator retries on a backoff, and a cause that clears itself — a webhook still starting, a secret created moments later — resolves without intervention.\n\n**Unhealthy:** A count that keeps climbing. Something the operator needs is not there and retrying will not make it appear. Each failure is also published as a Kubernetes event carrying the error's cause chain — see _Events -> Operator Health_.\n\nA count over the selected range rather than a rate, so the number answers \"how many\" directly. The rate over time is the `failed` series on _Reconciliation Passes -> Pass Outcomes_.", "id": 1025, "links": [], "title": "Failed Passes (Select Time Range)", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No metrics: the Materialize operator is not being scraped", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 25.75 }, { "color": "#F67E4B", "value": 50.5 }, { "color": "#DD3D2D", "value": 75.25 }, { "color": "#A50026", "value": 100 } ] } }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "failures-by-controller": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (controller, event_type) (\n rate(\n orchestratord_reconciliations_total{\n namespace=~\"$operatorNamespace\", outcome=\"failed\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{controller}} / {{event_type}}", "qryType": 1, "query": "sum by (controller, event_type) (\n rate(\n orchestratord_reconciliations_total{\n namespace=~\"$operatorNamespace\", outcome=\"failed\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Failing passes broken out by which controller failed and which of its entry points was running — the first question after \"something is failing\".**\n\n**Nominal:** Nothing.\n\n**Unhealthy:** `materialize` failing is the environment itself not reaching the state it was asked for. `balancer` or `console` failing leaves the environment running but unreachable or unmanageable, which is a different urgency.\n\n`event_type` is which entry point ran: `apply` is the ordinary path, and `cleanup` runs while a resource is being deleted. A `cleanup` failure is what leaves a resource stuck with its finalizer still attached.", "id": 1027, "links": [], "title": "Failed Passes by Controller", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No failing passes", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "hydrating-by-generation": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (generation) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) > bool 1e15\n )\n)\n", "legendFormat": "gen {{generation}}", "qryType": 1, "query": "sum by (generation) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) > bool 1e15\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Collections still rebuilding their in-memory state, split by generation — the panel that answers whether a new generation is ready to promote.**\n\n**Nominal:** Nothing for the serving generation. A new generation starts high and drains toward zero as its dataflows rebuild from persisted storage; that descent *is* the rollout making progress.\n\n**Degraded:** A new generation whose count is still falling. Normal, and the rate of descent is the honest estimate of how much longer it needs.\n\n**Unhealthy:** A count that stops falling while still above zero. Something cannot finish hydrating, and the rollout will not complete on its own. Confirm what with `SELECT * FROM mz_internal.mz_hydration_statuses WHERE NOT hydrated`.\n\n**A proxy, not a hydration state.** Self-managed Materialize exposes no hydration-state metric, so this counts collections parked at the wallclock-lag `u64::MAX` sentinel (`> 1e15`) — a collection that has not produced any results yet. Retire it once a real hydration metric exists. It is the same signal the environment overview reports as \"Currently Hydrating\", split by generation instead of summed across both sides of a rollout.\n\n**`> bool` rather than a filtering `>`.** A filtering comparison drops the series that do not match, so `count` emits no sample at all once a generation finishes hydrating — the line stops instead of reaching zero, and a stat reducing on the last non-null value goes on showing the last count it saw, indefinitely. `> bool` scores every collection 1 or 0 and `sum` adds them, so the series stays present and lands on zero, which is the descent this panel exists to show. The environment overview's unsplit version gets the same effect with `or vector(0)`; that appends a series carrying no labels, which is unusable once the panel groups by generation.\n\nA generation drops out only when it has no collections at all, which is true of a brand-new one for its first moments and of one that has been torn down.\n\n`instance_id!=\"\"` is load-bearing — it keeps this to collections attached to a compute instance.", "id": 1015, "links": [], "title": "Hydrating Collections by Generation", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No collections reporting for the selected generations" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "hydrating-now": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (generation) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) > bool 1e15\n )\n)\n", "legendFormat": "gen {{generation}}", "qryType": 1, "query": "sum by (generation) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) > bool 1e15\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Collections still rebuilding their in-memory state, split by generation — the panel that answers whether a new generation is ready to promote.**\n\n**Nominal:** Nothing for the serving generation. A new generation starts high and drains toward zero as its dataflows rebuild from persisted storage; that descent *is* the rollout making progress.\n\n**Degraded:** A new generation whose count is still falling. Normal, and the rate of descent is the honest estimate of how much longer it needs.\n\n**Unhealthy:** A count that stops falling while still above zero. Something cannot finish hydrating, and the rollout will not complete on its own. Confirm what with `SELECT * FROM mz_internal.mz_hydration_statuses WHERE NOT hydrated`.\n\n**A proxy, not a hydration state.** Self-managed Materialize exposes no hydration-state metric, so this counts collections parked at the wallclock-lag `u64::MAX` sentinel (`> 1e15`) — a collection that has not produced any results yet. Retire it once a real hydration metric exists. It is the same signal the environment overview reports as \"Currently Hydrating\", split by generation instead of summed across both sides of a rollout.\n\n**`> bool` rather than a filtering `>`.** A filtering comparison drops the series that do not match, so `count` emits no sample at all once a generation finishes hydrating — the line stops instead of reaching zero, and a stat reducing on the last non-null value goes on showing the last count it saw, indefinitely. `> bool` scores every collection 1 or 0 and `sum` adds them, so the series stays present and lands on zero, which is the descent this panel exists to show. The environment overview's unsplit version gets the same effect with `or vector(0)`; that appends a series carrying no labels, which is unusable once the panel groups by generation.\n\nA generation drops out only when it has no collections at all, which is true of a brand-new one for its first moments and of one that has been torn down.\n\n`instance_id!=\"\"` is load-bearing — it keeps this to collections attached to a compute instance.", "id": 1011, "links": [], "title": "Currently Hydrating", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE7733", "mode": "shades" }, "decimals": 0, "min": 0, "noValue": "No collections reporting for the selected generations" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value_and_name", "wideLayout": true } }, "version": "" } } }, "lag-by-generation": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (generation) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) < 1e9\n)\n", "legendFormat": "gen {{generation}}", "qryType": 1, "query": "max by (generation) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) < 1e9\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The worst lag in each generation — how far behind real time its most-lagged collection is.**\n\n**Nominal:** Seconds, and comparable between generations. A new generation that has hydrated should track the old one closely.\n\n**Unhealthy:** A new generation whose lag stays far above the serving one's. It has hydrated — it is producing results — but it is not keeping up, so promoting it would move traffic onto staler data.\n\nHydrating collections are excluded (`< 1e9`), since their sentinel would swamp every real reading. That makes this and _Hydration -> Hydrating Collections by Generation_ complementary: this one covers the collections that are *producing results*, and that one counts the collections that do not.\n\nThe metric is a summary carrying only `quantile` `0` and `1`, so `quantile=\"1\"` is the worst case rather than a percentile.", "id": 1017, "links": [], "title": "Worst-Case Lag by Generation", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No generations match the current filters", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lag-total-by-cluster": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (generation, instance_id) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) < 1e9\n )\n)\n", "legendFormat": "gen {{generation}} / {{instance_id}}", "qryType": 1, "query": "sum by (generation, instance_id) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) < 1e9\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Total lag split by generation *and* cluster — which cluster in which generation is carrying the lag.**\n\n**Nominal:** Every series low, or a new generation's series descending toward its counterpart in the old one.\n\n**Unhealthy:** One cluster's series flat and high while its siblings descend. That is a cluster-shaped problem — under-provisioned, or holding one expensive dataflow — rather than a rollout-shaped one, and it names the cluster to look at.\n\nThe breakdown behind `materialize.generations.lag.total`. The aggregate answers \"is this generation catching up\"; this answers \"where is what is left\", which is the next question whenever the answer to the first is \"not yet\".\n\nReading it during a rollout means comparing a cluster against *itself* across generations rather than against the other clusters: absolute totals depend on how many collections a cluster carries, so the pairing that matters is `gen N / u1` beside `gen N+1 / u1`.", "id": 1019, "links": [], "title": "Total Lag by Generation and Cluster", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No generations match the current filters", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lag-total-by-generation": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (generation) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) < 1e9\n )\n)\n", "legendFormat": "gen {{generation}}", "qryType": 1, "query": "sum by (generation) (\n max by (generation, instance_id, collection_id) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) < 1e9\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Every hydrated collection's lag in each generation, added together — how far behind the generation is in total.**\n\n**Nominal:** A new generation's total falling toward the serving generation's as it catches up.\n\n**Unhealthy:** A new generation whose total stops falling well above the old one's. It is producing results but is not closing the gap, so promoting it would move traffic onto staler data.\n\n**The panel that actually shows a rollout finishing.** The worst-case lag beside it is dominated by whichever single collection is furthest behind, so it stays high and jumpy while the generation is in fact catching up. The total falls steadily as collection after collection converges, which is the shape an operator is waiting to see before promoting.\n\nComplementary to _Hydration_ in the same way as the rest of this file: collections not yet producing results are excluded here (`< 1e9`) and counted there, so a rollout reads as one curve descending on that tab while this one descends here.\n\nDeduplicated across replicas by the inner `max`, like _Collections by Generation_ — a collection on two replicas would otherwise be added twice, and two replicas is the normal shape mid- rollout.", "id": 1018, "links": [], "title": "Total Lag by Generation", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No generations match the current filters", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lifecycle-events": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\"}\n | reportingcontroller=\"orchestratord.materialize.cloud\"\n | kind=\"Materialize\"\n | reason!=\"ReconciliationFailed\"\n | line_format \"{{.reason}} — {{.msg}}\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Every phase the Materialize resource moved through, as the operator reported it: `Applying`, `ReadyToPromote`, `WaitingForApproval`, `Promoting`, `Applied`, and the two that end a rollout badly, `RolloutTimeout` and `FailedDeploy`.**\n\n**Nominal:** A rollout reads as a short sequence ending in `Applied`. Nothing between rollouts, since a phase is only reported when the status actually changes.\n\n**Unhealthy:** `RolloutTimeout` or `FailedDeploy`. Both say the rollout stopped without reaching the requested state; neither says why, which is what _Operator Health -> Reconciliation Failures_ carries.\n\nThe resource's status conditions record only the phase it is in *now*, so a rollout that has moved on leaves no account of what it passed through. This is that account.\n\nSelected by `reportingcontroller` rather than by the operator's namespace: the event is filed against the `Materialize` resource, which lives in the environment's namespace rather than the operator's.\n\nOne line per transition rather than one per reconciliation pass: the operator publishes an event only when the status substantively moved, so a reconciler re-running on every watch event does not repeat itself here.\n\nA `WaitingForApproval` that persists is not a failure — the rollout is doing what it was configured to do and is waiting for a human to promote it.", "id": 1004, "links": [], "title": "Lifecycle Events", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No rollout activity in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "lifecycle-rate": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (reason) (\n count_over_time(\n {job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\"}\n | reportingcontroller=\"orchestratord.materialize.cloud\"\n | kind=\"Materialize\"\n | reason!=\"ReconciliationFailed\"\n [$__auto]\n )\n)\n", "legendFormat": "{{reason}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Lifecycle transitions over time, by phase — where a rollout got to, and when.**\n\nRead as a sequence rather than a rate: each point is one transition, and the interesting reading is which phases appear and in what order. A rollout that reaches `Promoting` and never reaches `Applied` stopped in promotion.", "id": 1003, "links": [], "title": "Lifecycle Transitions", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No rollout activity in this time range" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lifecycle-transitions": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (reason) (\n count_over_time(\n {job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\"}\n | reportingcontroller=\"orchestratord.materialize.cloud\"\n | kind=\"Materialize\"\n | reason!=\"ReconciliationFailed\"\n [$__auto]\n )\n)\n", "legendFormat": "{{reason}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Lifecycle transitions over time, by phase — where a rollout got to, and when.**\n\nRead as a sequence rather than a rate: each point is one transition, and the interesting reading is which phases appear and in what order. A rollout that reaches `Promoting` and never reaches `Applied` stopped in promotion.", "id": 1002, "links": [], "title": "Lifecycle Transitions", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE3377", "mode": "shades" }, "min": 0, "noValue": "No rollout activity in this time range" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value_and_name", "wideLayout": true } }, "version": "" } } }, "max-lag-now": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (generation) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) < 1e9\n)\n", "legendFormat": "gen {{generation}}", "qryType": 1, "query": "max by (generation) (\n label_replace(\n mz_dataflow_wallclock_lag_seconds{\n materialize_cloud_organization_name=~\"$environmentNameList\",\n pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\",\n instance_id=~\"$mzClusterList\",\n instance_id!=\"\",\n quantile=\"1\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n ) < 1e9\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The worst lag in each generation — how far behind real time its most-lagged collection is.**\n\n**Nominal:** Seconds, and comparable between generations. A new generation that has hydrated should track the old one closely.\n\n**Unhealthy:** A new generation whose lag stays far above the serving one's. It has hydrated — it is producing results — but it is not keeping up, so promoting it would move traffic onto staler data.\n\nHydrating collections are excluded (`< 1e9`), since their sentinel would swamp every real reading. That makes this and _Hydration -> Hydrating Collections by Generation_ complementary: this one covers the collections that are *producing results*, and that one counts the collections that do not.\n\nThe metric is a summary carrying only `quantile` `0` and `1`, so `quantile=\"1\"` is the worst case rather than a percentile.", "id": 1012, "links": [], "title": "Worst-Case Lag", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE7733", "mode": "shades" }, "min": 0, "noValue": "No generations match the current filters", "unit": "s" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value_and_name", "wideLayout": true } }, "version": "" } } }, "memory-by-generation": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (generation) (\n label_replace(\n container_memory_working_set_bytes{\n namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n)\n", "legendFormat": "gen {{generation}}", "qryType": 1, "query": "sum by (generation) (\n label_replace(\n container_memory_working_set_bytes{\n namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Memory used by each generation's pods.**\n\n**Nominal:** The new generation climbing as it rebuilds arrangements in memory, then levelling off near the old one's footprint.\n\n**Unhealthy:** A new generation that keeps climbing past the old one's footprint, or one whose pods are being OOM-killed — _Events -> Kubernetes Activity_ shows the restarts if so.\n\nWorking set rather than the container limit, so this is what the generation is really holding. Rehydration is the memory-hungriest part of a rollout, because both generations hold their arrangements at once.", "id": 1021, "links": [], "title": "Memory by Generation", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: cadvisor metrics are required (via kubelet)", "unit": "bytes" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "pass-duration": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.5, sum by (le) (\n rate(\n orchestratord_reconciliation_duration_seconds_bucket{\n namespace=~\"$operatorNamespace\"\n }\n [$__rate_interval]\n )\n))\n", "legendFormat": "p50", "qryType": 1, "query": "histogram_quantile(0.5, sum by (le) (\n rate(\n orchestratord_reconciliation_duration_seconds_bucket{\n namespace=~\"$operatorNamespace\"\n }\n [$__rate_interval]\n )\n))\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.9, sum by (le) (\n rate(\n orchestratord_reconciliation_duration_seconds_bucket{\n namespace=~\"$operatorNamespace\"\n }\n [$__rate_interval]\n )\n))\n", "legendFormat": "p90", "qryType": 1, "query": "histogram_quantile(0.9, sum by (le) (\n rate(\n orchestratord_reconciliation_duration_seconds_bucket{\n namespace=~\"$operatorNamespace\"\n }\n [$__rate_interval]\n )\n))\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99, sum by (le) (\n rate(\n orchestratord_reconciliation_duration_seconds_bucket{\n namespace=~\"$operatorNamespace\"\n }\n [$__rate_interval]\n )\n))\n", "legendFormat": "p99", "qryType": 1, "query": "histogram_quantile(0.99, sum by (le) (\n rate(\n orchestratord_reconciliation_duration_seconds_bucket{\n namespace=~\"$operatorNamespace\"\n }\n [$__rate_interval]\n )\n))\n" }, "version": "v0" }, "refId": "query-2" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long one reconciliation pass takes, at the 50th, 90th and 99th percentiles.**\n\n**Nominal:** Tens to hundreds of milliseconds. Reconciliation is a handful of Kubernetes API calls, not a long-running job.\n\n**Unhealthy:** Passes stretching into seconds, which points at the Kubernetes API rather than at Materialize — a slow or throttling API server, or an admission webhook the operator's writes have to wait on.\n\n**This is not how long a rollout takes.** A pass that is waiting on the new generation's pods returns promptly and asks to be run again rather than blocking, so this measures work done per pass, not the wall-clock length of the change. The rollout's own duration is the span between its first and last transition on the Events tab.\n\nThe histogram's buckets are deliberately coarse — 10ms, 50ms, 250ms, 1s, 5s, 30s — so a percentile here is the boundary of the bucket the value fell in rather than the value itself. Read it as an order of magnitude. That is the trade the operator made on purpose: finer buckets would cost several times the series for detail no operator question asks for.\n\nCovers the reconciler only, not the finalizer bookkeeping the controller does around it.", "id": 1028, "links": [], "title": "Pass Duration", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: the Materialize operator is not being scraped", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "pass-outcomes": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (outcome) (\n rate(\n orchestratord_reconciliations_total{namespace=~\"$operatorNamespace\"}\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{outcome}}", "qryType": 1, "query": "sum by (outcome) (\n rate(\n orchestratord_reconciliations_total{namespace=~\"$operatorNamespace\"}\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**What reconciliation passes concluded, by outcome. The shape of a rollout: `waiting` climbs while the new generation's pods come up, then gives way to `applied` when they are ready.**\n\n**Nominal:** `applied` with `waiting` alongside it during a change. Both are success.\n\n**Degraded:** `waiting` that stays high long after a change should have settled. The operator is still trying and the thing it is waiting for has not happened — usually pods that will not become ready.\n\n**Unhealthy:** A sustained `failed` rate. The pass is erroring rather than waiting, and _Steps -> Step Failures and Abandonments_ localizes which phase.\n\nThe five outcomes are the operator's own vocabulary: `applied` reached the desired state or found it already there; `waiting` made progress and asked to run again; `skipped` had nothing to act on, because what it manages is disabled or absent from the spec; `failed` returned an error; `abandoned` did not reach a conclusion.\n\n`abandoned` is rare at the pass level and is not a failure signal — it covers a pass cancelled by a leadership handoff or a shutdown as well as one that errored out.\n\n`apply` and `cleanup` passes are summed together here. Cleanup runs when a resource is being deleted, so it is rare, and splitting it out would double every series to distinguish a case that is usually absent.", "id": 1026, "links": [], "title": "Pass Outcomes", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: the Materialize operator is not being scraped", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "pods-now": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count by (generation) (\n group by (pod, generation) (\n label_replace(\n container_memory_working_set_bytes{\n namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n )\n)\n", "legendFormat": "gen {{generation}}", "qryType": 1, "query": "count by (generation) (\n group by (pod, generation) (\n label_replace(\n container_memory_working_set_bytes{\n namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n },\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Pods belonging to each generation — its environmentd and the cluster replicas standing behind it.**\n\n**Nominal:** A steady count per generation. During a rollout the new generation's count climbs as its replicas are created.\n\n**Unhealthy:** A new generation whose pod count stalls below the old one's. The pods it is missing never scheduled; _Events -> Kubernetes Activity_ names the reason, which is usually that the cluster has nowhere to put them.\n\nCounted from cAdvisor, so it counts pods that have *started*. A pod that is scheduled but not yet running does not appear, which is why a stalled rollout can show a count that simply stops climbing rather than one that drops.", "id": 1013, "links": [], "title": "Pods", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE7733", "mode": "shades" }, "decimals": 0, "min": 0, "noValue": "No generations match the current filters" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value_and_name", "wideLayout": true } }, "version": "" } } }, "reconciliation-failure-events": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\"}\n | reportingcontroller=\"orchestratord.materialize.cloud\"\n | reason=\"ReconciliationFailed\"\n | line_format \"{{.kind}}/{{.name}} — {{.msg}}\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Why the operator could not reconcile a resource. The event carries the error's whole cause chain, which is usually the actionable half — an admission webhook that is down, a secret that does not exist yet, a license key that will not parse.**\n\n**Nominal:** Nothing. A reconciliation that succeeds reports no failure.\n\n**Degraded:** A failure that appears once and stops. The operator retries on a backoff, and a cause that clears itself — a webhook still starting up, a resource created moments later — resolves without intervention.\n\n**Unhealthy:** Failures that keep arriving. The `Kind` says which resource is stuck: `Materialize` is the environment itself, `Balancer` and `Console` are the components around it, and a failure on those leaves the environment running but unreachable or unmanageable.\n\nRepeats of one failure aggregate into a single event carrying a count, rather than one line each. When the message changes — a missing secret first, the real cause once it appears — the operator files a fresh event, so a changing cause reads as several lines rather than one stale one.\n\nA `FailedDeploy` reports twice: once here with the cause, and once in _Rollout -> Lifecycle Events_ with the phase.", "id": 1006, "links": [], "title": "Reconciliation Failure Events", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No reconciliation failures in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "reconciliation-failure-rate": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (kind) (\n count_over_time(\n {job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\"}\n | reportingcontroller=\"orchestratord.materialize.cloud\"\n | reason=\"ReconciliationFailed\"\n [$__auto]\n )\n)\n", "legendFormat": "{{kind}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Reconciliation failures over time, by the kind of resource that failed.**\n\n**Nominal:** Flat at zero.\n\n**Unhealthy:** A sustained rate on any kind. The operator retries with a backoff that tops out around four minutes, so a steady low rate is a resource that has been stuck for as long as the line has been above zero.", "id": 1005, "links": [], "title": "Reconciliation Failures", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No reconciliation failures in this time range" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reconciliation-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (kind) (\n count_over_time(\n {job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\"}\n | reportingcontroller=\"orchestratord.materialize.cloud\"\n | reason=\"ReconciliationFailed\"\n [$__auto]\n )\n)\n", "legendFormat": "{{kind}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Reconciliation failures over time, by the kind of resource that failed.**\n\n**Nominal:** Flat at zero.\n\n**Unhealthy:** A sustained rate on any kind. The operator retries with a backoff that tops out around four minutes, so a steady low rate is a resource that has been stuck for as long as the line has been above zero.", "id": 1001, "links": [], "title": "Reconciliation Failures", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE3377", "mode": "shades" }, "min": 0, "noValue": "No reconciliation failures in this time range" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value_and_name", "wideLayout": true } }, "version": "" } } }, "reconciliation-rate": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n orchestratord_reconciliations_total{namespace=~\"$operatorNamespace\"}\n [$__rate_interval]\n )\n)\n", "legendFormat": "passes", "qryType": 1, "query": "sum(\n rate(\n orchestratord_reconciliations_total{namespace=~\"$operatorNamespace\"}\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Reconciliation passes per second across every controller — whether the loop is turning at all.**\n\n**Nominal:** A low steady rate, rising while something is changing. The controllers re-run on every watch event and on a backoff, so an idle operator still ticks over.\n\n**Unhealthy:** Flat at zero while work is outstanding. Check _Operator Status -> Reconciling Replicas_ first: an operator that lost the lease stops reconciling and this is what that looks like.\n\nNot a load signal. A high rate is normal during a rollout and says nothing about cost; it is the *absence* of passes that is diagnostic.", "id": 1024, "links": [], "title": "Reconciliation Rate", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#009988", "mode": "shades" }, "min": 0, "noValue": "No metrics: the Materialize operator is not being scraped", "unit": "reqps" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "reconciling-replicas": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n orchestratord_is_leader{namespace=~\"$operatorNamespace\"}\n)\n", "legendFormat": "leaders", "qryType": 1, "query": "sum(\n orchestratord_is_leader{namespace=~\"$operatorNamespace\"}\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How many operator replicas hold the leadership lease and are therefore reconciling. This should be exactly one.**\n\n**Nominal:** One. The lease is held, and something is acting on your resources.\n\n**Unhealthy:** Zero means no replica can take the lease and the operator is reconciling nothing — every other panel here goes quiet and nothing on the cluster changes, however healthy the pods look. The usual cause is the service account lacking permission on leases. More than one should not be possible; treat it as a stale reading rather than a real split.\n\nSummed across replicas on purpose. A non-leader exports the metric at zero rather than not at all, so this is a count of leaders and not a count of pods.", "id": 1022, "links": [], "title": "Reconciling Replicas", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No metrics: the Materialize operator is not being scraped", "thresholds": { "mode": "absolute", "steps": [ { "color": "#AB0003", "value": null }, { "color": "#A2F49B", "value": 1 }, { "color": "#AB0003", "value": 2 } ] } }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "step-activity": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (step) (\n rate(\n orchestratord_reconciliation_steps_total{namespace=~\"$operatorNamespace\"}\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{step}}", "qryType": 1, "query": "sum by (step) (\n rate(\n orchestratord_reconciliation_steps_total{namespace=~\"$operatorNamespace\"}\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Which phases of reconciliation are running, and how often. A rollout moves through these in order, so the set that is active says where the operator has got to.**\n\n**Nominal:** The steps of whichever controllers have work. `promote` and `teardown_generation` appear only around a deployment; the rest tick over with the loop.\n\nA step is the only level at which \"there was nothing to do\" is distinguishable from \"the desired state was reached\" — the reconciler interface reports both as success. That distinction is the `outcome` label, which this panel sums over; see _Steps -> Step Failures and Abandonments_ for the half that matters.", "id": 1030, "links": [], "title": "Step Activity", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: the Materialize operator is not being scraped", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "step-duration": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99, sum by (le, step) (\n rate(\n orchestratord_reconciliation_step_duration_seconds_bucket{\n namespace=~\"$operatorNamespace\"\n }\n [$__rate_interval]\n )\n))\n", "legendFormat": "{{step}}", "qryType": 1, "query": "histogram_quantile(0.99, sum by (le, step) (\n rate(\n orchestratord_reconciliation_step_duration_seconds_bucket{\n namespace=~\"$operatorNamespace\"\n }\n [$__rate_interval]\n )\n))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The slowest phase of a reconciliation pass, at the 99th percentile per step — where the time in a pass actually goes.**\n\n**Nominal:** Most steps in the low tens of milliseconds. The two that touch a generation's resources — `global_resources` and `generation_resources` — are the expensive ones, in the hundreds.\n\n**Unhealthy:** A step that leaves its usual band. Because each step is a group of Kubernetes API calls, a step that slows down names the resource kind whose writes are being held up.\n\nSame coarse buckets as the whole-pass histogram, and the same reading: an order of magnitude rather than a measurement. Steps and passes share one bucket set precisely so a step's latency reads against the pass it belongs to.", "id": 1029, "links": [], "title": "Step Duration (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: the Materialize operator is not being scraped", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "step-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (step, outcome) (\n rate(\n orchestratord_reconciliation_steps_total{\n namespace=~\"$operatorNamespace\", outcome=~\"failed|abandoned\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{step}} / {{outcome}}", "qryType": 1, "query": "sum by (step, outcome) (\n rate(\n orchestratord_reconciliation_steps_total{\n namespace=~\"$operatorNamespace\", outcome=~\"failed|abandoned\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Steps that did not complete, by step and by how they ended — the panel that turns \"reconciliation is failing\" into \"reconciliation is failing *here*\".**\n\n**Nominal:** Nothing.\n\n**Unhealthy:** Any sustained rate names the phase to investigate. Pair it with the matching event on _Events -> Operator Health_, which carries the cause; this panel says where and that one says why.\n\n**`abandoned` is not by itself a failure.** A step records it when it did not reach a conclusion, which covers an error propagating out of it *and* a pass cancelled by a leadership handoff or a shutdown — a `Drop` cannot tell those apart. So a burst of `abandoned` around an operator restart is expected. Alert on _Operator Status -> Failed Passes (Select Time Range)_, which is recorded from the reconciler's actual result and which a cancelled pass never reaches, and read this to locate the step.\n\nAn `abandoned` step alongside a `failed` pass is the useful pairing: the pass says it errored, and the step says where it was when it did.", "id": 1031, "links": [], "title": "Step Failures and Abandonments", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No incomplete steps", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "version-by-generation": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "group by (generation, mz_version) (\n label_replace(\n max_over_time(\n mz_compute_cluster_status{\n materialize_cloud_organization_name=~\"$environmentNameList\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n }\n [$__range]\n ),\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n)\n", "instant": true, "qryType": 1, "query": "group by (generation, mz_version) (\n label_replace(\n max_over_time(\n mz_compute_cluster_status{\n materialize_cloud_organization_name=~\"$environmentNameList\", pod=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\"\n }\n [$__range]\n ),\n \"generation\", \"$1\", \"pod\", \".*-(?:environmentd|gen)-([0-9]+)-[0-9]+\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "labelsToFields", "kind": "Transformation", "spec": { "options": { "keepLabels": [ "generation", "mz_version" ] } } }, { "group": "merge", "kind": "Transformation", "spec": { "options": {} } }, { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true, "Value": true, "mz_compute_cluster_status": true, "v2_mz_compute_cluster_status": true }, "indexByName": { "generation": 0, "mz_version": 1 } } } }, { "group": "sortBy", "kind": "Transformation", "spec": { "options": { "fields": {}, "sort": [ { "desc": true, "field": "generation" } ] } } } ] } }, "description": "**The Materialize version each generation is running — what the rollout is actually changing.**\n\n**Nominal:** One row between rollouts. Two during one, and the pair of versions is the upgrade you are performing: the older is what is serving now, the newer is what will serve once the new generation is promoted.\n\n**Unhealthy:** Two rows showing the *same* version. The rollout is replacing a generation with its own version, which is a restart rather than an upgrade — usually a forced rollout, and worth confirming that is what was intended before it costs a rehydration.\n\nRead off `compute_cluster_status`, whose `mz_version` label each generation's own environmentd reports, so the two sides genuinely disagree during a rollout rather than one overwriting the other. The string carries the build commit and the Helm chart version alongside the release.\n\n**Covers the whole selected time range, not the current instant.** A promoted generation is torn down, so a table evaluated at \"now\" would show only the survivor and a finished rollout would look like it never happened. `max_over_time` over the range is what keeps both sides present: widen the time picker to span a rollout and its old generation comes back, which is the same reason `$mzGenerationList` refreshes on a time-range change.\n\nEvaluated once rather than per step, so the table has one row per generation rather than one per generation per scrape.\n\nA generation appears here only once its environmentd is serving the SQL-derived metrics. A brand-new generation is therefore missing from this table for the first moments of its life, while it is already visible in _Rollout Status -> Pods_.", "id": 1014, "links": [], "title": "Version by Generation (Select Time Range)", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No generations match the current filters" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "warning-event-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\", level=\"WARN\"}\n | name=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\" or name!~\".*-(environmentd|gen)-([0-9]+)-[0-9]+\"\n | line_format \"{{.reason}} {{.kind}}/{{.name}} — {{.msg}}\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Kubernetes events the reporting component flagged as warnings — a pod that will not schedule, an image that will not pull, a container failing its probes.**\n\n**Nominal:** Nothing, or a brief flurry that stops on its own. Warnings during a rollout are expected while pods are replaced: a new pod is unschedulable for as long as it takes a node to appear.\n\n**Unhealthy:** Warnings that keep arriving after a rollout should have settled. Read the reason: `FailedScheduling` and `NotTriggerScaleUp` mean the cluster has nowhere to put the new pods, `BackOff` means they start and die, `Unhealthy` means they start and never become ready.\n\nWarning is the reporting component's own judgement, not this stack's. Kubernetes marks an event `Warning` when the component that emitted it thought something was wrong, so this is a filter on their opinion rather than a threshold of ours.", "id": 1008, "links": [], "title": "Warning Events", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No warnings in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "warning-events": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum(\n count_over_time(\n {job=\"loki.source.kubernetes_events\", namespace=~\"${operatorNamespace:regex}|${mzNamespaceList:regex}\", level=\"WARN\"}\n | name=~\".*-(environmentd|gen)-(${mzGenerationList:regex})-[0-9]+\" or name!~\".*-(environmentd|gen)-([0-9]+)-[0-9]+\"\n [$__auto]\n )\n)\n", "legendFormat": "warnings" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Warning events per interval across both namespaces, as one series — the at-a-glance answer to whether anything is complaining right now.**\n\n**Nominal:** Flat at zero, or a spike that returns to zero.\n\n**Unhealthy:** A level that does not come back down. Read _Kubernetes Activity -> Warning Events_ for which reasons make it up.", "id": 1000, "links": [], "title": "Warning Events", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE3377", "mode": "shades" }, "min": 0, "noValue": "No warnings in this time range" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } } }, "layout": { "kind": "TabsLayout", "spec": { "tabs": [ { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "warning-events" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reconciliation-failures" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lifecycle-transitions" } } } ], "maxColumnCount": 3, "rowHeightMode": "short" } }, "title": "Event Summary" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lifecycle-rate" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lifecycle-events" } } } ], "maxColumnCount": 2, "rowHeightMode": "tall" } }, "title": "Rollout" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reconciliation-failure-rate" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reconciliation-failure-events" } } } ], "maxColumnCount": 2, "rowHeightMode": "tall" } }, "title": "Operator Health" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "event-rate-by-reason" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "warning-event-feed" } } } ], "maxColumnCount": 2, "rowHeightMode": "tall" } }, "title": "Kubernetes Activity" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": true, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "all-event-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "All Events" } } ] } }, "title": "Events" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "active-generations" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "hydrating-now" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "max-lag-now" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "pods-now" } } } ], "maxColumnCount": 4, "rowHeightMode": "short" } }, "title": "Rollout Status" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "version-by-generation" } } } ], "maxColumnCount": 1, "rowHeightMode": "short" } }, "title": "Versions" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "hydrating-by-generation" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "collections-by-generation" } } } ], "maxColumnCount": 2, "rowHeightMode": "tall" } }, "title": "Hydration" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lag-by-generation" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lag-total-by-generation" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lag-total-by-cluster" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Freshness" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "cpu-by-generation" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "memory-by-generation" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Footprint" } } ] } }, "title": "Generations" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reconciling-replicas" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "environments-needing-update" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reconciliation-rate" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "failed-passes" } } } ], "maxColumnCount": 4, "rowHeightMode": "short" } }, "title": "Operator Status" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "pass-outcomes" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "failures-by-controller" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Reconciliation Passes" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "pass-duration" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "step-duration" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Duration" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "step-activity" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "step-failures" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Steps" } } ] } }, "title": "Reconciliation" } } ] } }, "links": [], "liveNow": false, "preload": false, "tags": [ "materialize", "mzmon", "upgrade" ], "timeSettings": { "autoRefresh": "", "autoRefreshIntervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "fiscalYearStartMonth": 0, "from": "now-6h", "hideTimepicker": false, "timezone": "browser", "to": "now" }, "title": "Materialize Upgrade", "variables": [ { "kind": "DatasourceVariable", "spec": { "allowCustomValue": false, "current": { "text": "", "value": "" }, "description": "Datasource for metrics queries", "hide": "dontHide", "includeAll": false, "label": "Metrics Datasource", "multi": false, "name": "metricsDatasource", "options": [], "pluginId": "prometheus", "refresh": "never", "regex": "", "skipUrlSync": false } }, { "kind": "DatasourceVariable", "spec": { "allowCustomValue": false, "current": { "text": "", "value": "" }, "description": "Datasource for log and event queries", "hide": "dontHide", "includeAll": false, "label": "Logs Datasource", "multi": false, "name": "logsDatasource", "options": [], "pluginId": "loki", "refresh": "never", "regex": "", "skipUrlSync": false } }, { "kind": "QueryVariable", "spec": { "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(orchestratord_is_leader, namespace)", "description": "The namespace the Materialize operator (orchestratord) runs in", "hide": "dontHide", "includeAll": false, "label": "Operator Namespace", "multi": false, "name": "operatorNamespace", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(orchestratord_is_leader, namespace)", "qryType": 1, "query": "label_values(orchestratord_is_leader, namespace)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".*", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(mz_compute_commands_total, materialize_cloud_organization_name)", "description": "The current environment to view", "hide": "dontHide", "includeAll": false, "label": "Environment", "multi": false, "name": "environmentNameList", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(mz_compute_commands_total, materialize_cloud_organization_name)", "qryType": 1, "query": "label_values(mz_compute_commands_total, materialize_cloud_organization_name)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(mz_compute_commands_total{ materialize_cloud_organization_name=~\"$environmentNameList\" }, materialize_cloud_organization_namespace)", "description": "The current materialize namespace where environments live", "hide": "hideVariable", "includeAll": true, "label": "Materialize Namespace", "multi": true, "name": "mzNamespaceList", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(mz_compute_commands_total{ materialize_cloud_organization_name=~\"$environmentNameList\" }, materialize_cloud_organization_namespace)", "qryType": 1, "query": "label_values(mz_compute_commands_total{ materialize_cloud_organization_name=~\"$environmentNameList\" }, materialize_cloud_organization_namespace)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": true, "sort": "alphabeticalAsc" } }, { "kind": "SwitchVariable", "spec": { "current": ".*", "description": "Whether to include materialize system clusters in the cluster list.", "disabledValue": "^[^s].*", "enabledValue": ".*", "hide": "inControlsMenu", "label": "Include System Clusters", "name": "includeSystemClusters", "skipUrlSync": true } }, { "kind": "QueryVariable", "spec": { "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "query_result(mz_compute_cluster_status{materialize_cloud_organization_name=~\"$environmentNameList\", compute_cluster_id=~\"$includeSystemClusters\" })", "description": "The cluster within the current environment to filter to", "hide": "inControlsMenu", "includeAll": true, "label": "Cluster", "multi": true, "name": "mzClusterList", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "query_result(mz_compute_cluster_status{materialize_cloud_organization_name=~\"$environmentNameList\", compute_cluster_id=~\"$includeSystemClusters\" })", "qryType": 1, "query": "query_result(mz_compute_cluster_status{materialize_cloud_organization_name=~\"$environmentNameList\", compute_cluster_id=~\"$includeSystemClusters\" })" }, "version": "v0" }, "refresh": "never", "regex": ".*compute_cluster_id=\\\"(?[^\\\"]+)\\\",.*compute_cluster_name=\\\"(?[^\\\"]+)\\\",.*", "regexApplyTo": "value", "skipUrlSync": false, "sort": "naturalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".*", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(mz_compute_commands_total{materialize_cloud_organization_name=~\"$environmentNameList\", instance_id=~\"$mzClusterList\"}, replica_id)", "description": "The replica within the current cluster to filter to", "hide": "inControlsMenu", "includeAll": true, "label": "Replica", "multi": true, "name": "mzReplicaList", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(mz_compute_commands_total{materialize_cloud_organization_name=~\"$environmentNameList\", instance_id=~\"$mzClusterList\"}, replica_id)", "qryType": 1, "query": "label_values(mz_compute_commands_total{materialize_cloud_organization_name=~\"$environmentNameList\", instance_id=~\"$mzClusterList\"}, replica_id)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "naturalAsc" } }, { "kind": "QueryVariable", "spec": { "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(mz_compute_commands_total{materialize_cloud_organization_name=~\"$environmentNameList\"}, pod)", "description": "The deployment generation(s) to filter to, for blue/green rollouts", "hide": "dontHide", "includeAll": true, "label": "Generation", "multi": true, "name": "mzGenerationList", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(mz_compute_commands_total{materialize_cloud_organization_name=~\"$environmentNameList\"}, pod)", "qryType": 1, "query": "label_values(mz_compute_commands_total{materialize_cloud_organization_name=~\"$environmentNameList\"}, pod)" }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "/.*-environmentd-(?[0-9]+)-[0-9]+/", "regexApplyTo": "value", "skipUrlSync": false, "sort": "numericalDesc" } }, { "datasource": { "name": "$metricsDatasource" }, "group": "", "kind": "AdhocVariable", "spec": { "allowCustomValue": true, "baseFilters": [ { "key": "namespace", "operator": "=~", "origin": "dashboard", "value": "$mzNamespaceList" } ], "defaultKeys": [], "description": "Adhoc filters to apply to all metrics queries", "enableGroupBy": false, "filters": [], "hide": "inControlsMenu", "label": "Advanced Metric Filter", "name": "metricAdhoc", "skipUrlSync": false } }, { "datasource": { "name": "$logsDatasource" }, "group": "", "kind": "AdhocVariable", "spec": { "allowCustomValue": true, "baseFilters": [], "defaultKeys": [], "description": "Adhoc filters to apply to all logs queries", "enableGroupBy": false, "filters": [], "hide": "inControlsMenu", "label": "Advanced Log Filter", "name": "logsAdhoc", "skipUrlSync": false } } ] } }