{ "apiVersion": "dashboard.grafana.app/v2", "kind": "Dashboard", "metadata": { "annotations": { "grafana.app/folder": "materialize", "monitoring.materialize.cloud/min-mz-version": "v26.24.0", "monitoring.materialize.cloud/rec-mz-version": "v26.50.0", "monitoring.materialize.cloud/sql-metric-prefix": "mz_", "monitoring.materialize.cloud/target-export": "generic" }, "name": "mz-mon-env-ingress" }, "spec": { "annotations": [ { "kind": "AnnotationQuery", "spec": { "builtIn": true, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "query": { "datasource": { "name": "-- Grafana --" }, "group": "grafana", "kind": "DataQuery", "spec": {}, "version": "v0" } } } ], "cursorSync": "Crosshair", "description": "How users reach a Materialize environment.\n\nRate, errors and latency at each hop: the cloud load balancer, where its metrics are pulled; balancerd, which every SQL and HTTPS connection passes through; the console, which serves the web app and forwards its API calls; and environmentd's SQL and HTTP APIs behind them both.", "editable": true, "elements": { "api-console-commands": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (application_name, status) (\n rate(mz_adapter_commands{materialize_cloud_organization_name=~\"$environmentNameList\", application_name=~\"web_console|web_console_shell\"}[$__rate_interval])\n)\n", "legendFormat": "{{application_name}} {{status}}", "qryType": 1, "query": "sum by (application_name, status) (\n rate(mz_adapter_commands{materialize_cloud_organization_name=~\"$environmentNameList\", application_name=~\"web_console|web_console_shell\"}[$__rate_interval])\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**SQL commands the console ran per second, by part of the console and outcome.**\n\n**Nominal:** `web_console` is the console's own queries, which it runs to draw every page. `web_console_shell` is what users type into the SQL shell.\n\n**Degraded:** `error` from `web_console` is the console failing to load data, which users see as empty or broken pages. `error` from the shell is usually users' own SQL errors.", "id": 1063, "links": [], "title": "Console SQL Commands", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "The console ran no SQL in this environment in this time range.", "unit": "ops" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "api-console-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le, application_name) (\n rate(mz_time_to_first_row_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\", application_name=~\"web_console|web_console_shell\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{application_name}}", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le, application_name) (\n rate(mz_time_to_first_row_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\", application_name=~\"web_console|web_console_shell\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long the slowest 1% of the console's queries take to return their first row.**\n\n**Nominal:** Steady for `web_console`, whose queries read Materialize's catalog. The shell follows whatever users run.\n\n**Degraded:** Slow `web_console` queries make every console page slow. They run on the `mz_catalog_server` cluster, so check its CPU first.", "id": 1064, "links": [], "title": "Console Query Time to First Row (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "The console ran no queries in this environment in this time range.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "api-http-errors": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (path, status) (rate(mz_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", source=\"external\", status=~\"5..\"}[$__rate_interval]))\n", "legendFormat": "{{path}} {{status}}", "qryType": 1, "query": "sum by (path, status) (rate(mz_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", source=\"external\", status=~\"5..\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**HTTP API requests per second that environmentd answered with a server error, by path and status code.**\n\n**Nominal:** Zero.\n\n**Degraded:** environmentd itself failed the request. A query error is not a server error: `/api/sql` reports those inside a `200` response.", "id": 1059, "links": [], "title": "HTTP Server Errors by Path", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "environmentd answered no HTTP request with a server error in this time range.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "api-http-requests": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (path) (rate(mz_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", source=\"external\"}[$__rate_interval]))\n", "legendFormat": "{{path}}", "qryType": 1, "query": "sum by (path) (rate(mz_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", source=\"external\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**HTTP API requests environmentd served per second, by path.**\n\n**Nominal:** `/api/sql` is SQL over HTTP, from the console and from HTTP clients. `/api/experimental/sql` is a WebSocket upgrade, one per streaming session.\n\nOnly requests that arrived through balancerd. The operator's own calls and the metric scrapes are left out.", "id": 1058, "links": [], "title": "HTTP Requests by Path", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No metrics from environmentd in this environment. Check the environment picker, and that Materialize is running.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "api-http-response": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le, path) (\n rate(mz_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\", source=\"external\", path!=\"/api/experimental/sql\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{path}}", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le, path) (\n rate(mz_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\", source=\"external\", path!=\"/api/experimental/sql\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The time the slowest 1% of HTTP API requests take, by path.**\n\n**Nominal:** For `/api/sql`, the slowest queries clients run over HTTP.\n\nWebSocket upgrades are left out: their time is the handshake, not the session.", "id": 1060, "links": [], "title": "HTTP Response Time (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No metrics from environmentd in this environment. Check the environment picker, and that Materialize is running.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "api-pgwire": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (status) (rate(mz_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\", source=\"external\"}[$__rate_interval]))\n", "legendFormat": "{{status}}", "qryType": 1, "query": "sum by (status) (rate(mz_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\", source=\"external\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**SQL connections environmentd finished per second, by outcome, as environmentd counts them.**\n\n**Nominal:** Close to balancerd's `pgwire` rate on _Balancer -> Connection Outcomes_. Each SQL connection is one connection at each hop.\n\nenvironmentd's `error` is any connection that ended in an error, whichever side caused it, so it does not match balancerd's.", "id": 1062, "links": [], "title": "SQL Connections Completed", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No metrics from environmentd in this environment. Check the environment picker, and that Materialize is running.", "unit": "cps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "api-sessions": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (session_type) (mz_active_sessions{materialize_cloud_organization_name=~\"$environmentNameList\"})\n", "legendFormat": "{{session_type}}", "qryType": 1, "query": "sum by (session_type) (mz_active_sessions{materialize_cloud_organization_name=~\"$environmentNameList\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Open SQL sessions, split into `system` (Materialize's internal probing — a few are always present) and `user` (client connections).**\n\n**Degraded:** A `user` count that climbs and never settles usually means a client is opening connections and never closing them; cross-check whether query activity rose to match, or whether connections are just accumulating idle.\n\nEnvironment-scoped — the cluster/replica selectors don't narrow it.", "id": 1061, "links": [], "title": "Open Sessions", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 0, "noValue": "No metrics from environmentd in this environment. Check the environment picker, and that Materialize is running.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-accepting": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "min by (pod) (mz_balancer_connection_accepting{materialize_cloud_organization_name=~\"$environmentNameList\"})\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "min by (pod) (mz_balancer_connection_accepting{materialize_cloud_organization_name=~\"$environmentNameList\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Whether each balancerd pod is in the load balancer's rotation: 1 in, 0 out.**\n\n**Nominal:** 1 everywhere, all the time.\n\n**Degraded:** A pod at 0 passed its high watermark. It keeps its connections and rejoins at its low watermark.", "id": 1036, "links": [], "title": "Taking New Connections by Pod", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "max": 1, "min": 0, "noValue": "balancerd does not report its capacity state. Older Materialize versions don't publish it.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-autoscaler-conditions": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_current_replicas{namespace=~\"$mzNamespaceList\"})\n >= bool\n max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas{namespace=~\"$mzNamespaceList\"})\n)\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n", "legendFormat": "at its maximum", "qryType": 1, "query": "(\n max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_current_replicas{namespace=~\"$mzNamespaceList\"})\n >= bool\n max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas{namespace=~\"$mzNamespaceList\"})\n)\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (namespace, horizontalpodautoscaler) (\n kube_horizontalpodautoscaler_status_condition{namespace=~\"$mzNamespaceList\", condition=\"ScalingActive\", status=\"false\"}\n)\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n", "legendFormat": "cannot read its metrics", "qryType": 1, "query": "max by (namespace, horizontalpodautoscaler) (\n kube_horizontalpodautoscaler_status_condition{namespace=~\"$mzNamespaceList\", condition=\"ScalingActive\", status=\"false\"}\n)\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (namespace, horizontalpodautoscaler) (\n kube_horizontalpodautoscaler_status_condition{namespace=~\"$mzNamespaceList\", condition=\"AbleToScale\", status=\"false\"}\n)\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n", "legendFormat": "cannot scale", "qryType": 1, "query": "max by (namespace, horizontalpodautoscaler) (\n kube_horizontalpodautoscaler_status_condition{namespace=~\"$mzNamespaceList\", condition=\"AbleToScale\", status=\"false\"}\n)\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n" }, "version": "v0" }, "refId": "query-2" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Whether balancerd's autoscaler is at its maximum, cannot read its metrics, or cannot scale. Each line is 1 while it holds.**\n\n**Nominal:** All three at 0.\n\n**Unhealthy:** `cannot read its metrics` holds the pods at their current count. The autoscaler cannot see the metric it scales on, usually because metrics-server or a custom metrics adapter is down, or because the pods set no CPU request. `cannot scale` means it cannot change the Deployment at all.", "id": 1046, "links": [], "title": "Autoscaler Conditions", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "max": 1, "min": 0, "noValue": "No metrics: kube-state-metrics is required", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-autoscaler-events": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{job=\"loki.source.kubernetes_events\", namespace=~\"$mzNamespaceList\"}\n | reportingcontroller=\"horizontal-pod-autoscaler\"\n | line_format \"{{.reason}} {{.name}} — {{.msg}}\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**What the environment's HorizontalPodAutoscalers did and why, newest first.**\n\n**Nominal:** `SuccessfulRescale` lines naming the metric that moved the replica count.\n\n**Degraded:** `FailedGetResourceMetric` or `FailedComputeMetricsReplicas` is an autoscaler that could not read its metric. A few while a pod starts are normal; a steady stream is not.", "id": 1047, "links": [], "title": "Autoscaler Events", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No autoscaler in this environment recorded an event in this time range." }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "balancer-autoscaler-metrics": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n max by (namespace, horizontalpodautoscaler, metric_name, metric_target_type) (\n kube_horizontalpodautoscaler_status_target_metric{namespace=~\"$mzNamespaceList\"}\n )\n / on (namespace, horizontalpodautoscaler, metric_name, metric_target_type)\n max by (namespace, horizontalpodautoscaler, metric_name, metric_target_type) (\n kube_horizontalpodautoscaler_spec_target_metric{namespace=~\"$mzNamespaceList\"} > 0\n )\n)\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n", "legendFormat": "{{metric_name}}", "qryType": 1, "query": "(\n max by (namespace, horizontalpodautoscaler, metric_name, metric_target_type) (\n kube_horizontalpodautoscaler_status_target_metric{namespace=~\"$mzNamespaceList\"}\n )\n / on (namespace, horizontalpodautoscaler, metric_name, metric_target_type)\n max by (namespace, horizontalpodautoscaler, metric_name, metric_target_type) (\n kube_horizontalpodautoscaler_spec_target_metric{namespace=~\"$mzNamespaceList\"} > 0\n )\n)\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Each metric balancerd's autoscaler scales on, as a share of its target.**\n\n**Nominal:** Below 100% while load is steady.\n\n**Degraded:** Well above 100%, the autoscaler adds pods until it reaches its maximum. A metric that stays above 100% at the maximum is load the pods cannot absorb.\n\nA utilization target is a share of each pod's request, so the CPU line is _CPU vs Request_ measured against the target.", "id": 1045, "links": [], "title": "Scaling Metrics Against Their Targets", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: kube-state-metrics is required", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-autoscaler-replicas": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_current_replicas{namespace=~\"$mzNamespaceList\"})\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n", "legendFormat": "{{horizontalpodautoscaler}} current", "qryType": 1, "query": "max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_current_replicas{namespace=~\"$mzNamespaceList\"})\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_desired_replicas{namespace=~\"$mzNamespaceList\"})\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n", "legendFormat": "{{horizontalpodautoscaler}} desired", "qryType": 1, "query": "max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_status_desired_replicas{namespace=~\"$mzNamespaceList\"})\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_min_replicas{namespace=~\"$mzNamespaceList\"})\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n", "legendFormat": "{{horizontalpodautoscaler}} minimum", "qryType": 1, "query": "max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_min_replicas{namespace=~\"$mzNamespaceList\"})\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n" }, "version": "v0" }, "refId": "query-2" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas{namespace=~\"$mzNamespaceList\"})\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n", "legendFormat": "{{horizontalpodautoscaler}} maximum", "qryType": 1, "query": "max by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_spec_max_replicas{namespace=~\"$mzNamespaceList\"})\nand on (namespace, horizontalpodautoscaler)\nmax by (namespace, horizontalpodautoscaler) (kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"})\n" }, "version": "v0" }, "refId": "query-3" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The replicas balancerd's HorizontalPodAutoscaler runs and wants, between the bounds it scales within.**\n\n**Nominal:** Current follows desired, between the minimum and the maximum.\n\n**Degraded:** Desired above current is a scale-up under way, or one whose new pods cannot be scheduled. Current at the maximum cannot grow any further: raise the maximum, or each pod's capacity.", "id": 1044, "links": [], "title": "Autoscaler Replicas", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No metrics: kube-state-metrics is required", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-capacity": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (mz_balancer_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (mz_balancer_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max(mz_balancer_connection_limit{materialize_cloud_organization_name=~\"$environmentNameList\"}) > 0\n", "legendFormat": "limit", "qryType": 1, "query": "max(mz_balancer_connection_limit{materialize_cloud_organization_name=~\"$environmentNameList\"}) > 0\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max(mz_balancer_connection_high_watermark{materialize_cloud_organization_name=~\"$environmentNameList\"}) > 0\n", "legendFormat": "high watermark", "qryType": 1, "query": "max(mz_balancer_connection_high_watermark{materialize_cloud_organization_name=~\"$environmentNameList\"}) > 0\n" }, "version": "v0" }, "refId": "query-2" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max(mz_balancer_connection_low_watermark{materialize_cloud_organization_name=~\"$environmentNameList\"}) > 0\n", "legendFormat": "low watermark", "qryType": 1, "query": "max(mz_balancer_connection_low_watermark{materialize_cloud_organization_name=~\"$environmentNameList\"}) > 0\n" }, "version": "v0" }, "refId": "query-3" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Each balancerd pod's open connections, against the limits it enforces.**\n\n**Nominal:** Well under the high watermark, where one is drawn. Without a limit or watermarks configured, only the pods are drawn.\n\n**Degraded:** A pod at its **high watermark** leaves the load balancer's rotation and takes no new connections until it falls to its **low watermark**. A pod at its **limit** refuses connections outright.\n\nThe limit and the watermarks are the same on every pod.", "id": 1035, "links": [], "title": "Open Connections Against the Limit", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-cpu": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"}[$__rate_interval]))\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (rate(container_cpu_usage_seconds_total{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max(\n max by (namespace, pod) (\n kube_pod_container_resource_requests{namespace=~\"$mzNamespaceList\", container=\"balancerd\", resource=\"cpu\"}\n )\n)\n", "legendFormat": "request", "qryType": 1, "query": "max(\n max by (namespace, pod) (\n kube_pod_container_resource_requests{namespace=~\"$mzNamespaceList\", container=\"balancerd\", resource=\"cpu\"}\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**CPU each balancerd pod uses, in cores, against the CPU each one requests.**\n\n**Nominal:** Below the request, and roughly even across pods.\n\n**Degraded:** One pod well above the others is carrying more connections. Compare _Open Connections by Pod_.", "id": 1041, "links": [], "title": "CPU by Pod", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: cadvisor metrics are required (via kubelet)", "unit": "cores" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-lifetime": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.50,\n sum by (le, source) (\n rate(mz_balancer_connection_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{source}} p50", "qryType": 1, "query": "histogram_quantile(0.50,\n sum by (le, source) (\n rate(mz_balancer_connection_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le, source) (\n rate(mz_balancer_connection_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{source}} p99", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le, source) (\n rate(mz_balancer_connection_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long connections stay open, from accept to close, by protocol.**\n\n**Nominal:** Whatever the clients do. HTTP API calls last seconds, and pooled SQL connections last hours.\n\n**Degraded:** A sudden drop for `pgwire` while clients did not change means something is cutting sessions short: an environmentd restart, or a load balancer idle timeout shorter than the clients expect.\n\nMeasured from one second up, so connections shorter than that all read as one second.", "id": 1040, "links": [], "title": "Connection Lifetime", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No connection timings from balancerd. Older Materialize versions don't publish them.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-logs": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n namespace=~\"$mzNamespaceList\",\n app=\"balancerd\",\n level=~\"WARN|WARNING|ERROR|CRITICAL|FATAL\"\n}\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**What balancerd logged at warning level or above.**\n\n**Nominal:** A shutdown notice each time a pod is replaced.\n\n**Degraded:** `failed to connect to upstream server` is environmentd refusing or not answering. Lines about the connection limit or the watermarks mark a pod running out of room.", "id": 1049, "links": [], "title": "Warnings and Errors", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "balancerd logged no warnings or errors in this time range." }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "balancer-memory": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (pod) (container_memory_working_set_bytes{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"})\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (pod) (container_memory_working_set_bytes{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"})\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max(container_spec_memory_limit_bytes{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"} > 0)\n", "legendFormat": "limit", "qryType": 1, "query": "max(container_spec_memory_limit_bytes{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"} > 0)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Memory each balancerd pod uses, against its limit.**\n\n**Nominal:** Flat, and well below the limit.\n\n**Unhealthy:** Steady growth toward the limit ends in an OOM kill, which drops the pod's connections.\n\nWorking set.", "id": 1042, "links": [], "title": "Memory by Pod", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: cadvisor metrics are required (via kubelet)", "unit": "bytes" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-no-autoscaler": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [], "queryOptions": {}, "transformations": [] } }, "description": "Why this tab has no autoscaler row, and how to add one.", "id": 1048, "links": [], "title": "No Autoscaler", "vizConfig": { "group": "text", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": {}, "overrides": [] }, "options": { "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false }, "content": "**No HorizontalPodAutoscaler scales balancerd in this environment.**\n\nThe Materialize operator sets balancerd's replica count from the Materialize resource's `balancerdReplicas`. To scale it with load instead, set `balancerdExternallyScaled: true` on the Materialize resource, so that the operator stops resetting the count, and point a HorizontalPodAutoscaler at the environment's `…-balancerd` Deployment. Materialize versions before v26.48.0 reset any count an autoscaler sets.\n\nThe Resources row shows the CPU an autoscaler would scale on, against each pod's request.", "mode": "markdown" } }, "version": "" } } }, "balancer-open-by-pod": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (mz_balancer_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (mz_balancer_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Open connections on each balancerd pod.**\n\n**Nominal:** Roughly even. The load balancer in front spreads new connections across the pods.\n\n**Degraded:** One pod holding most of them is normal for a while after a pod restarts, because existing connections stay where they are. A lasting imbalance usually means the load balancer is not routing to every pod.", "id": 1033, "links": [], "title": "Open Connections by Pod", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-outcomes": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (source, status) (rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n", "legendFormat": "{{source}} {{status}}", "qryType": 1, "query": "sum by (source, status) (rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Connections balancerd finished handling per second, by protocol and outcome.**\n\n**Nominal:** All `success`.\n\n**Degraded:** `error` is a connection balancerd or environmentd could not serve. The client's own failures, such as a wrong password or a dropped network, are `success` here.", "id": 1032, "links": [], "title": "Connection Outcomes", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "unit": "cps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-refused": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(rate(mz_balancer_connection_rejected_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n", "legendFormat": "connection limit", "qryType": 1, "query": "sum(rate(mz_balancer_connection_rejected_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (source) (rate(mz_balancer_pre_resolved_timeout_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n", "legendFormat": "{{source}} timed out before routing", "qryType": 1, "query": "sum by (source) (rate(mz_balancer_pre_resolved_timeout_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Connections balancerd closed before passing them on, per second, by reason.**\n\n**Nominal:** Zero for both.\n\n**Degraded:** **Connection limit:** a pod was full and refused the connection. More balancerd replicas, or a higher limit, are the fixes. **Timed out before routing:** a connection did not finish its setup in time. Usually clients that open a connection and send nothing, such as a port scanner or a misconfigured health check.", "id": 1037, "links": [], "title": "Refused Connections", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "unit": "cps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-setup": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.50,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{source}} p50", "qryType": 1, "query": "histogram_quantile(0.50,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.90,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{source}} p90", "qryType": 1, "query": "histogram_quantile(0.90,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{source}} p99", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-2" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long new connections wait before balancerd passes them to environmentd, by protocol, at three quantiles.**\n\n**Nominal:** Steady, with p99 a small multiple of p50.\n\n**Degraded:** A p99 far above p50 is a few slow connections rather than a slow balancerd, often clients on a slow network finishing TLS. Compare with _Time to Reach environmentd_.", "id": 1038, "links": [], "title": "Connection Setup Quantiles", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No connection timings from balancerd. Older Materialize versions don't publish them.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-summary-cpu": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(rate(container_cpu_usage_seconds_total{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"}[$__rate_interval]))\n/\nsum(\n max by (namespace, pod) (\n kube_pod_container_resource_requests{namespace=~\"$mzNamespaceList\", container=\"balancerd\", resource=\"cpu\"}\n )\n)\n", "legendFormat": "cpu", "qryType": 1, "query": "sum(rate(container_cpu_usage_seconds_total{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"}[$__rate_interval]))\n/\nsum(\n max by (namespace, pod) (\n kube_pod_container_resource_requests{namespace=~\"$mzNamespaceList\", container=\"balancerd\", resource=\"cpu\"}\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**CPU balancerd's pods use, as a share of the CPU they request.**\n\n**Nominal:** Well under 100%.\n\n**Degraded:** Above 100%, the pods use more than the scheduler set aside for them, so a busy node can starve them. Raise the request in the Materialize resource's `balancerdResourceRequirements`.\n\nA HorizontalPodAutoscaler's CPU utilization target is a share of this same request.", "id": 1029, "links": [], "title": "CPU vs Request", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: cadvisor and kube-state-metrics are required", "thresholds": { "mode": "absolute", "steps": [ { "color": "#CEFFFF", "value": null }, { "color": "#F94902", "value": 0.8 }, { "color": "#E40515", "value": 0.9 }, { "color": "#AB0003", "value": 1 } ] }, "unit": "percentunit" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "balancer-summary-memory": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max(\n max by (namespace, pod) (container_memory_working_set_bytes{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"})\n /\n max by (namespace, pod) (container_spec_memory_limit_bytes{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"} > 0)\n)\n", "legendFormat": "busiest pod", "qryType": 1, "query": "max(\n max by (namespace, pod) (container_memory_working_set_bytes{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"})\n /\n max by (namespace, pod) (container_spec_memory_limit_bytes{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"} > 0)\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Memory the busiest balancerd pod uses, as a share of its limit.**\n\n**Nominal:** Well under 100%.\n\n**Unhealthy:** A pod that reaches its limit is OOM-killed, and every connection it holds drops. Raise the limit in the Materialize resource's `balancerdResourceRequirements`.\n\nWorking set. Empty when balancerd has no memory limit.", "id": 1030, "links": [], "title": "Memory vs Limit", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "balancerd has no memory limit, or the kubelet's container metrics are not scraped.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#CEFFFF", "value": null }, { "color": "#F94902", "value": 0.8 }, { "color": "#E40515", "value": 0.9 }, { "color": "#AB0003", "value": 1 } ] }, "unit": "percentunit" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "balancer-summary-open": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(mz_balancer_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n", "legendFormat": "connections", "qryType": 1, "query": "sum(mz_balancer_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Connections balancerd is passing to environmentd right now, SQL and HTTPS together.**\n\n**Nominal:** Whatever the environment's clients keep open. Connection poolers and BI tools hold steady numbers.\n\nEvery open connection is also a session on environmentd. See _SQL and HTTP API -> Open Sessions_.", "id": 1026, "links": [], "title": "Open Connections", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#33BBEE", "mode": "shades" }, "decimals": 0, "min": 0, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "unit": "short" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "balancer-summary-ready": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n max by (namespace, deployment) (\n kube_deployment_status_replicas_ready{namespace=~\"$mzNamespaceList\", deployment=~\".+-balancerd\"}\n )\n)\n", "legendFormat": "ready", "qryType": 1, "query": "sum(\n max by (namespace, deployment) (\n kube_deployment_status_replicas_ready{namespace=~\"$mzNamespaceList\", deployment=~\".+-balancerd\"}\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**balancerd pods passing their readiness check.**\n\n**Nominal:** The replica count the Materialize resource, or an autoscaler, asks for.\n\n**Degraded:** Fewer means pods are starting, crashing, or waiting for a node. _Unavailable Pods_ counts the shortfall.\n\nkube-state-metrics reports each Deployment once per replica of its own, so the query keeps one copy.", "id": 1027, "links": [], "title": "Ready Pods", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No metrics: kube-state-metrics is required", "thresholds": { "mode": "absolute", "steps": [ { "color": "#AB0003", "value": null }, { "color": "#F19903", "value": 1 }, { "color": "#A2F49B", "value": 1 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "balancer-summary-restart": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "time() - max(container_start_time_seconds{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"})\n", "instant": true, "legendFormat": "newest pod", "qryType": 1, "query": "time() - max(container_start_time_seconds{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Time since the newest balancerd container started.**\n\n**Nominal:** Hours to days, or as long as it has been since the last rollout.\n\n**Degraded:** Minutes means a pod just restarted or was replaced, and the connections it held dropped. _Balancer -> Balancer Logs_ and the environment's events say why.", "id": 1031, "links": [], "title": "Since Last Restart", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No metrics: cadvisor metrics are required (via kubelet)", "thresholds": { "mode": "absolute", "steps": [ { "color": "#AB0003", "value": null }, { "color": "#AB0003", "value": 2 }, { "color": "#E40515", "value": 8 }, { "color": "#F94902", "value": 26 }, { "color": "#F6790B", "value": 80 }, { "color": "#F19903", "value": 240 }, { "color": "#E7B503", "value": 719 }, { "color": "#D5CE04", "value": 2152 }, { "color": "#BBE453", "value": 6443 }, { "color": "#A2F49B", "value": 19286 }, { "color": "#C6F7D6", "value": 57730 }, { "color": "#CEFFFF", "value": 172800 } ] }, "unit": "s" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "balancer-summary-unavailable": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n max by (namespace, deployment) (\n kube_deployment_status_replicas_unavailable{namespace=~\"$mzNamespaceList\", deployment=~\".+-balancerd\"}\n )\n)\n", "legendFormat": "unavailable", "qryType": 1, "query": "sum(\n max by (namespace, deployment) (\n kube_deployment_status_replicas_unavailable{namespace=~\"$mzNamespaceList\", deployment=~\".+-balancerd\"}\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**balancerd pods the Deployment wants that are not available.**\n\n**Nominal:** Zero.\n\n**Degraded:** A pod counted here is crashing, failing its readiness check, or waiting for a node. A rollout counts each new pod here until it is ready.", "id": 1028, "links": [], "title": "Unavailable Pods", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No metrics: kube-state-metrics is required", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 1.5 }, { "color": "#F67E4B", "value": 2 }, { "color": "#DD3D2D", "value": 2.5 }, { "color": "#A50026", "value": 3 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "balancer-throttling": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"}[$__rate_interval]))\n/\nsum by (pod) (rate(container_cpu_cfs_periods_total{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"}[$__rate_interval]))\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"}[$__rate_interval]))\n/\nsum by (pod) (rate(container_cpu_cfs_periods_total{namespace=~\"$mzNamespaceList\",container!=\"\",container!=\"POD\", container=\"balancerd\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Share of each balancerd pod's CPU periods in which the kernel held it back at its CPU limit.**\n\n**Nominal:** Zero.\n\n**Degraded:** A throttled pod delays every connection it carries. Compare _Connection Setup Quantiles_, and raise or remove the CPU limit in the Materialize resource's `balancerdResourceRequirements`.\n\nEmpty when balancerd has no CPU limit, because nothing throttles it.", "id": 1043, "links": [], "title": "CPU Throttling", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "balancerd has no CPU limit, so the kernel never throttles it.", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-upstream-connect": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.50,\n sum by (le, source) (\n rate(mz_balancer_upstream_connect_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{source}} p50", "qryType": 1, "query": "histogram_quantile(0.50,\n sum by (le, source) (\n rate(mz_balancer_upstream_connect_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le, source) (\n rate(mz_balancer_upstream_connect_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{source}} p99", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le, source) (\n rate(mz_balancer_upstream_connect_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long balancerd takes to open its own connection to environmentd, by protocol.**\n\n**Nominal:** A small part of the setup time, because both run in the same cluster.\n\n**Degraded:** environmentd is slow to accept connections: busy, starting up, or on an overloaded node. Failed attempts are not timed; they are on _Overview -> Connection Error Ratio_.", "id": 1039, "links": [], "title": "Time to Reach environmentd", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No connection timings from balancerd. Older Materialize versions don't publish them.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "balancer-waiting": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (source) (mz_balancer_pre_resolved_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n", "legendFormat": "{{source}}", "qryType": 1, "query": "sum by (source) (mz_balancer_pre_resolved_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Connections balancerd has accepted but not yet sent anywhere, by protocol.**\n\n**Nominal:** Zero or close to it. Each connection spends a few milliseconds here.\n\n**Degraded:** Connections are stuck during setup: waiting on a slow client's startup message, or on a lookup. Connections that stay too long are closed and counted on _Refused Connections_.", "id": 1034, "links": [], "title": "Connections Not Yet Routed", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 0, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "console-client-errors": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (location, status) (rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=~\"4..\"}[$__rate_interval]))\n", "legendFormat": "{{location}} {{status}}", "qryType": 1, "query": "sum by (location, status) (rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=~\"4..\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Console requests per second rejected as the client's mistake, by location and status code.**\n\n**Nominal:** Low and bursty. `401` from `api` is an expired sign-in, and `404` is a stale link or a bookmark.\n\n**Degraded:** A steady `401` stream means something keeps calling the API with credentials it does not have, such as a script or a forgotten browser tab.", "id": 1054, "links": [], "title": "Client Errors by Location", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No console request was rejected in this time range.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "console-connections": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (state) (mz_console_http_connections{materialize_cloud_organization_name=~\"$environmentNameList\"})\n", "legendFormat": "{{state}}", "qryType": 1, "query": "sum by (state) (mz_console_http_connections{materialize_cloud_organization_name=~\"$environmentNameList\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Client connections nginx holds open, by what each is doing.**\n\n**Nominal:** `waiting` (idle keep-alive connections) is most of them. `reading` and `writing` are requests in flight.\n\n`active` is the total of the other three.", "id": 1052, "links": [], "title": "Client Connections by State", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 0, "noValue": "No metrics from the console. Older Materialize versions don't publish them, and the console PodMonitor must be installed.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "console-logs": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{namespace=~\"$mzNamespaceList\", app=\"console\"}\n != \"HealthChecker\"\n != \"kube-probe\"\n |~ \"\\\" 5[0-9][0-9] |\\\\[(error|crit|alert|emerg)\\\\]\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Console requests that ended in a server error, and what nginx logged about them.**\n\n**Nominal:** Empty.\n\n**Degraded:** Access lines show the request and its status. nginx's own `[error]` lines follow with the reason, such as `connect() failed` or `upstream timed out` for the hop to balancerd.\n\nLoad balancer and Kubernetes health checks are left out.", "id": 1057, "links": [], "title": "Server Errors", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "The console logged no server errors in this time range." }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "console-response-p50": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.50,\n sum by (le, location) (\n rate(mz_console_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{location}}", "qryType": 1, "query": "histogram_quantile(0.50,\n sum by (le, location) (\n rate(mz_console_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The median time to serve a console request, by location.**\n\n**Nominal:** Milliseconds for `app` and `static`. `api` is the median query.", "id": 1055, "links": [], "title": "Response Time (p50)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No metrics from the console. Older Materialize versions don't publish them, and the console PodMonitor must be installed.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "console-response-p99": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le, location) (\n rate(mz_console_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{location}}", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le, location) (\n rate(mz_console_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The time the slowest 1% of console requests take, by location.**\n\n**Nominal:** Milliseconds for `app` and `static`. `api` follows the slowest queries users run from the console.\n\n**Degraded:** `app` and `static` slow together is the console pod: check its CPU on the Materialize Environment Overview dashboard, under _Kubernetes Workloads_.\n\nMeasured in buckets up to 10 seconds, so anything slower reads as 10.", "id": 1056, "links": [], "title": "Response Time (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No metrics from the console. Older Materialize versions don't publish them, and the console PodMonitor must be installed.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "console-server-errors": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (location, status) (rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=~\"5..\"}[$__rate_interval]))\n", "legendFormat": "{{location}} {{status}}", "qryType": 1, "query": "sum by (location, status) (rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=~\"5..\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Console requests per second that ended in a server error, by location and status code.**\n\n**Nominal:** Zero.\n\n**Degraded:** `502` and `504` from `api`: nginx could not reach balancerd, or it timed out. Check _Overview -> Connection Error Ratio_ next. `5xx` from `app` or `static` is the console pod itself.", "id": 1053, "links": [], "title": "Server Errors by Location", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No console request ended in a server error in this time range.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "console-status-class": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (class) (\n label_replace(\n rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]),\n \"class\", \"${1}xx\", \"status\", \"([0-9])..\"\n )\n)\n", "legendFormat": "{{class}}", "qryType": 1, "query": "sum by (class) (\n label_replace(\n rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]),\n \"class\", \"${1}xx\", \"status\", \"([0-9])..\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Console requests per second, by status class.**\n\n**Nominal:** Mostly `2xx` and `3xx`. Some `4xx` is normal: a session that expired gets `401` until the user signs in again.\n\n**Degraded:** Any `5xx`. See _Server Errors by Location_.", "id": 1050, "links": [], "title": "Requests by Status Class", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No metrics from the console. Older Materialize versions don't publish them, and the console PodMonitor must be installed.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "console-websockets": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=\"101\"}[$__rate_interval]))\n", "legendFormat": "sessions/s", "qryType": 1, "query": "sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=\"101\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**WebSocket sessions through the console that ended, per second, such as the SQL shell.**\n\n**Nominal:** One each time someone closes the shell or leaves a page that streams results.\n\nnginx counts a WebSocket when its session ends, so a shell left open for an hour reaches this panel an hour late. environmentd counts one when it opens, as `/api/experimental/sql` on _SQL and HTTP API -> HTTP Requests by Path_.\n\nEach one is also an `https` connection on balancerd, open for as long as the session.", "id": 1051, "links": [], "title": "WebSocket Sessions Closed", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No WebSocket session through the console ended in this time range.", "unit": "ops" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-addresses": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (service, ip, hostname) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"})\nor on (service)\nmax by (service) (kube_service_spec_type{namespace=~\"$mzNamespaceList\", type=\"LoadBalancer\"})\n", "format": "table", "instant": true, "qryType": 2, "query": "max by (service, ip, hostname) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"})\nor on (service)\nmax by (service) (kube_service_spec_type{namespace=~\"$mzNamespaceList\", type=\"LoadBalancer\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true, "Value": true, "__name__": true }, "indexByName": { "hostname": 2, "ip": 1, "service": 0 } } } } ] } }, "description": "**The environment's LoadBalancer Services, and the address each was given.**\n\n**Nominal:** One for balancerd and one for the console, where the deployment exposes them through Kubernetes. The AWS Terraform wrappers make their Network Load Balancer in Terraform instead, so there are none on those installs.\n\n**Degraded:** A Service with no address has not been given a load balancer yet. That is normal for a minute after it is created, and a fault after several: subnets missing their tags, no free public addresses, or a controller without permission.\n\nA Service without an address has no ingress series, so the query adds every LoadBalancer Service that lacks one, with its address columns empty.", "id": 1013, "links": [], "title": "Load Balancer Addresses", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No LoadBalancer Service in this environment. The AWS wrappers make their Network Load Balancer outside Kubernetes, so it has no row here; its provider row is below." }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "lb-aws-healthy": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "min by (target_group) (\n label_replace(\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"target_group\", \"$1\", \"dimension_TargetGroup\", \"targetgroup/([^/]+)/.*\"\n )\n)\n", "legendFormat": "{{target_group}}", "qryType": 1, "query": "min by (target_group) (\n label_replace(\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"target_group\", \"$1\", \"dimension_TargetGroup\", \"targetgroup/([^/]+)/.*\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**balancerd and console pods passing the Network Load Balancer's health check, per target group, at the worst moment of the last five minutes.**\n\n**Nominal:** One target per pod: the balancerd pods in `pgwire` and `http`, and the console pods in `console`.\n\n**Unhealthy:** **Zero is a target group the load balancer has nowhere to send.** An NLB whose targets are all unhealthy sends to all of them anyway, so connections may still work, but a check that never passes means the check itself is wrong: its port or path, or a security group that does not admit it.\n\nCloudWatch publishes health only per target group, so this needs the target groups pulled beside the load balancer.", "id": 1014, "links": [], "title": "Healthy Targets", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No target group of this environment is pulled. CloudWatch publishes health only per target group, so list the pgwire, http and console target groups beside the load balancer in values.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#AB0003", "value": null }, { "color": "#F19903", "value": 1 }, { "color": "#A2F49B", "value": 1 } ] }, "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-aws-new": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_new_flow_count_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n) / 60\n", "legendFormat": "{{load_balancer}}", "qryType": 1, "query": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_new_flow_count_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n) / 60\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**New connections per second through the Network Load Balancer, over the newest whole minute CloudWatch has published.**\n\n**Nominal:** The rate clients connect at. Health checks are not counted.\n\n**Degraded:** Connections arriving here and missing from balancerd's _Balancer -> Connection Outcomes_ are lost between the two.", "id": 1018, "links": [], "title": "New Connections", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "unit": "cps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-aws-open": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_active_flow_count_average[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n)\n", "legendFormat": "{{load_balancer}}", "qryType": 1, "query": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_active_flow_count_average[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Connections open through the Network Load Balancer in front of this environment, averaged over the newest minute.**\n\n**Nominal:** Close to balancerd's _Overview -> Open Connections_, plus any console connections.\n\nKept only for a load balancer whose target groups are pulled too, because those are what name the environment.", "id": 1017, "links": [], "title": "Open Connections", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-aws-resets": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_tcp_client_reset_count_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n)\n", "legendFormat": "client {{load_balancer}}", "qryType": 1, "query": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_tcp_client_reset_count_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_tcp_target_reset_count_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n)\n", "legendFormat": "target {{load_balancer}}", "qryType": 1, "query": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_tcp_target_reset_count_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_tcp_elb_reset_count_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n)\n", "legendFormat": "load balancer {{load_balancer}}", "qryType": 1, "query": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_tcp_elb_reset_count_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n)\n" }, "version": "v0" }, "refId": "query-2" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**TCP resets through the Network Load Balancer in the last five minutes, by which side sent them.**\n\n**Nominal:** Some from clients, which close connections abruptly all the time. Few or none from targets or the load balancer.\n\n**Degraded:** **Target** resets come from balancerd or the console: a pod restarting, or balancerd refusing a connection. **Load balancer** resets are the NLB itself: a connection idle past its timeout, or a target that went away. A pooled SQL client that drops its connections every 350 seconds is meeting the NLB's idle timeout, and needs TCP keepalives.\n\nFive-minute totals that end at the pull, so read them as \"any\", not \"how many\".", "id": 1016, "links": [], "title": "Connection Resets (per 5 min)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 0, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-aws-throughput": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_processed_bytes_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n) / 60\n", "legendFormat": "{{load_balancer}}", "qryType": 1, "query": "max by (load_balancer) (\n label_replace(\n last_over_time(aws_networkelb_processed_bytes_sum[15m])\n and on (dimension_LoadBalancer)\n last_over_time(aws_networkelb_healthy_host_count_minimum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"load_balancer\", \"$1\", \"dimension_LoadBalancer\", \"net/([^/]+)/.*\"\n )\n) / 60\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Bytes per second through the Network Load Balancer, both directions together, over the newest whole minute.**\n\n**Nominal:** Follows query results: most of it is data leaving Materialize.", "id": 1019, "links": [], "title": "Throughput", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "unit": "Bps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-aws-unhealthy": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (target_group) (\n label_replace(\n last_over_time(aws_networkelb_un_healthy_host_count_maximum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"target_group\", \"$1\", \"dimension_TargetGroup\", \"targetgroup/([^/]+)/.*\"\n )\n)\n", "legendFormat": "{{target_group}}", "qryType": 1, "query": "max by (target_group) (\n label_replace(\n last_over_time(aws_networkelb_un_healthy_host_count_maximum{dimension_TargetGroup=~\"targetgroup/.+-${environmentNameList}-(pgwire|http|console)/.+\"}[15m]),\n \"target_group\", \"$1\", \"dimension_TargetGroup\", \"targetgroup/([^/]+)/.*\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**balancerd and console pods failing the Network Load Balancer's health check, per target group, at the worst moment of the last five minutes.**\n\n**Nominal:** Zero.\n\n**Degraded:** A pod here that Kubernetes reports Ready is one the load balancer cannot check: usually the network between them, or a check aimed at the wrong port. balancerd fails its check on purpose while it is over its high watermark; _Balancer -> Taking New Connections by Pod_ says when.", "id": 1015, "links": [], "title": "Unhealthy Targets", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 3.25 }, { "color": "#F67E4B", "value": 5.5 }, { "color": "#DD3D2D", "value": 7.75 }, { "color": "#A50026", "value": 10 } ] }, "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-azure-backends": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (service) (\n (last_over_time(azure_microsoft_network_loadbalancers_dipavailability_average_count[15m]) == bool 100)\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n)\n", "legendFormat": "{{service}} healthy", "qryType": 1, "query": "sum by (service) (\n (last_over_time(azure_microsoft_network_loadbalancers_dipavailability_average_count[15m]) == bool 100)\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count by (service) (\n last_over_time(azure_microsoft_network_loadbalancers_dipavailability_average_count[15m])\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n)\n", "legendFormat": "{{service}} total", "qryType": 1, "query": "count by (service) (\n last_over_time(azure_microsoft_network_loadbalancers_dipavailability_average_count[15m])\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Backends behind each of the environment's load balancer frontends, and how many passed every health probe in the last five minutes.**\n\n**Nominal:** Healthy equals the number of nodes running that Service's pods, and total is every node in the pool. An AKS load balancer's backends are nodes, and the Materialize Services set `externalTrafficPolicy: Local`, so a node passes the probe only while it runs a balancerd or console pod.\n\n**Degraded:** Fewer healthy nodes than pods is a node the load balancer cannot probe. Zero healthy is a frontend with nowhere to send.", "id": 1022, "links": [], "title": "Nodes Passing the Health Probe", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-azure-new": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (service) (\n last_over_time(azure_microsoft_network_loadbalancers_syncount_total_count{dimensionDirection=\"In\"}[15m])\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n) / 60\n", "legendFormat": "{{service}}", "qryType": 1, "query": "sum by (service) (\n last_over_time(azure_microsoft_network_loadbalancers_syncount_total_count{dimensionDirection=\"In\"}[15m])\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n) / 60\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**New connections per second arriving at each of the environment's load balancer frontends, from the SYN packets Azure counts.**\n\n**Nominal:** The rate clients connect at.", "id": 1023, "links": [], "title": "New Connections by Service", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "unit": "cps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-azure-throughput": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (service) (\n last_over_time(azure_microsoft_network_loadbalancers_bytecount_total_bytes{dimensionDirection=\"In\"}[15m])\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n) / 60\n", "legendFormat": "{{service}} in", "qryType": 1, "query": "sum by (service) (\n last_over_time(azure_microsoft_network_loadbalancers_bytecount_total_bytes{dimensionDirection=\"In\"}[15m])\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n) / 60\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (service) (\n last_over_time(azure_microsoft_network_loadbalancers_bytecount_total_bytes{dimensionDirection=\"Out\"}[15m])\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n) / 60\n", "legendFormat": "{{service}} out", "qryType": 1, "query": "sum by (service) (\n last_over_time(azure_microsoft_network_loadbalancers_bytecount_total_bytes{dimensionDirection=\"Out\"}[15m])\n * on (dimensionFrontendIPAddress) group_left (service)\n label_replace(\n max by (service, ip) (kube_service_status_load_balancer_ingress{namespace=~\"$mzNamespaceList\"}),\n \"dimensionFrontendIPAddress\", \"$1\", \"ip\", \"(.+)\"\n )\n) / 60\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Bytes per second through each of the environment's load balancer frontends, by Service and direction.**\n\n**Nominal:** Follows query results: most of it leaves Materialize.", "id": 1024, "links": [], "title": "Throughput by Service", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "unit": "Bps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-gcp-packets": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (service) (\n (\n (idelta(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_ingress_packets_count[15m]) >= 0) or last_over_time(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_ingress_packets_count[10m])\n or\n (idelta(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_ingress_packets_count[15m]) >= 0) or last_over_time(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_ingress_packets_count[10m])\n )\n * on (forwarding_rule_name) group_left (service)\n label_replace(\n max by (service, uid) (kube_service_info{namespace=~\"$mzNamespaceList\"}),\n \"forwarding_rule_name\", \"a$1$2$3$4$5\", \"uid\", \"(.{8})-(.{4})-(.{4})-(.{4})-(.{11}).*\"\n )\n) / 60\n", "legendFormat": "{{service}} in", "qryType": 1, "query": "sum by (service) (\n (\n (idelta(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_ingress_packets_count[15m]) >= 0) or last_over_time(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_ingress_packets_count[10m])\n or\n (idelta(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_ingress_packets_count[15m]) >= 0) or last_over_time(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_ingress_packets_count[10m])\n )\n * on (forwarding_rule_name) group_left (service)\n label_replace(\n max by (service, uid) (kube_service_info{namespace=~\"$mzNamespaceList\"}),\n \"forwarding_rule_name\", \"a$1$2$3$4$5\", \"uid\", \"(.{8})-(.{4})-(.{4})-(.{4})-(.{11}).*\"\n )\n) / 60\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (service) (\n (\n (idelta(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_egress_packets_count[15m]) >= 0) or last_over_time(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_egress_packets_count[10m])\n or\n (idelta(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_egress_packets_count[15m]) >= 0) or last_over_time(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_egress_packets_count[10m])\n )\n * on (forwarding_rule_name) group_left (service)\n label_replace(\n max by (service, uid) (kube_service_info{namespace=~\"$mzNamespaceList\"}),\n \"forwarding_rule_name\", \"a$1$2$3$4$5\", \"uid\", \"(.{8})-(.{4})-(.{4})-(.{4})-(.{11}).*\"\n )\n) / 60\n", "legendFormat": "{{service}} out", "qryType": 1, "query": "sum by (service) (\n (\n (idelta(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_egress_packets_count[15m]) >= 0) or last_over_time(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_egress_packets_count[10m])\n or\n (idelta(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_egress_packets_count[15m]) >= 0) or last_over_time(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_egress_packets_count[10m])\n )\n * on (forwarding_rule_name) group_left (service)\n label_replace(\n max by (service, uid) (kube_service_info{namespace=~\"$mzNamespaceList\"}),\n \"forwarding_rule_name\", \"a$1$2$3$4$5\", \"uid\", \"(.{8})-(.{4})-(.{4})-(.{4})-(.{11}).*\"\n )\n) / 60\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Packets per second through each of the environment's load balancers, by Service and direction.**\n\n**Nominal:** Follows client traffic. Cloud Monitoring publishes no health or connection counts for these load balancers, so traffic is the whole of the provider's view.\n\n**Degraded:** Packets arriving with none leaving usually means the load balancer has no backend to send them to: check that the balancerd and console pods are Ready.\n\nCloud Monitoring publishes a minute only when it carried traffic, so an idle load balancer has gaps rather than zeros.", "id": 1020, "links": [], "title": "Packets by Service", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "unit": "pps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-gcp-throughput": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (service) (\n (\n (idelta(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_ingress_bytes_count[15m]) >= 0) or last_over_time(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_ingress_bytes_count[10m])\n or\n (idelta(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_ingress_bytes_count[15m]) >= 0) or last_over_time(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_ingress_bytes_count[10m])\n )\n * on (forwarding_rule_name) group_left (service)\n label_replace(\n max by (service, uid) (kube_service_info{namespace=~\"$mzNamespaceList\"}),\n \"forwarding_rule_name\", \"a$1$2$3$4$5\", \"uid\", \"(.{8})-(.{4})-(.{4})-(.{4})-(.{11}).*\"\n )\n) / 60\n", "legendFormat": "{{service}} in", "qryType": 1, "query": "sum by (service) (\n (\n (idelta(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_ingress_bytes_count[15m]) >= 0) or last_over_time(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_ingress_bytes_count[10m])\n or\n (idelta(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_ingress_bytes_count[15m]) >= 0) or last_over_time(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_ingress_bytes_count[10m])\n )\n * on (forwarding_rule_name) group_left (service)\n label_replace(\n max by (service, uid) (kube_service_info{namespace=~\"$mzNamespaceList\"}),\n \"forwarding_rule_name\", \"a$1$2$3$4$5\", \"uid\", \"(.{8})-(.{4})-(.{4})-(.{4})-(.{11}).*\"\n )\n) / 60\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (service) (\n (\n (idelta(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_egress_bytes_count[15m]) >= 0) or last_over_time(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_egress_bytes_count[10m])\n or\n (idelta(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_egress_bytes_count[15m]) >= 0) or last_over_time(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_egress_bytes_count[10m])\n )\n * on (forwarding_rule_name) group_left (service)\n label_replace(\n max by (service, uid) (kube_service_info{namespace=~\"$mzNamespaceList\"}),\n \"forwarding_rule_name\", \"a$1$2$3$4$5\", \"uid\", \"(.{8})-(.{4})-(.{4})-(.{4})-(.{11}).*\"\n )\n) / 60\n", "legendFormat": "{{service}} out", "qryType": 1, "query": "sum by (service) (\n (\n (idelta(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_egress_bytes_count[15m]) >= 0) or last_over_time(stackdriver_loadbalancing_googleapis_com_external_network_load_balancer_rule_loadbalancing_googleapis_com_l_3_external_egress_bytes_count[10m])\n or\n (idelta(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_egress_bytes_count[15m]) >= 0) or last_over_time(stackdriver_internal_tcp_lb_rule_loadbalancing_googleapis_com_l_3_internal_egress_bytes_count[10m])\n )\n * on (forwarding_rule_name) group_left (service)\n label_replace(\n max by (service, uid) (kube_service_info{namespace=~\"$mzNamespaceList\"}),\n \"forwarding_rule_name\", \"a$1$2$3$4$5\", \"uid\", \"(.{8})-(.{4})-(.{4})-(.{4})-(.{11}).*\"\n )\n) / 60\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Bytes per second through each of the environment's load balancers, by Service and direction.**\n\n**Nominal:** Follows query results: most of it leaves Materialize.\n\nAs with packets, an idle minute is a gap rather than a zero.", "id": 1021, "links": [], "title": "Throughput by Service", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No data for this environment's load balancer. Check that the pull names it, and on Infrastructure Networking -> Cloud Networking that the pull is reporting at all.", "unit": "Bps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "lb-no-pull": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [], "queryOptions": {}, "transformations": [] } }, "description": "Why this tab has no provider load balancer rows on it.", "id": 1025, "links": [], "title": "No Load Balancer Metrics", "vizConfig": { "group": "text", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": {}, "overrides": [] }, "options": { "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false }, "content": "**No load balancer metrics are being collected.**\n\nThe table above is what Kubernetes knows about the load balancer in front of balancerd and the console: that it exists, and its address. What it is doing — whether its targets are healthy, what it carries, who resets its connections — is published by the cloud provider, and the gateway pulls it only for load balancers named in values.\n\n- **AWS**: list the Network Load Balancer's ARN, with its pgwire, http and console target groups' ARNs, under `pipeline.metrics.provider.cloudwatch.nlb.loadBalancers`. From the Terraform wrappers, that is `provider_metrics.network_load_balancers`.\n- **Google Cloud**: list the balancerd and console forwarding rules under `pipeline.metrics.provider.gcp.loadBalancing.forwardingRules`, or the cluster's VPC network under `.networks`. From the Terraform wrappers, that is `provider_metrics.forwarding_rules`.\n- **Azure**: list the AKS cluster under `pipeline.metrics.provider.azure.loadBalancer.clusters`. From the Terraform wrappers, that is `provider_metrics.load_balancer_clusters`.\n\nThe Cloud Provider Metrics page of the documentation has the details.", "mode": "markdown" } }, "version": "" } } }, "overview-accepting": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "min(mz_balancer_connection_accepting{materialize_cloud_organization_name=~\"$environmentNameList\"})\n", "legendFormat": "state", "qryType": 1, "query": "min(mz_balancer_connection_accepting{materialize_cloud_organization_name=~\"$environmentNameList\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Whether every balancerd pod is taking new connections.**\n\n**Nominal:** Accepting. Without connection watermarks configured, balancerd always is.\n\n**Degraded:** At least one balancerd pod has passed its high watermark and asked the load balancer to stop sending it new connections. Its existing connections carry on. The other pods take the new ones, so this is capacity running short rather than an outage, unless every pod is out.\n\n_Balancer -> Open Connections Against the Limit_ shows which pod, and how close the rest are.", "id": 1002, "links": [], "title": "Taking New Connections", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "mappings": [ { "options": { "1": { "color": "#A2F49B", "index": 1, "text": "Yes" } }, "type": "value" }, { "options": { "0": { "color": "#F19903", "index": 2, "text": "At Capacity" } }, "type": "value" } ], "noValue": "balancerd does not report its capacity state. Older Materialize versions don't publish it.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#F19903", "value": null }, { "color": "#A2F49B", "value": 1 } ] } }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-balancer-error-ratio": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n sum(rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\", status=\"error\"}[$__rate_interval]))\n /\n sum(rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n) >= 0\nor\n0 * sum(rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n", "legendFormat": "errors", "qryType": 1, "query": "(\n sum(rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\", status=\"error\"}[$__rate_interval]))\n /\n sum(rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n) >= 0\nor\n0 * sum(rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The share of SQL and HTTPS connections balancerd could not serve.**\n\n**Nominal:** Zero. A client giving up, or typing the wrong password, does not count.\n\n**Degraded:** Connections are failing on the server side: environmentd is down or restarting, balancerd is at its connection limit, or it cannot look up where to send them. Users see `upstream server not available` or a connection reset.\n\n_Balancer -> Refused Connections_ separates the connection limit from the rest, and _Balancer -> Warnings and Errors_ has the reason for each one.", "id": 1000, "links": [], "title": "Connection Error Ratio", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.01 }, { "color": "#FDB366", "value": 0.02 }, { "color": "#F67E4B", "value": 0.03 }, { "color": "#DD3D2D", "value": 0.04 }, { "color": "#A50026", "value": 0.05 } ] }, "unit": "percentunit" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-balancer-errors": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (source) (rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\", status=\"error\"}[$__rate_interval]))\n", "legendFormat": "{{source}}", "qryType": 1, "query": "sum by (source) (rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\", status=\"error\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Connections per second that balancerd could not serve, by protocol.**\n\n**Nominal:** Zero.\n\n**Degraded:** Both protocols at once points at environmentd or at balancerd's own capacity. `https` alone, while SQL clients are fine, is usually a lookup failing for the HTTP listener.", "id": 1009, "links": [], "title": "Connection Errors by Protocol", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "unit": "cps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-connection-setup": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.50,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{source}} p50", "qryType": 1, "query": "histogram_quantile(0.50,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{source}} p99", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le, source) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long new connections wait before balancerd passes them to environmentd, by protocol.**\n\n**Nominal:** Steady. It includes each client's TLS handshake, so it follows how far away the clients are.\n\n**Degraded:** A rise across both protocols points at environmentd or at balancerd's own CPU. _Balancer -> Connection Setup Quantiles_ has more of them.", "id": 1011, "links": [], "title": "Connection Setup Time", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No connection timings from balancerd. Older Materialize versions don't publish them.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-connection-setup-p99": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "p99", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le) (\n rate(mz_balancer_connection_setup_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long the slowest 1% of new connections wait before balancerd starts passing their traffic to environmentd.**\n\n**Nominal:** Steady. It includes the client's TLS handshake, so clients far from the cluster take longer than clients inside it.\n\n**Degraded:** Users wait this long before their first query even starts. Compare with _Balancer -> Time to Reach environmentd_: when that is slow too, environmentd is slow to accept connections; when it is not, the time goes to TLS or to the lookup.\n\nCounts only connections that reached environmentd. Connections that failed are on _Connection Error Ratio_.", "id": 1001, "links": [], "title": "Connection Setup (p99)", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No connection timings from balancerd. Older Materialize versions don't publish them.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#F19903", "value": 0.5 }, { "color": "#AB0003", "value": 2 } ] }, "unit": "s" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-connections": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (source) (rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n", "legendFormat": "{{source}}", "qryType": 1, "query": "sum by (source) (rate(mz_balancer_connection_status{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Connections balancerd finished handling per second, by protocol.**\n\n**Nominal:** `pgwire` follows the SQL clients' connection habits: a connection pool reconnects rarely, a script reconnects for every query. `https` counts each HTTP API call and each console request that reaches the API.\n\nCounted when a connection closes, so a long session shows up when it ends rather than when it starts.", "id": 1007, "links": [], "title": "Connections by Protocol", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "unit": "cps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-console-error-ratio": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=~\"5..\"}[$__rate_interval]))\n /\n sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n) >= 0\nor\n0 * sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n", "legendFormat": "errors", "qryType": 1, "query": "(\n sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=~\"5..\"}[$__rate_interval]))\n /\n sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n) >= 0\nor\n0 * sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The share of console requests that ended in a server error (`5xx`).**\n\n**Nominal:** Zero.\n\n**Degraded:** `502` and `504` from the `api` location mean nginx could not reach balancerd, or balancerd could not reach environmentd: the console loads but cannot run queries. Errors from `app` or `static` mean the console itself is broken.\n\n_Console -> Server Errors by Location_ breaks this down by location and status code.", "id": 1003, "links": [], "title": "Console Error Ratio", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics from the console. Older Materialize versions don't publish them, and the console PodMonitor must be installed.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.01 }, { "color": "#FDB366", "value": 0.02 }, { "color": "#F67E4B", "value": 0.03 }, { "color": "#DD3D2D", "value": 0.04 }, { "color": "#A50026", "value": 0.05 } ] }, "unit": "percentunit" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-console-errors": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (location) (rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=~\"5..\"}[$__rate_interval]))\n", "legendFormat": "{{location}}", "qryType": 1, "query": "sum by (location) (rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\", status=~\"5..\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Console requests per second that ended in a server error, by location.**\n\n**Nominal:** Zero.\n\n**Degraded:** `api` alone means the hop to balancerd is failing while the console itself is up.", "id": 1010, "links": [], "title": "Console Server Errors by Location", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No console request ended in a server error in this time range.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-console-page-p99": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le) (\n rate(mz_console_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\", location=~\"app|static\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "p99", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le) (\n rate(mz_console_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\", location=~\"app|static\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long the slowest 1% of requests for the console's page and its scripts take to serve.**\n\n**Nominal:** Milliseconds. These are static files.\n\n**Degraded:** The console pod itself is struggling: CPU-throttled, or short of memory. API calls are left out because they include the time a query takes to run.", "id": 1004, "links": [], "title": "Console Page Load (p99)", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics from the console. Older Materialize versions don't publish them, and the console PodMonitor must be installed.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#F19903", "value": 0.5 }, { "color": "#AB0003", "value": 2 } ] }, "unit": "s" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-console-requests": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n", "legendFormat": "requests/s", "qryType": 1, "query": "sum(rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Requests the console serves per second.**\n\n**Nominal:** Bursts while someone has the console open, and close to zero otherwise. Health checks are not counted.", "id": 1006, "links": [], "title": "Console Requests", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#0077BB", "mode": "shades" }, "min": 0, "noValue": "No metrics from the console. Older Materialize versions don't publish them, and the console PodMonitor must be installed.", "unit": "reqps" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-console-response": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(0.99,\n sum by (le, location) (\n rate(mz_console_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n", "legendFormat": "{{location}}", "qryType": 1, "query": "histogram_quantile(0.99,\n sum by (le, location) (\n rate(mz_console_http_request_duration_seconds_bucket{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval])\n )\n) >= 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long the slowest 1% of console requests take, by location.**\n\n**Nominal:** Milliseconds for `app` and `static`. `api` includes the time each query takes to run, so it follows the workload.\n\n**Degraded:** `app` and `static` rising together means the console pod is struggling. `api` rising alone is the queries, or the hops behind the console: compare with _Connection Setup Time_.\n\nWebSocket sessions, such as the SQL shell, are not timed.", "id": 1012, "links": [], "title": "Console Response Time (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "noValue": "No metrics from the console. Older Materialize versions don't publish them, and the console PodMonitor must be installed.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-console-traffic": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (location) (rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n", "legendFormat": "{{location}}", "qryType": 1, "query": "sum by (location) (rate(mz_console_http_requests_total{materialize_cloud_organization_name=~\"$environmentNameList\"}[$__rate_interval]))\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Console requests per second, by what they asked for.**\n\n**Nominal:** `static` dominates while the page loads, and `api` while someone uses it.", "id": 1008, "links": [], "title": "Console Requests by Location", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "scaleDistribution": { "log": 10, "type": "log" } }, "decimals": 3, "noValue": "No metrics from the console. Older Materialize versions don't publish them, and the console PodMonitor must be installed.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-open-connections": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(mz_balancer_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n", "legendFormat": "connections", "qryType": 1, "query": "sum(mz_balancer_connection_active{materialize_cloud_organization_name=~\"$environmentNameList\"})\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Connections balancerd is passing to environmentd right now, SQL and HTTPS together.**\n\n**Nominal:** Whatever the environment's clients keep open. Connection poolers and BI tools hold steady numbers.\n\nEvery open connection is also a session on environmentd. See _SQL and HTTP API -> Open Sessions_.", "id": 1005, "links": [], "title": "Open Connections", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#0077BB", "mode": "shades" }, "decimals": 0, "min": 0, "noValue": "No metrics from balancerd in this environment. Check the environment picker, and that the balancerd PodMonitor is installed.", "unit": "short" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } } }, "layout": { "kind": "TabsLayout", "spec": { "tabs": [ { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "narrow", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-balancer-error-ratio" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-connection-setup-p99" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-accepting" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-console-error-ratio" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-console-page-p99" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-open-connections" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-console-requests" } } } ], "maxColumnCount": 7, "rowHeightMode": "short" } }, "title": "Verdict" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-connections" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-console-traffic" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Rate" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-balancer-errors" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-console-errors" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Errors" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-connection-setup" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-console-response" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Latency" } } ] } }, "title": "Overview" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-addresses" } } } ], "maxColumnCount": 1, "rowHeightMode": "standard" } }, "title": "Load Balancer Services" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingVariable", "spec": { "operator": "matches", "value": "cloudwatch_nlb", "variable": "cloudNetworkPullList" } } ], "visibility": "show" } }, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-aws-healthy" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-aws-unhealthy" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-aws-resets" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-aws-open" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-aws-new" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-aws-throughput" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "AWS Network Load Balancer" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingVariable", "spec": { "operator": "matches", "value": "gcp_load_balancing", "variable": "cloudNetworkPullList" } } ], "visibility": "show" } }, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-gcp-packets" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-gcp-throughput" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Google Cloud Load Balancer" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingVariable", "spec": { "operator": "matches", "value": "azure_load_balancer", "variable": "cloudNetworkPullList" } } ], "visibility": "show" } }, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-azure-backends" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-azure-new" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-azure-throughput" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Azure Load Balancer" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingVariable", "spec": { "operator": "notMatches", "value": "cloudwatch_nlb|gcp_load_balancing|azure_load_balancer", "variable": "cloudNetworkPullList" } } ], "visibility": "show" } }, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "lb-no-pull" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "No Load Balancer Metrics" } } ] } }, "title": "Load Balancer" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "narrow", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-summary-open" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-summary-ready" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-summary-unavailable" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-summary-cpu" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-summary-memory" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-summary-restart" } } } ], "maxColumnCount": 6, "rowHeightMode": "short" } }, "title": "Balancer Summary" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-outcomes" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-open-by-pod" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-waiting" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Connections" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-capacity" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-accepting" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-refused" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Capacity" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-setup" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-upstream-connect" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-lifetime" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Connection Timing" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-cpu" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-memory" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-throttling" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Resources" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingVariable", "spec": { "operator": "matches", "value": "balancerd", "variable": "balancerdAutoscaler" } } ], "visibility": "show" } }, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-autoscaler-replicas" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-autoscaler-metrics" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-autoscaler-conditions" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-autoscaler-events" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Autoscaling" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingVariable", "spec": { "operator": "notMatches", "value": "balancerd", "variable": "balancerdAutoscaler" } } ], "visibility": "show" } }, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-no-autoscaler" } } } ], "maxColumnCount": 1, "rowHeightMode": "standard" } }, "title": "No Autoscaler" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "balancer-logs" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "Balancer Logs" } } ] } }, "title": "Balancer" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "console-status-class" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "console-websockets" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "console-connections" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Traffic" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "console-server-errors" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "console-client-errors" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Console Errors" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "console-response-p50" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "console-response-p99" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Response Time" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "console-logs" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "Console Logs" } } ] } }, "title": "Console" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "api-http-requests" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "api-http-errors" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "api-http-response" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "HTTP API" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "api-sessions" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "api-pgwire" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "SQL Connections" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "api-console-commands" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "api-console-latency" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Console Queries" } } ] } }, "title": "SQL and HTTP API" } } ] } }, "links": [], "liveNow": false, "preload": false, "tags": [ "materialize", "mzmon", "network" ], "timeSettings": { "autoRefresh": "", "autoRefreshIntervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "fiscalYearStartMonth": 0, "from": "now-6h", "hideTimepicker": false, "timezone": "browser", "to": "now" }, "title": "Materialize Ingress (Balancer, Console)", "variables": [ { "kind": "DatasourceVariable", "spec": { "allowCustomValue": false, "current": { "text": "", "value": "" }, "description": "Datasource for metrics queries", "hide": "dontHide", "includeAll": false, "label": "Metrics Datasource", "multi": false, "name": "metricsDatasource", "options": [], "pluginId": "prometheus", "refresh": "never", "regex": "", "skipUrlSync": false } }, { "kind": "DatasourceVariable", "spec": { "allowCustomValue": false, "current": { "text": "", "value": "" }, "description": "Datasource for log and event queries", "hide": "dontHide", "includeAll": false, "label": "Logs Datasource", "multi": false, "name": "logsDatasource", "options": [], "pluginId": "loki", "refresh": "never", "regex": "", "skipUrlSync": false } }, { "kind": "QueryVariable", "spec": { "allValue": ".*", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(mz_compute_commands_total, materialize_cloud_organization_name)", "description": "The current environment to view", "hide": "dontHide", "includeAll": false, "label": "Environment", "multi": false, "name": "environmentNameList", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(mz_compute_commands_total, materialize_cloud_organization_name)", "qryType": 1, "query": "label_values(mz_compute_commands_total, materialize_cloud_organization_name)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(mz_compute_commands_total{ materialize_cloud_organization_name=~\"$environmentNameList\" }, materialize_cloud_organization_namespace)", "description": "The current materialize namespace where environments live", "hide": "hideVariable", "includeAll": true, "label": "Materialize Namespace", "multi": true, "name": "mzNamespaceList", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(mz_compute_commands_total{ materialize_cloud_organization_name=~\"$environmentNameList\" }, materialize_cloud_organization_namespace)", "qryType": 1, "query": "label_values(mz_compute_commands_total{ materialize_cloud_organization_name=~\"$environmentNameList\" }, materialize_cloud_organization_namespace)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": true, "sort": "alphabeticalAsc" } }, { "datasource": { "name": "$metricsDatasource" }, "group": "", "kind": "AdhocVariable", "spec": { "allowCustomValue": true, "baseFilters": [ { "key": "namespace", "operator": "=~", "origin": "dashboard", "value": "$mzNamespaceList" } ], "defaultKeys": [], "description": "Adhoc filters to apply to all metrics queries", "enableGroupBy": false, "filters": [], "hide": "inControlsMenu", "label": "Advanced Metric Filter", "name": "metricAdhoc", "skipUrlSync": false } }, { "kind": "QueryVariable", "spec": { "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(up{provider_pull=~\".+\"}, provider_pull)", "description": "Load balancer and NAT gateway pulls the gateway was found to be running", "hide": "hideVariable", "includeAll": true, "label": "Network Pulls", "multi": true, "name": "cloudNetworkPullList", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(up{provider_pull=~\".+\"}, provider_pull)", "qryType": 1, "query": "label_values(up{provider_pull=~\".+\"}, provider_pull)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": true, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"}, scaletargetref_name)", "description": "The HorizontalPodAutoscaler found scaling the environment's balancerd", "hide": "hideVariable", "includeAll": true, "label": "balancerd Autoscaler", "multi": true, "name": "balancerdAutoscaler", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"}, scaletargetref_name)", "qryType": 1, "query": "label_values(kube_horizontalpodautoscaler_info{namespace=~\"$mzNamespaceList\", scaletargetref_name=~\".+-balancerd\"}, scaletargetref_name)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": true, "sort": "alphabeticalAsc" } } ] } }