{ "apiVersion": "dashboard.grafana.app/v2", "kind": "Dashboard", "metadata": { "annotations": { "grafana.app/folder": "meta-o11y", "monitoring.materialize.cloud/min-mz-version": "v26.24.0", "monitoring.materialize.cloud/rec-mz-version": "v26.24.0", "monitoring.materialize.cloud/sql-metric-prefix": "mz_", "monitoring.materialize.cloud/target-export": "generic" }, "name": "mz-mon-infra-alloy" }, "spec": { "annotations": [ { "kind": "AnnotationQuery", "spec": { "builtIn": true, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "query": { "datasource": { "name": "-- Grafana --" }, "group": "grafana", "kind": "DataQuery", "spec": {}, "version": "v0" } } } ], "cursorSync": "Crosshair", "description": "Is telemetry collection healthy, and if not, which stage of it broke.\n\nThe collectors watching themselves: the per-node agents, the gateway, both pipelines, and Alloy's own logs. Everything here reaches the dashboard through the gateway, so a dashboard that is empty throughout most likely means the gateway is down.", "editable": true, "elements": { "alloy-event-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n job=\"loki.source.kubernetes_events\",\n namespace=~\"$alloyNamespace\"\n}\n |~ \"(?i)$logSearch\"\n | name=~\"(${alloyRole:regex}).*|.+-validate-(agent|gateway)(-.*)?\"\n | name=~\"$alloyPod\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Every Kubernetes event about the collectors, newest first.**\n\nMatched on the involved object's name, since events carry no `app` label. Choosing one collector narrows the feed to that pod's own events, which leaves out its DaemonSet's or Deployment's.\n\nThe validation Jobs are named after the release, as `-validate-agent` and `-validate-gateway`, rather than after the role. Both run together on every install and upgrade, so their events are shown whatever the role picker is set to.", "id": 1086, "links": [], "title": "All Events", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No events about the collectors in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "signature", "detailsMode": "inline", "displayedFields": [ "reason", "name", "msg" ], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": true, "wrapLogMessage": false } }, "version": "" } } }, "alloy-event-rate-by-kind": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (kind) (\n count_over_time(\n {\n job=\"loki.source.kubernetes_events\",\n namespace=~\"$alloyNamespace\"\n }\n |~ \"(?i)$logSearch\"\n | name=~\"(${alloyRole:regex}).*|.+-validate-(agent|gateway)(-.*)?\"\n | name=~\"$alloyPod\"\n [$__auto]\n )\n)\n", "legendFormat": "{{kind}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Kubernetes events about the collectors, per kind of object they concern.**\n\n**Nominal:** `Pod` and `ReplicaSet` during a gateway rollout, `Pod` alone during an agent rollout, and `HorizontalPodAutoscaler` when the gateway scales.\n\n**Degraded:** A steady stream of `HorizontalPodAutoscaler` events. The gateway is scaling up and down repeatedly, which restarts scrapes each time.", "id": 1084, "links": [], "title": "Event Rate by Object Kind", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No events about the collectors in this time range" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "alloy-event-rate-by-reason": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (reason) (\n count_over_time(\n {\n job=\"loki.source.kubernetes_events\",\n namespace=~\"$alloyNamespace\"\n }\n |~ \"(?i)$logSearch\"\n | name=~\"(${alloyRole:regex}).*|.+-validate-(agent|gateway)(-.*)?\"\n | name=~\"$alloyPod\"\n [$__auto]\n )\n)\n", "legendFormat": "{{reason}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Kubernetes events about the collectors' pods, DaemonSet, Deployment and autoscaler, and about the configuration-validation Jobs that run before each install and upgrade, per reason.**\n\n**Nominal:** Bursts during a rollout or a scale-up, such as `ScalingReplicaSet`, `Scheduled` and `Killing`, and quiet in between. Each upgrade adds a `SuccessfulCreate` and a `Completed` per validation Job.\n\n**Unhealthy:** A reason that repeats. `BackOff` is a collector that starts and dies, `Unhealthy` one whose probe fails, `FailedMount` a volume or certificate secret that did not attach, and `FailedScheduling` an agent with no room on a new node. `BackoffLimitExceeded` on a validation Job is a pipeline configuration that failed to validate, and the install or upgrade that ran it stopped there.\n\nCounts events as Loki received them. Kubernetes folds a repeat into one event with a rising `count`, so a tight loop reports fewer lines than it has occurrences.", "id": 1083, "links": [], "title": "Event Rate by Reason", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No events about the collectors in this time range" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "alloy-logs-errors-by-component": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n {\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n level=~\"WARN|WARNING|ERROR|CRITICAL|FATAL\"\n }\n |~ \"(?i)$logSearch\"\n | pod=~\"$alloyPod\"\n | logfmt component_id\n [$__auto]\n )\n) * 60\n", "legendFormat": "{{component_id}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Warning-and-worse lines per minute from the collectors, split by the pipeline component that logged them.**\n\n**Nominal:** Empty, or close to it.\n\n**Unhealthy:** One component dominating. Its name says which stage of the pipeline is failing: `loki.write` and `prometheus.remote_write` are deliveries, `otelcol.exporter` an extra destination, `discovery` and `prometheus.operator` finding targets.\n\nIndependent of the level picker, for the same reason as the warning feed.", "id": 1088, "links": [], "title": "Error Rate by Component", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No warnings in this time range", "unit": "suffix:logs/min" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "alloy-logs-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n level=~\"$logLevelList\"\n}\n |~ \"(?i)$logSearch\"\n | pod=~\"$alloyPod\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The collectors' own log feed, newest first, for the selected roles and levels.**\n\nThese lines reach Loki through the gateway. The agents' own lines are read by the gateway over the Kubernetes API, and the gateway's lines are read off disk by the agent on its node. A gateway that is down takes both with it, and `kubectl logs` is then the tool.\n\nAn empty search matches every line rather than none.", "id": 1092, "links": [], "title": "All Logs", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No lines match the current filters, or the gateway is not delivering logs" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "alloy-logs-rate": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (app, level) (\n rate(\n {\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n level=~\"$logLevelList\"\n }\n |~ \"(?i)$logSearch\"\n | pod=~\"$alloyPod\"\n [$__auto]\n )\n)\n", "legendFormat": "{{app}} {{level}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The collectors' own log lines per second, split by role and level.**\n\n**Nominal:** A low, steady baseline. The agents log each file they start or stop tailing, so pod churn shows up here.\n\n**Degraded:** `UNKNOWN` climbing. Those are lines Alloy did not write in its own format, most often the Go HTTP server reporting failed TLS handshakes on a listener.", "id": 1087, "links": [], "title": "Log Rate by Role and Level", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No lines match the current filters, or the gateway is not delivering logs", "unit": "suffix:logs/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "alloy-logs-warning-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n level=~\"WARN|WARNING|ERROR|CRITICAL|FATAL\"\n}\n |~ \"(?i)$logSearch\"\n | pod=~\"$alloyPod\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Warning-and-worse lines from the collectors, newest first.**\n\n**Nominal:** Quiet. Alloy logs component starts and stops at INFO and little above it.\n\n**Unhealthy:** A steady stream from one component. The `component_id` on each line names it.\n\nIndependent of the level picker, like the other warning feeds in this repository. Narrowing the selection to INFO would otherwise silently empty it.", "id": 1091, "links": [], "title": "Warnings and Errors", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No warnings in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "alloy-logs-warning-rate": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n {\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n level=~\"WARN|WARNING|ERROR|CRITICAL|FATAL\"\n }\n |~ \"(?i)$logSearch\"\n | pod=~\"$alloyPod\"\n [1m]\n )\n) * 60\n", "legendFormat": "warnings/min" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Warning-and-worse lines per minute from the selected collectors.**\n\n**Nominal:** Zero, or close to it.\n\n**Unhealthy:** A sustained non-zero rate. _Error Rate by Component_ says which stage, and the feed below says why.", "id": 1089, "links": [], "title": "Warning Rate", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#33BBEE", "mode": "shades" }, "min": 0, "noValue": "No warnings in this time range", "unit": "suffix:logs/min" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "alloy-volume-hidden-note": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [], "queryOptions": {}, "transformations": [] } }, "description": "Why the volume panels are not drawn over long ranges.", "id": 1090, "links": [], "title": "Volume (skipped)", "vizConfig": { "group": "text", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": {}, "overrides": [] }, "options": { "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false }, "content": "**Volume panels are hidden for ranges longer than 7 days.**\n\nCounting log lines means reading every one of them — Loki indexes labels, not counts — so these panels scan the whole selection rather than an index.\n\nShorten the time range, or narrow the namespace and app pickers, and they come back. The feeds below are unaffected at any range: they stop at the first page of matches. For counting over longer spans, [Explore](/explore) is the better tool.", "mode": "markdown" } }, "version": "" } } }, "alloy-warning-event-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n job=\"loki.source.kubernetes_events\",\n namespace=~\"$alloyNamespace\",\n level=\"WARN\"\n}\n |~ \"(?i)$logSearch\"\n | name=~\"(${alloyRole:regex}).*|.+-validate-(agent|gateway)(-.*)?\"\n | name=~\"$alloyPod\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Kubernetes warning events about the collectors, newest first.**\n\n**Nominal:** Nothing.\n\n**Unhealthy:** Warnings that keep arriving. The reason and message name the cause.\n\n`Warning` is the reporting component's own judgement, not a threshold of ours.", "id": 1085, "links": [], "title": "Warning Events", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No warning events about the collectors in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "signature", "detailsMode": "inline", "displayedFields": [ "reason", "name", "msg" ], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": true, "wrapLogMessage": false } }, "version": "" } } }, "cluster-health-score": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (pod) (\n cluster_node_gossip_health_score{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (pod) (\n cluster_node_gossip_health_score{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Each gateway replica's gossip health score, where zero is healthy.**\n\n**Nominal:** Zero.\n\n**Degraded:** Above zero. The replica is missing gossip messages, usually from CPU starvation or network trouble between replicas. _Resources -> CPU_ says whether it is being throttled.", "id": 1071, "links": [], "title": "Gossip Health Score", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Only the gateway runs clustered", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "cluster-members": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "min by (pod, state) (\n cluster_ready_for_traffic{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n * on (namespace, pod) group_left (state)\n cluster_node_info{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "min by (pod, state) (\n cluster_ready_for_traffic{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n * on (namespace, pod) group_left (state)\n cluster_node_info{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true }, "indexByName": { "Value": 2, "pod": 0, "state": 1 }, "renameByName": { "Value": "ready for traffic" } } } } ] } }, "description": "**Each gateway replica's own cluster state, and whether it considers the cluster ready for traffic.**\n\n**Nominal:** Every replica a `participant`, and every one ready.\n\n**Degraded:** A replica not ready. The cluster has fewer members than its configured minimum, and the replica's clustered components wait rather than scrape.\n\n`terminating` is a replica shutting down and handing its targets on. It is expected during a rollout and should clear within a minute.", "id": 1074, "links": [], "title": "Cluster Members", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "Only the gateway runs clustered", "unit": "bool_yes_no" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "cluster-peers": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n cluster_node_peers{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}} sees", "qryType": 1, "query": "sum by (pod) (\n cluster_node_peers{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count(\n cluster_node_info{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "gateways running", "qryType": 1, "query": "count(\n cluster_node_info{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How many cluster members each gateway replica can see, beside how many replicas are running.**\n\n**Nominal:** Every replica sees the same number, and it equals the number running.\n\n**Unhealthy:** A replica seeing fewer. It has lost gossip contact with its peers, and both sides assign themselves the targets they think are unowned. The result is targets scraped twice, and duplicate samples the store rejects.\n\nOnly the gateway runs clustered. The agent reads only its own node and publishes no cluster metrics.", "id": 1070, "links": [], "title": "Peers Seen", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "Only the gateway runs clustered", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "cluster-targets": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n prometheus_target_scrape_pool_targets{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n prometheus_target_scrape_pool_targets{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metrics targets each gateway replica is scraping.**\n\n**Nominal:** Roughly even. Clustering splits targets across replicas by hashing them, so the split is close but not exact.\n\n**Degraded:** One replica carrying all of them while others carry none. Those replicas are not in the cluster, or clustering is not enabled for the components that scrape.", "id": 1072, "links": [], "title": "Targets per Gateway", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Only the gateway runs clustered", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "cluster-transport-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n rate(\n cluster_transport_tx_packets_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n+ sum by (pod) (\n rate(\n cluster_transport_stream_tx_packets_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n rate(\n cluster_transport_tx_packets_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n+ sum by (pod) (\n rate(\n cluster_transport_stream_tx_packets_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Gossip messages per second each gateway replica failed to send to its peers.**\n\n**Nominal:** Zero.\n\n**Degraded:** A sustained rate. Replicas cannot reach each other on the clustering port, most often because a NetworkPolicy does not allow it.", "id": 1073, "links": [], "title": "Gossip Send Failures", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Only the gateway runs clustered", "unit": "suffix:messages/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "components-config-by-pod": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (app, pod, sha256) (\n alloy_config_hash{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n * on (namespace, pod) group_left ()\n alloy_config_last_load_successful{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "format": "table", "instant": true, "legendFormat": "last load succeeded", "qryType": 2, "query": "max by (app, pod, sha256) (\n alloy_config_hash{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n * on (namespace, pod) group_left ()\n alloy_config_last_load_successful{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (app, pod) (\n time()\n - alloy_resources_process_start_time_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "format": "table", "instant": true, "legendFormat": "uptime", "qryType": 2, "query": "max by (app, pod) (\n time()\n - alloy_resources_process_start_time_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [ { "group": "merge", "kind": "Transformation", "spec": { "options": {} } }, { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true }, "indexByName": { "Value #query-0": 5, "Value #query-1": 6, "app": 0, "last load succeeded": 3, "pod": 1, "sha256": 2, "uptime": 4 }, "renameByName": { "Value #query-0": "last load succeeded", "Value #query-1": "uptime", "sha256": "config hash" } } } } ] } }, "description": "**The configuration each collector is running, by content hash, whether its last load succeeded, and how long the collector has been up.**\n\n**Nominal:** One hash per role, every load successful, and uptimes that differ only by when each pod last rolled.\n\n**Degraded:** One collector with a much shorter uptime than its peers. It restarted on its own rather than in a rollout, and _Overview -> Collection Health_ counts the restarts.\n\n**Unhealthy:** A failed load. The collector is still running the hash shown, which is the last configuration that loaded, rather than the one it was given.\n\nThe chart restarts a collector to change its configuration, so a collector's uptime is also how long it has run the hash shown. Two hashes for one role is a rollout in progress. _Distinct Configurations_ beside this says whether it finished.", "id": 1064, "links": [], "title": "Configuration by Collector", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No data from the collectors. Check that the gateway is running." }, "overrides": [ { "matcher": { "id": "byName", "options": "last load succeeded" }, "properties": [ { "id": "unit", "value": "bool_yes_no" } ] }, { "matcher": { "id": "byName", "options": "uptime" }, "properties": [ { "id": "unit", "value": "dtdurations" } ] } ] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "components-distinct-configs": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count by (app) (\n count by (app, sha256) (\n alloy_config_hash{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n )\n)\n", "legendFormat": "{{app}}", "qryType": 1, "query": "count by (app) (\n count by (app, sha256) (\n alloy_config_hash{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Distinct configurations running per role.**\n\n**Nominal:** One per role.\n\n**Degraded:** Two for longer than a rollout takes. Some collectors of that role did not restart onto the new configuration, and the fleet is split.", "id": 1065, "links": [], "title": "Distinct Configurations", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "components-eval-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (le, app) (\n rate(\n alloy_component_evaluation_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "{{app}}", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (le, app) (\n rate(\n alloy_component_evaluation_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long a component evaluation takes, at the 99th percentile, per role.**\n\n**Nominal:** Milliseconds.\n\n**Degraded:** Seconds. Evaluations queue behind the slow one, and changes such as a new scrape target take that much longer to take effect.", "id": 1067, "links": [], "title": "Evaluation Latency (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "components-eval-queue": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (pod) (\n alloy_component_evaluation_queue_size{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (pod) (\n alloy_component_evaluation_queue_size{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Component evaluations waiting to run, per collector.**\n\n**Nominal:** Zero or one.\n\n**Degraded:** A queue that stays above a handful. The controller is not keeping up with changes to its inputs. _Evaluation Latency_ says whether one slow evaluation is the cause.", "id": 1069, "links": [], "title": "Evaluation Queue", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "components-eval-rate": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (app) (\n rate(\n alloy_component_evaluation_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{app}}", "qryType": 1, "query": "sum by (app) (\n rate(\n alloy_component_evaluation_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Component evaluations per second per role.**\n\n**Nominal:** A steady rate. Components re-evaluate when their inputs change, so a busier cluster evaluates more often.\n\n**Degraded:** A sustained step up with no change to the cluster. Some input is changing continuously, such as a discovery result that flaps, and every downstream component re-evaluates with it.", "id": 1066, "links": [], "title": "Evaluation Rate", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:evaluations/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "components-eval-slow": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (app, component_id) (\n rate(\n alloy_component_evaluation_slow_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\nor on (app)\n0 * sum by (app) (\n rate(\n alloy_component_evaluation_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{app}} {{component_id}}", "qryType": 1, "query": "sum by (app, component_id) (\n rate(\n alloy_component_evaluation_slow_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\nor on (app)\n0 * sum by (app) (\n rate(\n alloy_component_evaluation_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Time per second spent in component evaluations that Alloy classed as slow, per component.**\n\n**Nominal:** A flat zero per role.\n\n**Degraded:** Any component listed. It is holding up every evaluation queued behind it, and its name says where to look.\n\nAlloy records this only for an evaluation over its slowness threshold, so a healthy collector publishes no series of its own. The zero line comes from its ordinary evaluations instead.\n\nA flat zero means the collectors reported and there was nothing to show. No data means nothing was collected at all.", "id": 1068, "links": [], "title": "Slow Components", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "components-not-healthy": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod, health_type) (\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n health_type!=\"healthy\"\n }\n)\nor on (pod)\n0 * sum by (pod) (\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}} {{health_type}}", "qryType": 1, "query": "sum by (pod, health_type) (\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n health_type!=\"healthy\"\n }\n)\nor on (pod)\n0 * sum by (pod) (\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Components per collector in any state other than healthy.**\n\n**Nominal:** A flat zero per collector.\n\n**Unhealthy:** `unhealthy` is a component that failed to evaluate or failed while running, and it has stopped passing data on. `exited` is a component that stopped. `unknown` for more than a few seconds after startup is a component that never finished starting.\n\nAlloy publishes the count per state rather than per component. The collector's log under _Logs_ names the component.\n\nA flat zero means the collectors reported and there was nothing to show. No data means nothing was collected at all.", "id": 1062, "links": [], "title": "Components Not Healthy", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "components-running": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Components each collector is running.**\n\n**Nominal:** The same count for every collector of a role, since they share one configuration.\n\n**Degraded:** One collector running fewer than its peers. Part of its configuration failed to load, or it is running a different configuration.", "id": 1063, "links": [], "title": "Components Running", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ingest-flow-lines": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n loki_source_api_entries_written{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "lines/s", "qryType": 1, "query": "sum(\n rate(\n loki_source_api_entries_written{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second the gateway accepted on its log-push listener.**\n\n**Nominal:** Close to what the agents send, plus anything else in the cluster pushing logs to the gateway.\n\n**Unhealthy:** A fall to zero while the agents are running. They are failing to reach the gateway, and _Log Pipeline -> Delivery_ shows their side of it.", "id": 1050, "links": [], "title": "Log Lines Received", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#BBBBBB", "mode": "shades" }, "min": 0, "noValue": "Nothing is pushing logs to the gateway", "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "ingest-flow-refused": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n (\n sum(\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n route!=\"other\",\n status_code!~\"2..\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n + (\n sum(\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n route!=\"other\",\n status_code!~\"2..\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n)\nand on ()\n(\n sum(\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or\n sum(\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "refused/s", "qryType": 1, "query": "(\n (\n sum(\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n route!=\"other\",\n status_code!~\"2..\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n + (\n sum(\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n route!=\"other\",\n status_code!~\"2..\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n)\nand on ()\n(\n sum(\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or\n sum(\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Push requests per second the gateway answered with an error, across its log-push and remote-write listeners.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** A sustained rate. Something is sending data the gateway will not take, and that data is lost unless the sender retries successfully. _Ingest_ says which listener, which status, and which sender.\n\nCounts requests to the push routes that reached the gateway's HTTP layer. Requests to any other path, such as a probe sending `GET` to the push port and receiving `405`, are not refused pushes and are left out; the per-listener panels on _Ingest_ still show them. A sender whose TLS handshake fails never gets that far, and _Ingest -> Connections_ carries that case. OTLP refusals are counted in points rather than requests, and appear under _Ingest -> OTLP_.", "id": 1052, "links": [], "title": "Push Requests Refused", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No push requests reached the gateway", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.01 }, { "color": "#FDB366", "value": 2.5075 }, { "color": "#F67E4B", "value": 5.005 }, { "color": "#DD3D2D", "value": 7.5024999999999995 }, { "color": "#A50026", "value": 10 } ] }, "unit": "reqps" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "ingest-flow-samples": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_forwarded_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n component_id=~\"prometheus\\\\.receive_http\\\\..*\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "samples/s", "qryType": 1, "query": "sum(\n rate(\n prometheus_forwarded_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n component_id=~\"prometheus\\\\.receive_http\\\\..*\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Samples per second the gateway accepted over Prometheus remote write.**\n\n**Nominal:** Empty unless something remote-writes to the gateway. The chart points the Thanos ruler at it, so a default install carries the ruler's recorded series here.\n\n**Unhealthy:** A fall to zero while a sender is configured. _Remote-Write Senders_ below says whether the sender is failing.", "id": 1051, "links": [], "title": "Remote-Write Samples Received", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#BBBBBB", "mode": "shades" }, "min": 0, "noValue": "Nothing is remote-writing to the gateway", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "ingest-log-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (le) (\n rate(\n loki_source_api_request_duration_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "p99", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (le) (\n rate(\n loki_source_api_request_duration_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long the gateway takes to accept a log push, at the 99th percentile.**\n\n**Nominal:** Milliseconds.\n\n**Unhealthy:** Seconds. The gateway is slow to take data, usually because its own delivery to Loki is backed up, and every sender queues behind it.", "id": 1054, "links": [], "title": "Log Push Latency (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Nothing is pushing logs to the gateway", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ingest-log-requests": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (route, status_code) (\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{route}} {{status_code}}", "qryType": 1, "query": "sum by (route, status_code) (\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log push requests per second the gateway received, split by the status it returned.**\n\n**Nominal:** Every request a 2xx.\n\n**Unhealthy:** Non-2xx responses. The gateway is refusing its senders, most often because its own delivery to Loki is backed up. _Log Pipeline -> Delivery_ says whether it is.\n\nOnly requests that reached the HTTP layer. A sender whose TLS handshake fails never reaches this counter, and _Connections_ below carries that case.", "id": 1053, "links": [], "title": "Log Push Requests by Status", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Nothing is pushing logs to the gateway", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ingest-otlp-logs": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n otelcol_receiver_accepted_log_records_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "accepted", "qryType": 1, "query": "sum(\n rate(\n otelcol_receiver_accepted_log_records_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n otelcol_receiver_refused_log_records_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "refused", "qryType": 1, "query": "sum(\n rate(\n otelcol_receiver_refused_log_records_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log records per second the gateway accepted and refused over OTLP.**\n\n**Nominal:** Empty unless something in the cluster sends logs over OTLP. When something does, nothing refused.\n\n**Unhealthy:** Refused records. The gateway could not take them, usually under memory pressure, and the sender decides whether to retry.", "id": 1060, "links": [], "title": "OTLP Log Records", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Nothing has sent logs to the gateway over OTLP", "unit": "suffix:records/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ingest-otlp-metrics": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n otelcol_receiver_accepted_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "accepted", "qryType": 1, "query": "sum(\n rate(\n otelcol_receiver_accepted_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n otelcol_receiver_refused_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "refused", "qryType": 1, "query": "sum(\n rate(\n otelcol_receiver_refused_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metric points per second the gateway accepted and refused over OTLP.**\n\n**Nominal:** Empty unless something in the cluster sends metrics over OTLP. When something does, nothing refused.\n\n**Unhealthy:** Refused points. The gateway could not take them, usually under memory pressure, and the sender decides whether to retry.", "id": 1059, "links": [], "title": "OTLP Metric Points", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Nothing has sent metrics to the gateway over OTLP", "unit": "suffix:points/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ingest-rw-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (le) (\n rate(\n prometheus_receive_http_request_duration_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "p99", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (le) (\n rate(\n prometheus_receive_http_request_duration_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long the gateway takes to answer a remote-write request, at the 99th percentile.**\n\n**Nominal:** Milliseconds.\n\n**Unhealthy:** Seconds. Senders run out of parallelism waiting on the gateway, and their queues grow.", "id": 1057, "links": [], "title": "Remote-Write Latency (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Nothing is remote-writing to the gateway", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ingest-rw-requests": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (route, status_code) (\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{route}} {{status_code}}", "qryType": 1, "query": "sum by (route, status_code) (\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Remote-write requests per second the gateway answered, split by status code.**\n\n**Nominal:** Every request a 2xx.\n\n**Unhealthy:** 4xx is the gateway rejecting a batch as malformed, which the sender does not retry. 5xx is the gateway failing to take a batch, usually because it could not pass the samples on. The sender retries those, and _Remote-Write Senders_ shows the retries accumulating.\n\nOnly requests that reached the HTTP layer. A failed TLS handshake is under _Connections_ below.", "id": 1055, "links": [], "title": "Remote-Write Requests by Status", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Nothing is remote-writing to the gateway", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ingest-rw-samples": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_forwarded_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n component_id=~\"prometheus\\\\.receive_http\\\\..*\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "accepted", "qryType": 1, "query": "sum(\n rate(\n prometheus_forwarded_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n component_id=~\"prometheus\\\\.receive_http\\\\..*\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_api_remote_write_invalid_labels_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "dropped: invalid labels", "qryType": 1, "query": "sum(\n rate(\n prometheus_api_remote_write_invalid_labels_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Samples per second the gateway accepted over remote write, beside the samples it dropped for carrying invalid labels.**\n\n**Nominal:** Accepted tracks the senders. Dropped stays at zero.\n\n**Degraded:** Dropped samples. A sender is writing label names or values the gateway cannot store, and those samples are not kept.", "id": 1056, "links": [], "title": "Remote-Write Samples", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Nothing is remote-writing to the gateway", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ingest-rw-senders": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (job) (\n rate(\n prometheus_remote_storage_samples_total{\n url=~\".*alloy-gateway.*\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{job}} sent", "qryType": 1, "query": "sum by (job) (\n rate(\n prometheus_remote_storage_samples_total{\n url=~\".*alloy-gateway.*\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (job) (\n rate(\n prometheus_remote_storage_samples_failed_total{\n url=~\".*alloy-gateway.*\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{job}} failed", "qryType": 1, "query": "sum by (job) (\n rate(\n prometheus_remote_storage_samples_failed_total{\n url=~\".*alloy-gateway.*\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (job) (\n rate(\n prometheus_remote_storage_samples_retried_total{\n url=~\".*alloy-gateway.*\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{job}} retried", "qryType": 1, "query": "sum by (job) (\n rate(\n prometheus_remote_storage_samples_retried_total{\n url=~\".*alloy-gateway.*\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-2" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**What each remote-write sender pointed at the gateway reports about its own queue: samples sent, failed, and retried.**\n\n**Nominal:** Sent steady. Failed and retried at zero.\n\n**Unhealthy:** Retried close to sent. Almost every attempt is a retry, so little or nothing is arriving. This is the one view of remote-write ingest that sees a sender which cannot connect at all, since such a sender reaches no gateway counter.\n\nFound by the sender's own `url`, matched against `alloy-gateway`. Only senders this stack scrapes appear, so an external sender, or one addressing the gateway by IP, does not. The collector pickers do not narrow it, because the series belong to the sender.", "id": 1058, "links": [], "title": "Remote-Write Senders", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No scraped sender is remote-writing to the gateway", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ingest-tls-rejections": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (app) (\n rate(\n {\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\"\n }\n |= \"TLS handshake error\"\n | pod=~\"$alloyPod\"\n [$__auto]\n )\n) * 60\n", "legendFormat": "{{app}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**TLS handshakes per minute that a collector's listener refused, usually because the client presented a certificate the listener does not trust.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** A sustained rate. A client whose handshake fails never reaches any counter, so its data is missing from every other panel as well. The line names the client's address. `kubectl get pods -o wide` maps that address to a pod.\n\nRead from the collectors' own logs, since no metric records a failed handshake. Covers every listener a collector serves: log push, OTLP, and remote write into the gateway. Its most common cause is a client rolled onto a new certificate before or after the gateway trusts it.", "id": 1061, "links": [], "title": "Rejected TLS Handshakes", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No refused handshakes. If the Logs tab is empty too, the gateway's log is not arriving.", "unit": "suffix:handshakes/min" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-agents-file-lines": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n rate(\n loki_source_file_read_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n rate(\n loki_source_file_read_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Container log lines per second each agent is reading.**\n\n**Nominal:** Uneven across agents. The rate follows whichever workloads a node runs.\n\n**Unhealthy:** One agent at zero while its node runs workloads, next to a non-zero _Files Tailed_. The agent has the files open and is not reading them.", "id": 1019, "links": [], "title": "Container Log Lines Read", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No matches for the current filters", "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-agents-files": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n loki_source_file_files_active_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n loki_source_file_files_active_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Container log files each agent currently has open.**\n\n**Nominal:** A count per agent that tracks how many containers run on that node.\n\n**Unhealthy:** Zero on a node that is running workloads. That node's container logs are not being collected, and nothing downstream can report it.", "id": 1018, "links": [], "title": "Files Tailed", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No matches for the current filters", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-agents-journal-lines": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n rate(\n loki_source_journal_target_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n rate(\n loki_source_journal_target_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Node journal lines per second each agent is reading.**\n\n**Nominal:** A low, steady rate per node from the kubelet, the container runtime and the kernel.\n\n**Unhealthy:** One agent at zero while its peers read. The journal mount on that node is missing or unreadable, which the agent's own log states at startup.", "id": 1020, "links": [], "title": "Journal Lines Read", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No matches for the current filters", "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-delivery-dropped": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (app, reason) (\n rate(\n loki_write_dropped_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n) > 0\nor on (app)\n0 * sum by (app) (\n rate(\n loki_write_sent_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{app}} {{reason}}", "qryType": 1, "query": "sum by (app, reason) (\n rate(\n loki_write_dropped_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n) > 0\nor on (app)\n0 * sum by (app) (\n rate(\n loki_write_sent_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second a collector gave up delivering, split by collector role and reason.**\n\n**Nominal:** Zero on every reason.\n\n**Unhealthy:** `ingester_error` and `rate_limited` are the destination refusing the lines. `stream_limited` is Loki's per-stream limit. `queue_is_full` is the collector's own buffer overflowing, which means delivery has been slower than collection for a while. `line_too_long` is a single line above the destination's size limit.\n\nOnly reasons with drops are drawn, beside a zero line per role that is delivering at all. The same lines appear on the Loki dashboard's discard panels, seen from the receiving end, when Loki is the one refusing them.\n\nA flat zero means the collectors reported and there was nothing to show. No data means nothing was collected at all.", "id": 1022, "links": [], "title": "Lines Dropped at Delivery", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-delivery-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (le, host) (\n rate(\n loki_write_request_duration_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "{{host}}", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (le, host) (\n rate(\n loki_write_request_duration_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long log delivery requests take to be acknowledged, at the 99th percentile, per destination.**\n\n**Nominal:** Well under a second for the agent-to-gateway hop, and under a second into Loki.\n\n**Unhealthy:** A climb into seconds. The destination is saturated, and retries and drops follow if it persists.", "id": 1024, "links": [], "title": "Delivery Latency (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-delivery-requests": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (host, status_code) (\n rate(\n loki_write_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{host}} {{status_code}}", "qryType": 1, "query": "sum by (host, status_code) (\n rate(\n loki_write_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log delivery requests per second, split by destination and the status code the destination returned.**\n\n**Nominal:** Every request a 2xx.\n\n**Unhealthy:** `429` is the destination rate-limiting the sender. Other 4xx codes are the destination refusing the batch, which is not retried. 5xx codes and `-1`, a request that never received a response, are retried until the sender's buffer fills.\n\nCounts each request once. A panel built by summing the histogram's `_bucket` series instead counts every request once per bucket.", "id": 1021, "links": [], "title": "Delivery Requests by Status", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-delivery-retries": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (app, host) (\n rate(\n loki_write_batch_retries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{app}} to {{host}}", "qryType": 1, "query": "sum by (app, host) (\n rate(\n loki_write_batch_retries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log delivery batches per second a collector had to retry, per destination.**\n\n**Nominal:** Zero, or a background trickle.\n\n**Unhealthy:** A sustained rate. Retries precede drops. The collector is buffering because the destination is refusing or slow, and it starts discarding once the buffer fills.", "id": 1023, "links": [], "title": "Delivery Retries", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:batches/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-flow-lost": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n loki_write_dropped_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "lost/s", "qryType": 1, "query": "sum(\n rate(\n loki_write_dropped_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second the collectors gave up delivering, across every hop and every reason.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Anything sustained. These lines were collected and will never be stored. _Log Pipeline -> Delivery_ splits them by reason, which says whether the receiver refused them or the collector ran out of room.\n\nCounts the agent-to-gateway hop and the gateway-to-Loki hop alike. A rejection by Loki also appears on the Loki dashboard's discard panels, seen from the receiving end.", "id": 1016, "links": [], "title": "Log Lines Lost", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.01 }, { "color": "#FDB366", "value": 2.5075 }, { "color": "#F67E4B", "value": 5.005 }, { "color": "#DD3D2D", "value": 7.5024999999999995 }, { "color": "#A50026", "value": 10 } ] }, "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "logs-flow-propagation": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (le, app) (\n rate(\n loki_write_entry_propagation_latency_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "{{app}}", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (le, app) (\n rate(\n loki_write_entry_propagation_latency_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long a log line took from being written to being accepted by the next hop, at the 99th percentile, per collector role.**\n\n**Nominal:** A few seconds. Each hop batches before it sends, so the floor is the batch interval rather than zero.\n\n**Unhealthy:** Tens of seconds and climbing. Logs are arriving late, and an incident's explanatory lines surface after the investigation has moved on.\n\nMeasured from the line's own timestamp. The gateway's figure therefore includes the agent hop, and approximates end-to-end delay into Loki.", "id": 1017, "links": [], "title": "Time to Delivery (p99)", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#009988", "mode": "shades" }, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "s" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value_and_name", "wideLayout": true } }, "version": "" } } }, "logs-flow-read": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n loki_source_file_read_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n+ (\n sum(\n rate(\n loki_source_journal_target_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n)\n", "legendFormat": "lines/s", "qryType": 1, "query": "sum(\n rate(\n loki_source_file_read_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n+ (\n sum(\n rate(\n loki_source_journal_target_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second the agents are reading, from container log files and the node journal together.**\n\n**Nominal:** A steady rate that tracks how much the cluster's workloads are writing.\n\n**Unhealthy:** Zero while workloads are running. The agents are up and collecting nothing, which usually means pod discovery failed.", "id": 1014, "links": [], "title": "Lines Read", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#009988", "mode": "shades" }, "min": 0, "noValue": "No agent is reporting what it reads", "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "logs-flow-stored": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n loki_write_sent_entries_total{\n app=~\"$alloyRole\",\n app=\"alloy-gateway\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "lines/s", "qryType": 1, "query": "sum(\n rate(\n loki_write_sent_entries_total{\n app=~\"$alloyRole\",\n app=\"alloy-gateway\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second the gateway delivered to its log destinations.**\n\n**Nominal:** Close to what the agents read, plus Kubernetes events. The gateway also drops the debug tap's echo and anything its guards discard, so a small shortfall against the agents is expected.\n\n**Unhealthy:** A widening gap against _Lines Read_. Lines are being lost between reading and storage, and _Delivery_ below says at which hop.\n\nBoth this and _Lines Read_ are short-window rates, and the gateway sends in batches, so at any instant the two can differ by half or more. Over the range they agree to within a few percent: on a reference install, 17.7 lines per second read against 17.6 delivered over six hours.", "id": 1015, "links": [], "title": "Lines Delivered to Store", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#009988", "mode": "shades" }, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "logs-processing-guards": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (app, reason) (\n rate(\n loki_process_dropped_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n reason!~\"sampleDebug|skip self|skip loki|alloy loki.echo\"\n }\n [$__rate_interval]\n )\n) > 0\nor on (app)\n0 * sum by (app) (\n rate(\n loki_process_dropped_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{app}} {{reason}}", "qryType": 1, "query": "sum by (app, reason) (\n rate(\n loki_process_dropped_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n reason!~\"sampleDebug|skip self|skip loki|alloy loki.echo\"\n }\n [$__rate_interval]\n )\n) > 0\nor on (app)\n0 * sum by (app) (\n rate(\n loki_process_dropped_lines_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second a pipeline guard discarded before delivery, split by reason.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** `ratelimit_drop_stage` is a namespace writing faster than the gateway's per-namespace rate limit allows. `backlog > 2hr` and `processed time > 2hr old` are lines older than the store accepts, most often a backlog draining after an outage. Each of these lines is lost.\n\nExcludes the debug tap's sampling and its echo suppression, which drop copies that were never bound for a store. A reason not listed above is new, and counts as a loss until it is classified.\n\nA flat zero means the collectors reported and there was nothing to show. No data means nothing was collected at all.", "id": 1025, "links": [], "title": "Lines Discarded by Guards", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-processing-truncated": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod, field) (\n label_replace(\n rate(\n loki_process_cri_lines_truncated_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n ),\n \"field\", \"cri_reassembly\", \"\", \"\"\n )\n or\n rate(\n loki_process_truncated_fields_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{pod}} {{field}}", "qryType": 1, "query": "sum by (pod, field) (\n label_replace(\n rate(\n loki_process_cri_lines_truncated_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n ),\n \"field\", \"cri_reassembly\", \"\", \"\"\n )\n or\n rate(\n loki_process_truncated_fields_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Cuts per second the pipelines made to log lines and their fields, split by Alloy pod and by what was cut.**\n\n**Nominal:** Zero.\n\n**Degraded:** A steady rate. Some workload writes lines longer than a limit, and the tail of each is lost.\n\n`cri_reassembly` is an agent cutting a line the container runtime had split, when the reassembled line is longer than the agent reassembles. `line`, `label`, and `structured_metadata` are a size limit cutting a value, and the cut line ends in a suffix naming the rule: `[TRUNCATED:SAFETYCLAMP]` for the 1MiB clamp on the agent and the gateway, `[TRUNCATED:LINE]` for the gateway's line limit, and `[TRUNCATED:FIELDS]` for its fixed backstop. A line the clamp and the line limit both cut counts twice.", "id": 1026, "links": [], "title": "Truncated Lines", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No matches for the current filters", "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-flow-delay": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max(\n clamp_min(\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n - ignoring (url, remote_name) group_right (pod)\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n },\n 0\n ) < 31536000\n)\n", "legendFormat": "behind", "qryType": 1, "query": "max(\n clamp_min(\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n - ignoring (url, remote_name) group_right (pod)\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n },\n 0\n ) < 31536000\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The furthest behind any remote-write queue is sending, in seconds.**\n\n**Nominal:** Close to zero.\n\n**Unhealthy:** A delay that grows steadily. _Remote Write_ below says which queue and whether the destination is slow or refusing.\n\nThe maximum of `infra.alloy.metric_pipeline.send_delay` across every queue and replica.", "id": 1031, "links": [], "title": "Worst Send Delay", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE7733", "mode": "shades" }, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "s" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "metrics-flow-lost": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n (\n sum(\n rate(\n prometheus_remote_storage_samples_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n + (\n sum(\n rate(\n otelcol_exporter_send_failed_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n)\nand on ()\n(\n sum(\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or\n sum(\n rate(\n otelcol_exporter_sent_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "lost/s", "qryType": 1, "query": "(\n (\n sum(\n rate(\n prometheus_remote_storage_samples_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n + (\n sum(\n rate(\n otelcol_exporter_send_failed_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n)\nand on ()\n(\n sum(\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or\n sum(\n rate(\n otelcol_exporter_sent_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metric samples per second the gateway failed to deliver and will not retry, across every metrics destination.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** A sustained rate. A burst right after a gateway restart is usually the write-ahead log replaying samples the store already has, which it rejects as duplicates with `409 Conflict`. That burst is not a finding.\n\nRemote-write failures are samples the destination rejected outright, which are not retried. Recoverable errors are retried instead, and appear under _Metric Pipeline -> Remote Write_ as retries.\n\nEither kind of destination may be absent, since a deployment can run without the bundled store or without any OpenTelemetry exporter. Each contributes zero when absent, and the panel reads no data only when neither is reporting.", "id": 1030, "links": [], "title": "Samples Lost", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.01 }, { "color": "#FDB366", "value": 25.0075 }, { "color": "#F67E4B", "value": 50.004999999999995 }, { "color": "#DD3D2D", "value": 75.0025 }, { "color": "#A50026", "value": 100 } ] }, "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "metrics-flow-samples": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "samples/s", "qryType": 1, "query": "sum(\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Samples per second the gateway wrote to its remote-write destinations.**\n\n**Nominal:** A steady rate that tracks the number of targets and how much each exposes.\n\n**Unhealthy:** A sudden fall with no matching fall in targets. The gateway is collecting and not delivering, and _Send Delay_ grows alongside.", "id": 1029, "links": [], "title": "Samples Written", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE7733", "mode": "shades" }, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "metrics-flow-targets": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n prometheus_target_scrape_pool_targets{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "targets", "qryType": 1, "query": "sum(\n prometheus_target_scrape_pool_targets{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metrics targets the gateways are scraping, summed across replicas.**\n\n**Nominal:** A steady count that moves with the workloads running. Each target is scraped by one gateway replica, so the sum is the cluster's total.\n\n**Unhealthy:** A sudden drop. A monitor stopped matching its targets, or a gateway replica left the cluster and its share was not picked up.", "id": 1027, "links": [], "title": "Targets Scraped", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#EE7733", "mode": "shades" }, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "metrics-flow-targets-down": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count(up == 0)\nor\n0 * count(up)\n", "legendFormat": "down", "qryType": 1, "query": "count(up == 0)\nor\n0 * count(up)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metrics targets whose most recent scrape failed.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. Each target down is a source whose metrics are missing from every dashboard. _Scraping_ below lists them.\n\nCovers every target the gateway scrapes, so the collector pickers do not narrow it.", "id": 1028, "links": [], "title": "Targets Down", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 2 }, { "color": "#F67E4B", "value": 3 }, { "color": "#DD3D2D", "value": 4 }, { "color": "#A50026", "value": 5 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "metrics-heavy-largest": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "topk(\n 10,\n sum by (job) (\n scrape_samples_post_metric_relabeling\n )\n)\n", "legendFormat": "{{job}}", "qryType": 1, "query": "topk(\n 10,\n sum by (job) (\n scrape_samples_post_metric_relabeling\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The ten scrape jobs contributing the most samples per scrape, after relabelling.**\n\n**Nominal:** cAdvisor and kube-state-metrics near the top, since both describe every container in the cluster.\n\n**Degraded:** A job climbing steadily. Series count drives the gateway's memory, the store's size, and the bill of any metered destination, and this is where a cardinality problem first shows.\n\nCovers every target the gateway scrapes, so the collector pickers do not narrow it.", "id": 1038, "links": [], "title": "Largest Scrapes (Top 10)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:samples" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-heavy-slowest": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "topk(\n 10,\n max by (job) (\n scrape_duration_seconds\n )\n)\n", "legendFormat": "{{job}}", "qryType": 1, "query": "topk(\n 10,\n max by (job) (\n scrape_duration_seconds\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The ten scrape jobs taking longest to scrape, by their slowest target.**\n\n**Nominal:** Well under the scrape interval. SQL-backed exporters and the kubelet's cAdvisor endpoint are usually the slowest.\n\n**Degraded:** A job approaching its scrape timeout. When it passes it, that target reads as down with no fault on the target's side.\n\nCovers every target the gateway scrapes, so the collector pickers do not narrow it.", "id": 1037, "links": [], "title": "Slowest Scrapes (Top 10)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-otel-failed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n otelcol_exporter_send_failed_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}} metrics", "qryType": 1, "query": "sum by (component_id) (\n rate(\n otelcol_exporter_send_failed_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n otelcol_exporter_send_failed_log_records_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}} logs", "qryType": 1, "query": "sum by (component_id) (\n rate(\n otelcol_exporter_send_failed_log_records_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metric points and log records per second each OpenTelemetry exporter failed to deliver.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** A sustained rate. The gateway's log names the exporter and quotes the destination's error, which is usually an authentication, quota or schema problem on the destination's side.", "id": 1047, "links": [], "title": "Export Failures", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No OpenTelemetry destination is configured", "unit": "suffix:points/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-otel-filtered": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n otelcol_processor_filter_datapoints_filtered_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}}", "qryType": 1, "query": "sum by (component_id) (\n rate(\n otelcol_processor_filter_datapoints_filtered_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metric points per second each OpenTelemetry filter removed.**\n\n**Nominal:** Non-zero and deliberate. A destination's egress filter keeps only the series that destination is meant to receive, so a steady rate is the configuration working.\n\n**Degraded:** A step change nobody made. A filter started matching more than it was written to match, and the points it removes never reach the destination.", "id": 1049, "links": [], "title": "Points Filtered", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No OpenTelemetry destination is configured", "unit": "suffix:points/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-otel-queue": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (component_id, data_type) (\n otelcol_exporter_queue_size{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n / otelcol_exporter_queue_capacity{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{component_id}} {{data_type}}", "qryType": 1, "query": "max by (component_id, data_type) (\n otelcol_exporter_queue_size{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n / otelcol_exporter_queue_capacity{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How full each OpenTelemetry exporter's sending queue is, as a fraction of its capacity.**\n\n**Nominal:** Close to zero.\n\n**Unhealthy:** A queue approaching full. The destination is slower than the gateway's output, and new data is refused once the queue is full.", "id": 1048, "links": [], "title": "Export Queue Fill", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "No OpenTelemetry destination is configured", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-otel-sent": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n otelcol_exporter_sent_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}} metrics", "qryType": 1, "query": "sum by (component_id) (\n rate(\n otelcol_exporter_sent_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n otelcol_exporter_sent_log_records_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}} logs", "qryType": 1, "query": "sum by (component_id) (\n rate(\n otelcol_exporter_sent_log_records_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metric points and log records per second each OpenTelemetry exporter delivered.**\n\n**Nominal:** Empty unless an OpenTelemetry destination is configured. When one is, a steady line per exporter.\n\n**Unhealthy:** An exporter falling to zero. _Export Failures_ beside it says whether the destination is refusing.", "id": 1046, "links": [], "title": "Points Exported", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No OpenTelemetry destination is configured", "unit": "suffix:points/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-rw-delay": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "clamp_min(\n max by (pod, component_id) (\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n - ignoring (url, remote_name) group_right (pod)\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n ),\n 0\n) < 31536000\n", "legendFormat": "{{pod}} {{component_id}}", "qryType": 1, "query": "clamp_min(\n max by (pod, component_id) (\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n - ignoring (url, remote_name) group_right (pod)\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n ),\n 0\n) < 31536000\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How far behind collection each remote-write queue is sending, in seconds.**\n\n**Nominal:** Close to zero. The queue sends samples moments after they are collected.\n\n**Unhealthy:** A delay that grows steadily. The destination cannot accept samples as fast as the gateway collects them. The write-ahead log absorbs the backlog until the delay passes what the destination accepts as recent, and samples are lost from that point.\n\nThe newest sample written to the write-ahead log, minus the newest sample the queue has sent. Values above a year are the queue's initial zero before its first send, and are left out.", "id": 1040, "links": [], "title": "Send Delay", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-rw-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (le, component_id) (\n rate(\n prometheus_remote_storage_sent_batch_duration_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "{{component_id}}", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (le, component_id) (\n rate(\n prometheus_remote_storage_sent_batch_duration_seconds_bucket{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long remote-write batches take to be acknowledged, at the 99th percentile, per destination.**\n\n**Nominal:** Well under a second for the bundled store.\n\n**Unhealthy:** Seconds. The destination is saturated or far away. Shards rise to compensate until they reach their maximum.", "id": 1042, "links": [], "title": "Send Latency (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-rw-samples": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}} sent", "qryType": 1, "query": "sum by (component_id) (\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n prometheus_remote_storage_samples_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}} failed", "qryType": 1, "query": "sum by (component_id) (\n rate(\n prometheus_remote_storage_samples_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n prometheus_remote_storage_samples_retried_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}} retried", "qryType": 1, "query": "sum by (component_id) (\n rate(\n prometheus_remote_storage_samples_retried_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-2" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Samples per second each remote-write destination took, rejected, and had retried.**\n\n**Nominal:** Sent tracks collection. Failed and retried stay at zero.\n\n**Unhealthy:** Retried is the destination answering slowly or with a recoverable error. The samples are resent, and _Send Delay_ grows while they are. Failed is the destination rejecting samples outright, and those are lost.\n\nA burst of failures right after a gateway restart is usually the write-ahead log replaying samples the destination already has, which it rejects as duplicates.", "id": 1039, "links": [], "title": "Samples Sent, Failed and Retried", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-rw-shards": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (component_id) (\n prometheus_remote_storage_shards{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{component_id}} running", "qryType": 1, "query": "max by (component_id) (\n prometheus_remote_storage_shards{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (component_id) (\n prometheus_remote_storage_shards_desired{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{component_id}} desired", "qryType": 1, "query": "max by (component_id) (\n prometheus_remote_storage_shards_desired{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (component_id) (\n prometheus_remote_storage_shards_max{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{component_id}} max", "qryType": 1, "query": "max by (component_id) (\n prometheus_remote_storage_shards_max{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-2" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Parallel senders each remote-write queue is running, against how many it wants and how many it is allowed.**\n\n**Nominal:** Desired well below running. One or two shards carry a small cluster.\n\n**Unhealthy:** Desired at or above the maximum. The queue cannot keep up with collection however many senders it runs, so the destination is the bottleneck. _Send Delay_ confirms it.", "id": 1041, "links": [], "title": "Shards", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-scrape-by-monitor": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (scrape_job) (\n prometheus_target_scrape_pool_targets{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "sum by (scrape_job) (\n prometheus_target_scrape_pool_targets{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true }, "indexByName": { "Value": 1, "scrape_job": 0 }, "renameByName": { "Value": "targets", "scrape_job": "monitor" } } } }, { "group": "sortBy", "kind": "Transformation", "spec": { "options": { "fields": {}, "sort": [ { "field": "targets" } ] } } } ] } }, "description": "**Targets each ServiceMonitor, PodMonitor or scrape job currently matches, summed across gateway replicas.**\n\n**Nominal:** A non-zero count for every monitor whose workload is running.\n\n**Degraded:** Zero for a monitor whose workload is running. Its selector matches nothing, and that workload's metrics are not collected.\n\nZero is correct for the CNI monitors of vendors the cluster does not run. Those ship everywhere and match only their own dataplane.", "id": 1033, "links": [], "title": "Targets by Monitor", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "metrics-scrape-down": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (job, namespace, pod, instance) (\n up == 0\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "max by (job, namespace, pod, instance) (\n up == 0\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true, "Value": true }, "indexByName": { "instance": 3, "job": 0, "namespace": 1, "pod": 2 } } } } ] } }, "description": "**Every metrics target whose most recent scrape failed.**\n\n**Nominal:** Empty.\n\n**Unhealthy:** Any row. The target is discovered and its scrape is failing. The usual causes are a scheme mismatch, where the scrape speaks HTTPS to a plaintext listener or the reverse, a NetworkPolicy blocking the gateway, or the target itself not serving.", "id": 1032, "links": [], "title": "Targets Down", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No target is down. If Targets Scraped is empty too, nothing is being collected." }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "metrics-scrape-rejected": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_target_scrapes_exceeded_sample_limit_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "sample limit", "qryType": 1, "query": "sum(\n rate(\n prometheus_target_scrapes_exceeded_sample_limit_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_target_scrapes_exceeded_body_size_limit_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "body size limit", "qryType": 1, "query": "sum(\n rate(\n prometheus_target_scrapes_exceeded_body_size_limit_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_target_scrape_pool_exceeded_target_limit_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "target limit", "qryType": 1, "query": "sum(\n rate(\n prometheus_target_scrape_pool_exceeded_target_limit_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-2" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_target_scrape_pool_exceeded_label_limits_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "label limits", "qryType": 1, "query": "sum(\n rate(\n prometheus_target_scrape_pool_exceeded_label_limits_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-3" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Whole scrapes per second the gateway discarded for breaking a configured limit.**\n\n**Nominal:** Zero on every series.\n\n**Unhealthy:** Any. A scrape over its sample or body-size limit is dropped entirely, so the target's metrics go missing without the target reading as down. A scrape pool over its target or label limit is dropped the same way.", "id": 1034, "links": [], "title": "Rejected Scrapes", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:scrapes/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-scrape-rejected-samples": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_target_scrapes_sample_out_of_order_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "out of order", "qryType": 1, "query": "sum(\n rate(\n prometheus_target_scrapes_sample_out_of_order_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_target_scrapes_sample_duplicate_timestamp_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "duplicate timestamp", "qryType": 1, "query": "sum(\n rate(\n prometheus_target_scrapes_sample_duplicate_timestamp_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_target_scrapes_sample_out_of_bounds_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "out of bounds", "qryType": 1, "query": "sum(\n rate(\n prometheus_target_scrapes_sample_out_of_bounds_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-2" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Individual samples per second the gateway discarded from otherwise successful scrapes.**\n\n**Nominal:** Zero on every series.\n\n**Degraded:** Out-of-order and duplicate-timestamp samples come from a target exposing the same series twice, or from two targets relabelled onto one series. Out-of-bounds samples carry timestamps too far from now to store.", "id": 1035, "links": [], "title": "Rejected Samples", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-scrape-samples": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n prometheus_forwarded_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n component_id=~\"prometheus\\\\.(scrape|operator)\\\\..*\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}}", "qryType": 1, "query": "sum by (component_id) (\n rate(\n prometheus_forwarded_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n component_id=~\"prometheus\\\\.(scrape|operator)\\\\..*\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Samples per second the gateway scraped, split by the component that scraped them.**\n\n**Nominal:** The monitor-driven scrapes carry most of it, with the kubelet's cAdvisor scrape next.\n\n**Unhealthy:** A component at zero that should not be. For a `prometheus.operator` component, its monitors stopped matching anything, and _Targets by Monitor_ above says which.\n\nSamples pushed into the gateway rather than scraped by it are on _Ingest_.", "id": 1036, "links": [], "title": "Samples Scraped by Source", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-wal-errors": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_remote_write_wal_corruptions_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "corruptions", "qryType": 1, "query": "sum(\n rate(\n prometheus_remote_write_wal_corruptions_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_tsdb_wal_writes_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "failed writes", "qryType": 1, "query": "sum(\n rate(\n prometheus_tsdb_wal_writes_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_remote_write_wal_checkpoint_creations_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "failed checkpoints", "qryType": 1, "query": "sum(\n rate(\n prometheus_remote_write_wal_checkpoint_creations_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-2" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_tsdb_wal_truncations_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "failed truncations", "qryType": 1, "query": "sum(\n rate(\n prometheus_tsdb_wal_truncations_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-3" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Write-ahead log failures per second: corruptions, failed writes, and failed checkpoints or truncations.**\n\n**Nominal:** Zero on every series.\n\n**Unhealthy:** Any. The gateway's buffer against a slow destination is itself failing, most often because the volume under it is full.", "id": 1045, "links": [], "title": "Write-Ahead Log Errors", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:errors/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-wal-series": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n prometheus_remote_write_wal_storage_active_series{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n prometheus_remote_write_wal_storage_active_series{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Series each gateway replica is holding in its write-ahead log.**\n\n**Nominal:** Roughly even across replicas, since clustering splits targets between them.\n\n**Degraded:** Steady growth with no new workloads. A target is producing new series faster than old ones go stale, which is a cardinality problem. _Heaviest Targets_ names the job.", "id": 1043, "links": [], "title": "Active Series", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "metrics-wal-size": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n prometheus_tsdb_wal_storage_size_bytes{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n prometheus_tsdb_wal_storage_size_bytes{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Bytes each gateway replica's write-ahead log occupies on disk.**\n\n**Nominal:** A sawtooth. The log grows between checkpoints and shrinks at each one.\n\n**Degraded:** Growth with no sawtooth. The log is not being truncated, most often because a remote-write queue is far behind and the log is holding samples it has not sent.", "id": 1044, "links": [], "title": "Write-Ahead Log Size", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "bytes" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-agents-missing": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "clamp_min(\n sum(\n kube_daemonset_status_desired_number_scheduled{\n namespace=~\"$alloyNamespace\",\n daemonset=\"alloy-agent\"\n }\n )\n - (\n sum(\n up{\n app=\"alloy-agent\",\n namespace=~\"$alloyNamespace\"\n }\n )\n or vector(0)\n ),\n 0\n)\n", "legendFormat": "agents", "qryType": 1, "query": "clamp_min(\n sum(\n kube_daemonset_status_desired_number_scheduled{\n namespace=~\"$alloyNamespace\",\n daemonset=\"alloy-agent\"\n }\n )\n - (\n sum(\n up{\n app=\"alloy-agent\",\n namespace=~\"$alloyNamespace\"\n }\n )\n or vector(0)\n ),\n 0\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Nodes the agent DaemonSet is scheduled onto whose agent is not reporting.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Anything sustained. Each missing agent is a node whose container logs and journal are not being collected. Nothing downstream reports this, because Loki cannot miss lines it was never sent.\n\nDesired pods come from kube-state-metrics, and reporting pods from the gateway's scrape of each agent. A brief 1 during a DaemonSet rollout is the rollout replacing one agent at a time.\n\nReads the DaemonSet named `alloy-agent`, which is the chart's fixed `fullnameOverride`. The role and pod pickers do not apply, since the count is about the whole DaemonSet.", "id": 1001, "links": [], "title": "Agents Not Reporting", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No metrics: kube-state-metrics is required", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 1.5 }, { "color": "#F67E4B", "value": 2 }, { "color": "#DD3D2D", "value": 2.5 }, { "color": "#A50026", "value": 3 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-alert-history": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count by (alertname) (\n max without (ruler_replica) (\n ALERTS{component=~\"alloy|monitoring\", alertstate=\"firing\"}\n )\n)\n", "legendFormat": "{{alertname}}", "qryType": 1, "query": "count by (alertname) (\n max without (ruler_replica) (\n ALERTS{component=~\"alloy|monitoring\", alertstate=\"firing\"}\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Which alerts about collection were firing over the selected time range.**\n\n**Nominal:** Empty.\n\nA line is the number of firing instances of that alert. Read it beside the tab that alert is about to see what changed when it started.", "id": 1008, "links": [], "title": "Alerts Firing Over Time", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No alert about collection fired in this range", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-alerts": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max without (ruler_replica, tenant_id, cluster, audience) (\n label_replace(\n label_replace(\n ALERTS{component=~\"alloy|monitoring\"},\n \"job\", \"\", \"job\", \"thanos-ruler|loki-ruler\"\n ),\n \"instance\", \"\", \"instance\", \"thanos-ruler|loki-ruler\"\n )\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "max without (ruler_replica, tenant_id, cluster, audience) (\n label_replace(\n label_replace(\n ALERTS{component=~\"alloy|monitoring\"},\n \"job\", \"\", \"job\", \"thanos-ruler|loki-ruler\"\n ),\n \"instance\", \"\", \"instance\", \"thanos-ruler|loki-ruler\"\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true, "Value": true }, "indexByName": { "alertname": 0, "alertstate": 1, "app": 6, "component": 3, "job": 5, "namespace": 7, "pod": 4, "severity": 2 }, "renameByName": { "alertname": "alert", "alertstate": "state" } } } } ] } }, "description": "**Every alert about collection that is firing or pending now, with the target or collector it is about.**\n\n**Nominal:** Empty.\n\n**Unhealthy:** Any firing row. A pending row has met its condition and is waiting out its `for` duration; it fires if the condition holds.\n\n`ALERTS` carries a rule's labels and not its annotations, so the summary and the advice are a click away: each alert name links to its entry under Common Alerts. Alloy dropping log lines is on the Loki dashboard instead, as `alloy-log-drops`, because its usual cause is Loki refusing writes.\n\nAn alert about a collector names it by `pod`; an alert about a scrape target names the target's `job`. A rule that names no `job` or `instance` of its own is given the ruler's, which says nothing about the alert, so those cells read as empty.", "id": 1007, "links": [], "title": "Alerts", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No alert about collection is firing or pending, or none reached Thanos. Collection Alerts Firing says which." }, "overrides": [ { "matcher": { "id": "byType", "options": "string" }, "properties": [ { "id": "noValue", "value": "—" } ] }, { "matcher": { "id": "byName", "options": "alert" }, "properties": [ { "id": "links", "value": [ { "targetBlank": true, "title": "What this alert means", "url": "https://materializeinc.github.io/materialize-monitoring/reference/common-alerts/#${__value.raw}" } ] } ] } ] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "overview-alerts-firing": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count(\n max without (ruler_replica) (\n ALERTS{component=~\"alloy|monitoring\", alertstate=\"firing\"}\n )\n)\nor\n0 * (\n (\n (\n time()\n - max(\n prometheus_rule_group_last_evaluation_timestamp_seconds{\n app=\"thanos\",\n container=\"ruler\"\n }\n )\n < 300\n )\n and on ()\n (\n min(\n max by (pod) (\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=\"thanos\",\n container=\"ruler\"\n }\n )\n - on (pod)\n max by (pod) (\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=\"thanos\",\n container=\"ruler\"\n }\n )\n )\n < 300\n )\n )\n unless on ()\n (\n (\n max(\n (\n time()\n - max by (rule_group, user) (\n loki_prometheus_rule_group_last_evaluation_timestamp_seconds{\n app=\"loki\",\n container=\"ruler\"\n }\n > 0\n )\n )\n - on (rule_group, user)\n max by (rule_group, user) (\n loki_prometheus_rule_group_interval_seconds{\n app=\"loki\",\n container=\"ruler\"\n }\n )\n )\n >= 300\n )\n or on ()\n (\n max(\n max by (pod, tenant) (\n loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds{\n app=\"loki\",\n container=\"ruler\"\n }\n )\n - on (pod, tenant)\n max by (pod, tenant) (\n loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=\"loki\",\n container=\"ruler\"\n }\n )\n )\n >= 300\n )\n )\n)\n", "instant": true, "legendFormat": "alerts", "qryType": 1, "query": "count(\n max without (ruler_replica) (\n ALERTS{component=~\"alloy|monitoring\", alertstate=\"firing\"}\n )\n)\nor\n0 * (\n (\n (\n time()\n - max(\n prometheus_rule_group_last_evaluation_timestamp_seconds{\n app=\"thanos\",\n container=\"ruler\"\n }\n )\n < 300\n )\n and on ()\n (\n min(\n max by (pod) (\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=\"thanos\",\n container=\"ruler\"\n }\n )\n - on (pod)\n max by (pod) (\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=\"thanos\",\n container=\"ruler\"\n }\n )\n )\n < 300\n )\n )\n unless on ()\n (\n (\n max(\n (\n time()\n - max by (rule_group, user) (\n loki_prometheus_rule_group_last_evaluation_timestamp_seconds{\n app=\"loki\",\n container=\"ruler\"\n }\n > 0\n )\n )\n - on (rule_group, user)\n max by (rule_group, user) (\n loki_prometheus_rule_group_interval_seconds{\n app=\"loki\",\n container=\"ruler\"\n }\n )\n )\n >= 300\n )\n or on ()\n (\n max(\n max by (pod, tenant) (\n loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds{\n app=\"loki\",\n container=\"ruler\"\n }\n )\n - on (pod, tenant)\n max by (pod, tenant) (\n loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=\"loki\",\n container=\"ruler\"\n }\n )\n )\n >= 300\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Alerts about collection that are firing now.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. _Alerts_ below names each one, and links to what it means.\n\nReads zero only while the Thanos ruler has evaluated a rule and written its results back within five minutes. Where a Loki ruler runs, every Loki rule group also has to be no more than five minutes overdue, and written back: the Loki rulers divide the groups between them, so one stalled ruler leaves its groups unevaluated while the others carry on. A ruler that has stopped evaluating fires nothing, and one whose write-back has stalled fires alerts that never reach Thanos, so either would read as no alerts. An instant query, because over a range the last zero from before a ruler went silent would still be shown. The Loki ruler matters here more than anywhere: `alloy-gateway-refusing-scrapes` is matched in the gateway's log, because a gateway in that state discards its own metrics too.", "id": 1000, "links": [], "title": "Collection Alerts Firing", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "Alerts are not reaching Thanos, so none can show. The Thanos dashboard's Ruler tab covers the Thanos ruler, and the Loki dashboard's Logs tab the Loki ruler.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 1.5 }, { "color": "#F67E4B", "value": 2 }, { "color": "#DD3D2D", "value": 2.5 }, { "color": "#A50026", "value": 3 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-config-failed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count(\n alloy_config_last_load_successful{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n } == 0\n)\nor\n0 * count(\n alloy_config_last_load_successful{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "collectors", "qryType": 1, "query": "count(\n alloy_config_last_load_successful{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n } == 0\n)\nor\n0 * count(\n alloy_config_last_load_successful{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Collectors whose most recent attempt to load their configuration failed.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. A collector that fails to load a new configuration keeps running the last one that loaded, so nothing stops and nothing looks wrong. A change that was rolled out is simply not in effect on that collector.\n\nThe collector's log carries the parse or evaluation error. _Components -> Configuration_ shows which configuration each collector is running.", "id": 1003, "links": [], "title": "Failed Config Loads", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 1.5 }, { "color": "#F67E4B", "value": 2 }, { "color": "#DD3D2D", "value": 2.5 }, { "color": "#A50026", "value": 3 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-lines-delivered": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (app, host) (\n rate(\n loki_write_sent_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{app}} to {{host}}", "qryType": 1, "query": "sum by (app, host) (\n rate(\n loki_write_sent_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second each collector delivered, split by where it sent them.**\n\n**Nominal:** The agents' line into the gateway close to the gateway's line into Loki. The gateway also forwards Kubernetes events and the agents' own logs, so its line sits slightly higher.\n\n**Unhealthy:** A line falling to zero while collectors are running. Whatever that hop has buffered is dropped once the buffer fills.", "id": 1012, "links": [], "title": "Log Lines Delivered", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-log-lines-lost": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n loki_write_dropped_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "lost/s", "qryType": 1, "query": "sum(\n rate(\n loki_write_dropped_entries_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second the collectors gave up delivering, across every hop and every reason.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Anything sustained. These lines were collected and will never be stored. _Log Pipeline -> Delivery_ splits them by reason, which says whether the receiver refused them or the collector ran out of room.\n\nCounts the agent-to-gateway hop and the gateway-to-Loki hop alike. A rejection by Loki also appears on the Loki dashboard's discard panels, seen from the receiving end.", "id": 1004, "links": [], "title": "Log Lines Lost", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.01 }, { "color": "#FDB366", "value": 2.5075 }, { "color": "#F67E4B", "value": 5.005 }, { "color": "#DD3D2D", "value": 7.5024999999999995 }, { "color": "#A50026", "value": 10 } ] }, "unit": "suffix:lines/s" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-pushes-refused": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n (\n sum(\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n route!=\"other\",\n status_code!~\"2..\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n + (\n sum(\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n route!=\"other\",\n status_code!~\"2..\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n)\nand on ()\n(\n sum(\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or\n sum(\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "refused/s", "qryType": 1, "query": "(\n (\n sum(\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n route!=\"other\",\n status_code!~\"2..\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n + (\n sum(\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n route!=\"other\",\n status_code!~\"2..\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n)\nand on ()\n(\n sum(\n rate(\n loki_source_api_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or\n sum(\n rate(\n prometheus_receive_http_request_duration_seconds_count{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Push requests per second the gateway answered with an error, across its log-push and remote-write listeners.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** A sustained rate. Something is sending data the gateway will not take, and that data is lost unless the sender retries successfully. _Ingest_ says which listener, which status, and which sender.\n\nCounts requests to the push routes that reached the gateway's HTTP layer. Requests to any other path, such as a probe sending `GET` to the push port and receiving `405`, are not refused pushes and are left out; the per-listener panels on _Ingest_ still show them. A sender whose TLS handshake fails never gets that far, and _Ingest -> Connections_ carries that case. OTLP refusals are counted in points rather than requests, and appear under _Ingest -> OTLP_.", "id": 1006, "links": [], "title": "Push Requests Refused", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No push requests reached the gateway", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.01 }, { "color": "#FDB366", "value": 2.5075 }, { "color": "#F67E4B", "value": 5.005 }, { "color": "#DD3D2D", "value": 7.5024999999999995 }, { "color": "#A50026", "value": 10 } ] }, "unit": "reqps" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-restarts": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (namespace, pod) (\n increase(\n kube_pod_container_status_restarts_total{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n [$__range]\n )\n) > 0\nand on (namespace, pod)\nmax_over_time(\n up{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__range]\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "sum by (namespace, pod) (\n increase(\n kube_pod_container_status_restarts_total{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n [$__range]\n )\n) > 0\nand on (namespace, pod)\nmax_over_time(\n up{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__range]\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true, "namespace": true }, "indexByName": { "Value": 1, "pod": 0 } } } } ] } }, "description": "**Container restarts per collector over the selected time range.**\n\n**Nominal:** None.\n\n**Unhealthy:** Any restart on the gateway. Each one interrupts every scrape and every delivery in flight, and replays the write-ahead log on the way back up. _Resources -> Memory_ says whether memory was the cause.\n\nOnly pods that were running at some point in the range appear.", "id": 1010, "links": [], "title": "Container Restarts", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No restarts in this range. If Collector Scrape Health is empty too, nothing is being collected.", "unit": "short" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "overview-samples-delivered": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}}", "qryType": 1, "query": "sum by (component_id) (\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (component_id) (\n rate(\n otelcol_exporter_sent_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{component_id}}", "qryType": 1, "query": "sum by (component_id) (\n rate(\n otelcol_exporter_sent_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metric samples per second the gateway delivered, split by destination.**\n\n**Nominal:** A steady line per destination, tracking the number of targets scraped. An extra destination that filters what it sends carries less than the bundled store.\n\n**Unhealthy:** A destination falling to zero. _Metric Pipeline -> Remote Write_ and _Metric Pipeline -> OpenTelemetry Exporters_ say whether it is refusing or unreachable.", "id": 1013, "links": [], "title": "Samples Delivered", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-samples-lost": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n (\n sum(\n rate(\n prometheus_remote_storage_samples_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n + (\n sum(\n rate(\n otelcol_exporter_send_failed_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n)\nand on ()\n(\n sum(\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or\n sum(\n rate(\n otelcol_exporter_sent_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "lost/s", "qryType": 1, "query": "(\n (\n sum(\n rate(\n prometheus_remote_storage_samples_failed_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n + (\n sum(\n rate(\n otelcol_exporter_send_failed_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or vector(0)\n )\n)\nand on ()\n(\n sum(\n rate(\n prometheus_remote_storage_samples_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n or\n sum(\n rate(\n otelcol_exporter_sent_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metric samples per second the gateway failed to deliver and will not retry, across every metrics destination.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** A sustained rate. A burst right after a gateway restart is usually the write-ahead log replaying samples the store already has, which it rejects as duplicates with `409 Conflict`. That burst is not a finding.\n\nRemote-write failures are samples the destination rejected outright, which are not retried. Recoverable errors are retried instead, and appear under _Metric Pipeline -> Remote Write_ as retries.\n\nEither kind of destination may be absent, since a deployment can run without the bundled store or without any OpenTelemetry exporter. Each contributes zero when absent, and the panel reads no data only when neither is reporting.", "id": 1005, "links": [], "title": "Samples Lost", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.01 }, { "color": "#FDB366", "value": 25.0075 }, { "color": "#F67E4B", "value": 50.004999999999995 }, { "color": "#DD3D2D", "value": 75.0025 }, { "color": "#A50026", "value": 100 } ] }, "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-scrape": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (app, pod) (\n up{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "max by (app, pod) (\n up{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true }, "indexByName": { "Value": 2, "app": 0, "pod": 1 } } } } ] } }, "description": "**Whether the gateway can scrape each collector's own metrics endpoint.**\n\n**Nominal:** 1 for every collector.\n\n**Unhealthy:** 0 for any of them. Every other panel about that collector is then empty, and an empty panel is indistinguishable from a quiet one.\n\nA collector missing from the table entirely is not being discovered. If every row is missing, the gateway is not writing metrics at all, and the check is `kubectl get pods` for the gateway.", "id": 1009, "links": [], "title": "Collector Scrape Health", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "overview-unhealthy-components": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n health_type!=\"healthy\"\n }\n)\nor\n0 * sum(\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "components", "qryType": 1, "query": "sum(\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\",\n health_type!=\"healthy\"\n }\n)\nor\n0 * sum(\n alloy_component_controller_running_components{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Pipeline components that Alloy reports as anything other than healthy, across the selected collectors.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Anything sustained. An unhealthy component is still part of the running pipeline, and data routed through it is not flowing. _Components -> Health_ says which collector, and the collector's own log says which component and why.\n\nA few seconds of `unknown` while a collector starts is normal. Alloy publishes the count per health state, not per component, so naming the component takes the log.", "id": 1002, "links": [], "title": "Unhealthy Components", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 2 }, { "color": "#F67E4B", "value": 3 }, { "color": "#DD3D2D", "value": 4 }, { "color": "#A50026", "value": 5 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-versions": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count by (app, version) (\n alloy_build_info{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "count by (app, version) (\n alloy_build_info{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true }, "indexByName": { "Value": 2, "app": 0, "version": 1 }, "renameByName": { "Value": "collectors" } } } } ] } }, "description": "**The Alloy version each role is running, and how many collectors run it.**\n\n**Nominal:** One version per role.\n\n**Degraded:** Two versions for one role outside a rollout. Part of the fleet did not restart onto the new image.", "id": 1011, "links": [], "title": "Running Versions", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "resources-cpu": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (namespace, pod) (\n rate(\n container_cpu_usage_seconds_total{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n [$__rate_interval]\n )\n)\n/ max by (namespace, pod) (\n kube_pod_container_resource_limits{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\",\n resource=\"cpu\"\n }\n)\nand on (namespace, pod)\nup{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n}\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (namespace, pod) (\n rate(\n container_cpu_usage_seconds_total{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n [$__rate_interval]\n )\n)\n/ max by (namespace, pod) (\n kube_pod_container_resource_limits{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\",\n resource=\"cpu\"\n }\n)\nand on (namespace, pod)\nup{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n}\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**CPU each collector is using, as a fraction of its CPU limit.**\n\n**Nominal:** Well below 1.\n\n**Degraded:** Sustained near 1. The collector is throttled, and scrapes and deliveries take longer. _CPU Throttling_ beside this shows how often.\n\nEmpty for a collector with no CPU limit set.", "id": 1078, "links": [], "title": "CPU Used Against Limit", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "No metrics: cadvisor and kube-state-metrics are required", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-fds": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (pod) (\n process_open_fds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n / process_max_fds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (pod) (\n process_open_fds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n / process_max_fds{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Open file descriptors in each collector, as a fraction of its limit.**\n\n**Nominal:** Well below 1.\n\n**Unhealthy:** Approaching 1. The agent holds a descriptor for every log file it tails, and once none are left it stops opening new ones, so new containers on that node go uncollected.", "id": 1081, "links": [], "title": "File Descriptors Against Limit", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-gomemlimit": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (pod) (\n (\n go_memstats_sys_bytes{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n - go_memstats_heap_released_bytes{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n )\n / go_gc_gomemlimit_bytes{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (pod) (\n (\n go_memstats_sys_bytes{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n - go_memstats_heap_released_bytes{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n )\n / go_gc_gomemlimit_bytes{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Memory the Go runtime holds in each collector, as a fraction of the soft limit `GOMEMLIMIT` sets for it.**\n\n**Nominal:** Below about 0.8.\n\n**Degraded:** Sustained near 1. The garbage collector is running almost continuously to stay under the limit, which spends CPU the pipeline needed and slows everything down. The remedy is more memory for the collector, with `GOMEMLIMIT` raised in step, or less for it to hold.\n\nThe chart sets `GOMEMLIMIT` to about 80% of each collector's memory limit, so the runtime slows down before the kernel kills it. Heap in use alone understates this figure, because the limit also covers the runtime's stacks and metadata. A collector with no `GOMEMLIMIT` reads close to zero.", "id": 1076, "links": [], "title": "Go Memory Against GOMEMLIMIT", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-goroutines": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (pod) (\n go_goroutines{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (pod) (\n go_goroutines{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Concurrent goroutines in each collector.**\n\n**Nominal:** Flat, at a level set by the collector's workload. The gateway runs several times the agent's count.\n\n**Degraded:** Steady growth over hours. Something inside the collector is starting work it never finishes, and memory grows with it.", "id": 1080, "links": [], "title": "Goroutines", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-memory": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (namespace, pod) (\n container_memory_working_set_bytes{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n)\n/ max by (namespace, pod) (\n kube_pod_container_resource_limits{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\",\n resource=\"memory\"\n }\n)\nand on (namespace, pod)\nup{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n}\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (namespace, pod) (\n container_memory_working_set_bytes{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n)\n/ max by (namespace, pod) (\n kube_pod_container_resource_limits{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\",\n resource=\"memory\"\n }\n)\nand on (namespace, pod)\nup{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n}\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Memory each collector is using, as a fraction of its memory limit.**\n\n**Nominal:** Well below 1.\n\n**Unhealthy:** Approaching 1. The kernel kills the container at the limit, and every scrape and delivery in flight is lost with it.\n\nReads the container's working set, which is what the limit is enforced against. The process's resident set is larger, because it counts file pages the kernel can reclaim, and reads as closer to the limit than the collector really is.", "id": 1075, "links": [], "title": "Memory Used Against Limit", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "No metrics: cadvisor and kube-state-metrics are required", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-memory-limiter": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n rate(\n otelcol_processor_memory_limiter_refused_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\nor\n0 * sum by (pod) (\n rate(\n otelcol_processor_memory_limiter_accepted_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n rate(\n otelcol_processor_memory_limiter_refused_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\nor\n0 * sum by (pod) (\n rate(\n otelcol_processor_memory_limiter_accepted_metric_points_total{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Metric points per second the gateway's memory limiter refused.**\n\n**Nominal:** A flat zero per gateway.\n\n**Unhealthy:** Any. The gateway is over the memory threshold its limiter guards, and it is refusing data rather than risk being killed. The refused points are lost unless their sender retries, and scraped points have no sender to retry them.\n\nThe limiter publishes a refusal series only once it has refused something. The zero line comes from the points it accepted instead.\n\nA flat zero means the collectors reported and there was nothing to show. No data means nothing was collected at all.\n\n**A gateway that is refusing usually disappears from this panel rather than rising on it.** Its own `/metrics` is one of the targets the gateways share out, and when the refusing pod holds that target it refuses its own counters with everything else. A gateway pod that is running but missing here is the case to suspect; the `alloy-gateway-refusing-scrapes` alert reads the pod's logs instead, which do not pass through the limiter.", "id": 1077, "links": [], "title": "Memory Limiter Refusals", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from the collectors. Check that the gateway is running.", "unit": "suffix:points/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-terminations": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (namespace, pod, reason) (\n kube_pod_container_status_last_terminated_reason{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n) == 1\nand on (namespace, pod)\nup{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n}\n", "format": "table", "instant": true, "qryType": 2, "query": "max by (namespace, pod, reason) (\n kube_pod_container_status_last_terminated_reason{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n) == 1\nand on (namespace, pod)\nup{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n}\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true, "Value": true, "namespace": true }, "indexByName": { "pod": 0, "reason": 1 } } } } ] } }, "description": "**Why each collector's container last stopped, for those that have.**\n\n**Nominal:** Empty.\n\n**Unhealthy:** `OOMKilled`. The collector reached its memory limit. _Go Memory Against GOMEMLIMIT_ says whether the runtime saw it coming.", "id": 1082, "links": [], "title": "Last Termination Reason", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No recorded termination. If Memory Used Against Limit is empty too, nothing is being collected." }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "resources-throttling": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (namespace, pod) (\n rate(\n container_cpu_cfs_throttled_periods_total{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n [$__rate_interval]\n )\n)\n/ sum by (namespace, pod) (\n rate(\n container_cpu_cfs_periods_total{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n [$__rate_interval]\n )\n)\nand on (namespace, pod)\nup{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n}\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (namespace, pod) (\n rate(\n container_cpu_cfs_throttled_periods_total{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n [$__rate_interval]\n )\n)\n/ sum by (namespace, pod) (\n rate(\n container_cpu_cfs_periods_total{\n namespace=~\"$alloyNamespace\",\n container=\"alloy\"\n }\n [$__rate_interval]\n )\n)\nand on (namespace, pod)\nup{\n app=~\"$alloyRole\",\n namespace=~\"$alloyNamespace\",\n pod=~\"$alloyPod\"\n}\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The fraction of scheduling periods in which each collector was held back by its CPU limit.**\n\n**Nominal:** Close to zero.\n\n**Degraded:** A sustained few percent or more. Collection is bursty, since every scrape interval starts a wave of work, so a collector can be throttled while its average use looks low. The bursts are what the limit clips.", "id": 1079, "links": [], "title": "CPU Throttling", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "No metrics: cadvisor metrics are required (via kubelet)", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } } }, "layout": { "kind": "TabsLayout", "spec": { "tabs": [ { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "narrow", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-alerts-firing" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-agents-missing" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-unhealthy-components" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-config-failed" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-log-lines-lost" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-samples-lost" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-pushes-refused" } } } ], "maxColumnCount": 7, "rowHeightMode": "short" } }, "title": "Verdict" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-alerts" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-alert-history" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Alerts" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-scrape" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-restarts" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-versions" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Collection Health" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-lines-delivered" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-samples-delivered" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Throughput" } } ] } }, "title": "Overview" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "narrow", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-flow-read" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-flow-stored" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-flow-lost" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-flow-propagation" } } } ], "maxColumnCount": 4, "rowHeightMode": "short" } }, "title": "Flow" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-agents-files" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-agents-file-lines" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-agents-journal-lines" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Agents" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-delivery-requests" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-delivery-dropped" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-delivery-retries" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-delivery-latency" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Delivery" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-processing-guards" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-processing-truncated" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Processing" } } ] } }, "title": "Log Pipeline" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "narrow", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-flow-targets" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-flow-targets-down" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-flow-samples" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-flow-lost" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-flow-delay" } } } ], "maxColumnCount": 5, "rowHeightMode": "short" } }, "title": "Flow" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-scrape-down" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-scrape-by-monitor" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-scrape-rejected" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-scrape-rejected-samples" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-scrape-samples" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Scraping" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-heavy-slowest" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-heavy-largest" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Heaviest Targets" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-rw-samples" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-rw-delay" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-rw-shards" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-rw-latency" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Remote Write" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-wal-series" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-wal-size" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-wal-errors" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Write-Ahead Log" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": true, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-otel-sent" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-otel-failed" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-otel-queue" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "metrics-otel-filtered" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "OpenTelemetry Exporters" } } ] } }, "title": "Metric Pipeline" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "narrow", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-flow-lines" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-flow-samples" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-flow-refused" } } } ], "maxColumnCount": 3, "rowHeightMode": "short" } }, "title": "Flow" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-log-requests" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-log-latency" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Log Push" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-rw-requests" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-rw-samples" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-rw-latency" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-rw-senders" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Remote Write" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-otlp-metrics" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-otlp-logs" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "OTLP" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ingest-tls-rejections" } } } ], "maxColumnCount": 1, "rowHeightMode": "standard" } }, "title": "Connections" } } ] } }, "title": "Ingest" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "components-not-healthy" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "components-running" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Health" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "components-config-by-pod" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "components-distinct-configs" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Configuration" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "components-eval-rate" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "components-eval-latency" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "components-eval-slow" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "components-eval-queue" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Evaluation" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "cluster-peers" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "cluster-health-score" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "cluster-targets" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "cluster-transport-failures" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "cluster-members" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Clustering" } } ] } }, "title": "Components" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-memory" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-gomemlimit" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-memory-limiter" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Memory" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-cpu" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-throttling" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "CPU" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-goroutines" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-fds" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-terminations" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Runtime" } } ] } }, "title": "Resources" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-event-rate-by-reason" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-event-rate-by-kind" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Activity" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-warning-event-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "Warning Events" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-event-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "All Events" } } ] } }, "title": "Events" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingTimeRangeSize", "spec": { "value": "7d" } } ], "visibility": "show" } }, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-logs-rate" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-logs-errors-by-component" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-logs-warning-rate" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Volume" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingTimeRangeSize", "spec": { "value": "7d" } } ], "visibility": "hide" } }, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-volume-hidden-note" } } } ], "maxColumnCount": 1, "rowHeightMode": "short" } }, "title": "Volume" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-logs-warning-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "Warnings" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "alloy-logs-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "All Logs" } } ] } }, "title": "Logs" } } ] } }, "links": [], "liveNow": false, "preload": false, "tags": [ "infrastructure", "mzmon", "meta-monitoring" ], "timeSettings": { "autoRefresh": "", "autoRefreshIntervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "fiscalYearStartMonth": 0, "from": "now-6h", "hideTimepicker": false, "timezone": "browser", "to": "now" }, "title": "Alloy Meta Monitoring", "variables": [ { "kind": "DatasourceVariable", "spec": { "allowCustomValue": false, "current": { "text": "", "value": "" }, "description": "Datasource for metrics queries", "hide": "dontHide", "includeAll": false, "label": "Metrics Datasource", "multi": false, "name": "metricsDatasource", "options": [], "pluginId": "prometheus", "refresh": "never", "regex": "", "skipUrlSync": false } }, { "kind": "DatasourceVariable", "spec": { "allowCustomValue": false, "current": { "text": "", "value": "" }, "description": "Datasource for log and event queries", "hide": "dontHide", "includeAll": false, "label": "Logs Datasource", "multi": false, "name": "logsDatasource", "options": [], "pluginId": "loki", "refresh": "never", "regex": "", "skipUrlSync": false } }, { "kind": "QueryVariable", "spec": { "allValue": ".+", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(up{app=~\"alloy-agent|alloy-gateway\"}, namespace)", "description": "Namespace(s) the collectors run in", "hide": "dontHide", "includeAll": true, "label": "Alloy Namespace", "multi": true, "name": "alloyNamespace", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(up{app=~\"alloy-agent|alloy-gateway\"}, namespace)", "qryType": 1, "query": "label_values(up{app=~\"alloy-agent|alloy-gateway\"}, namespace)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": "alloy-agent|alloy-gateway", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(up{app=~\"alloy-agent|alloy-gateway\", namespace=~\"$alloyNamespace\"}, app)", "description": "Which collectors to read: the per-node agents, the gateway, or both", "hide": "dontHide", "includeAll": true, "label": "Role", "multi": true, "name": "alloyRole", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(up{app=~\"alloy-agent|alloy-gateway\", namespace=~\"$alloyNamespace\"}, app)", "qryType": 1, "query": "label_values(up{app=~\"alloy-agent|alloy-gateway\", namespace=~\"$alloyNamespace\"}, app)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".+", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(up{app=~\"$alloyRole\", namespace=~\"$alloyNamespace\"}, pod)", "description": "Which collector pods to read", "hide": "dontHide", "includeAll": true, "label": "Collector", "multi": true, "name": "alloyPod", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(up{app=~\"$alloyRole\", namespace=~\"$alloyNamespace\"}, pod)", "qryType": 1, "query": "label_values(up{app=~\"$alloyRole\", namespace=~\"$alloyNamespace\"}, pod)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".*", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(level)", "description": "The severity level(s) to include", "hide": "dontHide", "includeAll": true, "label": "Level", "multi": true, "name": "logLevelList", "options": [], "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "label": "level", "refId": "LokiVariableQueryEditor-VariableQuery", "stream": "{app=~\"$alloyRole\", namespace=~\"$alloyNamespace\"}", "type": 1 }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "TextVariable", "spec": { "current": { "text": "", "value": "" }, "description": "Case-insensitive text to match anywhere in the log line", "hide": "dontHide", "label": "Search", "name": "logSearch", "query": "", "skipUrlSync": false } }, { "datasource": { "name": "$metricsDatasource" }, "group": "", "kind": "AdhocVariable", "spec": { "allowCustomValue": true, "baseFilters": [ { "key": "namespace", "operator": "=~", "origin": "dashboard", "value": "$alloyNamespace" } ], "defaultKeys": [], "description": "Adhoc filters to apply to all metrics queries", "enableGroupBy": false, "filters": [], "hide": "inControlsMenu", "label": "Advanced Metric Filter", "name": "metricAdhoc", "skipUrlSync": false } }, { "datasource": { "name": "$logsDatasource" }, "group": "", "kind": "AdhocVariable", "spec": { "allowCustomValue": true, "baseFilters": [], "defaultKeys": [], "description": "Adhoc filters to apply to all logs queries", "enableGroupBy": false, "filters": [], "hide": "inControlsMenu", "label": "Advanced Log Filter", "name": "logsAdhoc", "skipUrlSync": false } } ] } }