{ "apiVersion": "dashboard.grafana.app/v2", "kind": "Dashboard", "metadata": { "annotations": { "grafana.app/folder": "meta-o11y", "monitoring.materialize.cloud/min-mz-version": "v26.24.0", "monitoring.materialize.cloud/rec-mz-version": "v26.24.0", "monitoring.materialize.cloud/sql-metric-prefix": "mz_", "monitoring.materialize.cloud/target-export": "generic" }, "name": "mz-mon-infra-thanos" }, "spec": { "annotations": [ { "kind": "AnnotationQuery", "spec": { "builtIn": true, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "query": { "datasource": { "name": "-- Grafana --" }, "group": "grafana", "kind": "DataQuery", "spec": {}, "version": "v0" } } } ], "cursorSync": "Crosshair", "description": "Is the metrics store healthy, and if not, which part of it broke.\n\nThe monitoring stack watching itself: remote writes into the receivers, queries through Thanos Query, the ruler, the compactor, and object storage, and any Thanos alert firing now. Thanos answers every panel here, so a dashboard that is empty throughout most likely means Thanos Query is down; the Logs tab reads Loki and still works.", "editable": true, "elements": { "logs-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n component=~\"$thanosComponent\",\n level=~\"$logLevelList\"\n}\n |~ \"(?i)$logSearch\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Thanos's own log feed, newest first, for the selected components and levels.**\n\nThese logs are stored in Loki, not Thanos, so this feed keeps working when Thanos does not. That makes it the place to start when every other panel is empty.", "id": 1063, "links": [], "title": "All Logs", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No lines match the current filters" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "logs-rate-by-component": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (component) (\n rate(\n {\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n component=~\"$thanosComponent\",\n level=~\"$logLevelList\"\n }\n |~ \"(?i)$logSearch\"\n [$__auto]\n )\n)\n", "legendFormat": "{{component}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Thanos's own log lines per second, split by component.**\n\n**Nominal:** A steady, low baseline.\n\n**Unhealthy:** A sharp step up confined to one component. That is a single process in a retry loop.", "id": 1059, "links": [], "title": "Log Rate by Component", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No lines match the current filters", "unit": "suffix:logs/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "logs-warning-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n component=~\"$thanosComponent\",\n level=~\"WARN|WARNING|ERROR|CRITICAL|FATAL\"\n}\n != \"PromQL info:\"\n |~ \"(?i)$logSearch\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Warning-and-worse lines from Thanos, newest first.**\n\n**Nominal:** Quiet, apart from occasional retries.\n\n**Unhealthy:** A steady stream from one component. A compactor that halted logs the reason here once and then goes quiet, so look back to when it stopped.\n\nIndependent of the level picker, like the other warning feeds in this repository.\n\nLeaves out the ruler's `PromQL info:` lines. The ruler logs PromQL's informational annotations at WARN, such as `rate()` over a counter whose name lacks `_total`, once per evaluation. On a reference install that is four lines a minute about rules that are working.", "id": 1062, "links": [], "title": "Warnings and Errors", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No warnings in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "logs-warning-rate": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n {\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n component=~\"$thanosComponent\",\n level=~\"WARN|WARNING|ERROR|CRITICAL|FATAL\"\n }\n != \"PromQL info:\"\n |~ \"(?i)$logSearch\"\n [1m]\n )\n) * 60\n", "legendFormat": "warnings/min" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Warning-and-worse lines per minute from Thanos, across the selected components.**\n\n**Nominal:** Zero, or close to it.\n\n**Unhealthy:** A sustained non-zero rate. The feed below says which component and why.\n\nLeaves out the ruler's `PromQL info:` lines, for the reason the warning feed gives.", "id": 1060, "links": [], "title": "Warning Rate", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#33BBEE", "mode": "shades" }, "min": 0, "noValue": "No warnings in this time range", "unit": "suffix:logs/min" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-alert-history": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count by (alertname) (\n max by (alertname, job, pod) (\n ALERTS{alertname=~\"Thanos.*\", alertstate=\"firing\"}\n )\n)\n", "legendFormat": "{{alertname}}", "qryType": 1, "query": "count by (alertname) (\n max by (alertname, job, pod) (\n ALERTS{alertname=~\"Thanos.*\", alertstate=\"firing\"}\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Which Thanos alerts were firing over the selected time range.**\n\n**Nominal:** Empty.\n\nA line is the number of firing instances of that alert. Read it beside the tab that alert is about to see what changed when it started.", "id": 1007, "links": [], "title": "Alerts Firing Over Time", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No alert about Thanos fired in this range", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-alerts": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (alertname, alertstate, severity, job, pod) (\n ALERTS{alertname=~\"Thanos.*\"}\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "max by (alertname, alertstate, severity, job, pod) (\n ALERTS{alertname=~\"Thanos.*\"}\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true, "Value": true }, "indexByName": { "alertname": 0, "alertstate": 1, "job": 3, "pod": 4, "severity": 2 }, "renameByName": { "alertname": "alert", "alertstate": "state" } } } } ] } }, "description": "**Every Thanos alert that is firing or pending now, with the process it is about.**\n\n**Nominal:** Empty.\n\n**Unhealthy:** Any firing row. A pending row has met its condition and is waiting out its `for` duration; it fires if the condition holds.\n\nThe rules are the upstream Thanos mixin's, installed by the Thanos subchart. `ALERTS` carries a rule's labels and not its annotations, so the table has no runbook link. The mixin's runbook, , has a section for each alert name. Rows without a pod are about a whole component, such as a compactor that has not run.", "id": 1006, "links": [], "title": "Alerts", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No alert about Thanos is firing or pending, or none reached Thanos. Thanos Alerts Firing says which." }, "overrides": [ { "matcher": { "id": "byType", "options": "string" }, "properties": [ { "id": "noValue", "value": "—" } ] } ] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "overview-alerts-firing": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count(\n max by (alertname, job, pod) (\n ALERTS{alertname=~\"Thanos.*\", alertstate=\"firing\"}\n )\n)\nor\n0 * (\n (\n time()\n - max(\n prometheus_rule_group_last_evaluation_timestamp_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n )\n < 300\n )\n and on ()\n (\n min(\n max by (pod) (\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n )\n - on (pod)\n max by (pod) (\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n )\n )\n < 300\n )\n)\n", "instant": true, "legendFormat": "alerts", "qryType": 1, "query": "count(\n max by (alertname, job, pod) (\n ALERTS{alertname=~\"Thanos.*\", alertstate=\"firing\"}\n )\n)\nor\n0 * (\n (\n time()\n - max(\n prometheus_rule_group_last_evaluation_timestamp_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n )\n < 300\n )\n and on ()\n (\n min(\n max by (pod) (\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n )\n - on (pod)\n max by (pod) (\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n )\n )\n < 300\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Thanos alerts firing now.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. _Alerts_ below names each one. The upstream Thanos mixin's runbook, , has a section for each alert name.\n\nReads zero only while a ruler has evaluated a rule in the last five minutes and written its results back within five minutes. An instant query, because over a range the last zero from before a ruler went silent would still be shown. A ruler that has stopped evaluating fires nothing, and one whose write-back has stalled fires alerts that never reach Thanos, so either reads as no alerts. An empty reading means the alert path itself needs checking: _Ruler -> Evaluation_ and _Ruler -> Write-back_ say which part.", "id": 1000, "links": [], "title": "Thanos Alerts Firing", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "Alerts are not reaching Thanos, so none can show. The Ruler tab's Evaluation and Write-back rows say why.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 1.5 }, { "color": "#F67E4B", "value": 2 }, { "color": "#DD3D2D", "value": 2.5 }, { "color": "#A50026", "value": 3 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-compactor": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max(\n thanos_compact_halted{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n", "legendFormat": "compactor", "qryType": 1, "query": "max(\n thanos_compact_halted{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Whether the compactor has halted.**\n\n**Nominal:** Running.\n\n**Unhealthy:** Halted. The compactor stops for good on an error it does not risk repeating, most often overlapping blocks in the bucket. While it is halted, retention is not enforced and the bucket grows, and queries over long ranges read ever more small blocks. Its log names the cause, and a restart is needed after the cause is fixed.", "id": 1005, "links": [], "title": "Compactor", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "mappings": [ { "options": { "0": { "color": "#A2F49B", "index": 1, "text": "Running" } }, "type": "value" }, { "options": { "1": { "color": "#AB0003", "index": 2, "text": "Halted" } }, "type": "value" } ], "noValue": "Compactor not collected", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#AB0003", "value": 1 } ] } }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-components-down": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count(\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n } == 0\n)\nor\n0 * count(\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\n", "legendFormat": "processes", "qryType": 1, "query": "count(\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n } == 0\n)\nor\n0 * count(\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Thanos processes whose metrics endpoint the gateway cannot scrape.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. Every panel about that process is then empty, and an empty panel reads the same as a quiet one. _Collection Health_ below names it.", "id": 1001, "links": [], "title": "Processes Down", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No Thanos process found. Check the namespace picker.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 1.25 }, { "color": "#F67E4B", "value": 1.5 }, { "color": "#DD3D2D", "value": 1.75 }, { "color": "#A50026", "value": 2 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-query-errors": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\",\n code=~\"5..\"\n }\n [$__rate_interval]\n )\n )\n or\n 0 * sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\"\n }\n [$__rate_interval]\n )\n )\n)\n/ sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "5xx share", "qryType": 1, "query": "(\n sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\",\n code=~\"5..\"\n }\n [$__rate_interval]\n )\n )\n or\n 0 * sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\"\n }\n [$__rate_interval]\n )\n )\n)\n/ sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The share of PromQL queries Thanos Query answered with a server error.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Anything sustained. Every dashboard and every rule evaluation goes through Thanos Query, so its errors show up as empty panels everywhere and as alerts that do not fire.\n\nA client error, such as a query that does not parse, is a 4xx and is not counted.", "id": 1003, "links": [], "title": "Query Errors (5xx)", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No queries in this range", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.001 }, { "color": "#FDB366", "value": 0.013250000000000001 }, { "color": "#F67E4B", "value": 0.025500000000000002 }, { "color": "#DD3D2D", "value": 0.037750000000000006 }, { "color": "#A50026", "value": 0.05 } ] }, "unit": "percentunit" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-query-rate": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (handler) (\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{handler}}", "qryType": 1, "query": "sum by (handler) (\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**PromQL queries per second Thanos Query answered, split by instant and range queries.**\n\n**Nominal:** A steady floor from the rulers' instant queries, with range queries on top whenever someone has a dashboard open.\n\n**Unhealthy:** Instant queries falling to zero. The rulers evaluate every rule as an instant query, so no instant queries means no rule is being evaluated.", "id": 1012, "links": [], "title": "Queries Answered", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-restarts": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (container, pod) (\n increase(\n kube_pod_container_status_restarts_total{\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__range]\n )\n) > 0\nand on (pod)\nmax by (pod) (\n max_over_time(\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__range]\n )\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "sum by (container, pod) (\n increase(\n kube_pod_container_status_restarts_total{\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__range]\n )\n) > 0\nand on (pod)\nmax by (pod) (\n max_over_time(\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__range]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true }, "indexByName": { "Value": 2, "container": 0, "pod": 1 } } } } ] } }, "description": "**Container restarts per Thanos pod over the selected time range.**\n\n**Nominal:** None.\n\n**Unhealthy:** Any restart of a receiver. A receiver replays its write-ahead log on the way back up, which can take minutes, and its peers carry its share of writes until it rejoins. _Resources_ says whether memory was the cause.\n\nOnly pods that were running at some point in the range appear.", "id": 1009, "links": [], "title": "Container Restarts", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No restarts in this range. If Scrape Health is empty too, nothing is being collected.", "unit": "short" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "overview-rule-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_rule_evaluation_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "failures/s", "qryType": 1, "query": "sum(\n rate(\n prometheus_rule_evaluation_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Alerting and recording rules the rulers failed to evaluate, per second.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Anything sustained. A rule that fails to evaluate cannot fire, so its alert stays silent whatever the condition it watches. _Ruler -> Evaluation_ names the rule group.", "id": 1004, "links": [], "title": "Failed Rule Evaluations", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No ruler is reporting", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.001 }, { "color": "#FDB366", "value": 0.025750000000000002 }, { "color": "#F67E4B", "value": 0.0505 }, { "color": "#DD3D2D", "value": 0.07525000000000001 }, { "color": "#A50026", "value": 0.1 } ] }, "unit": "suffix:/s" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "overview-samples-ingested": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n thanos_receive_write_samples_sum{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n code=~\"2..\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "accepted", "qryType": 1, "query": "sum(\n rate(\n thanos_receive_write_samples_sum{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n code=~\"2..\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Samples per second the receivers accepted from remote writes.**\n\n**Nominal:** A steady line, close to what the gateway reports sending. It moves with the number of scrape targets, not with traffic.\n\n**Unhealthy:** A drop while the gateway is running. Either the gateway has stopped sending or the receivers have stopped accepting, and _Writes_ says which.\n\nCounts each sample once, on the receiver that took the request. Each receiver then stores a copy for every replica, which is why the receivers' own databases append more than this.", "id": 1011, "links": [], "title": "Samples Ingested", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#0077BB", "mode": "shades" }, "min": 0, "noValue": "No remote writes reached the receivers", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "overview-scrape": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (container, pod) (\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "max by (container, pod) (\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true }, "indexByName": { "Value": 2, "container": 0, "pod": 1 } } } } ] } }, "description": "**Whether the gateway can scrape each Thanos process's own metrics endpoint.**\n\n**Nominal:** 1 for every process.\n\n**Unhealthy:** 0 for any of them. Every other panel about that process is then empty.\n\nA process missing from the table entirely is not being discovered. If every row is missing, either Thanos is not running in the selected namespace or Thanos Query cannot answer, and the check is `kubectl get pods` in that namespace.", "id": 1008, "links": [], "title": "Scrape Health", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No Thanos targets found at all. Check the namespace picker, or the Logs tab.", "unit": "short" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "overview-versions": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count by (container, version) (\n thanos_build_info{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "count by (container, version) (\n thanos_build_info{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true }, "indexByName": { "Value": 2, "container": 0, "version": 1 }, "renameByName": { "Value": "pods" } } } } ] } }, "description": "**The Thanos version each process is running, and how many pods run it.**\n\n**Nominal:** One version throughout.\n\n**Degraded:** Two versions outside a rollout. Part of the deployment did not restart onto the new image.", "id": 1010, "links": [], "title": "Running Versions", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "short" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "overview-write-errors": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\",\n code=~\"5..\"\n }\n [$__rate_interval]\n )\n )\n or\n 0 * sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n )\n)\n/ sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "5xx share", "qryType": 1, "query": "(\n sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\",\n code=~\"5..\"\n }\n [$__rate_interval]\n )\n )\n or\n 0 * sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n )\n)\n/ sum(\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The share of remote writes the receivers answered with a server error.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Anything sustained. The gateway retries a server error, so a short burst costs nothing, but a sustained one fills the gateway's queue and it starts dropping samples. _Writes_ says whether replication or the local database is the cause.", "id": 1002, "links": [], "title": "Write Errors (5xx)", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No remote writes reached the receivers", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 0.001 }, { "color": "#FDB366", "value": 0.013250000000000001 }, { "color": "#F67E4B", "value": 0.025500000000000002 }, { "color": "#DD3D2D", "value": 0.037750000000000006 }, { "color": "#A50026", "value": 0.05 } ] }, "unit": "percentunit" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "reads-block-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n rate(\n thanos_bucket_store_block_load_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n + ignoring (operation)\n rate(\n thanos_bucket_store_block_drop_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n rate(\n thanos_bucket_store_block_load_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n + ignoring (operation)\n rate(\n thanos_bucket_store_block_drop_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Blocks per second each store gateway failed to load or to drop.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. A block that fails to load is not served. Its log names the block, and a block the compactor has since deleted clears on its own.", "id": 1033, "links": [], "title": "Block Load and Drop Failures", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No store gateway is reporting", "unit": "suffix:/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reads-blocks-loaded": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n thanos_bucket_store_blocks_loaded{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n thanos_bucket_store_blocks_loaded{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Blocks each store gateway has loaded from object storage and can serve.**\n\n**Nominal:** The same on every store gateway, rising each time a receiver uploads and falling each time the compactor merges or deletes.\n\n**Unhealthy:** One store gateway well below the others. It cannot see part of the bucket, and queries it answers are missing that data.", "id": 1032, "links": [], "title": "Blocks Loaded", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No store gateway is reporting", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reads-concurrency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n thanos_query_concurrent_gate_queries_in_flight{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\"\n }\n)\n", "legendFormat": "in flight", "qryType": 1, "query": "sum(\n thanos_query_concurrent_gate_queries_in_flight{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n thanos_query_concurrent_gate_queries_max{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\"\n }\n)\n", "legendFormat": "limit", "qryType": 1, "query": "sum(\n thanos_query_concurrent_gate_queries_max{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\"\n }\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Queries Thanos Query is evaluating at once, against the most it allows.**\n\n**Nominal:** Well below the limit.\n\n**Degraded:** At the limit. Further queries wait their turn, which shows up as slow dashboards and late rule evaluations rather than as errors.", "id": 1027, "links": [], "title": "Concurrent Queries", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reads-dropped": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (container) (\n rate(\n thanos_store_selects_dropped_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{container}} selects", "qryType": 1, "query": "sum by (container) (\n rate(\n thanos_store_selects_dropped_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (container) (\n rate(\n thanos_bucket_store_queries_dropped_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{container}} series", "qryType": 1, "query": "sum by (container) (\n rate(\n thanos_bucket_store_queries_dropped_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Store requests per second refused for exceeding a limit, by process.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. The query that sent it gets a partial or failed response. A single query asking for too many series is the usual cause.", "id": 1031, "links": [], "title": "Requests Refused by Limits", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reads-endpoints": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "min by (store_type) (\n sum by (pod, store_type) (\n thanos_store_nodes_grpc_connections{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\"\n }\n )\n)\n", "legendFormat": "{{store_type}}", "qryType": 1, "query": "min by (store_type) (\n sum by (pod, store_type) (\n thanos_store_nodes_grpc_connections{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\"\n }\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Stores each Thanos Query instance is connected to, by kind, taking the worst-connected instance.**\n\n**Nominal:** One per receiver, at least one store gateway, and one per ruler.\n\n**Unhealthy:** Fewer receivers than are running. Thanos Query answers without the missing store and says nothing, so recent data from it is absent from every panel. The receivers replicate, so one missing receiver loses nothing while the others hold its series.\n\nThe store gateway is reached through its Service, so it counts once however many replicas run.", "id": 1028, "links": [], "title": "Stores Connected", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reads-index-cache": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (item_type) (\n rate(\n thanos_store_index_cache_hits_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n)\n/ sum by (item_type) (\n rate(\n thanos_store_index_cache_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{item_type}}", "qryType": 1, "query": "sum by (item_type) (\n rate(\n thanos_store_index_cache_hits_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n)\n/ sum by (item_type) (\n rate(\n thanos_store_index_cache_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"storegateway\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The share of index lookups the store gateways answered from their cache, by kind of index entry.**\n\n**Nominal:** High, after the store gateways have been running a while.\n\n**Degraded:** Low and staying low. The cache is too small for the working set, so each query fetches index data from object storage again.", "id": 1034, "links": [], "title": "Index Cache Hit Ratio", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "No index lookups in this range", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reads-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (handler, le) (\n rate(\n http_request_duration_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "{{handler}}", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (handler, le) (\n rate(\n http_request_duration_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\",\n handler=~\"query|query_range\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long Thanos Query took to answer instant and range queries, at the 99th percentile.**\n\n**Nominal:** Under a second for instant queries. Range queries depend on the range and on how much has to come from object storage.\n\n**Degraded:** Instant queries in seconds. Every rule evaluation is an instant query, so slow instant queries delay every alert.", "id": 1026, "links": [], "title": "Query Latency (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No queries in this range", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reads-requests": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (handler, code) (\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{handler}} {{code}}", "qryType": 1, "query": "sum by (handler, code) (\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"query\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Requests per second Thanos Query answered, by endpoint and status code.**\n\n**Nominal:** Mostly `query` from the rulers, `query_range` from dashboards, and the label endpoints from variable pickers.\n\n**Unhealthy:** 5xx on any endpoint. A 422 is a query Thanos could not evaluate, such as one that matched too much data, and is the caller's to fix.", "id": 1025, "links": [], "title": "Requests by Endpoint and Status", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reads-series-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (container, le) (\n rate(\n grpc_server_handling_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\",\n grpc_service=\"thanos.Store\",\n grpc_method=\"Series\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "{{container}}", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (container, le) (\n rate(\n grpc_server_handling_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\",\n grpc_service=\"thanos.Store\",\n grpc_method=\"Series\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long each kind of store took to answer a request for series, at the 99th percentile.**\n\n**Nominal:** Receivers fastest, because they answer from memory. The store gateway is slower, because it can fetch from object storage.\n\n**Degraded:** The store gateway in seconds. Its index cache is too small, or the bucket is slow; _Store Gateway_ below and _Storage -> Object Storage_ tell those apart.", "id": 1030, "links": [], "title": "Series Request Latency (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No series requests in this range", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "reads-store-errors": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "(\n sum by (container) (\n rate(\n grpc_server_handled_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\",\n grpc_service=\"thanos.Store\",\n grpc_code=~\"Unknown|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\"\n }\n [$__rate_interval]\n )\n )\n or\n 0 * sum by (container) (\n rate(\n grpc_server_handled_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\",\n grpc_service=\"thanos.Store\"\n }\n [$__rate_interval]\n )\n )\n)\n/ sum by (container) (\n rate(\n grpc_server_handled_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\",\n grpc_service=\"thanos.Store\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{container}}", "qryType": 1, "query": "(\n sum by (container) (\n rate(\n grpc_server_handled_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\",\n grpc_service=\"thanos.Store\",\n grpc_code=~\"Unknown|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded\"\n }\n [$__rate_interval]\n )\n )\n or\n 0 * sum by (container) (\n rate(\n grpc_server_handled_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\",\n grpc_service=\"thanos.Store\"\n }\n [$__rate_interval]\n )\n )\n)\n/ sum by (container) (\n rate(\n grpc_server_handled_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\",\n grpc_service=\"thanos.Store\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The share of store requests each process failed with a server error.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Anything sustained. Thanos Query treats a failed store as a partial response, so the query succeeds with that store's data missing.", "id": 1029, "links": [], "title": "Store Request Errors", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-cpu": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (container, pod) (\n rate(\n container_cpu_usage_seconds_total{\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\nand on (pod)\nmax by (pod) (\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (container, pod) (\n rate(\n container_cpu_usage_seconds_total{\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\nand on (pod)\nmax by (pod) (\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**CPU each Thanos pod is using, in cores.**\n\n**Nominal:** Steady for receivers and rulers, bursty for the compactor and the store gateway.\n\n**Degraded:** A receiver pinned high. Writes slow down, and the gateway's queue grows.", "id": 1056, "links": [], "title": "CPU", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: cadvisor metrics are required (via kubelet)", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-goroutines": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (container, pod) (\n go_goroutines{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (container, pod) (\n go_goroutines{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Goroutines in each Thanos process.**\n\n**Nominal:** Flat.\n\n**Degraded:** Climbing without end. Requests are piling up inside the process, most often in Thanos Query or a store waiting on something slow.", "id": 1057, "links": [], "title": "Goroutines", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-memory": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (container, pod) (\n container_memory_working_set_bytes{\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\nand on (pod)\nmax by (pod) (\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (container, pod) (\n container_memory_working_set_bytes{\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\nand on (pod)\nmax by (pod) (\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Memory each Thanos pod is using, by working set.**\n\n**Nominal:** Receivers well above everything else, because they hold the last two hours in memory, and flat between uploads.\n\n**Unhealthy:** A receiver climbing toward its node's capacity. A receiver that runs out of memory is killed and replays its write-ahead log on restart, which takes longer the larger it has grown.\n\nThe working set is what the kernel's memory limit is enforced against. The chart sets no memory limit on Thanos by default, so the ceiling is the node's.", "id": 1055, "links": [], "title": "Memory", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No metrics: cadvisor metrics are required (via kubelet)", "unit": "bytes" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "resources-terminations": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (container, pod, reason) (\n kube_pod_container_status_last_terminated_reason{\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\nand on (pod)\nmax by (pod) (\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\"\n }\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "max by (container, pod, reason) (\n kube_pod_container_status_last_terminated_reason{\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n)\nand on (pod)\nmax by (pod) (\n up{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true, "Value": true }, "indexByName": { "container": 0, "pod": 1, "reason": 2 } } } } ] } }, "description": "**Why each Thanos container last stopped, for those that have stopped since their pod started.**\n\n**Nominal:** Empty.\n\n**Unhealthy:** `OOMKilled`. The container ran out of memory, and _Memory_ beside this shows what it was holding.", "id": 1058, "links": [], "title": "Last Termination Reason", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No recorded termination. If Memory is empty too, nothing is being collected." }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "ruler-alerts-dropped": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n thanos_alert_queue_alerts_dropped_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "queue full", "qryType": 1, "query": "sum(\n rate(\n thanos_alert_queue_alerts_dropped_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n thanos_alert_sender_alerts_dropped_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "send failed", "qryType": 1, "query": "sum(\n rate(\n thanos_alert_sender_alerts_dropped_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Alerts per second the rulers dropped, before sending and while sending.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. A dropped alert is a notification nobody receives. Dropped before sending means the queue was full; dropped while sending means no Alertmanager accepted it.", "id": 1042, "links": [], "title": "Alerts Dropped", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No ruler is reporting", "unit": "suffix:alerts/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-alerts-sent": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (alertmanager) (\n rate(\n thanos_alert_sender_alerts_sent_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{alertmanager}}", "qryType": 1, "query": "sum by (alertmanager) (\n rate(\n thanos_alert_sender_alerts_sent_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Alert notifications per second the rulers sent, per Alertmanager.**\n\n**Nominal:** Zero when nothing is firing, and otherwise steady: a ruler resends every firing alert on each evaluation.\n\nEach ruler sends to every Alertmanager replica, and the replicas agree among themselves on who notifies, so every line should match.", "id": 1041, "links": [], "title": "Alerts Sent by Alertmanager", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No ruler is reporting", "unit": "suffix:alerts/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-config-failed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count(\n thanos_rule_config_last_reload_successful{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n } == 0\n)\nor\n0 * count(\n thanos_rule_config_last_reload_successful{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n", "legendFormat": "rulers", "qryType": 1, "query": "count(\n thanos_rule_config_last_reload_successful{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n } == 0\n)\nor\n0 * count(\n thanos_rule_config_last_reload_successful{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Rulers whose most recent attempt to reload their rule files failed.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. One bad rule rejects the whole set, so a ruler in this state is running stale rules. Its log names the file and the rule.", "id": 1040, "links": [], "title": "Config Reload Failures", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No ruler is reporting", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 1.25 }, { "color": "#F67E4B", "value": 1.5 }, { "color": "#DD3D2D", "value": 1.75 }, { "color": "#A50026", "value": 2 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "ruler-evaluations": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n rate(\n prometheus_rule_evaluations_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n rate(\n prometheus_rule_evaluations_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Rules evaluated per second, per ruler.**\n\n**Nominal:** A flat line on each ruler. The rule groups run on fixed intervals, so this changes only when rules are added or removed.\n\n**Unhealthy:** A ruler at zero. Nothing it holds is being evaluated, and its alerts cannot fire.", "id": 1035, "links": [], "title": "Rule Evaluations", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No ruler is reporting", "unit": "suffix:/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (group) (\n label_replace(\n rate(\n prometheus_rule_evaluation_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n ),\n \"group\", \"$1\", \"rule_group\", \".*;(.*)\"\n )\n) > 0\n", "legendFormat": "{{group}}", "qryType": 1, "query": "sum by (group) (\n label_replace(\n rate(\n prometheus_rule_evaluation_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n ),\n \"group\", \"$1\", \"rule_group\", \".*;(.*)\"\n )\n) > 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Rule evaluations per second that failed, by rule group.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. Every rule in the group is affected. Thanos Query being down or slow fails every group at once; a single group failing on its own has a rule that no longer evaluates, and the ruler's log has the error.", "id": 1036, "links": [], "title": "Failed Evaluations by Group", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No rule group failed to evaluate in this range. If Rule Evaluations is empty too, no ruler is reporting.", "unit": "suffix:/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-loaded": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n thanos_rule_loaded_rules{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n thanos_rule_loaded_rules{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Rules each ruler has loaded.**\n\n**Nominal:** The same number on every ruler.\n\n**Unhealthy:** Rulers disagreeing, or a drop to zero. A ruler that rejected its rule files keeps the last set that loaded, so a rule added since is not running there. _Config Reload Failures_ beside this says whether that happened.", "id": 1039, "links": [], "title": "Rules Loaded", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No ruler is reporting", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-missed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (group) (\n label_replace(\n increase(\n prometheus_rule_group_iterations_missed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n ),\n \"group\", \"$1\", \"rule_group\", \".*;(.*)\"\n )\n) > 0\n", "legendFormat": "{{group}}", "qryType": 1, "query": "sum by (group) (\n label_replace(\n increase(\n prometheus_rule_group_iterations_missed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n ),\n \"group\", \"$1\", \"rule_group\", \".*;(.*)\"\n )\n) > 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Evaluations skipped because the previous one was still running, by rule group.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. The group takes longer than its interval, so its rules run less often than they claim to and their alerts fire late.", "id": 1037, "links": [], "title": "Missed Evaluations by Group", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No rule group missed an evaluation in this range", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-queue": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n thanos_alert_queue_length{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n", "legendFormat": "waiting", "qryType": 1, "query": "sum(\n thanos_alert_queue_length{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n thanos_alert_queue_capacity{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n", "legendFormat": "capacity", "qryType": 1, "query": "sum(\n thanos_alert_queue_capacity{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Alerts waiting to be sent, against the queue's capacity.**\n\n**Nominal:** Close to zero.\n\n**Degraded:** Climbing toward capacity. Alertmanager is slow to accept, and after the queue is full alerts are dropped.", "id": 1043, "links": [], "title": "Alert Queue", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No ruler is reporting", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-remote-write-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n rate(\n prometheus_remote_storage_samples_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n rate(\n prometheus_remote_storage_samples_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Samples per second each ruler failed to write back, after retries.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. Those recorded samples and alert states are lost.", "id": 1046, "links": [], "title": "Write-back Failures", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No ruler is reporting", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-remote-write-lag": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (pod) (\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n- ignoring (remote_name, url) group_right ()\nmax by (pod, remote_name, url) (\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (pod) (\n prometheus_remote_storage_highest_timestamp_in_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n- ignoring (remote_name, url) group_right ()\nmax by (pod, remote_name, url) (\n prometheus_remote_storage_queue_highest_sent_timestamp_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How far behind each ruler is in writing its results back, in seconds.**\n\n**Nominal:** A few seconds.\n\n**Unhealthy:** Climbing. Recorded series and the `ALERTS` series reach Thanos late, so panels built on them, including the alert panels on this dashboard, show stale values. The rulers write through the gateway, so _Alloy Meta Monitoring_ is where to look next.", "id": 1045, "links": [], "title": "Write-back Lag", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-sender-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (alertmanager, le) (\n rate(\n thanos_alert_sender_latency_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "{{alertmanager}}", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (alertmanager, le) (\n rate(\n thanos_alert_sender_latency_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long each Alertmanager took to accept a batch of alerts, at the 99th percentile.**\n\n**Nominal:** Milliseconds.\n\n**Degraded:** Seconds. Notifications arrive late, and the queue starts to fill.", "id": 1044, "links": [], "title": "Alertmanager Latency (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No alerts were sent in this range", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "ruler-slowest": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "topk(\n 10,\n max by (group) (\n label_replace(\n prometheus_rule_group_last_duration_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n / prometheus_rule_group_interval_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n },\n \"group\", \"$1\", \"rule_group\", \".*;(.*)\"\n )\n )\n)\n", "format": "table", "instant": true, "qryType": 2, "query": "topk(\n 10,\n max by (group) (\n label_replace(\n prometheus_rule_group_last_duration_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n }\n / prometheus_rule_group_interval_seconds{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"ruler\"\n },\n \"group\", \"$1\", \"rule_group\", \".*;(.*)\"\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [ { "group": "organize", "kind": "Transformation", "spec": { "options": { "excludeByName": { "Time": true }, "indexByName": { "Value": 1, "group": 0 }, "renameByName": { "Value": "share of interval" } } } }, { "group": "sortBy", "kind": "Transformation", "spec": { "options": { "fields": {}, "sort": [ { "desc": true, "field": "share of interval" } ] } } } ] } }, "description": "**The ten rule groups whose last evaluation took the largest share of their interval.**\n\n**Nominal:** Small fractions.\n\n**Degraded:** Any group near 1. It is about to start missing evaluations.", "id": 1038, "links": [], "title": "Slowest Groups (share of interval)", "vizConfig": { "group": "table", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "custom": { "align": "auto", "filterable": true, "inspect": false }, "noValue": "No ruler is reporting", "unit": "percentunit" }, "overrides": [] }, "options": { "cellHeight": "sm", "footer": { "countRows": false, "reducer": [], "show": false }, "frameIndex": 0, "showHeader": true, "showTypeIcons": false } }, "version": "" } } }, "storage-backlog": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n thanos_compact_todo_compactions{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n", "legendFormat": "compactions", "qryType": 1, "query": "sum(\n thanos_compact_todo_compactions{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n thanos_compact_todo_compaction_blocks{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n", "legendFormat": "blocks to compact", "qryType": 1, "query": "sum(\n thanos_compact_todo_compaction_blocks{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n thanos_compact_todo_downsample_blocks{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n", "legendFormat": "blocks to downsample", "qryType": 1, "query": "sum(\n thanos_compact_todo_downsample_blocks{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n" }, "version": "v0" }, "refId": "query-2" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n thanos_compact_todo_deletion_blocks{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n", "legendFormat": "blocks to delete", "qryType": 1, "query": "sum(\n thanos_compact_todo_deletion_blocks{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n)\n" }, "version": "v0" }, "refId": "query-3" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Work the compactor has planned and not yet done: compactions, the blocks they would merge, blocks waiting to be downsampled, and blocks waiting to be deleted.**\n\n**Nominal:** Close to zero between runs, with a brief rise each time receivers upload.\n\n**Unhealthy:** Climbing and staying up. The compactor is halted or cannot keep up, and long-range queries read ever more small blocks.", "id": 1047, "links": [], "title": "Compaction Backlog", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Compactor not collected", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "storage-blocks": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (state) (\n thanos_blocks_meta_synced{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n) > 0\n", "legendFormat": "{{state}}", "qryType": 1, "query": "max by (state) (\n thanos_blocks_meta_synced{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n) > 0\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Blocks the compactor found in the bucket on its last pass, by state.**\n\n**Nominal:** `loaded` growing slowly, with `marked-for-deletion` and `duplicate` rising and falling as compactions replace blocks.\n\n**Unhealthy:** `loaded` climbing without end. Blocks are not being merged; read the backlog beside this.\n\nStates with no blocks are left out.", "id": 1050, "links": [], "title": "Blocks in the Bucket by State", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Compactor not collected", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "storage-compactions": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (resolution) (\n increase(\n thanos_compact_group_compactions_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [1h]\n )\n)\n", "legendFormat": "{{resolution}} ms", "qryType": 1, "query": "sum by (resolution) (\n increase(\n thanos_compact_group_compactions_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [1h]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Compactions per hour, by the resolution of the blocks merged.**\n\n**Nominal:** Raw-resolution compactions every couple of hours, as receivers upload. The downsampled resolutions appear only after data is old enough to downsample.\n\n**Unhealthy:** None at all over several hours while receivers are uploading. Read _Compactor_ on the Overview tab first.\n\nResolution is in milliseconds: 0 is raw data, 300000 is five-minute downsampling and 3600000 is one-hour.", "id": 1048, "links": [], "title": "Compactions per Hour by Resolution", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Compactor not collected", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "storage-compactor-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n thanos_compact_group_compactions_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "compaction", "qryType": 1, "query": "sum(\n rate(\n thanos_compact_group_compactions_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n thanos_compact_downsample_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "downsampling", "qryType": 1, "query": "sum(\n rate(\n thanos_compact_downsample_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n thanos_compact_garbage_collection_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "garbage collection", "qryType": 1, "query": "sum(\n rate(\n thanos_compact_garbage_collection_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-2" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n thanos_compact_block_cleanup_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "block deletion", "qryType": 1, "query": "sum(\n rate(\n thanos_compact_block_cleanup_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"compactor\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-3" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Compactor operations per second that failed, by kind: compaction, downsampling, garbage collection, and deleting blocks.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. A failed compaction is retried, and repeated failures halt the compactor. The compactor's log names the block.", "id": 1049, "links": [], "title": "Compactor Failures", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "Compactor not collected", "unit": "suffix:/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "storage-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (container) (\n rate(\n thanos_objstore_bucket_operation_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\n/ sum by (container) (\n rate(\n thanos_objstore_bucket_operations_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{container}}", "qryType": 1, "query": "sum by (container) (\n rate(\n thanos_objstore_bucket_operation_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\n/ sum by (container) (\n rate(\n thanos_objstore_bucket_operations_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The share of object-storage requests that failed, by process.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Anything sustained. Every process failing at once is the bucket, its credentials, or the network to it. One process alone is that process's configuration.", "id": 1053, "links": [], "title": "Failure Ratio", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "max": 1, "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "percentunit" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "storage-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (container, operation, le) (\n rate(\n thanos_objstore_bucket_operation_duration_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "{{container}} {{operation}}", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (container, operation, le) (\n rate(\n thanos_objstore_bucket_operation_duration_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long object-storage requests took, at the 99th percentile, by process and operation.**\n\n**Nominal:** Tens of milliseconds for small reads, longer for uploads, which carry whole blocks.\n\n**Degraded:** Reads in seconds. The store gateway answers slowly, and so does every query over older data.", "id": 1054, "links": [], "title": "Latency (p99)", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "storage-operations": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (container, operation) (\n rate(\n thanos_objstore_bucket_operations_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{container}} {{operation}}", "qryType": 1, "query": "sum by (container, operation) (\n rate(\n thanos_objstore_bucket_operations_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Object-storage requests per second, by process and operation.**\n\n**Nominal:** Mostly reads from the store gateway and the compactor. Uploads are bursts, every two hours from each receiver and after each compaction.", "id": 1052, "links": [], "title": "Operations", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "storage-upload-age": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "time()\n- max by (container, pod) (\n thanos_objstore_bucket_last_successful_upload_time{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n } > 0\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "time()\n- max by (container, pod) (\n thanos_objstore_bucket_last_successful_upload_time{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=~\"$thanosComponent\"\n } > 0\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Time since each pod last uploaded to object storage.**\n\n**Nominal:** Under about two hours for each receiver, which uploads a block every two hours. The compactor uploads after each compaction.\n\n**Unhealthy:** A receiver well over two hours. Its recent data exists only on its own disk. The upstream alert `ThanosReceiveNoUpload` fires on this.\n\nOnly pods that have uploaded at least once since they started appear. The store gateway and the ruler never upload.", "id": 1051, "links": [], "title": "Time Since Last Upload", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No process has uploaded to object storage since it started", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "thanos-volume-hidden-note": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [], "queryOptions": {}, "transformations": [] } }, "description": "Why the volume panels are not drawn over long ranges.", "id": 1061, "links": [], "title": "Volume (skipped)", "vizConfig": { "group": "text", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": {}, "overrides": [] }, "options": { "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false }, "content": "**Volume panels are hidden for ranges longer than 7 days.**\n\nCounting log lines means reading every one of them — Loki indexes labels, not counts — so these panels scan the whole selection rather than an index.\n\nShorten the time range, or narrow the namespace and app pickers, and they come back. The feeds below are unaffected at any range: they stop at the first page of matches. For counting over longer spans, [Explore](/explore) is the better tool.", "mode": "markdown" } }, "version": "" } } }, "writes-config-failed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "count(\n thanos_receive_config_last_reload_successful{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n } == 0\n)\nor\n0 * count(\n thanos_receive_config_last_reload_successful{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n)\n", "legendFormat": "receivers", "qryType": 1, "query": "count(\n thanos_receive_config_last_reload_successful{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n } == 0\n)\nor\n0 * count(\n thanos_receive_config_last_reload_successful{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Receivers whose most recent attempt to reload the hashring configuration failed.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. The receiver keeps routing by the last configuration that loaded, so nothing stops, but a change to the receivers is not in effect on it.", "id": 1024, "links": [], "title": "Hashring Reload Failures", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No data from Thanos. Check Collection Health on the Overview tab, or the Logs tab if that is empty too.", "thresholds": { "mode": "absolute", "steps": [ { "color": "#A2F49B", "value": null }, { "color": "#FEDA8B", "value": 1 }, { "color": "#FDB366", "value": 1.25 }, { "color": "#F67E4B", "value": 1.5 }, { "color": "#DD3D2D", "value": 1.75 }, { "color": "#A50026", "value": 2 } ] }, "unit": "short" }, "overrides": [] }, "options": { "colorMode": "background", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } }, "writes-database-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_tsdb_compactions_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "compaction", "qryType": 1, "query": "sum(\n rate(\n prometheus_tsdb_compactions_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_tsdb_head_truncations_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n+ sum(\n rate(\n prometheus_tsdb_wal_truncations_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "truncation", "qryType": 1, "query": "sum(\n rate(\n prometheus_tsdb_head_truncations_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n+ sum(\n rate(\n prometheus_tsdb_wal_truncations_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_tsdb_reloads_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n+ sum(\n rate(\n prometheus_tsdb_checkpoint_creations_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "reload or checkpoint", "qryType": 1, "query": "sum(\n rate(\n prometheus_tsdb_reloads_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n+ sum(\n rate(\n prometheus_tsdb_checkpoint_creations_failed_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-2" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_tsdb_wal_corruptions_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "WAL corruption", "qryType": 1, "query": "sum(\n rate(\n prometheus_tsdb_wal_corruptions_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-3" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Failures per second in the receivers' local databases, by kind: compactions, truncations, reloads and checkpoints, and write-ahead log corruptions.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. Each one means data in memory is not reaching disk or a block, and the receiver's log has the error.\n\nOne expression per kind rather than one over a name pattern: a rate over several metric names loses the names, and the series collide.", "id": 1017, "links": [], "title": "Database Failures", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No receiver is reporting", "unit": "suffix:/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-forward": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (result) (\n rate(\n thanos_receive_forward_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{result}}", "qryType": 1, "query": "sum by (result) (\n rate(\n thanos_receive_forward_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Requests per second each receiver forwarded to the peer that owns the series, by outcome.**\n\n**Nominal:** All success.\n\n**Unhealthy:** Errors. Usually one peer is down or restarting; _Hashring Members_ says which receivers the others can see.", "id": 1019, "links": [], "title": "Forwarded Requests", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No receiver is reporting", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-hashring": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (pod) (\n thanos_receive_hashring_nodes{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (pod) (\n thanos_receive_hashring_nodes{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How many receivers each receiver counts in the hashring, the table that assigns each series to its replicas.**\n\n**Nominal:** The same number on every receiver, equal to the receivers running.\n\n**Unhealthy:** Receivers disagreeing. Each one routes writes by its own copy of the hashring, so a disagreement sends writes to a peer that is not expecting them.", "id": 1020, "links": [], "title": "Hashring Members", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "decimals": 0, "min": 0, "noValue": "No receiver is reporting", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-head-series": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n prometheus_tsdb_head_series{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n prometheus_tsdb_head_series{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Series each receiver holds in memory, for the last two hours of data.**\n\n**Nominal:** The same on every receiver, when each one is a replica for everything, and changing only as targets come and go.\n\n**Degraded:** A step up that does not come back down. A new label value is creating series faster than they expire, and receiver memory grows with it.\n\nThis is the number that sizes a receiver's memory. With the replication factor equal to the number of receivers, each one holds every series.", "id": 1015, "links": [], "title": "Series in Memory", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No receiver is reporting", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-latency": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.50,\n sum by (le) (\n rate(\n http_request_duration_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "p50", "qryType": 1, "query": "histogram_quantile(\n 0.50,\n sum by (le) (\n rate(\n http_request_duration_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "histogram_quantile(\n 0.99,\n sum by (le) (\n rate(\n http_request_duration_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n )\n)\n", "legendFormat": "p99", "qryType": 1, "query": "histogram_quantile(\n 0.99,\n sum by (le) (\n rate(\n http_request_duration_seconds_bucket{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How long the receivers took to answer a remote write, at the median and the 99th percentile.**\n\n**Nominal:** Tens of milliseconds at p99.\n\n**Degraded:** A p99 in seconds. A write is answered only after enough replicas have it, so one slow receiver slows every write it is a replica for.", "id": 1014, "links": [], "title": "Write Latency", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No remote writes reached the receivers", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-pending": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "max by (pod) (\n thanos_receive_pending_write_requests{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "max by (pod) (\n thanos_receive_pending_write_requests{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Remote-write requests each receiver is handling at once.**\n\n**Nominal:** A handful.\n\n**Degraded:** Climbing steadily. The receiver is taking requests faster than it finishes them, usually because a peer it replicates to is slow.", "id": 1021, "links": [], "title": "Writes in Progress", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No receiver is reporting", "unit": "short" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-rejected": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_tsdb_out_of_order_samples_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "out of order", "qryType": 1, "query": "sum(\n rate(\n prometheus_tsdb_out_of_order_samples_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_tsdb_out_of_bound_samples_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "out of bounds", "qryType": 1, "query": "sum(\n rate(\n prometheus_tsdb_out_of_bound_samples_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-1" } }, { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n prometheus_tsdb_too_old_samples_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "too old", "qryType": 1, "query": "sum(\n rate(\n prometheus_tsdb_too_old_samples_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-2" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Samples per second the receivers' databases refused, by reason.**\n\n**Nominal:** Zero.\n\n**Degraded:** A steady rate of out-of-order samples. Two senders are writing the same series, or the gateway is retrying writes that had already landed.\n\n**Unhealthy:** Out-of-bounds or too-old samples. The sender's clock is wrong, or it is replaying data older than the receiver still accepts, and those samples are lost.", "id": 1016, "links": [], "title": "Samples Rejected by the Database", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No receiver is reporting", "unit": "suffix:samples/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-replication": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (result) (\n rate(\n thanos_receive_replications_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{result}}", "qryType": 1, "query": "sum by (result) (\n rate(\n thanos_receive_replications_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Replication requests per second between receivers, by outcome.**\n\n**Nominal:** All success.\n\n**Unhealthy:** Errors. A receiver could not hand a sample to a peer it is a replica for. Writes still succeed while enough replicas have the sample, so this degrades before anything fails.", "id": 1018, "links": [], "title": "Replication", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No receiver is reporting", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-requests": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (code) (\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{code}}", "qryType": 1, "query": "sum by (code) (\n rate(\n http_requests_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\",\n handler=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Remote-write requests per second the receivers answered, by status code.**\n\n**Nominal:** Almost all 200.\n\n**Degraded:** A steady 409. The receiver refused samples it already had or that arrived out of order, which happens when the gateway retries a request that had partly succeeded. A trickle is harmless; a steady share means samples are being lost, and _Samples Rejected by the Database_ says why.\n\n**Unhealthy:** 5xx. The gateway retries those, and its queue grows until it drops.", "id": 1013, "links": [], "title": "Write Requests by Status", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No remote writes reached the receivers", "unit": "reqps" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-upload-age": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "time()\n- max by (pod) (\n thanos_objstore_bucket_last_successful_upload_time{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n } > 0\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "time()\n- max by (pod) (\n thanos_objstore_bucket_last_successful_upload_time{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n } > 0\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Time since each receiver last uploaded a block to object storage.**\n\n**Nominal:** Under about two hours. Each receiver uploads a block every two hours.\n\n**Unhealthy:** Well over two hours. That receiver's recent data exists only on its own disk. The upstream alert `ThanosReceiveNoUpload` fires on this.\n\nOnly receivers that have uploaded at least once since they started appear.", "id": 1022, "links": [], "title": "Time Since Last Upload", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No receiver has uploaded a block since it started", "unit": "s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "writes-upload-failures": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "sum by (pod) (\n rate(\n thanos_shipper_upload_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n", "legendFormat": "{{pod}}", "qryType": 1, "query": "sum by (pod) (\n rate(\n thanos_shipper_upload_failures_total{\n app=\"thanos\",\n namespace=~\"$thanosNamespace\",\n container=\"receive\"\n }\n [$__rate_interval]\n )\n)\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Failed block uploads to object storage per second, per receiver.**\n\n**Nominal:** Zero.\n\n**Unhealthy:** Any. The block stays on the receiver's disk and is retried, but until it uploads, the receiver is the only place that data exists. A receiver that keeps failing fills its volume. _Storage -> Object Storage_ says whether the bucket is refusing everyone.", "id": 1023, "links": [], "title": "Block Upload Failures", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No receiver is reporting", "unit": "suffix:/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } } }, "layout": { "kind": "TabsLayout", "spec": { "tabs": [ { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "narrow", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-alerts-firing" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-components-down" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-write-errors" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-query-errors" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-rule-failures" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-compactor" } } } ], "maxColumnCount": 6, "rowHeightMode": "short" } }, "title": "Verdict" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-alerts" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-alert-history" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Alerts" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-scrape" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-restarts" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-versions" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Collection Health" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-samples-ingested" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "overview-query-rate" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Throughput" } } ] } }, "title": "Overview" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-requests" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-latency" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Requests" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-head-series" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-rejected" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-database-failures" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Database" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-replication" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-forward" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-hashring" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-pending" } } } ], "maxColumnCount": 4, "rowHeightMode": "standard" } }, "title": "Replication" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-upload-age" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-upload-failures" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "writes-config-failed" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Uploads and Configuration" } } ] } }, "title": "Writes" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-requests" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-latency" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-concurrency" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Thanos Query" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-endpoints" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-store-errors" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-series-latency" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-dropped" } } } ], "maxColumnCount": 4, "rowHeightMode": "standard" } }, "title": "Stores" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-blocks-loaded" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-block-failures" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "reads-index-cache" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Store Gateway" } } ] } }, "title": "Reads" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-evaluations" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-failures" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-missed" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Evaluation" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-slowest" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-loaded" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-config-failed" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Rules" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-alerts-sent" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-alerts-dropped" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-queue" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-sender-latency" } } } ], "maxColumnCount": 4, "rowHeightMode": "standard" } }, "title": "Notifications" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-remote-write-lag" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "ruler-remote-write-failures" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Write-back" } } ] } }, "title": "Ruler" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "storage-backlog" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "storage-compactions" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "storage-compactor-failures" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Compactor" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "wide", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "storage-blocks" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "storage-upload-age" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Blocks" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "storage-operations" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "storage-failures" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "storage-latency" } } } ], "maxColumnCount": 3, "rowHeightMode": "standard" } }, "title": "Object Storage" } } ] } }, "title": "Storage" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-memory" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-cpu" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Usage" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-goroutines" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "resources-terminations" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Runtime" } } ] } }, "title": "Resources" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingTimeRangeSize", "spec": { "value": "7d" } } ], "visibility": "show" } }, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-rate-by-component" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-warning-rate" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Volume" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingTimeRangeSize", "spec": { "value": "7d" } } ], "visibility": "hide" } }, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "thanos-volume-hidden-note" } } } ], "maxColumnCount": 1, "rowHeightMode": "short" } }, "title": "Volume" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-warning-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "Warnings" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "logs-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "All Logs" } } ] } }, "title": "Logs" } } ] } }, "links": [], "liveNow": false, "preload": false, "tags": [ "infrastructure", "mzmon", "meta-monitoring" ], "timeSettings": { "autoRefresh": "", "autoRefreshIntervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "fiscalYearStartMonth": 0, "from": "now-6h", "hideTimepicker": false, "timezone": "browser", "to": "now" }, "title": "Thanos Meta Monitoring", "variables": [ { "kind": "DatasourceVariable", "spec": { "allowCustomValue": false, "current": { "text": "", "value": "" }, "description": "Datasource for metrics queries", "hide": "dontHide", "includeAll": false, "label": "Metrics Datasource", "multi": false, "name": "metricsDatasource", "options": [], "pluginId": "prometheus", "refresh": "never", "regex": "", "skipUrlSync": false } }, { "kind": "DatasourceVariable", "spec": { "allowCustomValue": false, "current": { "text": "", "value": "" }, "description": "Datasource for log and event queries", "hide": "dontHide", "includeAll": false, "label": "Logs Datasource", "multi": false, "name": "logsDatasource", "options": [], "pluginId": "loki", "refresh": "never", "regex": "", "skipUrlSync": false } }, { "kind": "QueryVariable", "spec": { "allValue": ".+", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(up{app=\"thanos\"}, namespace)", "description": "Namespace(s) the metrics store runs in", "hide": "dontHide", "includeAll": true, "label": "Thanos Namespace", "multi": true, "name": "thanosNamespace", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(up{app=\"thanos\"}, namespace)", "qryType": 1, "query": "label_values(up{app=\"thanos\"}, namespace)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".+", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(up{app=\"thanos\", namespace=~\"$thanosNamespace\"}, container)", "description": "Which Thanos processes to read, on the panels that span several", "hide": "dontHide", "includeAll": true, "label": "Component", "multi": true, "name": "thanosComponent", "options": [], "query": { "datasource": { "name": "${metricsDatasource}" }, "group": "prometheus", "kind": "DataQuery", "spec": { "expr": "label_values(up{app=\"thanos\", namespace=~\"$thanosNamespace\"}, container)", "qryType": 1, "query": "label_values(up{app=\"thanos\", namespace=~\"$thanosNamespace\"}, container)" }, "version": "v0" }, "refresh": "never", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".*", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(level)", "description": "The severity level(s) to include", "hide": "dontHide", "includeAll": true, "label": "Level", "multi": true, "name": "logLevelList", "options": [], "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "label": "level", "refId": "LokiVariableQueryEditor-VariableQuery", "stream": "{app=\"thanos\", namespace=~\"$thanosNamespace\"}", "type": 1 }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "TextVariable", "spec": { "current": { "text": "", "value": "" }, "description": "Case-insensitive text to match anywhere in the log line", "hide": "dontHide", "label": "Search", "name": "logSearch", "query": "", "skipUrlSync": false } }, { "datasource": { "name": "$metricsDatasource" }, "group": "", "kind": "AdhocVariable", "spec": { "allowCustomValue": true, "baseFilters": [ { "key": "namespace", "operator": "=~", "origin": "dashboard", "value": "$thanosNamespace" } ], "defaultKeys": [], "description": "Adhoc filters to apply to all metrics queries", "enableGroupBy": false, "filters": [], "hide": "inControlsMenu", "label": "Advanced Metric Filter", "name": "metricAdhoc", "skipUrlSync": false } }, { "datasource": { "name": "$logsDatasource" }, "group": "", "kind": "AdhocVariable", "spec": { "allowCustomValue": true, "baseFilters": [], "defaultKeys": [], "description": "Adhoc filters to apply to all logs queries", "enableGroupBy": false, "filters": [], "hide": "inControlsMenu", "label": "Advanced Log Filter", "name": "logsAdhoc", "skipUrlSync": false } } ] } }