{ "apiVersion": "dashboard.grafana.app/v2", "kind": "Dashboard", "metadata": { "annotations": { "grafana.app/folder": "infra", "monitoring.materialize.cloud/min-mz-version": "v26.24.0", "monitoring.materialize.cloud/rec-mz-version": "v26.24.0", "monitoring.materialize.cloud/sql-metric-prefix": "mz_", "monitoring.materialize.cloud/target-export": "generic" }, "name": "mz-mon-infra-logs" }, "spec": { "annotations": [ { "kind": "AnnotationQuery", "spec": { "builtIn": true, "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "query": { "datasource": { "name": "-- Grafana --" }, "group": "grafana", "kind": "DataQuery", "spec": {}, "version": "v0" } } } ], "cursorSync": "Crosshair", "description": "Logs and Kubernetes events for the platform a Materialize deployment runs on.\n\nThe monitoring stack, the Kubernetes system components, and the node journal.", "editable": true, "elements": { "event-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{job=\"loki.source.kubernetes_events\", namespace=~\"$logNamespaceList\", namespace!~\"$excludeMaterialize\"}\n |~ \"(?i)$logSearch\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Every Kubernetes event in the selected namespaces, newest first.**\n\nThe general-purpose event feed, scoped by the same namespace picker as the logs beside it rather than by a deployment's own namespaces. Turning the Materialize-only switch off reaches `kube-system` and the rest of the cluster, which is where the answer lives when the question is about nodes or storage rather than about Materialize.\n\nDistinct from the rollout feed on the upgrade dashboard: this one carries no generation or reporting-controller filter, because a general browser should not quietly drop events for belonging to the wrong side of a rollout.\n\nThe consuming panel renders `reason`, `msg` and `name` as columns rather than showing the raw line, so this query deliberately does **not** `line_format`: displayed fields supersede the line, and formatting one would be work thrown away. The rollout feeds on the upgrade dashboard do reformat, because they show the line.", "id": 1012, "links": [], "title": "All Events", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No events in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "signature", "detailsMode": "inline", "displayedFields": [ "reason", "name", "msg" ], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": true, "wrapLogMessage": false } }, "version": "" } } }, "event-rate-by-namespace": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (namespace) (\n count_over_time(\n {job=\"loki.source.kubernetes_events\", namespace=~\"$logNamespaceList\", namespace!~\"$excludeMaterialize\"}\n |~ \"(?i)$logSearch\"\n [$__auto]\n )\n)\n", "legendFormat": "{{namespace}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Event rate by namespace — where in the cluster things are happening.**\n\nMost useful with the Materialize-only switch off, which is when this stops being one or two flat lines and starts saying whether the activity is in the deployment, the monitoring stack, or the platform underneath both.", "id": 1010, "links": [], "title": "Event Rate by Namespace", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No events in this time range" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "event-rate-by-reason": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (reason) (\n count_over_time(\n {job=\"loki.source.kubernetes_events\", namespace=~\"$logNamespaceList\", namespace!~\"$excludeMaterialize\"}\n |~ \"(?i)$logSearch\"\n [$__auto]\n )\n)\n", "legendFormat": "{{reason}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**How often each kind of event is being reported, by reason — the shape of what the cluster is doing.**\n\n**Nominal:** Quiet, apart from bursts that coincide with a deliberate change.\n\n**Unhealthy:** A reason that fires at a steady rate is a loop rather than a transition.\n\nCounts events as Loki received them. Kubernetes aggregates a repeat into one object with a rising `count` rather than sending it again, so a tight loop reports fewer lines than it has occurrences.", "id": 1009, "links": [], "title": "Event Rate by Reason", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No events in this time range" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "log-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n namespace=~\"$logNamespaceList\",\n namespace!~\"$excludeMaterialize\",\n app=~\"$logAppList\",\n component=~\"$logComponentList\",\n container=~\"$logContainerList\",\n level=~\"$logLevelList\",\n job=~\"$logJobList\"\n}\n |~ \"(?i)$logSearch\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The log feed for the selected namespaces, apps, components, containers and levels, newest first.**\n\nThe tab's primary panel. `component` and `container` are what this adds over the Materialize feed: the first isolates one process of a multi-process workload, the second reaches workloads that publish no `app` label.\n\nAn empty search matches every line rather than none, so the panel is useful before anything is typed.", "id": 1005, "links": [], "title": "All Logs", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No logs match the current filters" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "log-rate-by-component": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (app, component) (\n rate(\n {\n namespace=~\"$logNamespaceList\",\n namespace!~\"$excludeMaterialize\",\n app=~\"$logAppList\",\n component=~\"$logComponentList\",\n container=~\"$logContainerList\",\n level=~\"$logLevelList\",\n job=~\"$logJobList\"\n }\n |~ \"(?i)$logSearch\"\n [$__auto]\n )\n)\n", "legendFormat": "{{app}} / {{component}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second by application and sub-component — which process of which workload is doing the talking.**\n\n**Nominal:** A steady baseline. Loki's own canary dominates on a quiet cluster, which is expected: it exists to write and read a line on a loop.\n\n**Unhealthy:** A sharp step up confined to one component. That is the shape of a single process in a retry loop, and it is invisible on a panel that only splits by application.\n\nLegends read `app / component`. A workload with no sub-components reports an empty one, which is why the component picker's \"All\" is `.*` rather than `.+` — the stricter form would drop them.", "id": 1000, "links": [], "title": "Log Rate by Component", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No logs match the current filters", "unit": "suffix:logs/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "log-rate-by-namespace": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (namespace) (\n rate(\n {\n namespace=~\"$logNamespaceList\",\n namespace!~\"$excludeMaterialize\",\n app=~\"$logAppList\",\n component=~\"$logComponentList\",\n container=~\"$logContainerList\",\n level=~\"$logLevelList\",\n job=~\"$logJobList\"\n }\n |~ \"(?i)$logSearch\"\n [$__auto]\n )\n)\n", "legendFormat": "{{namespace}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Log lines per second by namespace — where in the cluster the volume is.**\n\nThe coarsest useful split, and the one that answers \"is this the platform or the workload\" before any of the finer pickers are touched.", "id": 1001, "links": [], "title": "Log Rate by Namespace", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No logs match the current filters", "unit": "suffix:logs/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "node-log-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n unit=~\"$logUnitList\",\n level=~\"$logLevelList\"\n}\n |~ \"(?i)$logSearch\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**The node journal, newest first — `kubelet`, `containerd`, the node problem detector, and the rest of what systemd runs on each node.**\n\n**Unreachable from any namespace-scoped dashboard.** These lines come from the node, not from a pod, so they carry no `namespace`, `app` or `container` at all — a selector requiring any of those excludes them by construction. That is the gap this tab exists to close.\n\n`unit` is the anchor here, the way `job` is for container logs: a journal selector has no namespace matcher to lean on, so its \"All\" is `.+`.\n\nThe node a line came from is structured metadata, not a label. Filter it after a `|`, or add it through the advanced filter.", "id": 1008, "links": [], "title": "Node Journal", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No node journal logs — is journal collection enabled?" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "node-rate-by-unit": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum by (unit) (\n rate(\n {\n unit=~\"$logUnitList\",\n level=~\"$logLevelList\"\n }\n |~ \"(?i)$logSearch\"\n [$__auto]\n )\n)\n", "legendFormat": "{{unit}}" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Node journal lines per second by systemd unit.**\n\n**Nominal:** A steady murmur, dominated by the cloud provider's guest agent.\n\n**Unhealthy:** A step up in `kubelet` or `containerd`. Both narrate their retries, so a rising rate there is usually a node struggling to reconcile what it has been asked to run.\n\nSplit by unit rather than by node: which *thing* is talking is the first question, and the node is one `|` away once you know.", "id": 1006, "links": [], "title": "Journal Rate by Unit", "vizConfig": { "group": "timeseries", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "min": 0, "noValue": "No node journal logs — is journal collection enabled?", "unit": "suffix:logs/s" }, "overrides": [] }, "options": { "legend": { "calcs": [ "max", "mean", "lastNotNull" ], "displayMode": "table", "placement": "bottom", "showLegend": true }, "tooltip": { "mode": "single", "sort": "asc" } } }, "version": "" } } }, "node-warning-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n unit=~\"$logUnitList\",\n level=~\"WARN|WARNING|ERROR|CRITICAL|FATAL\"\n}\n |~ \"(?i)$logSearch\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Warning-and-worse lines from the node journal.**\n\n**Nominal:** Quiet. Most journal lines never carry a level the pipeline recognizes and land as `UNKNOWN`, so what does reach `WARN` or `ERROR` is worth reading.\n\n**Unhealthy:** `kubelet` or `containerd` complaining repeatedly. Those two are the node's control plane for everything Kubernetes runs on it, so a sustained complaint there precedes pods that will not start or will not stop.\n\nIndependent of the level picker, like the other warning panels.", "id": 1007, "links": [], "title": "Node Warnings and Errors", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No warnings in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "volume-hidden-note": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [], "queryOptions": {}, "transformations": [] } }, "description": "Why the volume panels are not drawn over long ranges.", "id": 1003, "links": [], "title": "Volume (skipped)", "vizConfig": { "group": "text", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": {}, "overrides": [] }, "options": { "code": { "language": "plaintext", "showLineNumbers": false, "showMiniMap": false }, "content": "**Volume panels are hidden for ranges longer than 7 days.**\n\nCounting log lines means reading every one of them — Loki indexes labels, not counts — so these panels scan the whole selection rather than an index.\n\nShorten the time range, or narrow the namespace and app pickers, and they come back. The feeds below are unaffected at any range: they stop at the first page of matches. For counting over longer spans, [Explore](/explore) is the better tool.", "mode": "markdown" } }, "version": "" } } }, "warning-event-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{job=\"loki.source.kubernetes_events\", namespace=~\"$logNamespaceList\", namespace!~\"$excludeMaterialize\", level=\"WARN\"}\n |~ \"(?i)$logSearch\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Kubernetes events the reporting component flagged as warnings, across the selected namespaces.**\n\n**Nominal:** Nothing, or a burst that stops on its own.\n\n**Unhealthy:** Warnings that keep arriving. The reason names the class: `FailedScheduling` and `NotTriggerScaleUp` mean the cluster has nowhere to put a pod, `BackOff` means it starts and dies, `Unhealthy` means it starts and never becomes ready, `FailedMount` means storage did not attach.\n\n`Warning` is the reporting component's own judgement rather than a threshold of ours — this filters on their opinion.", "id": 1011, "links": [], "title": "Warning Events", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No warning events in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "signature", "detailsMode": "inline", "displayedFields": [ "reason", "name", "msg" ], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": true, "wrapLogMessage": false } }, "version": "" } } }, "warning-feed": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "{\n namespace=~\"$logNamespaceList\",\n namespace!~\"$excludeMaterialize\",\n app=~\"$logAppList\",\n component=~\"$logComponentList\",\n container=~\"$logContainerList\",\n level=~\"WARN|WARNING|ERROR|CRITICAL|FATAL\",\n job=~\"$logJobList\"\n}\n |~ \"(?i)$logSearch\"\n" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Warning-and-worse lines from the platform, newest first.**\n\n**Nominal:** Quiet, or a burst that stops on its own.\n\n**Unhealthy:** A steady stream. Read which component: the monitoring stack complaining about itself is the case worth acting on first, since everything else you are looking at depends on it.\n\nDeliberately independent of the level picker, like its Materialize counterpart: this panel answers \"is anything wrong\", and narrowing the selection to `INFO` would silently zero it.", "id": 1004, "links": [], "title": "Warnings and Errors", "vizConfig": { "group": "logs", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "noValue": "No warnings in this time range" }, "overrides": [] }, "options": { "dedupStrategy": "exact", "detailsMode": "inline", "displayedFields": [], "enableInfiniteScrolling": true, "enableLogDetails": true, "fontSize": "small", "prettifyLogMessage": false, "showCommonLabels": false, "showControls": true, "showFieldSelector": true, "showLabels": false, "showLevel": true, "showLogContextToggle": false, "showTime": true, "sortOrder": "Descending", "syntaxHighlighting": true, "timestampResolution": "ms", "unwrappedColumns": false, "wrapLogMessage": true } }, "version": "" } } }, "warning-rate": { "kind": "Panel", "spec": { "data": { "kind": "QueryGroup", "spec": { "queries": [ { "kind": "PanelQuery", "spec": { "hidden": false, "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "expr": "sum(\n rate(\n {\n namespace=~\"$logNamespaceList\",\n namespace!~\"$excludeMaterialize\",\n app=~\"$logAppList\",\n component=~\"$logComponentList\",\n container=~\"$logContainerList\",\n level=~\"WARN|WARNING|ERROR|CRITICAL|FATAL\",\n job=~\"$logJobList\"\n }\n |~ \"(?i)$logSearch\"\n [$__auto]\n )\n) * 60\n", "legendFormat": "warnings/min" }, "version": "v0" }, "refId": "query-0" } } ], "queryOptions": {}, "transformations": [] } }, "description": "**Warning-and-worse lines per minute across the platform, as one series.**\n\n**Nominal:** Flat, or a spike that returns to baseline.\n\n**Unhealthy:** A level that does not come back down. _Volume -> Log Rate by Component_ says which process, and the feed below says what.\n\nReported per **minute**: warnings are rare enough on a healthy platform that a per-second rate spends its life showing a small fraction.\n\nIndependent of the level picker, for the same reason as the feed.", "id": 1002, "links": [], "title": "Average Warning Rate", "vizConfig": { "group": "stat", "kind": "VizConfig", "spec": { "fieldConfig": { "defaults": { "color": { "fixedColor": "#33BBEE", "mode": "shades" }, "min": 0, "noValue": "No warnings in this time range", "unit": "suffix:logs/min" }, "overrides": [] }, "options": { "colorMode": "none", "graphMode": "area", "justifyMode": "auto", "orientation": "auto", "percentChangeColorMode": "standard", "reduceOptions": { "calcs": [], "values": false }, "showPercentChange": false, "textMode": "value", "wideLayout": true } }, "version": "" } } } }, "layout": { "kind": "TabsLayout", "spec": { "tabs": [ { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingTimeRangeSize", "spec": { "value": "7d" } } ], "visibility": "show" } }, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "log-rate-by-component" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "log-rate-by-namespace" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "warning-rate" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Volume" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "conditionalRendering": { "kind": "ConditionalRenderingGroup", "spec": { "condition": "and", "items": [ { "kind": "ConditionalRenderingTimeRangeSize", "spec": { "value": "7d" } } ], "visibility": "hide" } }, "hideHeader": true, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "volume-hidden-note" } } } ], "maxColumnCount": 1, "rowHeightMode": "short" } }, "title": "Volume" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "warning-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "Warnings" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "log-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "All Logs" } } ] } }, "title": "Logs" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "node-rate-by-unit" } } } ], "maxColumnCount": 1, "rowHeightMode": "standard" } }, "title": "Journal Volume" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "node-warning-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "Node Warnings" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "node-log-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "All Node Logs" } } ] } }, "title": "Nodes" } }, { "kind": "TabsLayoutTab", "spec": { "layout": { "kind": "RowsLayout", "spec": { "rows": [ { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "event-rate-by-reason" } } }, { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "event-rate-by-namespace" } } } ], "maxColumnCount": 2, "rowHeightMode": "standard" } }, "title": "Activity" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "warning-event-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "Warnings" } }, { "kind": "RowsLayoutRow", "spec": { "collapse": false, "hideHeader": false, "layout": { "kind": "AutoGridLayout", "spec": { "columnWidthMode": "standard", "fillScreen": false, "items": [ { "kind": "AutoGridLayoutItem", "spec": { "element": { "kind": "ElementReference", "name": "event-feed" } } } ], "maxColumnCount": 1, "rowHeightMode": "tall" } }, "title": "All Events" } } ] } }, "title": "Events" } } ] } }, "links": [], "liveNow": false, "preload": false, "tags": [ "infrastructure", "mzmon", "logs" ], "timeSettings": { "autoRefresh": "", "autoRefreshIntervals": [ "5s", "10s", "30s", "1m", "5m", "15m", "30m", "1h", "2h", "1d" ], "fiscalYearStartMonth": 0, "from": "now-6h", "hideTimepicker": false, "timezone": "browser", "to": "now" }, "title": "Infrastructure Logs and Events", "variables": [ { "kind": "DatasourceVariable", "spec": { "allowCustomValue": false, "current": { "text": "", "value": "" }, "description": "Datasource for log and event queries", "hide": "dontHide", "includeAll": false, "label": "Logs Datasource", "multi": false, "name": "logsDatasource", "options": [], "pluginId": "loki", "refresh": "never", "regex": "", "skipUrlSync": false } }, { "kind": "QueryVariable", "spec": { "allValue": ".+", "allowCustomValue": true, "current": { "selected": true, "text": [ ".+" ], "value": [ ".+" ] }, "definition": "label_values(namespace)", "description": "The namespace(s) to read logs from", "hide": "dontHide", "includeAll": true, "label": "Namespace", "multi": true, "name": "logNamespaceList", "options": [], "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "label": "namespace", "refId": "LokiVariableQueryEditor-VariableQuery", "stream": "", "type": 1 }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "SwitchVariable", "spec": { "current": ".*materialize.*|mz-.*|environment-.*", "description": "Drop the Materialize namespaces, so the platform's own logs and events are not buried under the deployment's.", "disabledValue": "a^", "enabledValue": ".*materialize.*|mz-.*|environment-.*", "hide": "dontHide", "label": "Exclude Materialize", "name": "excludeMaterialize", "skipUrlSync": false } }, { "kind": "QueryVariable", "spec": { "allValue": ".*", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(app)", "description": "The application(s) to read logs from", "hide": "dontHide", "includeAll": true, "label": "App", "multi": true, "name": "logAppList", "options": [], "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "label": "app", "refId": "LokiVariableQueryEditor-VariableQuery", "stream": "{namespace=~\"$logNamespaceList\"}", "type": 1 }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".*", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(component)", "description": "The sub-component(s) to read logs from", "hide": "dontHide", "includeAll": true, "label": "Component", "multi": true, "name": "logComponentList", "options": [], "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "label": "component", "refId": "LokiVariableQueryEditor-VariableQuery", "stream": "{namespace=~\"$logNamespaceList\"}", "type": 1 }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".*", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(level)", "description": "The severity level(s) to include", "hide": "dontHide", "includeAll": true, "label": "Level", "multi": true, "name": "logLevelList", "options": [], "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "label": "level", "refId": "LokiVariableQueryEditor-VariableQuery", "stream": "{namespace=~\"$logNamespaceList\"}", "type": 1 }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".+", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(unit)", "description": "The systemd unit(s) to read node journal logs from", "hide": "dontHide", "includeAll": true, "label": "Unit", "multi": true, "name": "logUnitList", "options": [], "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "label": "unit", "refId": "LokiVariableQueryEditor-VariableQuery", "stream": "", "type": 1 }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".*", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(container)", "description": "The container(s) to read logs from", "hide": "inControlsMenu", "includeAll": true, "label": "Container", "multi": true, "name": "logContainerList", "options": [], "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "label": "container", "refId": "LokiVariableQueryEditor-VariableQuery", "stream": "{namespace=~\"$logNamespaceList\"}", "type": 1 }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "QueryVariable", "spec": { "allValue": ".+", "allowCustomValue": true, "current": { "text": "", "value": "" }, "definition": "label_values(job)", "description": "The collection job(s) to read logs from", "hide": "inControlsMenu", "includeAll": true, "label": "Job", "multi": true, "name": "logJobList", "options": [], "query": { "datasource": { "name": "${logsDatasource}" }, "group": "loki", "kind": "DataQuery", "spec": { "label": "job", "refId": "LokiVariableQueryEditor-VariableQuery", "stream": "{namespace=~\"$logNamespaceList\"}", "type": 1 }, "version": "v0" }, "refresh": "onTimeRangeChanged", "regex": "", "regexApplyTo": "value", "skipUrlSync": false, "sort": "alphabeticalAsc" } }, { "kind": "TextVariable", "spec": { "current": { "text": "", "value": "" }, "description": "Case-insensitive text to match anywhere in the log line", "hide": "dontHide", "label": "Search", "name": "logSearch", "query": "", "skipUrlSync": false } }, { "datasource": { "name": "$logsDatasource" }, "group": "", "kind": "AdhocVariable", "spec": { "allowCustomValue": true, "baseFilters": [], "defaultKeys": [], "description": "Adhoc filters to apply to all logs queries", "enableGroupBy": false, "filters": [], "hide": "inControlsMenu", "label": "Advanced Log Filter", "name": "logsAdhoc", "skipUrlSync": false } } ] } }