Available Dashboards#

The dashboards come in two families: those scoped to a Materialize environment (env-*), and those scoped to the platform underneath it (infra-*). The infra-* dashboards that watch the monitoring stack itself, rather than anything it collects, are filed in the Meta Observability folder. Each one below has its own download links and its own compatibility annotations.

If you install the dashboards charts, you don’t need to download anything. Each chart installs every dashboard it carries by default, and the Grafana Operator path keeps them in sync rather than importing a point-in-time copy. materialize-monitoring-dashboards carries the env-* dashboards, materialize-monitoring-dashboards-meta carries the dashboards in the Meta Observability folder, and materialize-monitoring-dashboards-infra carries the rest of the infra-* dashboards. Download them when you are putting them into a Grafana you run yourself; Importing Dashboards covers the ways to do that.

Formats#

Every dashboard has a Grafana dashboard schema v2 render, which needs Grafana 12 or later. For more information, see Grafana compatibility. env-top also has a Datadog render, which Datadog describes. Each table below lists every format, so you can see which ones a dashboard has no render for. Grafana 10 and 11 (schema v1) renders are not planned. The rest of the Datadog set, and the Google Cloud Monitoring and Honeycomb sets, are tracked on the roadmap. Common Queries is the query material to build one yourself.

Checking your Grafana version#

Navigate to the Grafana instance and click on the “Help” menu (represented by a question mark icon) in the left sidebar. Selecting it will show the current version.

Materialize environment (env-*)#

Materialize Environment Overview (env-top)#

The high-level summary, and the one to open first. Its six tabs — Summary, Kubernetes Workloads, Connections / Activity, Cluster Objects / Replicas, Compute Objects, and Sources and Sinks — are there to catch the more obvious problems and point at what needs a closer look, rather than to answer any one of them in depth. The Summary tab counts the alerts firing about the environment beside its health numbers. Below them, it lists every alert firing or pending in the environment’s namespace. Each alert’s name links to its entry under Common Alerts. It reads metrics only, and scopes to one environment at a time, with further pickers for cluster and replica.

Materialize Logs and Events (env-logs)#

What the Materialize workloads said, as opposed to what they measured, across two tabs: Logs and Events. This one is Loki-only — it defines no metrics datasource, and its namespace, app, and level pickers are discovered from Loki — so it keeps working when the metrics pipeline is the thing being investigated. The namespace picker discovers every namespace and merely defaults to the Materialize ones, so the monitoring stack and kube-system are one selection away.

Materialize Upgrade (env-upgrade)#

What happened during an upgrade, from the operator’s own account of the rollout alongside what the cluster did underneath it. The Events tab carries Kubernetes events from the operator and environment namespaces, Generations splits a blue/green rollout into its two sides so the question a rollout actually poses — has the new generation caught up yet — can be asked at all, and Reconciliation is the operator’s control loop as metrics. It needs both a metrics and a logs datasource.

The operator signals this dashboard reads require Materialize v26.41.0. On an older release it degrades unevenly rather than going dark, and compatibility has the per-tab detail.

Materialize Persist (Storage) (env-persist)#

Object storage, as a Materialize environment experiences it, across five tabs: Overview, Operations, Compaction, Storage, and Object Store Internals. Persist is Materialize’s storage layer: every durable object is stored as data files in the persist bucket, and every read, write, and delete Materialize makes against the bucket is measured by the process that made it. Those measurements are identical on S3, GCS, Azure Blob, and S3-compatible stores, and need no cloud credentials.

The Overview tab answers whether the bucket is failing Materialize: failed operations, read and write latency, write stalls, and compaction failures, with the store’s own error text from the logs beneath them. The Storage tab is Materialize’s own accounting of what the stored data is for — current data, data kept only for a reader still looking at an older version, and data nothing refers to. The bucket’s own size, as the provider bills it, is on infra-cloud. The Object Store Internals tab is the store’s own account, with a section for each store it finds:

Ceph run by Rook

Ceph’s health verdict and active health checks, disks down, free space, and the object gateway’s requests, failures, and latency. The section appears when you scrape Rook’s Ceph manager and rook-ceph-exporter.

GCS

Each bucket’s live data, object count, and average object size, and how much its data and object count change each day. Two donuts split the selected buckets’ billed bytes and objects into live objects, noncurrent versions, and soft-deleted objects. The section appears when the gateway pulls the bucket from Cloud Monitoring, under pipeline.metrics.provider.gcp.gcs.buckets. The GCS Bucket picker opens on every pulled bucket. The environment’s bucket is the one whose Live Data tile reads the same as the environment’s. Google measures a bucket once a day, so the section’s charts show the last 14 days whatever time range you pick.

On S3, Azure Blob, or any other store, the tab says what it would show. The dashboard needs both a metrics and a logs data source.

Materialize Consensus (Metadata) (env-consensus)#

The metadata database, as a Materialize environment experiences it, across five tabs: Overview, Operations, Connections, State and Cleanup, and Database Internals. Persist records the current state of every durable object in the metadata database and commits a new version of that record on every change, and the timestamp oracle keeps every query’s timestamps in the same database. Both clients are measured here, the same way on RDS, Cloud SQL, Azure Flexible Server, CNPG, and CockroachDB.

The Overview tab’s verdict row separates the common failures: connection errors, which on a new install are almost always the metadata backend URL, its credentials, or the network path; a rising commit tail, which is the database slowing down; and calls queued for a pooled connection. The Connections tab shows each process’s connection pool, which is what the database counts against its max_connections. State and Cleanup shows whether the table persist writes to is growing, which happens when a reader holds old versions that cleanup cannot delete. Its logs row carries the database’s own error text, which is what distinguishes connection refused from remaining connection slots are reserved. The Database Internals tab is the database’s own account, in a section per source. The PostgreSQL section reads the chart’s PostgreSQL exporter, on any PostgreSQL, managed or not: connections against the ceiling PostgreSQL refuses them at, the oldest open transaction, locks, transaction-ID headroom, the size of each database on the instance, dead rows, and autovacuum. The CloudNativePG (CNPG) section reads what a CNPG cluster publishes about itself, adding which instance is primary, replication, and backups. Each section appears when its source is scraped, and has a picker for the database it reads. With neither, the tab says how to set each up. It needs both a metrics and a logs datasource.

Materialize Ingress (Balancer, Console) (env-ingress)#

How users reach a Materialize environment, across five tabs: Overview, Load Balancer, Balancer, Console, and SQL and HTTP API. Every SQL client, HTTP API call, and console session arrives through a cloud load balancer and enters balancerd, and the console’s own API calls pass through the console’s nginx first. Both forward to environmentd, which serves the SQL and HTTP APIs. Each tab measures one of those hops with its rate, errors, and latency, so you can tell which hop a complaint like “users can’t connect” or “the console is slow” belongs to.

The Overview tab’s verdict row reads each hop’s error ratio and latency, and whether every balancerd pod is taking new connections. Below it, balancerd and the console sit side by side for each signal: when both fail, look behind them at environmentd, and when one fails, look at that one. The Load Balancer tab shows the address Kubernetes gave each LoadBalancer Service, and what the cloud provider reports about the load balancer in front of this environment: target health and connection resets on AWS, health probes on Azure, and traffic on all three clouds. Its provider rows appear when the gateway pulls that load balancer’s metrics, and otherwise the tab says how to turn the pull on; see Cloud Provider Metrics. The Balancer tab adds each pod’s connections against its limit and watermarks, the time spent reaching environmentd, and balancerd’s warnings. It opens with sparkline stats for open connections, ready and unavailable pods, CPU against the pods’ request, memory against their limit, and the time since the last restart, and drills into each pod’s CPU, memory, and throttling further down. Where a HorizontalPodAutoscaler scales balancerd, an Autoscaling row shows its replicas, the metrics it scales on against their targets, the conditions that stop it, and its events. Without one, the tab says how balancerd’s replica count is set and how to hand it to an autoscaler. The Console tab splits every request by what it asked for, the page itself or an API call, and carries nginx’s server-error lines. The SQL and HTTP API tab is environmentd’s side of the same traffic, including the SQL the console runs.

balancerd’s connection timings, its server-side error count, and every console panel need a Materialize version that publishes them. On older versions, those panels say so, and the SQL and HTTP API tab is unaffected. It needs both a metrics and a logs datasource.

Infrastructure (infra-*)#

Infrastructure Node Detail (infra-nodes)#

Everything about one node, one node at a time: Summary, CPU, Memory & Swap, Network, Storage, Pods, and Logs & Events. The Summary tab is deliberately kubectl describe node for someone who cannot run it — identity and capacity, utilization sparklines, how much of the node the scheduler has already promised, and the conditions, cordon state, and taints that explain a node accepting no work. It needs both a metrics and a logs datasource, the latter for the node journal.

Infrastructure Logs and Events (infra-logs)#

The same question as env-logs, asked of the platform a Materialize deployment runs on rather than of Materialize: the monitoring stack, the Kubernetes system components, and the node journal, across Logs, Nodes, and Events tabs. Also Loki-only. Where env-logs opens on the Materialize namespaces, this one subtracts them, so each dashboard answers for one side of the deployment.

Infrastructure Networking (infra-net)#

How traffic moves through the cluster, and what stops it, across six tabs: Overview, Kubernetes, CNI, Node Networking, Cloud Networking, and Security. Pod and Service traffic come from cAdvisor and kube-state-metrics and are the same everywhere. The nodes’ own interfaces, connection tables, and kernel receive path are read across the whole fleet rather than one machine at a time, which is the difference between this dashboard’s Node Networking tab and infra-nodes.

The CNI tab adapts to the cluster it is open on. A cluster’s container network interface differs per cloud, and the metrics describing each share no names with the others. The scrape configs label every series they collect with the dataplane they came from, the dashboard discovers that label into its Dataplane picker, and each vendor’s rows render only where that vendor was found. A cluster whose CNI exports nothing gets a single row explaining which case it is in, because that is not always a fault: GKE Dataplane V2 runs Cilium and disables the agent’s Prometheus endpoint, so nothing can be collected from it.

The Security tab is split the same way and for the same reason. Which NetworkPolicy objects exist comes from kube-state-metrics and is available everywhere; which packets a policy actually dropped can only come from the CNI. A cluster showing policies and no enforcement metrics has not demonstrated that any of them work.

Cloud Networking reads the network around the cluster from both sides. Kubernetes says which load balancers were created and the address each was given, on every cluster. The cloud provider says whether a load balancer has a healthy backend and what it carries, and whether the NAT gateway pods leave through is running out of source ports. That half comes from the gateway’s provider pulls, for the load balancers and NAT gateways named in values, and each pull’s rows render only where it runs. A NAT gateway out of source ports makes a Materialize source stall and reconnect with no cause inside the cluster, and this tab is where it shows.

Infrastructure Cloud Provider (infra-cloud)#

What the cloud provider reports about the metadata database and the buckets a Materialize deployment depends on, across three tabs: Metadata Database, Object Storage, and Collection. It reads the CloudWatch, Cloud Monitoring, and Azure Monitor metrics the gateway pulls when cloud provider metrics are enabled, and renders rows only for the provider it finds. Provider collection is off by default; with none configured, each tab explains what enabling it adds.

The Metadata Database tab leads with the headroom every provider publishes — CPU against the 60% steady-state target Materialize’s sizing guidance sets, open connections beside the number Materialize holds, and transaction-ID consumption — followed by the provider’s own storage, memory, and disk I/O rows. The Object Storage tab draws each bucket’s billed size beside the data Materialize accounts for; on GCS it also splits live data from noncurrent versions and soft-deleted objects, which a persist bucket can accumulate in quantity. Provider data is minutes old, and bucket size a day old, so this dashboard says why a dependency is struggling rather than whether it is; env-consensus and env-persist answer the second question and are the ones to open first.

Infrastructure Autoscaling (infra-autoscaling)#

Whether the cluster’s nodes are keeping up with its pods, on any cloud, across six tabs: Overview, Node Pools, Pending Pods, Workload Autoscaling, Cloud Capacity, and Events. Open it when a Materialize cluster replica will not start, since that is usually a pod waiting for a node.

It is built from what every cluster has, so it reads the same whether Karpenter, GKE’s or AKS’s cluster autoscaler adds the nodes:

• Pods waiting for a node, and why. The scheduler’s own explanation for each pod it could not place, followed by what the autoscaler did about it.
• Nodes by pool, instance type and zone. Also how full each pool is to the scheduler: requests against allocatable, which is what decides whether a pod fits, rather than usage.
• HorizontalPodAutoscalers. Their current, desired, minimum and maximum replicas, and any that cannot read the metric they scale on.
• What the cloud says about capacity. EC2 status checks and node group sizes, Compute Engine quota, and AKS’s own autoscaler gauges, when cloud provider metrics are enabled for them.

The per-pool panels need kube-state-metrics to publish node labels, which the chart configures; on an install from before that they say so.

Karpenter (infra-karpenter)#

How Karpenter adds, replaces and removes the cluster’s nodes on EKS, across five tabs: Overview, Provisioning, Disruption, Controller, and Events and Logs. It needs the Karpenter ServiceMonitor, which the self-managed Terraform’s karpenter module creates, and renders nothing but an explanation on a cluster without Karpenter.

• Provisioning: what Karpenter launched and what EC2 refused, and which instance types EC2 briefly stopped offering in a zone. Also how long each stage of a launch took, from EC2 accepting it to the node being ready.
• Disruption: consolidation, drift and expiry, and what blocks them. On a Materialize cluster that is mostly the do-not-disrupt annotation the operator puts on every Materialize pod, since moving a replica means rehydrating it. That reading is the design working; a PodDisruptionBudget blocking the same nodes for hours is not.

Meta monitoring (infra-*)#

These dashboards watch the monitoring stack rather than anything it collects. Their subject can take its own instrumentation down with it, so an empty panel on any of them can mean the component that would have reported a problem is the one that failed. Every panel states its own empty-state text for that reason. An alert count reads zero only while the rulers that evaluate its alerts are evaluating and writing back, and is empty otherwise.

Loki Meta Monitoring (infra-loki)#

Whether the log store is healthy, and which half of it broke if not, across five tabs: Overview, Writes, Reads, Storage, and Logs. The Overview tab leads with any alert about Loki firing or pending now, then with scrape health and with the canary. The canary writes a line and reads it back, and is the only end-to-end check on the store. It needs both a metrics and a logs datasource.

Alloy Meta Monitoring (infra-alloy)#

Whether telemetry collection is healthy, and which stage of it broke if not, across eight tabs: Overview, Log Pipeline, Metric Pipeline, Ingest, Components, Resources, Events, and Logs. The Overview tab leads with any alert about collection firing or pending now, such as a lost scrape target or a gateway refusing what it scraped. The two pipeline tabs follow the data: a log line from a node through the agent and the gateway into Loki, and a sample from a scrape target through the gateway into each metrics destination. Ingest covers what other senders push into the gateway over log push, Prometheus remote write, and OTLP, including remote-write senders’ own view of their queues. Components covers configuration loads, uptime, pipeline health, and the gateway’s clustering; Resources measures each collector against its CPU, memory, and GOMEMLIMIT limits; Events is what Kubernetes reported about the collectors’ pods and workloads, and about the configuration-validation Jobs that run before every install and upgrade. It needs both a metrics and a logs datasource.

Everything on this dashboard reaches Grafana through the gateway. The gateway scrapes Alloy’s own metrics and forwards Alloy’s own logs, so a dashboard that is empty throughout most likely means the gateway is down. kubectl get pods -l app.kubernetes.io/name=alloy-gateway in the monitoring namespace is the check that does not depend on it.

Thanos Meta Monitoring (infra-thanos)#

Whether the metrics store is healthy, and which part of it broke if not. Its tabs are Overview, Writes, Reads, Ruler, Storage, Resources, and Logs. The Overview tab leads with any Thanos alert firing or pending now, then with scrape health. The alerts are the upstream Thanos mixin’s, which the Thanos subchart installs and the Thanos ruler evaluates. Writes follows a sample through the receivers to object storage. Reads covers Thanos Query and the stores it fans out to, including whether every store is connected. Ruler covers rule evaluation, notifications to Alertmanager, and the results written back. Storage covers the compactor, and object storage as each Thanos process sees it. It needs both a metrics and a logs datasource.

Thanos answers every metrics panel on this dashboard, including the ones about Thanos Query. A dashboard that is empty throughout most likely means Thanos Query is down. The Logs tab reads Loki rather than Thanos, so it keeps working, and it is where a halted compactor says why it halted.

What the annotations mean#

Each render carries its compatibility in the dashboard’s own metadata.annotations, which is what the Annotations column above shows. The manifests the Grafana Operator applies carry the same annotations.