# materialize-monitoring Documentation > materialize-monitoring Documentation -------------------------------------------------------------------------------- title: "materialize-monitoring Documentation" url: https://materializeinc.github.io/materialize-monitoring/index.md -------------------------------------------------------------------------------- # materialize-monitoring Documentation `materialize-monitoring` is first-class observability for Materialize deployments: metrics, logs, events, dashboards, and alerts, packaged as a Helm chart and a Terraform module. It's a one-stop shop if you want one, and you can turn off any piece of it if you already run your own. If you install Materialize with the Terraform modules, the stack comes up with the cluster by default from `materialize-terraform-self-managed` v11 onward. To opt out, set `enable_observability = false`. Nothing here is required to run Materialize. For Materialize itself, see the [Materialize documentation](https://materialize.com/docs/). From v1.0.0, a breaking change to the committed surface goes through a deprecation cycle and lands only in a major release. For what you can rely on, see the [stability guarantees]({{< relref "reference/stability.md" >}}). For what's planned, see the [Roadmap]({{< relref "reference/development/roadmap.md" >}}), and for what has shipped, see the [Changelog]({{< relref "reference/changelog.md" >}}). ## Start here * [Vision]({{< relref "vision.md" >}}) — what the project is for, where it is going, and why. * [Getting Started]({{< relref "getting-started/overview.md" >}}) — the installation paths, and how to choose between them. * [Installing via Terraform]({{< relref "getting-started/terraform.md" >}}) — the recommended path: observability comes up with the cluster. * [Installing via Helm]({{< relref "getting-started/helm.md" >}}) — the full-fidelity surface, for when Terraform is not how you deploy. * [Dependencies]({{< relref "getting-started/dependencies.md" >}}) — what has to exist in the cluster before any of it installs. * [Production Best Practices]({{< relref "operating/production-best-practices/checklist.md" >}}) — the checklist before this runs anywhere that matters. ## How it works * [Architecture]({{< relref "architecture.md" >}}) — the umbrella chart, the components it bundles, and how telemetry moves between them. * [Observability glossary]({{< relref "o11y-glossary.md" >}}) — the vocabulary the rest of these pages assume. ## By signal | | | |---|---| | **Metrics** | [Collecting]({{< relref "metrics/collecting/overview.md" >}}) — the four ways metrics get in — plus [scraping]({{< relref "metrics/scraping.md" >}}), [storing]({{< relref "metrics/storing.md" >}}) in Thanos, and [querying]({{< relref "metrics/querying.md" >}}) them back out | | **Logs & Events** | The [Alloy agent/gateway split]({{< relref "logs-and-events/architecture.md" >}}), [collecting]({{< relref "logs-and-events/collecting.md" >}}), [storing]({{< relref "logs-and-events/storing.md" >}}) in Loki, [querying]({{< relref "logs-and-events/querying.md" >}}), and [rules]({{< relref "logs-and-events/rules.md" >}}) | | **Alerting** | The [alert architecture]({{< relref "alerting/architecture.md" >}}) from ruler to receiver, [configuring]({{< relref "alerting/configuring.md" >}}) the rule evaluators, [alert channels]({{< relref "alerting/channels.md" >}}), [maintenance windows]({{< relref "alerting/maintenance.md" >}}), and [configuring alerting through Terraform]({{< relref "alerting/terraform.md" >}}) | | **Dashboards** | [Available dashboards]({{< relref "dashboards/all.md" >}}) and [importing the Grafana set]({{< relref "dashboards/grafana/importing.md" >}}), the [Grafana Operator]({{< relref "dashboards/grafana/grafana-operator.md" >}}) path that keeps it in sync, [how Grafana is wired]({{< relref "dashboards/grafana/architecture.md" >}}), [authentication]({{< relref "dashboards/grafana/auth.md" >}}), and [Datadog]({{< relref "dashboards/datadog.md" >}}) | Metrics → Rules is a stub, so this page doesn't link to it. It appears in the sidebar because the section exists, and the [Roadmap]({{< relref "reference/development/roadmap.md" >}}) tracks the work behind it. ## Working with the data * [Observability tools]({{< relref "o11y-tools/overview.md" >}}) — Grafana in a browser, the Grafana MCP server for AI agents, `gcx`, and the backend APIs, compared by audience, access, ease of use, and stability. ## Operating the stack * [Production Best Practices]({{< relref "operating/production-best-practices/checklist.md" >}}) — sizing, retention, replication, disruption budgets, and durability, each tagged by who owns it. * [Securing]({{< relref "operating/securing.md" >}}) — network policy, in-cluster TLS, and exposing Grafana. * [Upgrading]({{< relref "operating/upgrading.md" >}}) and [Uninstalling]({{< relref "operating/uninstalling.md" >}}) — including the teardown ordering that avoids a finalizer deadlock. * [o11y Troubleshooting]({{< relref "operating/o11y-troubleshooting.md" >}}) — when the monitoring itself is the thing that is broken. ## Reference * [materialize-monitoring values]({{< relref "reference/helm/materialize-monitoring-values.md" >}}) — the generated Helm values reference. * [Terraform variables]({{< relref "reference/terraform/materialize-monitoring-variables.md" >}}) — the generated module variable reference. * [Compatibility]({{< relref "reference/compatibility.md" >}}) — supported versions of Materialize, Grafana, GKE, and the Terraform modules. * [Common Alerts]({{< relref "reference/common-alerts.md" >}}) — the bundled alerting rules, and the page each alert's `runbook_url` points at. * [Common Queries]({{< relref "reference/common-queries.md" >}}) — the PromQL behind the dashboard panels. * [List of Metrics]({{< relref "reference/list-metrics.md" >}}) — the metrics the dashboards depend on. * [Custom Resource Definitions]({{< relref "reference/crds.md" >}}) — the custom resources the stack reads and relies on. * [Changelog]({{< relref "reference/changelog.md" >}}) — per-component release history. ## For contributors * [Contributing]({{< relref "reference/development/contributing.md" >}}) — the contributor guide, conventions, and the pre-commit wiring. * [Roadmap]({{< relref "reference/development/roadmap.md" >}}) — the current source of truth for what is built, in flight, and planned next. * [Repository Layout]({{< relref "reference/development/repo-layout.md" >}}) — where things live in the repo. * [Versioning]({{< relref "reference/development/versioning.md" >}}) and [Releasing]({{< relref "reference/development/releasing.md" >}}) — the per-component version streams and the release automation. * [Design Docs]({{< relref "reference/development/design-docs/overview.md" >}}) — the decisions behind the larger pieces. ## Getting help Please [reach out for Support](https://materialize.com/docs/support/), or open an issue on [GitHub](https://github.com/MaterializeInc/materialize-monitoring/issues). -------------------------------------------------------------------------------- title: "Vision" url: https://materializeinc.github.io/materialize-monitoring/vision/index.md -------------------------------------------------------------------------------- # Vision `materialize-monitoring` exists so that a Materialize deployment somebody else operates is as observable as one Materialize operates itself. It packages metrics, logs, events, dashboards, and alerts as a Helm chart and a Terraform module. It installs with the cluster by default. For a customer who already runs their own observability stack, it reduces to nothing but a documented metric contract. This page states the direction and the reasoning behind it. The [Roadmap](../reference/development/roadmap/) is the source of truth for status, and it is more current than this page by construction. ## The problem Materialize used to run in one place, operated by one team, watched by dashboards that team maintained by hand. Self-managed changed the shape of the question. A deployment now runs in a customer's Kubernetes cluster, on infrastructure the customer owns. The operator watching it has often never seen a Materialize incident before. Three costs follow from that, and the project exists to pay them down. | Cost | What it looks like | |---|---| | **Divergence** | Materialize Cloud and self-managed grew separate dashboards, separate rule sets, separate collection pipelines, and separate metric names for the same measurement | | **Invisibility** | A self-managed install is opaque between escalations, so an escalation starts by asking the customer to gather evidence that was never collected | | **No default** | The recommended path was a point-in-time dashboard copy and a legacy scrape config, vendored into the installer and maintained by nobody | The common cause is that observability was a deliverable attached to each deployment rather than a product with a version. A customer could not upgrade it, Materialize could not test it, and neither party could say what it guaranteed. ## What the project is **One stack, consumed several ways, with every component replaceable.** That sentence carries three commitments, and most of the design follows from holding all three at once. ### One stack There is a single set of pipelines, dashboards, scrape configurations, and alert definitions, generated from a single query registry. Dashboards are Rust source rendered against the Grafana schemas, not JSON maintained by hand. Alert and dashboard expressions come from the same registry, so a metric an alert depends on cannot quietly diverge from the panel that explains it. ### Several ways in | Path | Audience | Shape | |---|---|---| | **Helm** | An operator who deploys with Helm, or who wants the full lever set | The umbrella chart, with profiles as sized deltas from a medium default | | **Terraform module** | A customer installing Materialize with the published modules | On by default from `materialize-terraform-self-managed` v11, opt out with `enable_observability = false` | | **Per-cloud wrappers** | AWS, GCP, and Azure installs | Downstream modules that wrap the common module and supply what only the cloud knows | | **The metric contract alone** | A customer with a mature stack of their own | Documented metrics, queries, and scrape configurations, with none of the workloads | A fourth consumer is proposed rather than built: Materialize Cloud, deploying the same chart through Pulumi. ### Every component replaceable Loki, Thanos, Grafana, Alertmanager, and the Alloy pipelines are the bundled implementation of an interface, not the interface. Each one can be switched off in favour of something a customer already runs. The posture has a real cost, and the project pays it deliberately. Composability means the project cannot assume any particular backend exists when it designs a feature. That is why the read path below counts as a change of posture rather than a routine addition. ## Who this is for The stack has four audiences, and they ask different questions of it. | Reader | What they are trying to do | |---|---| | **A self-managed operator** | Keep a Materialize deployment healthy without a Materialize engineer beside them | | **An SRE or platform engineer** | Fit Materialize into an observability practice that already exists, on that practice's terms | | **A field engineer** | Answer a question about a deployment they cannot log into | | **Materialize support and engineering** | Diagnose an escalation from evidence that was already being collected | The maintaining team is small, and the audience above is not. That asymmetry is the argument for treating this as a collaborative repository rather than one team's internal tooling. The people who meet a failure mode are usually not the people who write the dashboard for it. A panel, an alert, a query, or a runbook contributed by whoever hit the problem carries context that a metric list cannot supply afterwards. Contributions from field engineers, SREs, and customers are encouraged on the same footing as contributions from the maintaining team. The [contributor guide](../reference/development/contributing/) covers the conventions. The single-registry design is what makes a small contribution worth making. A query added to the registry reaches the dashboard, the published documentation, and any alert referencing it. One contribution is therefore not one panel. ## Why a deployment turns it on Nothing here is required to run Materialize, and every component of it can be switched off. The case for turning it on therefore has to be made per deployment, and what makes it is the set of moments a deployment produces on its own. | The moment | The question it poses | Where the answer comes from otherwise | |---|---|---| | An upgrade has been running longer than expected | Is it progressing, or is it stuck | A version count, and a decision made on nerve | | A replica is resized, or a workload is resharded | Has the new replica caught up, and at what cost | A SQL session against the environment, repeated until it looks settled | | A source falls behind | Is it the source, the network, or the cluster reading it | Introspection queries against an environment that is already struggling | | The deployment stops accepting connections | Is Materialize unhealthy, or is it the network or the balancer in front of it | Logs gathered by hand once the incident is over | | An escalation opens | What was happening in the hour before it broke | A request to the customer to collect evidence nobody was collecting | | The platform underneath misbehaves | Is it the node, the CNI, the object store, or the metadata database | `kubectl`, and cluster access the person asking may not have | | The cloud bill arrives | What does Materialize cost, and which part of it is largest | An estimate | Every one of these arrives whether or not anyone prepared for it. What the stack changes is whether the evidence was being collected before the question was asked, because most of it cannot be gathered afterwards. How much of each row is answered today is the roadmap's subject, and several are answered only in part. ## Why Materialize invests in it The first question is asked by whoever installs the stack. This one is asked internally, and it has a different answer. | Reason | The evidence behind it | |---|---| | **Adoption decisions turn on operability** | An upgrade no operator could read blocked a production adoption decision, which is the case behind [the upgrade work](#from-displaying-metrics-to-answering-questions) | | **Support cost scales with blindness** | An install nobody can see is an install whose every escalation starts by asking the customer to go gather evidence | | **The alternative is the same work once per deployment** | The log alerts that detect panics and correctness violations exist as per-region copies, drifted from each other, maintained by clicking | | **Two implementations get fixed once between them** | Cloud and self-managed maintain two versions of nearly the same query set, and twenty-six of those queries are identical SQL under identical names | | **It has stopped being an accessory** | The console's [read path](#the-read-path-where-this-stops-being-optional) makes a PromQL and a LogQL endpoint load-bearing rather than optional | One argument sits outside the table. An operator evaluating self-managed asks how they will run it, and that answer is part of the product rather than an appendix to it. The stack is what makes the answer demonstrable rather than a claim. ## Where it stands today The platform is built and in use. The following is the surface 1.0 ships, at altitude. The [Changelog](../reference/changelog/) itemizes it, and the [Roadmap](../reference/development/roadmap/) lists what is still to do. | Area | State | |---|---| | **Collection** | Alloy agent and gateway, node-exporter, kube-state-metrics, kubelet cAdvisor, self-disabling CNI monitors for the two CNIs that publish anything, and provider metrics pulled from CloudWatch, Cloud Monitoring and Azure Monitor | | **Storage** | Loki and Thanos bundled and sized by profile, with fan-out to OTLP, Datadog, Google Cloud Monitoring, and additional Prometheus remote-write destinations | | **Dashboards** | Thirteen shipped. Five are for a Materialize user: an environment overview, logs, upgrades, and the object store and metadata database Materialize depends on. Eight are for whoever runs the cluster underneath: logs, nodes, networking, autoscaling, Karpenter, the cloud provider, and Loki and Alloy watching themselves | | **Alerting** | Thanos and Loki rulers notifying a highly available Alertmanager, thirty-four alerts in the default set, severity-to-receiver presets, and a Terraform surface | | **Packaging** | Chart, CRDs chart and dashboards chart published to GHCR, a Terraform module on the same version stream, per-component SemVer, a generated changelog, and a written deprecation policy | | **Security** | NetworkPolicy on every workload by default, opt-in in-cluster mTLS with a four-stage rollout, and chart and image scanning in CI | | **Qualification** | Terraform render checks, two `kind` end-to-end tiers, and a Rust assertion suite verified against real EKS and GKE clusters | Two gaps matter more than the rest, and both shape the next phase. **Alerting is in place and thin.** A default rule set is evaluated and routed on every install. Most of the ported rule set still awaits triage, and no shipped alert has a runbook behind it yet. **The four questions a Materialize user asks first are still unanswered.** Hydration, freshness, sources, and sinks each need instrumentation that lives upstream in the Materialize repository rather than here. ## Where it is going Each theme below is a question this stack cannot answer yet. The [Roadmap](../reference/development/roadmap/) tracks the individual items behind each one, with their status and their designs. ### From displaying metrics to answering questions The shipped overview is organized by subsystem, which suits a reader who already knows which subsystem is at fault. The next set is organized by symptom and by operation. A troubleshooting dashboard becomes the entry point, and every panel on it links onward into logs or into the matching drilldown. Day 2 operations are weighted above Day 1, because a running deployment fails during a change far more often than during an install. Upgrades is the worked example and the sharpest case. An operator watching a long upgrade with no way to tell whether it is progressing has a production adoption decision blocked on that uncertainty. The requirement that came out of it is the requirement for the whole family. A dashboard has to answer *is it stuck* and *what do I do about it*, rather than display a version count. The shipped upgrade dashboard answers the first question from the rollout's own account of itself, and the second is outstanding. ### Alerting: from shipping rules to paging a human Every component needed to alert is in the chart, and no two of them are connected. Closing that is the highest-value work on this page. Three separate problems share the workstream. Nothing evaluates rules, so the rule set is documentation. The rule set is Materialize Cloud's rule set, and a third of it names components a stock self-managed install does not run. Log-derived alerting has never been code anywhere at Materialize. The rules that detect panics and correctness violations exist only as per-region copies clicked into a Grafana instance. The proposed design takes four positions. | Position | Why | |---|---| | Two evaluators, one notifier | PromQL and LogQL need different evaluators, and an operator needs one place where silences and grouping live | | Severity belongs to the alert, urgency belongs to the deployment | One rule set cannot otherwise serve both a customer for whom Materialize is critical infrastructure and one who is evaluating it | | A rule declares what it requires, not who operates it | A CockroachDB rule is for a deployment running CockroachDB, which a self-managed customer may well be | | Customer-specific alerting composes out of values | Alert rules filed under individual customer names are what happens when it does not | Alert names become a surface the project owes a deprecation cycle on from the first release that ships rules. That decision is free only until the alerting path exists, so it is settled as part of the alerting design rather than after it. ### The services Materialize does not run Every dashboard in the stack stops at the cluster boundary. The two services a Materialize deployment cannot run without and does not run itself are the metadata database and the object store. Neither has a dashboard, a working alert, or a collection path beyond what its clients happen to publish. The central claim of the proposed design is that **the client's measurement of a dependency is the service level indicator**. The dependency's own telemetry is the diagnosis. Materialize, Loki, and Thanos already time and count every call they make to both dependencies, on every cloud, with no credentials. That vantage point can ship on by default. Everything flavour-specific sits behind a normalized contract. That is what keeps seven plausible database flavours and five object stores from multiplying the dashboard and alert set. The field evidence changed the design more than the analysis did. Two observed failures were the metadata database exhausting disk and CPU. One was a neighbouring project consuming a shared database instance, which no amount of client-side measurement can attribute. The most common class was a day-0 setup failure, which is a different problem with a different answer. The alerts that describe the first pair are well calibrated, carry correct remediation text, and could not have fired for two independent reasons. ### What a deployment costs Every other workstream measures what a deployment is doing. This one measures what it costs, which is the question a self-managed operator is asked by whoever owns the cloud bill. The proposed shape is an optional cost component, off by default. Deploying a cost exporter is the easy half. Making the number mean something is the feature, and two problems stand in the way. The default price for a node is a public list price rather than what the customer was billed, so every figure has to state where it came from. A cost exporter attributes spend to namespaces and pods, while a Materialize operator asks about clusters, replicas, and cost centers. The demand behind it is specific rather than a general appeal to efficiency. Five questions drive it: - Right-sizing a replica. - Comparing machine classes across node generations. - Deciding whether to shard a workload. - Apportioning Materialize across the departments using it. - Holding an honest total-cost conversation. The last one shapes the rest. A total that quietly excludes the object store, the metadata database, or the monitoring stack is not a total. ### The read path, where this stops being optional Every workstream above is about collecting telemetry, and every consumer of it so far is a Grafana the project deploys. The [tenant-scoped query API](../reference/development/design-docs/20260916-tenant-query-api/) is about reading it from somewhere else. The two readers are the Materialize console and a customer's own Grafana. This is the one place the composability posture is genuinely in tension with the product. The console renders environment metrics from SQL against the environment itself. That path keeps a short window of history. It is also unavailable precisely when the environment is. Reading PromQL and LogQL instead makes a deployment without those endpoints a deployment with a broken console. The design resolves the tension by mandating an interface rather than an implementation. A PromQL endpoint and a LogQL endpoint carrying the documented label contract, reachable through a tenant-scoped proxy, is what a deployment owes. Thanos and Loki are the bundled implementation of it, and a customer already running an equivalent points the proxy at theirs. One consequence reaches the rest of the project. The metric and label contract becomes load-bearing for a product surface rather than for dashboards alone. **Three planned features assume a scheduled reader against the metrics backend, and none of those readers exists.** Rule evaluation has to read in order to evaluate, this proxy serves reads to the console, and cost attribution issues queries on a schedule. All three were sized against a backend whose only reader is a Grafana with a human in front of it. That sizing question is common to all three, and it should be settled once rather than inside whichever lands first. ### Deployments Materialize cannot see Two workstreams share a channel and differ in what authorizes using it. **[Bring-your-own-cloud](../reference/development/design-docs/20260813-byoc-observability/)** covers environments Materialize operates. They run in the customer's own cloud account, and Materialize needs enough telemetry to operate them. A reduced copy of telemetry crosses into the control plane, and the customer's full-fidelity copy always stays with them. Metrics cross selected by importance tier, and logs cross as an allowlisted, redacted, level-filtered subset. A pair of gateways enforces that boundary rather than ad-hoc network configuration. Redaction attaches to the destination, so the reduced copy is a fork of the customer's stream rather than a downgrade of it. **[Call-home](../reference/development/design-docs/20260917-call-home-self-managed/)** covers installs Materialize does not operate. A self-managed deployment is invisible between escalations. The channel is the same, and the feature is consent. A bring-your-own-cloud customer bought an operated service, so telemetry crossing the boundary is what they purchased. A self-managed customer bought software they run themselves, and every byte that leaves is a concession. The design is therefore a bounded, monotone, locally-visible ladder: `off`, `heartbeat`, `alerts`, `metrics`, `diagnostics`. It defaults to off. A preview mode and a local egress meter let the customer verify the bound rather than trust it. Drafting it produced one finding that belongs on this page. **Receiving a signal creates an obligation.** Collecting alerts nobody is paged on is worse than collecting nothing. The customer took a disclosure risk on the assumption that someone is watching. A stated response model is a prerequisite for the alerts level rather than a follow-up to it. ### One stack instead of two The divergence in the table at the top of this page is a cost the project is now in a position to remove. Two changes do it from opposite ends, and both are Cloud-side work rather than changes to this chart. **Cloud becomes the chart's third consumer.** Cloud takes the Alloy pipelines and both backends, with Grafana staying external because it is operated outside this project. Logs lead rather than metrics. The log path already works end to end, which makes it the cheapest way to prove the chart against real traffic. The gateway-to-gateway shape it adopts is the same shape bring-your-own-cloud builds redaction on. **The metric-naming divergence is closed at its source.** Cloud and self-managed today maintain two implementations of nearly the same query set. Twenty-six of those queries are identical SQL under identical names. Every dashboard in this repository threads a prefix parameter for no reason other than to paper over which process served the metric. Both paths also run expensive SQL on the scrape path against the replicas they are trying to observe. The goal is one path per metric, named consistently, with no SQL on the scrape path, reached over an authenticated connection. ### 1.0, and what the number promises `materialize-monitoring` stamped 1.0 on 2026-10-06, and from that release a breaking change rides only a major. The label and metric contract, the profile semantics, and the alert names carry a deprecation cycle; chart value paths don't. The number was stamped before broad adoption on purpose. Once enough customers have dashboards built on these labels, the contract is frozen in practice whether or not it is frozen on paper. Surfaces are graded by how much control the project has over them and by how a break presents. Alerts fail silently and get the most care. Metrics fail visibly and get the least. See [Stability Guarantees and Deprecation Policy](../reference/stability/) for what a consumer can rely on today. ## What has to come from elsewhere The metric and label contract is the public interface for everything on this page. Part of it is instrumented upstream in the Materialize repository rather than here. That dependency shapes the dashboard roadmap directly. | Upstream ask | What it unblocks | |---|---| | Object and `_info` metrics carrying names and parent references | Delivered. Every other metric now has a stable join target for names | | Native source and sink status metrics | The sources and sinks drilldowns | | Native hydration and frontier signals | The hydration and freshness drilldowns | | Label-family harmonization | Queries that do not have to know which of three label forms a deployment emits | | `balancerd` and `console` metrics | The two components a user connects *through*, which today are observable only as logs | | Operator reconciliation metrics and lifecycle events | Most of the upgrade dashboard, which ships with those panels empty until it lands | The pattern is worth naming. This project can build a collection path, a dashboard, and an alert for any signal that exists. It cannot instrument what does not emit, and the gap between those two facts is where most of the blocked work on the roadmap sits. ## Where the detail lives | Question | Page | |---|---| | What is built, in flight, and planned next | [Roadmap](../reference/development/roadmap/) | | How the pieces fit together | [Architecture](../architecture/) | | What the vocabulary on these pages means | [Observability glossary](../o11y-glossary/) | | What a consumer can rely on | [Stability Guarantees and Deprecation Policy](../reference/stability/) | | How to install it | [Getting Started](../getting-started/overview/) | | Why a larger piece is shaped the way it is | [Design Docs](../reference/development/design-docs/overview/) | -------------------------------------------------------------------------------- title: "Architecture" url: https://materializeinc.github.io/materialize-monitoring/architecture/index.md -------------------------------------------------------------------------------- # Architecture Overview This section provides an overview of the architecture of `materialize-monitoring` and its components. ## `materialize-monitoring` Helm Umbrella Chart An Umbrella Helm Chart is a Helm Chart that orchestrates the installation of multiple dependent charts. The `materialize-monitoring` Helm Chart is an Umbrella Chart that orchestrates the installation of the following dependent charts: - `alloy-agent` (Grafana Alloy, Agent DaemonSet): o11y Pipelines - `alloy-gateway` (Grafana Alloy, Gateway Deployment): o11y Pipelines - `metrics-server` (metrics-server): cAdvisor/container runtime Metrics - `kube-state-metrics` (kube-state-metrics): Kubernetes Metrics - `node-exporter` (node-exporter): Node Metrics - `loki` (Grafana Loki): Default Logging Infrastructure - `thanos` (Thanos): Default Metrics Storage and Querying Infrastructure - `grafana` (Grafana): Default Dashboarding and Visualization Infrastructure - `grafana-operator` (Grafana Operator): Dashboards-as-Code Infrastructure - `alertmanager` (Prometheus Alertmanager): Default Alerting Infrastructure In addition to these dependent charts, `materialize-monitoring` also provides many opionated configurations such as o11y pipelines, Grafana dashboards, Scrape configurations, and Prometheus recording and alerting rules. ## `alloy-agent`: Grafana Alloy Agent DaemonSet `alloy-agent` is a [Grafana Alloy](https://grafana.com/docs/alloy/latest/introduction/) Agent DaemonSet that runs on every node in the cluster and is responsible for collecting logs from the node and forwarding them to the [`alloy-gateway`](#alloy-gateway-grafana-alloy-gateway-deployment). The agent does **not** collect its own logs. Those arrive by a second, independent route: the gateway reads them straight from the Kubernetes API — see [the agent log sidechannel](#the-agent-log-sidechannel) below. ### The agent log sidechannel Every other log in this stack reaches Loki because an agent tailed a file on its node and pushed it to the gateway. That makes the agent a single point of failure for exactly one set of logs: **its own**. When an agent OOMs, crash-loops on bad config, wedges on a full disk, or simply cannot reach the gateway, the logs explaining why are the ones it was about to send. The failure erases its own evidence, and the operator sees silence where they need a cause. So the gateway tails the agent DaemonSet's logs directly from the Kubernetes API, using `loki.source.kubernetes` against a `discovery.kubernetes` selector scoped to those pods. The two paths share nothing: a different process, on a different host, over a different transport, with a different failure domain. It also covers the windows where the agent legitimately is not running — a first install, a DaemonSet rollout, a node that has just come up — during which there is no file to tail yet. Three properties are deliberate: - **Always on, not a failover.** A path that only activates during an incident is a path nobody notices is broken. Running it constantly means its health is visible on an ordinary day, and there is no failure detection to get wrong. - **The agents do not also collect their own logs.** With the sidechannel always on, an agent tailing its own log file would deliver every line twice. The agent's discovery drops its own pods, so there is one path rather than two. - **Scoped to the agent DaemonSet only, server-side.** The label selector goes to the API server, so the watch never returns other pods. The target set is one pod per node emitting alloy's own modest output — which is what makes reading logs through the API server affordable here. This is **not** a general log-collection mechanism: pointed at application pods it would put the cluster's entire log volume through the API server. The gateway runs several replicas, so the tail is clustered — otherwise each replica would tail every agent pod and every line would land in Loki once per replica. ## `alloy-gateway`: Grafana Alloy Gateway Deployment `alloy-gateway` is a [Grafana Alloy](https://grafana.com/docs/alloy/latest/introduction/) Gateway Deployment that is responsible for the main observability pipeline processing and forwarding. Logging responsibilities of `alloy-gateway` include: * Receiving logs on multiple endpoints, so that the [`alloy-agent`](#alloy-agent-grafana-alloy-agent-daemonset), an application, or even another `alloy-gateway` (a chained-gateway topology) can send to it: * A [`loki.source.api`](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.source.api/) component receives Loki push traffic on `alloy-gateway.$namespace:3100`. * An [`otelcol.receiver.otlp`](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.receiver.otlp/) component receives OTLP logs on `alloy-gateway.$namespace:4317` (gRPC) and `:4318` (HTTP). * A [`loki.source.kubernetes_events`](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.source.kubernetes_events/) component collects Kubernetes events and processes them as logs. * A [`loki.process`](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/) pipeline performs log processing — level normalization, [cardinality](../o11y-glossary/#observability-foundations) reduction, and structured-metadata extraction. * A [`loki.write`](https://grafana.com/docs/alloy/latest/reference/components/loki/loki.write/) component forwards logs to log storage (e.g., [Grafana Loki](#loki-grafana-loki)) — or, in a [remote-only topology](../logs-and-events/architecture/#alternative-topologies), to an external OTLP or Loki destination. Metrics responsibilities of `alloy-gateway` include: * [`prometheus.operator.servicemonitors`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.operator.servicemonitors/) and [`prometheus.operator.podmonitors`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.operator.podmonitors/) components read ServiceMonitors and PodMonitors in order to determine what targets to scrape for metrics and then scrapes those targets. * A [`prometheus.enrich`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.enrich/) pipeline performs metric processing and enrichment on scraped metrics. * A [`prometheus.remote_write`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.remote_write/) component forwards metrics to metric storage (e.g., [Thanos](#thanos-thanos)). * An [`otelcol.exporter.otlp`](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.exporter.otlp/) component supports forwarding to an external OTLP endpoint (e.g., Honeycomb, Datadog, New Relic, etc.) for metrics and logs. Alloy supports further customization to integrate with an existing observability infrastructure. ## `metrics-server`: Container Metrics API [`metrics-server`](https://github.com/kubernetes-sigs/metrics-server) is a Kubernetes Metrics API implementation that collects resource usage metrics from the kubelet on each node and exposes them via the Kubernetes Metrics API. Do note that the `metrics-server` is primarily intended for decision-based components (like Horizontal Pod Autoscaler) and does not store historical metrics data. Nonetheless, Materialize relies on cluster-local metrics about its containers so this is required to not rely on external metrics sources for this data. ## `kube-state-metrics`: Kubernetes Metrics [`kube-state-metrics`](https://kubernetes.io/docs/concepts/cluster-administration/kube-state-metrics/) is a service that listens to the Kubernetes API server and generates metrics about the state of the objects in the cluster (e.g., Deployments, Pods, Services, etc.). This does not provide information about resource usage of individual containers. ## `node-exporter`: Node Metrics [`node-exporter`](https://github.com/prometheus/node_exporter) is a Prometheus exporter that collects hardware and OS metrics from the nodes in the cluster. It runs as a DaemonSet, is scraped by [`alloy-gateway`](#alloy-gateway-grafana-alloy-gateway-deployment) through a ServiceMonitor, and covers the layer nothing else in the stack sees: swap, memory pressure, node-level network counters, disk I/O, and conntrack. Two things about it are worth knowing before you change anything: - **It is a separate DaemonSet on purpose.** Alloy's `prometheus.exporter.unix` *is* node_exporter, and folding it into the agent would be one workload fewer. Keeping them apart keeps their resource limits apart, so a metrics regression cannot starve log collection — the signal you most need during the incident it caused. See [the Terraform modules design doc](../reference/development/design-docs/20260803-terraform-modules/) for the full argument. - **Collectors are an allowlist, not the upstream defaults.** The chart passes `--collector.disable-defaults` and names each collector it wants, so a new default-on collector in a future node_exporter release cannot silently join the cardinality budget. The list, and the reasoning behind every inclusion and exclusion, is in the [values reference](../reference/helm/materialize-monitoring-values/) under Node Exporter. The pods run with `hostNetwork: true`, which the network collectors require — `netdev`, `netclass`, `netstat`, `sockstat` and `conntrack` all read namespaced files under `/proc/net`, and in a pod network namespace they would report the pod's traffic rather than the node's. A consequence worth stating plainly: the exporter listens on the node's own interfaces, so a NetworkPolicy is not what keeps port 9100 private. Most CNIs, Cilium and Calico among them, do not apply pod NetworkPolicy to host-networked pods. The chart ships one anyway — it is enforced where the CNI supports it, and it documents intent everywhere else — but the node firewall or security group is the real control. ## `loki`: Grafana Loki [Grafana Loki](https://grafana.com/docs/loki/latest/) is a horizontally scalable log-aggregation system, included in `materialize-monitoring` as its default logging backend. It runs in **microservice / distributed** mode, where each component scales independently. For the full component breakdown — diagrams, the read/write paths, the hash ring, day-2 operations, and alternative topologies — see [Logs & Events > Logging Architecture](../logs-and-events/architecture/). At a glance: * **Write path** — `alloy-gateway` pushes processed logs to the **Loki Distributor**, which fans each stream across **Loki Ingesters** through a consistent hash ring (replication factor 3); ingesters flush chunks and the TSDB index to object storage. * **Read path** — Grafana issues LogQL through the **Loki Gateway** (nginx) to the **Loki Query Frontend**, which queues and splits work for **Loki Queriers** (assisted by the **Loki Query Scheduler** and **Loki Index Gateway**); queriers read recent data from ingesters and historical data from object storage. The gateway also routes Grafana's rule list to the **Loki Ruler**. * **Backend / maintenance** — the **Loki Compactor** (singleton) compacts the index and enforces retention; the **Loki Ruler** evaluates LogQL alerting and recording rules, sending alerts to [Alertmanager](#alertmanager-prometheus-alertmanager) and recording-rule samples back through `alloy-gateway` to the long-term metric store ([Thanos](#thanos-thanos)). Loki writes all durable data to object storage (S3-compatible, GCS, or Azure Blob). ## `thanos`: Thanos [Thanos](https://thanos.io/tip/thanos/getting-started.md/) is a highly available Prometheus setup with long-term storage capabilities. Thanos is included in `materialize-monitoring` as its default metrics storage and querying backend. For the full flow, with diagrams of collection, the write path, the block lifecycle, the read path and rule evaluation, see [Metrics > Metrics Architecture](../metrics/architecture/). Refer to [Thanos Design](https://thanos.io/tip/thanos/design.md/) for more details on the architecture of Thanos. The Thanos Receiver path includes: * A `thanos-receive` statefulset that receive metrics in Prometheus Remote Write format. * The `Shipper` subcomponent writes metrics to the object storage layer. * The `Store API` subcomponent provides an API for querying recent metrics. The Thanos Query path includes: * An optional `thanos-query-frontend` deployment is an optional caching and fan-out layer for queries. * A `thanos-query` scalable deployment that receives queries. * The `Query API` subcomponent handles PromQL queries. * The `Store API` component is used for gRPC internal communication between components. * A `thanos-storegateway` deployment that serves metrics from the object storage layer. Additional components include: * A `thanos-compactor` singleton deployment that operates against the storage layer to compact, manage retention, and downsample metrics. * A `thanos-ruler` deployment that runs the `Ruler` component for alerting and recording rules. * The `Ruler` subcomponent evaluates alerting and recording rules by querying `thanos-query`, and remote-writes the results to `alloy-gateway`. ## `grafana`: Grafana [Grafana](https://grafana.com/docs/grafana/latest/) is a multi-platform open source analytics and interactive visualization web application. Grafana is included in `materialize-monitoring` as its main dashboarding and visualization tool. Grafana is mainly deployed as a Deployment and is recommended to be backed with a compatible database for durability and scalability. We use `grafana-operator` to manage resources on a Grafana deployment. ## `grafana-operator`: Grafana Operator [Grafana Operator](https://grafana.github.io/grafana-operator/docs/) is a Kubernetes Operator that manages Grafana instances and their resources (e.g., Dashboards, Datasources, etc.) as Kubernetes Custom Resources. The operator itself is just a simple Kubernetes Deployment named `grafana-operator` that watches for Grafana Custom Resources and applies them to the Grafana instance. It manages these kinds of resources: - A `Grafana` defines how to set up a Grafana instance or connect to an existing Grafana instance. - A `GrafanaManifest` defines a k8s-style (12+) Grafana Dashboard that can be applied to the Grafana instance. - A `GrafanaDashboard` defines an old-style (<12) Grafana Dashboard that can be applied to the Grafana instance. - A `GrafanaDatasource` defines a Grafana Datasource that can be applied to the Grafana instance. We typically configure a datasource for Thanos and Loki. ## `alertmanager`: Prometheus Alertmanager [Prometheus Alertmanager](https://prometheus.io/docs/alerting/latest/alertmanager/) routes, groups, deduplicates and silences the alerts both rulers send, and delivers them to receivers. It is the single notification surface: the Thanos Ruler (PromQL) and the Loki Ruler (LogQL) both notify it. * A `StatefulSet` of two replicas that gossip silences and the notification log over `9094`, spread across zones, each with its own small volume. * Both rulers resolve the headless Service and send every alert to every replica, because gossip does not replicate alerts. * The chart renders the configuration from the `alerting` values key into the `alertmanager-config` Secret, and a config-reloader sidecar applies changes in place. * Grafana reads alerts and silences through an Alertmanager datasource, and the Alloy gateway scrapes Alertmanager's own metrics. See [Alert Architecture](../alerting/architecture/) for the full design. -------------------------------------------------------------------------------- title: "Observability glossary" url: https://materializeinc.github.io/materialize-monitoring/o11y-glossary/index.md -------------------------------------------------------------------------------- # Glossary of observability terms A reference for the vocabulary you'll see across these docs. The audience is a Materialize customer or operator: comfortable with SQL, conversant in DBA terminology, working knowledge of Kubernetes. We don't define general database or Kubernetes terms (table, namespace, pod) — only what's specific to monitoring, the Materialize internals you'll meet in panels, or this stack's deployment vocabulary. ## Observability foundations ##### Cardinality {#cardinality} The number of unique time series (or log streams) a metric produces. The dominant cost-and-stability lever for any Prometheus-shaped stack: a single ill-chosen label that takes on millions of values can balloon storage and crash query backends. The materialize-monitoring pipeline applies a cardinality-reduction policy at the gateway layer before metrics egress to expensive backends — see [Operating > Tuning](../operating/tuning/). ##### Error {#error} A signal that something failed — an exception, a 5xx response, a panic. Errors surface in three forms across this stack: log lines at `ERROR` or `WARN` level (see [Log level](#log-level)), increments on error counters (`*_errors_total`), and elevated rates that trip alerting rules. Per the [RED method](https://grafana.com/blog/2018/08/02/the-red-method-how-to-instrument-your-services/), error rate is one of the three primary indicators (alongside Rate and Duration) worth tracking for any request-serving system. ##### I/O {#io} Input/output — work performed against an external resource (network, disk, another process). I/O metrics in this stack come from [cAdvisor](#cadvisor) (`container_network_*_bytes_total`, `container_fs_*`), [node-exporter](#node-exporter) (block-device counters), and Materialize's own metrics (source bytes received, sink throughput). Disk and network I/O are common bottlenecks for streaming systems. ##### Label {#label} A key-value pair attached to a metric or log line describing one dimension of what was measured. For example, `mz_query_total{cluster_id="u1", query_kind="select"}` has two labels. Labels are how you filter and group in [PromQL](#promql)/[LogQL](#logql) — and the multiplicative source of [cardinality](#cardinality). ##### Label budget {#label-budget} A target ceiling on the number of distinct values a given label may take — or, more broadly, on the total [cardinality](#cardinality) a metric is allowed to produce. Treated as a contract between metric producers and the platform: producers commit to staying under budget, the platform commits to scraping and storing what fits. When a budget is exceeded, this stack's pipeline can drop the offending labels into structured metadata or refuse them entirely. See [Operating > Tuning](../operating/tuning/) for the policy applied here. ##### Latency (generically) {#latency} The time elapsed between a request being initiated and a response being received. Almost always reported as a [histogram](#histogram-metric) and queried at p50/p90/p99 — averages hide the tail behavior that matters operationally. Materialize-specific note: read-query latency is exposed as [peek](#peek) latency; there is no metric named "query latency." ##### Observability / o11y {#observability} The practice of inferring a running system's internal state from the signals it externalizes — metrics, logs, and events. "o11y" is the conventional shorthand: the letter `o`, eleven characters, the letter `y`. ##### Quantile / Percentile {#quantile} A statistical summary describing the value below which a given fraction of observations fall. Latency panels typically show p50 (median), p90, and p99; "p99 = 250ms" means 99% of requests completed in 250ms or less. Computed in [PromQL](#promql) via `histogram_quantile()` against a [histogram metric](#histogram-metric). Don't conflate with averages — averages hide tail behavior, which is usually the thing operators care about. ##### Series / time series {#series} A unique combination of metric name + label values, sampled over time. `mz_query_total{cluster_id="u1"}` and `mz_query_total{cluster_id="u2"}` are two distinct series of the same metric. ##### Swap (memory) {#swap} A region of disk the kernel uses as overflow when physical RAM is exhausted. Swapping is generally undesirable for performance-critical workloads — disk is orders of magnitude slower than RAM, so an apparently-running process that's actually swapping can appear catastrophically slow. Most production Kubernetes deployments disable swap entirely; if a node has it enabled, watch `node_memory_SwapUsed_bytes` for early warning of memory pressure. ##### Throughput {#throughput} The rate at which a system processes work, usually expressed as units per second — bytes/sec, queries/sec, rows/sec. Reported as a [counter](#counter-metric) in metrics and queried via `rate(...)`. Sources have ingest throughput; sinks have output throughput; queries have peek-rate throughput. ## Metrics ##### Counter Metric {#counter-metric} A metric that only increases (or resets to zero on process restart). Examples: total bytes received, total queries served. Always wrap a counter in `rate(...)` or `increase(...)` in [PromQL](#promql) — the raw value of a counter is rarely directly useful. ##### Gauge Metric {#gauge-metric} A metric that can go up or down at any time. Examples: current memory usage, currently active connections. Read the value directly; no rate wrapper needed. ##### Histogram Metric {#histogram-metric} A metric that records the distribution of observed values across configured bucket boundaries. Latency, request-size, and other "spread matters" metrics are almost always histograms. Query a histogram via `histogram_quantile(quantile, sum by (le) (rate(_bucket[...])))`. ##### Metric {#metric} A numeric measurement, identified by a name and a set of labels, that is scraped or pushed at fixed intervals into [Prometheus](#prometheus) (or another time-series backend). The shape of a metric is captured by its type — counter, gauge, or histogram — and its label set determines its [cardinality](#cardinality). Distinct from a [log line](#log) (text, unstructured-by-default, indexed by stream) and a *trace span* (causal, request-scoped). ##### Recording rule {#recording-rule} A precomputed [PromQL](#promql) expression evaluated at scrape interval and stored as a new series. Two main uses: speed up expensive queries that appear in many dashboards/alerts, and reduce [cardinality](#cardinality) before metrics egress to expensive backends. See [Metrics > Rules](../metrics/rules/). ##### Sample Metric {#sample-metric} One (timestamp, value) data point in a [series](#series). A series is, mechanically, an ordered list of samples. ## Logs and events ##### Event {#event} A discrete occurrence with associated metadata — a deployment, a pod restart, an alert firing. Kubernetes emits events on its API; some of these (notably Pod state changes) flow through this stack alongside container logs. ##### Log {#log} A line (or structured record) emitted by a running process. This stack collects container logs from Materialize's processes (`environmentd`, `clusterd`) as well as the operator and any sidecars. Structured logging (key=value or JSON) queries far better in [LogQL](#logql) than free-form text. ##### Log level {#log-level} The severity class attached to a log line: typically `TRACE`, `DEBUG`, `INFO`, `WARN`, `ERROR` (some systems add `FATAL` above `ERROR`). Materialize emits at `INFO` by default; production filtering is usually `WARN`/`ERROR` for storage cost, dropping to `INFO` or `DEBUG` for active debugging. Configurable per-process via the standard Rust `RUST_LOG`/log-filter conventions. ##### Log Stream {#log-stream} The [Loki](#loki) analogue of a Prometheus [series](#series): a unique combination of labels under which log entries are ordered by time. As with metrics, label [cardinality](#cardinality) is the cost driver — keep volatile attributes (request_id, trace_id) in the *log body*, not as Loki labels. ## Dashboards and queries ##### Dashboard {#dashboard} A collection of panels backed by queries, typically laid out in tabs and rows. This stack ships dashboards for both [Grafana](#grafana) and [Datadog](#datadog); see [Dashboards](../dashboards/). ##### Dashboard Panel {#dashboard-panel} A single visualization on a dashboard — a time series, table, donut, single-stat, etc. Most panels are backed by one or more queries. ##### LogQL {#logql} Loki's query language. Borrows [PromQL](#promql)'s selector syntax (`{job="mz", container="environmentd"}`) and adds line-filter operators (`|=`, `|~`) and parsing pipelines (`| json | line_format ...`). ##### PromQL {#promql} Prometheus's query language. Used by [Grafana](#grafana), [Alertmanager](#alertmanager), [recording rules](#recording-rule), and [alerting rules](#alerting-rule) across this stack. PromQL operates on time series: `rate(metric[5m])` computes a per-second rate; `sum by (label) (...)` aggregates. ## Alerting ##### Alert {#alert} A condition expressed as a [PromQL](#promql) (or [LogQL](#logql)) query that, when persistently true for a configured duration (`for: 5m`), signals that something needs attention. An alert *fires*, gets *routed* by [Alertmanager](#alertmanager) to [notification channels](#notification-channel), and is eventually *resolved* (or [silenced](#silence)). ##### Alerting rule {#alerting-rule} The [PromQL](#promql)/[LogQL](#logql) expression plus its severity, labels, and `for:` duration that defines when an alert fires. Distinct from a [recording rule](#recording-rule) in that an alerting rule's output is a firing/not-firing condition, not a new metric. ##### Alertmanager {#alertmanager} The component that receives fired alerts from [Prometheus](#prometheus), groups and deduplicates them, applies routing trees, fans out to [notification channels](#notification-channel), and handles silences and inhibitions. ##### Maintenance window / Silence {#silence} A time-bounded suppression of alerts matching a label selector. Used during planned work to prevent expected anomalies from paging the on-call. ##### Notification channel {#notification-channel} A destination [Alertmanager](#alertmanager) fans alerts to: PagerDuty, Slack, email, OpsGenie, generic webhook, etc. Configured in Alertmanager's routing tree. ##### SLA (Service Level Agreement) {#sla} A commitment made to a customer or downstream consumer about service reliability, often financially backed. Typically looser than the internal [SLO](#slo) that supports it — you might commit 99.9% to a customer (SLA) while targeting 99.95% internally (SLO) to leave buffer for absorbing routine incidents without breaching the agreement. ##### SLO (Service Level Objective) {#slo} A target reliability number expressed as a percentage over a time window — e.g., "99.9% of peek queries complete in under 250ms over a 30-day rolling window." SLOs are computed from SLIs (Service Level Indicators — the raw measurements feeding the objective) and typically drive *error budgets* and paging policy. Distinct from an [alerting rule](#alerting-rule): an SLO is a goal, an alerting rule is a condition that triggers when the goal is at risk. ## Collection and pipeline ##### Alloy {#alloy} [Grafana Alloy](https://grafana.com/docs/alloy/) is the collector at the heart of this stack. It scrapes metrics, ingests logs, applies [relabeling](#relabeling), and exports to one or more backends ([Prometheus](#prometheus) remote-write, [Loki](#loki) push, [OTLP](#otlp), the [Datadog Agent](#datadog)). Replaces older agents like Grafana Agent and Promtail. ##### Monitoring Profile {#monitoring-profile} A preset values file under `charts/materialize-monitoring/profiles/`, applied with `helm -f`, that configures the chart for a particular deployment shape — object-storage backend, split namespaces, an external Grafana, Loki sizing, and so on. The bundled backend stack ([Loki](#loki), [Thanos](#thanos), [Grafana](#grafana), [Alertmanager](#alertmanager), plus the [Alloy](#alloy) pipeline) is enabled by default through the `tags.default` group; profiles layer on top. Distinct from a *profile* in the continuous-profiling sense (Pyroscope, etc.). ##### Prometheus Exporter {#prometheus-exporter} A small process that translates some other system's state into [Prometheus](#prometheus) metrics on an HTTP endpoint. [`kube-state-metrics`](#kube-state-metrics), [`node-exporter`](#node-exporter), and [`cAdvisor`](#cadvisor) are common examples. Materialize itself exports its metrics natively from `environmentd` and `clusterd` — no separate exporter is required for the Materialize surface. ##### Relabeling {#relabeling} Rewriting label keys and values during ingestion. Used to normalize divergent label naming across deployment modes (self-managed vs. Cloud vs. BYOC) and to drop or coalesce high-[cardinality](#cardinality) labels before metrics egress to expensive backends. ##### Remote write {#remote-write} The [Prometheus](#prometheus) protocol for pushing metrics to a downstream backend (another Prometheus, [Thanos](#thanos), Mimir, [Datadog](#datadog), Honeycomb, etc.). The `prometheus-remote-write` profile is the most common shape for customers forwarding metrics to a paid backend. ##### Scrape {#scrape} The act of pulling metrics from an HTTP `/metrics` endpoint. [Prometheus](#prometheus) and [Alloy](#alloy) both scrape; targets are discovered either statically (in config) or dynamically via [ServiceMonitor / PodMonitor](#servicemonitor) resources. ##### Telemetry Pipeline {#telemetry-pipeline} The [Alloy](#alloy) configuration that describes how telemetry flows from sources to backends — scrape targets, relabeling rules, filters, and routing. This stack ships several reference pipelines, each parameterized as a [monitoring profile](#monitoring-profile). ## Stack components ##### Datadog / Datadog Agent {#datadog} A commercial monitoring backend and its in-cluster collector. The `datadog-agent` profile assumes a customer-managed Datadog Agent and configures [Alloy](#alloy) to feed it via OpenMetrics or DogStatsD. ##### Grafana {#grafana} The dashboard frontend in the bundled stack and the canonical target for the open-source dashboard outputs. Queries [Prometheus](#prometheus), [Thanos](#thanos), and [Loki](#loki); renders panels. ##### Loki {#loki} A log-storage backend modeled after [Prometheus](#prometheus). Indexes only labels (not log contents), which keeps it cheap at high volume but forces discipline about what becomes a label vs. what stays in the body. ##### OpenTelemetry / OTLP {#otlp} The vendor-neutral telemetry standard and its wire protocol. The `otlp` profile lets customers running OTel-native backends (Honeycomb-class) receive metrics and logs without standing up [Prometheus](#prometheus). ##### Prometheus {#prometheus} The de-facto open-source metrics database. Pull-based scraping, time-series storage, [PromQL](#promql), alerting rules — the conceptual foundation most of this vocabulary sits on top of. ##### Thanos {#thanos} A long-term-storage and global-query layer that sits behind [Prometheus](#prometheus). Stores blocks in S3-compatible object storage and provides a unified query view across multiple Prometheus instances; ships as part of the bundled stack for customers wanting cross-cluster federation. ## Cloud Technologies Managed metric/log backends offered by the cloud providers. Relevant when a customer wants to forward telemetry into their existing cloud console rather than run the bundled stack. The recurring theme: the managed-Prometheus options ([GMP](#gmp), [AMP](#amp)) are sample-billed, PromQL-native, and cardinality-tolerant, while the native custom-metric surfaces ([GCM](#gcm)'s `workload.googleapis.com/` domain, [CloudWatch](#cloudwatch)) bill per metric and reward the same [cardinality](#cardinality) discipline as [Datadog](#datadog). ##### AMP (Amazon Managed Service for Prometheus) {#amp} AWS's managed, Prometheus-compatible metrics store. Ingests via Prometheus [remote-write](#remote-write) authenticated with AWS SigV4 (hence the `sigv4` auth option on the remote-write destination), retains ~150 days, and is queried with PromQL directly or through [Grafana](#grafana). The AWS analogue of [GMP](#gmp); like GMP it doesn't scrape — you point [Alloy](#alloy) (or a Prometheus agent) at it. ##### CloudWatch {#cloudwatch} AWS's native monitoring service — metrics, logs, alarms, and dashboards in one product. The AWS analogue of [GCO](#gco) (spanning both [GCM](#gcm) and [GCL](#gcl)). Custom metrics are billed per-metric and per-API-call, so prefer [AMP](#amp) for high-volume Prometheus-shaped data and reserve CloudWatch custom metrics for a small, low-[cardinality](#cardinality) set. ##### GCL (Google Cloud Logging) {#gcl} Google Cloud's managed log store (formerly Stackdriver Logging), the logging half of [GCO](#gco). The OTel googlecloud exporter can route logs here in the same pipeline that sends metrics to [GCM](#gcm). The GCP analogue of [Loki](#loki), or of AWS [CloudWatch](#cloudwatch) Logs. ##### GCM (Google Cloud Monitoring) {#gcm} Google Cloud's managed metrics product (formerly Stackdriver Monitoring), the metrics half of [GCO](#gco) — both the store and the query/dashboard/alerting surface. Metrics live under type domains: `workload.googleapis.com/` (written by the OTel googlecloud exporter — per-MiB ingestion billing, a metric-descriptor model with a ~30-label cap and a per-minute descriptor-creation throttle) and `prometheus.googleapis.com/` (written by [GMP](#gmp)). Both domains are dashboardable and alertable within GCM. [Cardinality](#cardinality) drives cost — see [Operating > Tuning](../operating/tuning/). ##### GCO (Google Cloud Observability) {#gco} Google's umbrella observability suite (formerly Stackdriver, then "Google Cloud's operations suite") — spans [GCM](#gcm) (metrics), [GCL](#gcl) (logging), plus Trace, Profiler, and Error Reporting. When someone says "Stackdriver," they mean some part of GCO. The GCP analogue of AWS [CloudWatch](#cloudwatch). ##### GMP (Google Managed Service for Prometheus) {#gmp} Google's managed, Prometheus-compatible metrics store, backed by Monarch. Not a separate product from [GCM](#gcm) but an ingestion path *into* it, landing data under the `prometheus.googleapis.com/` domain where it's queryable by PromQL and usable in GCM dashboards and alerts. Billed per sample ingested — cheaper and more cardinality-tolerant than the `workload.googleapis.com/` domain — retains 24 months, and (like [AMP](#amp)) doesn't scrape: you push to it via [remote-write](#remote-write) or [OTLP](#otlp). The natural long-term store for GCP-centralized customers, replacing a self-run [Thanos](#thanos). The gateway's Google Cloud destination writes here, over OTLP to the Telemetry API. ## Kubernetes integration ##### cAdvisor {#cadvisor} Container Advisor — collects per-container resource usage (CPU, memory, network) from the kubelet. Surfaces metrics like `container_cpu_usage_seconds_total` and `container_memory_working_set_bytes`. Built into the kubelet; no separate install required. ##### CRD / CR {#crd} *Custom Resource Definition (CRD)* — a Kubernetes object that extends the API with a new resource type. *Custom Resource (CR)* — an instance of a CRD. This stack relies on CRDs from the [Prometheus Operator](#prometheus-operator) (`ServiceMonitor`, `PodMonitor`, `PrometheusRule`) and the Grafana Operator (`Grafana`, `Dashboard`, `Datasource`); the `prometheus-crds` subchart ships the Prometheus set for clusters that don't already run the operator. ##### kube-state-metrics {#kube-state-metrics} An exporter that surfaces metrics about Kubernetes API objects: pod readiness, deployment replicas desired vs. available, PVC status, etc. Different from `metrics-server`, which provides resource utilization for the horizontal pod autoscaler. ##### Kubernetes Probe {#kubernetes-probe} A Kubernetes health-check mechanism on a container. Three flavors: a *liveness probe* restarts the container when it fails (the process is wedged); a *readiness probe* removes the pod from Service load-balancing (the process is up but not ready to serve); a *startup probe* delays liveness/readiness during slow boots (e.g., during long [hydration](#hydration)). Probe failures surface through [kube-state-metrics](#kube-state-metrics) and pod events. ##### Kubernetes Sidecar {#kubernetes-sidecar} A secondary container running in the same Kubernetes pod as a primary application container, sharing the pod's network namespace and (often) volumes. In observability, sidecars commonly inject log shippers or proxies — though this stack prefers a DaemonSet (one [Alloy](#alloy) per node) over per-pod sidecars for collector workloads, since the DaemonSet shape scales with nodes rather than pods. ##### node-exporter {#node-exporter} An exporter that surfaces per-host (node-level) metrics — CPU utilization, memory, disk usage, filesystem stats, block-device counters, network counters — pulled from the Linux kernel. Typically deployed as a DaemonSet (one pod per node). Distinct from [cAdvisor](#cadvisor), which surfaces per-*container* metrics from the kubelet; node-exporter sees the whole machine. ##### Prometheus Operator {#prometheus-operator} A Kubernetes operator that manages [Prometheus](#prometheus), [Alertmanager](#alertmanager), and the CRDs (`ServiceMonitor`, `PodMonitor`, `PrometheusRule`, etc.) that drive their configuration. Optional — [Alloy](#alloy) alone can scrape without it — but ubiquitous in Kubernetes monitoring stacks. ##### ServiceMonitor / PodMonitor {#servicemonitor} Custom resources owned by the [Prometheus Operator](#prometheus-operator) that tell Prometheus/Alloy what to scrape. ServiceMonitors target Services; PodMonitors target Pods directly. This stack does **not** ship ServiceMonitors for Materialize itself — those live with the Materialize application chart — but [Alloy](#alloy) consumes them when present. ## Deployment models ##### DIY scraping {#diy-scraping} The "I'll scrape `environmentd` and `clusterd` myself" path. Customers who already operate a complete observability stack can ignore most of this repo and just consume the [documented metrics surface](../reference/list-metrics/). Source-side defaults are kept minimal specifically so this path stays viable. ##### Materialize Cloud {#materialize-cloud} Materialize's fully managed offering. The Cloud team operates its own observability stack built on this repo's primitives plus a private overlay; customers on Cloud don't run monitoring components themselves. ##### Self-managed {#self-managed} Customer-hosted Materialize, installed via the Materialize Terraform module or the Helm operator. The primary audience for this monitoring stack. ## Materialize concepts These terms come from the Materialize database itself; many show up as labels on metrics or in panel descriptions. Listed here because the dashboards lean on them and because some of the Materialize-side vocabulary is not what a DBA from a row-store background would expect. ##### Arrangement {#arrangement} An in-memory index of intermediate state that a [dataflow](#dataflow) maintains so it can answer queries quickly. Arrangements are why `SELECT ... FROM ` can be sub-millisecond — the result is already maintained on the cluster. Compute-cluster memory usage is often dominated by arrangement size. ##### Cluster (Materialize) {#cluster} A named pool of compute resources within a Materialize environment. **Not a Kubernetes cluster.** Workloads (indexes, materialized views, sources, sinks, subscribes) are placed on a Materialize cluster; the cluster has one or more [replicas](#replica) providing the actual capacity. In metric labels you'll see this as `cluster_id` or `instance_id`. ##### clusterd {#clusterd} The Materialize process that runs the compute side of a [replica](#replica) — executes [dataflows](#dataflow), maintains [arrangements](#arrangement), serves [peeks](#peek). Typically multiple `clusterd` pods per replica (one per [worker](#worker)). ##### Console {#console} Materialize's web application — accessible at `console.materialize.com` for Cloud customers, and at the chart-configured ingress in self-managed installs. Manages clusters and their objects (sources, sinks, materialized views, indexes), provides a SQL shell for ad-hoc queries, and renders its own built-in dashboards focused on user-object activity. Distinct from the [Grafana](#grafana) dashboards this stack ships, which are pulled from Prometheus and lean toward component- and infrastructure-level views. ##### Dataflow {#dataflow} The streaming execution plan Materialize compiles a query (or materialized view) into. Built on differential dataflow; each operator processes updates and emits updates. Long-lived dataflows back materialized views and indexes; short-lived dataflows back ad-hoc reads. ##### Environment / environmentd {#environment} A Materialize environment is the top-level tenancy unit; typically one per customer or per dev/staging/prod. `environmentd` is the control-plane process that owns the catalog, accepts SQL connections, schedules dataflows onto clusters, and emits most environment-scoped metrics. ##### Hydration {#hydration} The process of bringing a [dataflow](#dataflow)'s initial state up to current before it can serve queries. A newly created materialized view or index is *hydrating* until it catches up to upstream sources. Hydration time depends on data volume and source freshness; the *Hydration* row on the compute dashboard surfaces this. ##### Index {#index} A maintained data structure that lets queries read a subset of a collection without a full scan. In Materialize, indexes are always backed by an [arrangement](#arrangement) on a specific cluster — creating an index commits compute and memory to that cluster. ##### Materialized view {#materialized-view} A query whose result is incrementally maintained by a [dataflow](#dataflow) as inputs change. The user writes a SQL query once; Materialize keeps the answer continuously up-to-date. ##### Peek {#peek} Materialize's internal term for a read query against a materialized view or indexed table — the differential-dataflow operation that turns "give me the current value of this query" into bytes. Latency metrics like `v2_mz_compute_replica_peek_duration_seconds_*` describe peek latency, which is what users perceive as query latency. **There is no metric named "query latency"** — look for *peek*. ##### Replica (Materialize) {#replica} A unit of compute capacity inside a Materialize cluster, sized in [workers](#worker). Multiple replicas in one cluster provide redundancy. **Not a Kubernetes replica** — though under the hood each Materialize replica is implemented as one or more Kubernetes pods. ##### s2 / mz_catalog_server {#catalog-server} The Materialize system cluster that runs internal catalog and introspection workloads. It dominates many environment-scoped panels (peek counts, arrangement maintenance, hydration) because it's continuously busy maintaining the catalog. Its noise floor is business-as-usual, not an anomaly — flagged here because confusing s2 activity for user activity is a common false positive. ##### Sink {#sink} A streaming write from Materialize to an external system — Kafka, Iceberg, Postgres, MySQL. Sinks have throughput, lag, and (for transactional sinks like Iceberg) commit-latency metrics. ##### Source {#source} A streaming read into Materialize from an external system — Kafka, Postgres CDC, MySQL CDC, webhooks. Some source types (Postgres, MySQL) fan out into multiple [subsources](#subsource), one per replicated table. ##### Subsource {#subsource} A child of a multi-table source — e.g., the per-table replication streams of a Postgres source. Important wrinkle: the `source_id` label on `mz_source_bytes_received` is actually the *subsource* id; the parent id lives in `parent_source_id`. Aggregate by `parent_source_id` to get per-primary-source rates. ##### Worker {#worker} A single thread of execution inside a [`clusterd`](#clusterd) replica. Replica size determines worker count. Per-worker metrics can surface skew or hot-spot issues but multiply [cardinality](#cardinality), so dashboards typically aggregate to the cluster or replica level by default; per-worker breakdowns exist where the panel's whole point is detecting drift. -------------------------------------------------------------------------------- title: "Overview" url: https://materializeinc.github.io/materialize-monitoring/getting-started/overview/index.md -------------------------------------------------------------------------------- # Getting Started with materialize-monitoring There are a lot of ways to get started with `materialize-monitoring`. The contents of this chapter describe how to install and configure `materialize-monitoring` in various environments. ## `materialize-terraform-self-managed` Terraform Module If you stand up Materialize with the Terraform modules, observability comes up with the cluster — the modules create the buckets and workload identity, then install these charts at a pinned version. Since `materialize-terraform-self-managed` **v11** this is **opt-out rather than opt-in**: `enable_observability` defaults to `true` in every example root, on AWS, GCP, and Azure. Set it to `false` if you do not want the stack. See [Terraform Installation](../terraform/) — including the [tfvars reference](../terraform/#tfvars-reference). ## `materialize-monitoring` Helm Chart If you have a cluster with `materialize-operator` installed, you can install the `materialize-monitoring` Helm chart in the same cluster to set up your monitoring infrastructure. The Helm chart is the full-fidelity surface: the Terraform modules are a thin layer over it that adds the cloud resources Helm cannot create, and every chart value stays reachable from Terraform through `additional_values`. See [Helm Installation](../helm/) for instructions on how to install `materialize-monitoring` via Helm. ## Ad-hoc install for troubleshooting If you need a Grafana with metrics in it on a cluster you don't run day to day, such as during a day-1 install, one profile installs the whole stack with one replica of everything and no volumes. It isn't for production. See [Ad-hoc install](../adhoc/). ## Going to Production Whichever path you take, [Production Best Practices](../../operating/production-best-practices/checklist/) is the checklist for a real deployment — sizing, retention, replication, disruption budgets, and object-storage durability — with each item tagged by who owns it. ## Getting Help Please [reach out for Support](https://materialize.com/docs/support/). -------------------------------------------------------------------------------- title: "Dependencies" url: https://materializeinc.github.io/materialize-monitoring/getting-started/dependencies/index.md -------------------------------------------------------------------------------- # materialize-monitoring Dependencies What has to exist before `materialize-monitoring` installs, and who provides it. Almost everything on this page is the `[consumer]` half of the [shared responsibility model]({{< relref "../operating/production-best-practices/checklist.md" >}}#shared-responsibility-model): cloud resources, secrets, and cluster facilities the chart consumes **by name** but never creates. That is worth walking before the first install rather than after, because the common failure mode is not a rejected install — it is a chart that renders cleanly and a workload that crash-loops or sits `Pending` minutes later. > [!TIP] > Installing with the [Terraform modules]({{< relref "terraform.md" >}})? They create most of this for you — buckets, identity bindings, the Grafana database, the load balancer. > Skip to [If you install with Terraform](#terraform) for what is left. ## At a glance | Dependency | Needed when | Provided by | |---|---|---| | [A supported Kubernetes](#kubernetes) — v1.34+ | Always | Your cluster | | [CRDs](#crds) — Prometheus Operator + Grafana Operator | Always | The `materialize-monitoring-crds` chart | | [Object storage](#object-storage) + credentials | Unless both backends are external | You, or Terraform | | [A StorageClass the nodes can attach](#persistent-volumes) | Unless every volume is disabled | Your cluster | | [Materialize and `materialize-operator`](#materialize) | To scrape Materialize at all | You | | [The `materialize-sql-monitor` Secret](#the-sql-metrics-credential) | For the SQL-derived metrics | You | | [PostgreSQL](#postgresql-for-grafana) | For a durable, multi-replica Grafana | You, or Terraform | | [cert-manager](#cert-manager) | Only with `certificates.enabled` | You | | [Zone labels on nodes](#zone-labels) | Unless you apply `no-zone-spread` | Your cluster | ## Kubernetes {#kubernetes} **We support the non-EOL Kubernetes releases — v1.34 and newer.** Helm 3 is assumed throughout. The charts declare `kubeVersion: ">=1.27.0-0"`, but that is a floor rather than a support statement: older releases *may* work and are not supported. Cloud distributions offering extended support past a release's upstream EOL are supported best-effort and may run into issues. See [Compatibility]({{< relref "../reference/compatibility.md" >}}). Capacity matters more than version. The chart defaults target a **medium** install and request roughly **33Gi of memory and 9 CPU** in total, which no small cluster and no standard CI runner will schedule — the pods simply sit `Pending`, which reads like a chart bug rather than a capacity one. Pick a [sizing profile]({{< relref "helm.md" >}}#sizing) that matches the cluster you actually have. ## CRDs {#crds} Two sets of Custom Resource Definitions have to exist before the main chart installs: * **Prometheus Operator CRDs** — `ServiceMonitor`, `PodMonitor`, `ScrapeConfig`, `PrometheusRule`, and friends. These describe what gets scraped; the Alloy gateway reads them. * **Grafana Operator CRDs** — `Grafana`, `GrafanaDatasource`, `GrafanaManifest`, `GrafanaDashboard`, and friends. The chart creates these to provision Grafana and its dashboards. A second chart, `materialize-monitoring-crds`, installs them, so their lifecycle is managed separately from the stack that uses them. Install it first, then install the main chart with `--skip-crds`: ```bash helm install mzmon-crds oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-crds --namespace monitoring ``` `--skip-crds` on the main chart is not optional tidiness. The bundled Grafana Operator ships its own copy of the Grafana CRDs and offers no way to opt out of them, so without it a fresh install creates those definitions behind the CRDs chart's back and the two releases contend for the same cluster-scoped objects. To check what a cluster already has: ```bash kubectl get crds | grep -E 'monitoring\.coreos\.com|grafana\.integreatly\.org' ``` [Custom Resource Definitions]({{< relref "../reference/crds.md" >}}) is the full inventory — every kind, what the stack does with each, how to disable the ones you do not need, and the two groups (Google Managed Prometheus, cert-manager) that this chart deliberately does not install. ## Object storage {#object-storage} Loki and Thanos both keep their data in object storage. You need it unless **both** backends are external — pointing logs at an existing Loki and metrics at an existing Prometheus-compatible store leaves nothing to persist locally. **Two buckets (or containers) are the norm, one per backend.** Loki and Thanos want different lifecycle rules, and per-bucket IAM keeps each backend out of the other's data. ### Granting access On a managed Kubernetes service, grant access through **workload identity** rather than static keys. The binding is per-cloud, and the trust-policy subjects have to match the chart's rendered ServiceAccount names: * [Logs & Events → Storing]({{< relref "../logs-and-events/storing.md" >}}#granting-object-storage-access-workload-identity) — IRSA, GKE Workload Identity, and Entra Workload ID * [Metrics → Storing]({{< relref "../metrics/storing.md" >}}#authentication) — the same three for Thanos Without a cloud identity provider there is no identity to bind, so credentials become **static keys supplied as a Secret**. That is the documented escape hatch rather than the recommended path; both backends read them by reference, and the Terraform module exposes them as `object_storage_access_key_id` / `object_storage_secret_access_key`. Any S3-compatible endpoint works. This repository's own tier-2 E2E runs against [rustfs](https://github.com/rustfs/rustfs), so a self-hosted MinIO, Ceph, or rustfs is a supported shape rather than an untested one. > [!WARNING] > **Naming the backend is the step that most often goes wrong.** > The chart's defaults are S3-shaped, so any other backend has to be named in three load-bearing places, and none of them fails softly: the client is chosen by name and then validated against a config that was never populated, so the component crash-loops. > See [Selecting the backend]({{< relref "../logs-and-events/storing.md" >}}#selecting-the-backend). The example profiles do all of it correctly and are the shortest path: | Profile | Backend | |---|---| | `profiles/aws-example.values.yaml` | S3 + IRSA | | `profiles/gcp-example.values.yaml` | GCS + GKE Workload Identity | | `profiles/azure-example.values.yaml` | Blob + Entra Workload ID | | `profiles/aws-amp-fanout.values.yaml` | S3 for logs; metrics to the bundled Thanos **and** Amazon Managed Prometheus at once | ## Persistent volumes {#persistent-volumes} Most of this stack is volumeless on purpose — durability comes from the replication factor or from object storage, and a volume that cannot cross availability zones makes a zone failure worse rather than safer. Four workloads do claim one, so a **StorageClass has to exist**: | Workload | What the volume holds | |---|---| | Alertmanager | Silences and the notification log — the only state here that is genuinely not reconstructible | | The Loki ruler | Its remote-write WAL, buffering recording-rule samples while the metric store is unreachable | | The Thanos Store Gateway | The on-disk index-header cache: cheap to lose, slow to rebuild | | The Thanos Compactor | Scratch for the block group under compaction — tens of GiB, more than a typical node's allocatable ephemeral storage | Use `profiles/storage-class.values.yaml` (one anchor to edit) or the Terraform module's `storage_class` variable, which carries the same map. > [!WARNING] > **A class that cannot serve the nodes is not the same as a missing class.** > On GCP's C4 and N4 machine families, which accept only Hyperdisk, *every* StorageClass GKE creates by default is Persistent Disk and none of them will attach — the PVC binds and the pod never starts. Changing the class later does not move the volumes: `volumeClaimTemplates` are immutable, so the old PVCs must be deleted first, discarding their contents. ## Materialize {#materialize} The stack runs happily with no Materialize in the cluster — it just has nothing to scrape. To collect Materialize metrics you need `materialize-operator` installed, and the chart needs to know where it and the environments live: ```yaml materialize-operator: namespace: materialize materialize-system: namespace: materialize-environment materialize: namespaces: [] # empty means every namespace the chart can read ``` Version requirements are in [Compatibility]({{< relref "../reference/compatibility.md" >}}). In short: the scrapers need the `environmentd` labels introduced in Materialize **v26.24.0**, and the dashboards want the `mz_object_info` metric introduced in **v26.29.0** — without it they degrade gracefully rather than break. ### The SQL metrics credential {#the-sql-metrics-credential} The `materialize-sql` scrapers read SQL-derived metrics from `environmentd` as the built-in, password-less `mz_support` role. Prometheus Operator `basicAuth` can only reference a Secret — it has no inline fields — so this one Secret has to exist, in the namespace the scrapers run in: ```bash kubectl create secret generic materialize-sql-monitor \ --namespace materialize \ --from-literal=username=mz_support \ --from-literal=password= ``` The empty `password` is deliberate, and not a placeholder. **This endpoint does not validate passwords** — the username selects the role the queries run as, and the password field is never read. It has to be present only because Alloy's scrape-config generation rejects an absent password reference with `resource name may not be empty`. Set `materialize.environmentdSQL.secret.create: false` if you provision the Secret yourself. See [Scraping]({{< relref "../metrics/scraping.md" >}}#authenticating-the-sql-metrics-endpoint), and [Securing]({{< relref "../operating/securing.md" >}}#materialize-metrics-endpoint) for the access-control consequence. ## Conditional dependencies ### PostgreSQL for Grafana {#postgresql-for-grafana} Grafana keeps its own state — users, service accounts and tokens, annotations, dashboard versions and permissions, alert-rule state — in a database separate from the observability data in Thanos and Loki. The chart default is SQLite on an `emptyDir`, which loses that state on every pod restart. SQLite on a PersistentVolume survives restarts but tolerates one writer, capping you at a single replica. **An external PostgreSQL is the only shape that lifts both constraints**, which is why it is the production one — and why exposing Grafana without it turns a bundled extra nobody depended on into a primary interface that silently discards what users build in it. `profiles/grafana-postgres.values.yaml` assembles it. Three things it does not create: 1. A database and a user that **owns** it — Grafana runs its own schema migrations at startup, so a read/write-only grant fails the migration. 2. A Secret holding the password, in the namespace the Grafana *pod* runs in. 3. Network reach from that pod to the database. > [!INFO] > RDS and Cloud SQL both support IAM database authentication, but Grafana reads its password once at startup and has no refresh hook, so the 15-minute (RDS) or one-hour (Cloud SQL) token expiry breaks the first reconnect. Use a static password. The Terraform modules provision a dedicated small PostgreSQL instance for this, so on that path it is handled. ### cert-manager {#cert-manager} Only needed with `certificates.enabled`, which **defaults to `false`** — with it off, cert-manager is not a dependency of this chart in any sense and nothing TLS-related renders. With it on, the chart renders `Certificate` resources for in-cluster mTLS, and optionally a self-signed root. This is gated on a value rather than an API-capability probe, deliberately: enabling it without cert-manager present fails at apply time with a resource name in the error, which is a better outcome than silently rendering nothing. See [Securing]({{< relref "../operating/securing.md" >}}#certificates). ### Reaching Grafana {#reaching-grafana} Grafana is the only component in this stack meant for a human, and the only one the chart will help you expose. An Ingress or LoadBalancer Service is a prerequisite for anyone actually using it, and **DNS, TLS, and an identity provider remain yours** on every path, including Terraform. `profiles/grafana-ingress.values.yaml` assembles the shape — internal by default, public gated behind an enforced allowlist. Pair it with a persistence profile above; the render-time checks will tell you if you exposed Grafana without authentication or without a durable backend. See [Securing]({{< relref "../operating/securing.md" >}}) for what each exposure combination is graded as. ## Cluster-shape prerequisites These are properties of the cluster rather than things you install. Each one produces a *quiet* failure, which is why they are listed. ### Zone labels {#zone-labels} Thanos Receive and Loki's ingesters spread **hard** across zones (`DoNotSchedule`), so that an un-spread pod goes `Pending` and the autoscaler provisions the missing zone. On a cluster whose nodes carry **no zone label, or only one zone**, those pods stay `Pending` permanently. Apply `profiles/no-zone-spread.values.yaml` — see [Production Best Practices]({{< relref "../operating/production-best-practices/stack-wide.md" >}}#fewer-than-two-zones). ### Kubelet certificates {#kubelet-certificates} Container metrics arrive because the Alloy gateway scrapes `/metrics/cadvisor` on every kubelet, on by default. A distribution that signs kubelet certificates with a CA the pods do not trust **fails that scrape quietly** — nothing errors at install, the Kubernetes dashboards are simply empty. `kind` is the known case. Check `up{job="cadvisor"}` when bringing up a new cluster, and set `pipeline.metrics.kubelet.tlsInsecureSkipVerify: true` only where you have to; leaving verification on is correct everywhere real. ### A cluster that already serves the metrics API {#metrics-api} `tags.default` installs `metrics-server`, and the `v1beta1.metrics.k8s.io` APIService it registers is cluster-singleton. Most managed clusters already provide it — GKE always, EKS via addon, AKS by default. Set `metrics-server.enabled: false` there. `metrics-server` is deliberately left on the upstream `system-cluster-critical` priority class, because it backs the metrics API that HPAs and the Materialize Console read: it is cluster plumbing rather than monitoring. ### GKE managed collection {#gke} On GKE, managed collection is on by default from v1.27 and provides the `PodMonitoring` CRDs the GMP scrapers use. Container metrics do not depend on it. The gateway scrapes each kubelet's `/metrics/cadvisor` endpoint itself, which yields a fuller set than GMP's default collection, so the GCP-optimized dashboards carry the same metric families as the standard ones. See [Container metrics from the kubelet]({{< relref "../metrics/scraping.md" >}}#kubelet). ## If you install with Terraform {#terraform} The [Terraform modules]({{< relref "terraform.md" >}}) cover the cloud-resource dependencies above: buckets, the IAM role or service account with its workload-identity binding, the Grafana database, an internal load balancer, and the CRDs chart. What stays yours on that path: * **DNS, TLS, and an identity provider** for Grafana. * **The `materialize-sql-monitor` Secret**, if you want the SQL-derived metrics. * **Cluster shape** — zone labels, a StorageClass the nodes can attach, capacity for the sizing tier you pick. * **Rolling Alloy after a pipeline or filter change.** The module hashes the composed values to do this for you; on the Helm path it is yours, and it is the one place the chart cannot own its own rollout. See [Production Best Practices]({{< relref "../operating/production-best-practices/alloy.md" >}}#restart-after-a-config-change). ## Next steps Once the dependencies are in place: [Installing via Terraform]({{< relref "terraform.md" >}}) or [Installing via Helm]({{< relref "helm.md" >}}). -------------------------------------------------------------------------------- title: "Terraform" url: https://materializeinc.github.io/materialize-monitoring/getting-started/terraform/index.md -------------------------------------------------------------------------------- # Installing via Terraform For teams standing up self-managed Materialize with [`materialize-terraform-self-managed`](https://github.com/MaterializeInc/materialize-terraform-self-managed). The observability stack comes up with the cluster, from the same modules, with no separate install step. > [!SUCCESS] > **Shipped, and on by default.** Since `materialize-terraform-self-managed` **v11** the monitoring stack is **opt-out rather than opt-in** — `enable_observability` defaults to `true` in every example root. > All three clouds are wired: AWS, GCP, and Azure. The legacy Prometheus + Grafana modules are retired. ## What you get One module per cloud creates the storage and identity the stack needs, then installs the [Helm charts](../helm/) at a pinned version: | | | |---|---| | **Metrics** | Thanos, backed by object storage, with a Prometheus-compatible query API | | **Logs** | Loki, backed by object storage | | **Collection** | Alloy — an agent DaemonSet on every node, and a gateway for shaping and egress | | **Dashboards** | The released Grafana dashboard set, via grafana-operator | | **Alerting** | Alertmanager with the bundled rules. No receiver until one is configured; see [Alerting](#alerting) | | **Grafana state** | A dedicated small PostgreSQL instance, so what users build in the UI survives a restart | Cloud-side, each backend gets the following: - **AWS:** a bucket and an IAM role. - **GCP:** a bucket and a Google service account with a Workload Identity binding. - **Azure:** a container in a shared storage account, and a user-assigned managed identity with a federated credential. Unless you turn it off, the module also creates one small PostgreSQL instance for Grafana; see [Reaching Grafana](#reaching-grafana). Grafana gets an internal L4 load balancer with the rest of the stack on all three clouds. DNS, TLS, and an identity provider are still yours — see [Reaching Grafana](#reaching-grafana). ## Usage Observability is a module block in each example root, gated on one variable: ```hcl module "monitoring" { count = var.enable_observability ? 1 : 0 source = "../../modules/monitoring" prefix = var.name_prefix project_id = var.project_id region = var.region namespace = "monitoring" create_namespace = false # the operator module creates it node_selector = local.generic_node_labels materialize_instance_namespace = local.materialize_instance_namespace materialize_operator_namespace = local.materialize_operator_namespace depends_on = [module.operator, module.gke, module.generic_nodepool, module.coredns] } ``` If you start from an example root, that block is already there and already on. To opt out, set this in your `terraform.tfvars`: ```hcl enable_observability = false ``` See the [tfvars reference](#tfvars-reference) below for the rest. ## tfvars reference Variables you would realistically set in a `terraform.tfvars` at an example root. Everything else has a default that suits the supported topology. ### Turning it off | Variable | Default | Notes | |---|---|---| | `enable_observability` | `true` in both `simple` and `enterprise`, on all three clouds | The switch, on by default since v11. Everything below is inert without it | ### Sizing and placement These are set on the `monitoring` module block rather than as root variables, so change them there or add a matching root variable. | Variable | Default | Notes | |---|---|---| | `sizing` | `"medium"` | `small`, `medium`, or `large`. The chart's defaults *are* medium, so that tier applies no overlay. Start at `small` for dev or a constrained node pool | | `node_selector` | `{}` | Reaches every centralized workload. **Not** the Alloy agent DaemonSet, which must run on every node — see [Scheduling](#scheduling) | | `tolerations` | `[]` | Reaches the agent too, since tolerations widen rather than narrow where a pod may run | | `namespace` | `"monitoring"` | Also the namespace half of every workload-identity subject | | `create_namespace` | `false` | The operator module already creates `monitoring` | ### Storage and retention | Variable | Default | Notes | |---|---|---| | `storage_class` | `null` (cluster default) | Reaches the four PVC-backed workloads — Alertmanager, the Loki ruler, the Thanos Store Gateway and Compactor. **Required on GCP C4/N4 node pools** — see below. Loki's ingesters and Thanos Receive are unaffected; they use node-local `emptyDir` by design | | `bucket_force_destroy` | `false` | Allows `terraform destroy` to delete non-empty buckets. Leave false outside throwaway environments | | `enable_bucket_versioning` | `true` | Versioning is the disaster-recovery primitive — neither Loki nor Thanos has a native snapshot | | `logs_retention_days` | `null` | Bucket-level expiry for logs. Off by default; Loki's compactor already enforces retention | | `metrics_retention_days` | `null` | Off by default, and **leave it off** unless you have a reason. Thanos keeps blocks per downsampling resolution (raw 30d / 5m 90d / 1h 365d), and a bucket rule expiring sooner deletes blocks the compactor still references | #### StorageClass on GCP C4 and N4 node pools The **C4** and **N4** machine families accept only Hyperdisk. They cannot attach Persistent Disk of any type, and GKE's default `standard-rwo` class is `pd-balanced`, so every PVC-backed workload hangs: ``` AttachVolume.Attach failed for volume "pvc-...": pd-balanced disk type cannot be used by c4-standard-8 machine type, badRequest ``` The other classes GKE creates by default — `premium-rwo` (`pd-ssd`) and `standard` (`pd-standard`) — are Persistent Disk too, so none of them work either. GKE does not create a Hyperdisk class for you: ```yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: hyperdisk-balanced provisioner: pd.csi.storage.gke.io parameters: type: hyperdisk-balanced volumeBindingMode: WaitForFirstConsumer allowVolumeExpansion: true ``` Then `storage_class = "hyperdisk-balanced"`. > [!WARNING] > Switching the class on an existing install does not migrate the volumes. > A StatefulSet's `volumeClaimTemplates` are immutable, so the old PVCs have to be deleted before the new class takes effect — and deleting them discards their contents. > That is cheap for these five (Thanos re-downloads blocks from object storage, and the Alertmanager and ruler volumes hold local state, not the rules themselves), but it is not a no-op. ### Extra metrics destinations Thanos is always the primary metric store. These fan out in addition to it, from the Alloy gateway. | Variable | Default | Notes | |---|---|---| | `enable_google_cloud_metrics` | `false` | GCP only. Export to Google Cloud Managed Service for Prometheus, over OTLP to the Telemetry API. Creates a service account with `roles/monitoring.metricWriter` and binds the gateway to it | | `google_cloud_metrics_min_importance` | `"recommended"` | `essential`, `recommended`, `extended`, `diagnostic`, or `all`. Each tier includes the ones below it | | `google_cloud_metrics_prefix` | `null` | Deprecated and ignored. Every metric is `prometheus.googleapis.com//`, so there is no prefix to choose | The project needs the `telemetry.googleapis.com` API enabled. `min_importance` is a **cost control**, not a filter for convenience. Cloud Monitoring bills these metrics per sample ingested, so the tier you pick sets the bill. `recommended` covers the dashboards; `all` sends the full surface and is a diagnostic setting, not a steady state. [Metrics > Storing](../../metrics/storing/#gcm) has the prices and a measured install. The tiers come from the same `metric-tiers.yaml` the chart uses, so a Terraform-selected tier and a Helm-selected one always mean the same set. See [Metrics > Storing](../../metrics/storing/) for what each tier contains. > [!INFO] > Authentication is ADC only — there is no key-file path. > Without the Workload Identity binding the module creates, the exporter falls back to the node's service account, which works only if that account happens to hold `roles/monitoring.metricWriter`. **Datadog and generic OTLP** (Honeycomb, Grafana Cloud, your own collector) are inputs on the `materialize-monitoring` module itself — `datadog_metrics` / `datadog_api_key` and `otlp_metrics` / `otlp_auth_header_secrets` / `otlp_auth_bearer_token`. Neither needs cloud resources, so they are not surfaced as flat wrapper variables the way Google Cloud Monitoring is; set them on the wrapper's `monitoring` module block. Credentials reach the gateway through a Secret the module creates rather than through the Helm values, and rotating one rolls the gateway. See [Metrics > Storing](../../metrics/storing/#through-terraform) and the [module README](https://github.com/MaterializeInc/materialize-monitoring/blob/main/terraform/modules/materialize-monitoring/README.md#metric-destinations). ### Alerting Alertmanager and the bundled rules install with the stack. A default install configures no receiver, so every alert reaches `mzmon-null`, which notifies nobody, until one is set. Which rules install and where alerts go are inputs on the `materialize-monitoring` module itself: | Input | Default | Configures | |---|---|---| | `alert_rules` | `{}` | Which bundled rules install, their overrides, and the namespaces and infrastructure workloads they read | | `alerting` | `{}` | The routing preset, receivers, routes, inhibit rules, time intervals and templates | | `alerting_receiver_secrets` | `{}` | Receiver credentials. Sensitive, and delivered as the `alertmanager-receivers` Secret rather than as Helm values | | `alertmanager_namespace` | `namespace` | Where that Secret is created. `alertmanager` under the chart's `split-namespace` profile | ```hcl alerting = { preset = "critical-infrastructure" receivers = { oncall = { class = "page" config = { pagerduty_configs = [{ routing_key_file = "/etc/alertmanager/secrets/alertmanager-receivers/pagerduty-key" }] } } platform = { class = ["high", "normal"] config = { slack_configs = [{ channel = "#platform-alerts", api_url_file = "/etc/alertmanager/secrets/alertmanager-receivers/slack-url" }] } } } } alerting_receiver_secrets = { "pagerduty-key" = var.pagerduty_routing_key "slack-url" = var.platform_slack_webhook } ``` Like the Datadog and OTLP inputs, they need no cloud resources and pass through the per-cloud wrappers under the same names; set them on the wrapper's `monitoring` module block. [Configuring Alerting through Terraform](../../alerting/terraform/) covers each in full, and [Alert Channels](../../alerting/channels/) has the receiver configurations for PagerDuty, Slack, Opsgenie, email and the rest. ### Integration | Variable | Default | Notes | |---|---|---| | `install_metrics_server` | `false` | The operator module installs metrics-server. If you set `install_metrics_server = false` **there**, set this `true` in the same change — the Materialize Console depends on the metrics API | | `materialize_instance_namespace` | `"materialize-environment"` | Scopes scrape targets | | `materialize_operator_namespace` | `"materialize"` | | | `grafana_admin_password` | `null` | Generated when unset. Read it with `terraform output -raw grafana_admin_password` | ### Grafana persistence and exposure Set on the wrapper's `monitoring` module block, with root variables for them in the examples. See [Reaching Grafana](#reaching-grafana) for what each one implies. | Variable | Default | Notes | |---|---|---| | `grafana_host` | `null` | Example root variable. Optional on all three: the load balancer exists regardless, and this only makes `root_url` correct | | `grafana_load_balancer` | `null` | The wrapper's own input. Internal by default; public requires `ingress_cidr_blocks`, and refuses `0.0.0.0/0`. Its `ip` field pre-allocates the address, which on GCP and Azure is what makes `grafana_url` known at plan time | | `grafana_database_ssl_mode` | `"require"` | `verify-full` also authenticates the server, but needs a CA bundle mounted through `additional_values` | ### Anything else | Variable | Default | Notes | |---|---|---| | `additional_values` | `[]` | Raw Helm values YAML, appended last so it wins over everything the modules compute. The supported way to reach any chart setting the modules do not model | ```hcl additional_values = [ yamlencode({ thanos = { receive = { replicaCount = 5 } } }), ] ``` The full input and output list is in the [Terraform variable reference](../../reference/terraform/materialize-monitoring-variables/). Every chart value is reachable through `additional_values`; see the [Helm values reference](../../reference/helm/materialize-monitoring-values/). ## Outputs ```console $ terraform output grafana_url "http://grafana.monitoring.svc.cluster.local" $ terraform output -raw grafana_admin_password ``` | Output | | |---|---| | `grafana_url` | Conditional: the external URL once Grafana is exposed, the in-cluster Service otherwise. See [Reaching Grafana](#reaching-grafana) | | `grafana_admin_password` | Sensitive | | `grafana_database_endpoint` | `host:port` of the database backing Grafana's state, or null while it is on SQLite. Wrapper modules only | | `grafana_database_password` | Sensitive. Generated when the wrapper creates the instance | | `metrics_url` | Thanos Query. Prometheus-API-compatible, so anything that spoke to the old `prometheus_url` works against it | | `logs_url` | Loki read endpoint | ## Reaching Grafana Out of the box Grafana is `ClusterIP`, so access is a port-forward: ```bash kubectl -n monitoring port-forward svc/grafana 3000:80 ``` Log in as `admin` with the password from `terraform output -raw grafana_admin_password`. That is the right default, not a gap: the only account is the generated admin, and every datasource behind Grafana reads every metric in Thanos and every log in the tenant. Exposing it is a deliberate step with three parts. ### 1. Give it somewhere to keep state **Do this first.** Grafana stores users, service accounts and tokens, annotations, dashboard versions and permissions, preferences, and alert-rule state in a database of its own, and the chart default is SQLite on an `emptyDir` — lost on every restart, upgrade, and reschedule. An exposed Grafana that silently discards everything its users build in it is worse than an unreachable one. The per-cloud wrapper modules provision a **dedicated** small PostgreSQL instance for it, and the examples turn that on wherever `enable_observability` is on: | Variable | Default | Notes | |---|---|---| | `grafana_database` | set in the examples | The wrapper's own input: the networking facts its cloud needs. The examples always fill it in, so it follows `enable_observability`; set it `null` on the module block to opt out | | `grafana_database_host` and friends | `null` | Point at a database you already run instead. Mutually exclusive with `grafana_database` | Dedicated rather than a database inside the Materialize instance, for reasons that differ per cloud and happen to agree — RDS has no API to add a database to an existing instance, and an Azure Flexible Server has one administrator login and no ARM resource for extra roles. The default sizes are the smallest each cloud offers, which is enough: Grafana's state is small and its query rate is a handful per page load. > [!INFO] > Switching an existing install from SQLite to PostgreSQL **does not carry state over** — Grafana has no migration between the two. Export what matters through the HTTP API first. ### 2. Put a load balancer in front All three clouds take the same variable, `grafana_load_balancer`, and produce an L4 load balancer. How it is built differs, because the deterministic option does: | Cloud | Result | Built from | |---|---|---| | AWS | Network Load Balancer | `aws_lb` and a `TargetGroupBinding` onto a `ClusterIP` Service, all Terraform resources | | GCP | passthrough Network Load Balancer | the chart's `LoadBalancer` Service | | Azure | Azure Load Balancer | the chart's `LoadBalancer` Service | On AWS that means the address is a Terraform attribute rather than something read back after apply, so `grafana_url` is known at plan time — and the allowlist is security-group rules on the NLB rather than `loadBalancerSourceRanges`. The examples pass it unconditionally, alongside the Materialize console and balancerd load balancers, and `host` is optional — the load balancer answers on an address of its own either way, and the hostname only fixes `root_url`. > [!WARNING] > **The load balancer does not terminate TLS.** An L4 load balancer passes bytes through, so TLS terminates at the Grafana pod or not at all — and until Grafana has a certificate it serves plain HTTP, session cookie and admin password included. > > The modules can issue that certificate and have Grafana serve it; see [Giving Grafana a certificate](#grafana-certificate) below. Until it is in place, treat exposure as internal-only, and **do not set `security.cookie_secure`** — it marks the cookie `Secure`, the browser then refuses to send it over the plain-HTTP connection that works, and login stops working entirely. L7 — an ALB, a GCP Application Load Balancer, an Azure Application Gateway — is the intended end state for public exposure, because a WAF and authentication at the edge are the two things L4 cannot do. It is deferred rather than rejected: Azure has no ingress-controller module yet, and the chart's Gateway API support is still BETA, so adopting Ingress now would mean migrating twice. See [Ingress and Service are not interchangeable](../../dashboards/grafana/architecture/#ingress-and-service-are-not-interchangeable). On AWS the Load Balancer Controller has to already be installed — it reconciles the `TargetGroupBinding` that attaches the NLB's target group to the Grafana Service — and the examples install it as `module.aws_lbc`. Both are **internal by default**, and both read the `internal_load_balancer` and `ingress_cidr_blocks` variables the Materialize load balancers already use, with the presence check enforced by a `validation` block copied from the repo's `nlb` module rather than left as a convention. ```hcl # At an example root enable_observability = true # Optional on all three — the load balancer answers on an address of its own # either way, and this only fixes `root_url` and the certificate name. grafana_host = "grafana.example.com" # Public instead of internal. Narrow the allowlist yourself: it defaults to # ["0.0.0.0/0"], inherited from the variable the Materialize load balancers use, # which is only a sensible default while the load balancer is internal. internal_load_balancer = false ingress_cidr_blocks = ["203.0.113.0/24"] ``` **DNS is the one thing the modules cannot do for you.** Neither the Ingress nor the Service publishes the hostname, and Terraform has no view of your zone. An ACME challenge against a name that does not resolve is the usual way this gets noticed. ### 3. Give Grafana a certificate {#grafana-certificate} Behind an L4 load balancer the certificate has to exist in the cluster, because the pod is the only thing left that can terminate TLS. The module issues it from cert-manager: ```hcl certificates_enabled = true issuer_ref = { name = "letsencrypt-prod", kind = "ClusterIssuer" } grafana_external_dns_names = ["grafana.example.com"] ``` `issuer_ref` covers both the browser-facing certificate and the in-cluster ones, unless `internal_issuer_ref` overrides the latter. Leaving it null with `certificates_enabled = true` bootstraps a self-signed root scoped to the release — right for the internal mesh, but meaningless to a browser, so a public name needs a real issuer. Setting `grafana_external_dns_names` with no `issuer_ref` issues nothing, and the plan says so. That combination is a **validation error**, not a silent no-op, precisely because it looks configured. If instead an L7 load balancer terminates with a cloud-managed certificate (ACM, Google Certificate Manager, Azure Key Vault), the key never enters the cluster — leave this empty and attach the certificate by annotation through `grafana.service.annotations` in `additional_values`. With the certificate issued, Grafana serves HTTPS from it, with the settings the chart's `grafana-https` profile carries: - The module mounts the certificate's Secret and sets `grafana.ini.server.protocol` to `https`, `root_url` to `https://` and the first name, and `security.cookie_secure`. - The Grafana Service moves to port 443, so `https://grafana.example.com` needs no port. Point the load balancer's listener at 443 if your wrapper sets one. - The probes and Grafana's ServiceMonitor switch to HTTPS. The scrape verifies the certificate against the first name and the system's public roots. For a private `issuer_ref`, set `grafana.serviceMonitor.tlsConfig.ca` to its CA in `additional_values`. - grafana-operator connects to Grafana's Service over HTTPS without verifying the certificate, because the certificate doesn't name the Service. - The `grafana_url` output becomes `https://`. A `root_url` in `additional_values` still wins over the module's. It has to start with `https://`, or the render fails. See [Terminating TLS](../../dashboards/grafana/architecture/#terminating-tls). Separately, `internal_tls` (`off` / `encrypt` / `present` / `authenticate`) moves the **in-cluster** hops off plaintext. It also requires `certificates_enabled`. On a fresh install go straight to `authenticate`; on a running stack step through `present` first, or a server that rolls before its writers stops ingestion until they catch up. ### 4. Configure an identity provider Until you do, the generated admin password is the whole of the access control. Everything under `grafana.ini` is passed through verbatim, so any provider Grafana supports is reachable — through `additional_values` on the Terraform path. The chart warns at render time when an exposed Grafana has none. See [Authentication](../../dashboards/grafana/auth/) for the wiring, including why client secrets cannot live in `grafana.ini` and how to map an IdP group claim onto Grafana roles. ## Scheduling `node_selector` reaches every centralized workload — Loki, Thanos, Grafana, Alertmanager, kube-state-metrics, and the Alloy gateway — but deliberately **not** the Alloy agent. The agent is a DaemonSet whose job is to collect from every node. A node selector would confine it to one workload pool and silently stop collecting logs and node metrics from everywhere else. `tolerations` do reach it, because tolerations widen where a pod may run, which is exactly what a DaemonSet wants. ## Capacity The stack is meaningfully larger than the Prometheus + Grafana pair it replaces: microservice Loki, Thanos, Grafana, Alertmanager, kube-state-metrics, and two Alloy roles. If your node pool is sized for the old shape, the first apply lands unschedulable pods. Either grow the pool or start at `sizing = "small"`. ## Going to production The modules cover the cloud-resource half of a production deployment — buckets, workload identity, version pinning — but not the sizing, retention, and capacity decisions that only you can make. [Production Best Practices](../../operating/production-best-practices/checklist/) is the checklist, tagged by owner. Start with [what the Terraform path already handles](../../operating/production-best-practices/checklist/#terraform-consumer), then work the items still tagged `[operator]`. Four that catch people out on a first production install: - **A default StorageClass must exist.** Several components are PVC-backed and the modules do not create one. - **Retention is enforced in-cluster, not by the bucket.** `metrics_retention_days` defaults to off for a reason — see the [Thanos checklist](../../operating/production-best-practices/thanos/). - **Nothing is notified until a receiver exists.** Alertmanager and the bundled rules install, but a default install routes every alert to `mzmon-null`. See [Alerting](#alerting). - **Grafana has no identity provider until you configure one.** The modules give it a durable database and can put a load balancer in front of it, but not an IdP — see [Authentication](../../dashboards/grafana/auth/) and the [Grafana checklist](../../operating/production-best-practices/grafana/). ## Pinning a different version Each wrapper module pins the common module to a released tag: ```hcl # In {aws,gcp,azure}/modules/monitoring/main.tf source = "github.com/MaterializeInc/materialize-monitoring//terraform/modules/materialize-monitoring?ref=materialize-monitoring/v0.17.0" ``` To try an unreleased change, point `source` at a branch instead — `?ref=my-branch`. The ref is the only version you set. The module reads its chart version from the `Chart.yaml` shipped beside it, so the two cannot disagree — `chart_version` exists only to pin something different deliberately. A local path works too, but it **must be relative**. Terraform copies a module referenced by an absolute path into `.terraform/modules/` without the chart directory beside it, and the sizing profiles stop resolving. The module raises a precondition rather than failing quietly. ## Migrating from the previous stack The `prometheus` and `grafana` modules are replaced. Applying the change: - **Destroys** the `prometheus` and `grafana` releases and their PersistentVolumeClaims. Up to 15 days of local Prometheus data goes with them — there is no backfill, and collection restarts at install. Anything hand-created in the old Grafana does not carry over. - Replaces the `prometheus_url` output with `metrics_url` and `logs_url`. - Creates new cloud resources: buckets and identities per backend. `enable_observability` keeps its name and defaults. ## Tearing it down `terraform destroy` works unaided now: the chart's `pre-delete` hook removes the Grafana custom resources while grafana-operator is still running to clear its finalizers, and blocks until they are gone. The module already orders its two releases so the main release is destroyed before the CRDs release, and the hook covers the ordering *within* a release that Terraform cannot control. Two cases still deadlock: - **A release applied before the hook existed.** Helm runs pre-delete hooks from the stored release manifest, so bump the module and `terraform apply` once before destroying. - **Removing the CRDs first**, which takes the resource types with it and leaves the hook failing on an unknown type. In either case, delete them by hand first: ```bash kubectl -n monitoring delete grafanadatasources,grafanamanifests --all ``` See [Operating > Uninstalling](../../operating/uninstalling/) for the mechanism and for recovering a teardown that is already stuck. ## If you are not using Terraform The [Helm charts](../helm/) are the full-fidelity surface, and everything above is a thin layer over them. Terraform's job is the part Helm cannot do: creating the buckets, granting workload identity, and pinning a version. -------------------------------------------------------------------------------- title: "Helm" url: https://materializeinc.github.io/materialize-monitoring/getting-started/helm/index.md -------------------------------------------------------------------------------- # Installing via Helm If you are not using the terraform module to install `materialize-monitoring`, you can use the provided Helm charts to install the monitoring stack in your Kubernetes cluster. For a short-lived install to troubleshoot a cluster, see [Ad-hoc install](../adhoc/) instead. > [!INFO] > The [Terraform modules](../terraform/) are the recommended path: they create the buckets and workload-identity bindings, derive every backend key from one `object_storage` object, and roll Alloy when its config changes. > > The charts are the full-fidelity surface — everything Terraform does is a layer over them — so reach for Helm when you need a setting the modules do not model, or when Terraform is not how you deploy. > > What you take on by installing directly is the `[consumer]` half of the [shared responsibility model](../../operating/production-best-practices/checklist/#shared-responsibility-model): cloud resources, secrets, version pinning, and the Alloy rollout below. ## Dependency: Installing CRDs `materialize-monitoring` relies on several Custom Resource Definitions (CRDs) to function properly. A second `materialize-monitoring-crds` Helm chart is provided to install these CRDs separately from the main `materialize-monitoring` chart, which is recommended to manage the lifecycle of these CRDs separately from the main chart. Install it first, then install `materialize-monitoring` with `--skip-crds`: ```bash helm install mzmon-crds oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-crds --namespace monitoring ``` The bundled Grafana Operator ships its own copy of the Grafana CRDs and offers no way to opt out of them, so without `--skip-crds` a fresh install of the main chart would create them behind the CRDs chart's back. See [Dependencies](../dependencies) for the full list of CRDs. ## Dependency: Setting Up Storage You will likely need to set up storage for your metrics and logs before you can start using `materialize-monitoring`. The specific steps for setting up storage will depend on your environment and the storage solution you choose. If you are using both external metric storage and external log storage, you will not need an object storage bucket. Two buckets (or containers) are the norm, one per backend — Loki and Thanos want different lifecycle rules, and per-bucket IAM keeps each backend out of the other's data. ### Cloud Managed Kubernetes Service (AWS EKS, Google Cloud GKE, Azure AKS, etc.) Grant access through **workload identity** rather than static keys. The binding is per-cloud and documented step by step, including the trust-policy subjects that must match the chart's rendered ServiceAccount names: - [Logs & Events > Storing](../../logs-and-events/storing/#granting-object-storage-access-workload-identity) — IRSA, GKE Workload Identity, and Entra Workload ID, in tabs - [Metrics > Storing](../../metrics/storing/#authentication) — the same three for Thanos Then name the backend in your values. **This is the step that most often goes wrong**: the chart's defaults are S3-shaped, so any other backend has to be named in three load-bearing places and none of them fails softly — the client is chosen by name and validated against a config that was never populated, so the component crash-loops. See [Selecting the backend](../../logs-and-events/storing/#selecting-the-backend). The example profiles do all of it correctly and are the shortest path: | Profile | Backend | |---|---| | `profiles/aws-example.values.yaml` | S3 + IRSA | | `profiles/gcp-example.values.yaml` | GCS + GKE Workload Identity | | `profiles/azure-example.values.yaml` | Blob + Entra Workload ID | | `profiles/aws-amp-fanout.values.yaml` | S3 for logs; metrics to the bundled Thanos **and** Amazon Managed Prometheus at once | > [!TIP] > Read the Azure profile's header before copying it. Entra Workload ID needs a pod label as well as the ServiceAccount annotation, and the two subcharts take that label under different keys — `loki.loki.podLabels` for Loki, `thanos.global.commonLabels` for Thanos, which has no `podLabels` of its own. ### On-Premises Kubernetes Cluster with Access to Cloud Object Storage (S3, GCS, Azure Blob Storage, etc.) Without a cloud identity provider there is no workload identity to bind, so credentials become **static keys supplied as a Secret** — the documented escape hatch rather than the recommended path. Both backends read them by reference: - Loki takes them under `loki.loki.storage.object_store.`; prefer supplying the values through `loki..extraEnvFrom` (a `secretRef`) over inlining them, since inline values render into a ConfigMap in plaintext. - Thanos takes them inside `thanos.global.objstore.config`, which becomes a Secret when `createSecret: true`. Any S3-compatible endpoint works, including Ceph RGW, MinIO, and rustfs. The chart's own tier-2 E2E runs against [rustfs](https://github.com/rustfs/rustfs). `profiles/s3-compatible.values.yaml` sets up both backends for one of these stores. Its header has the commands that create the two Secrets it reads: the Thanos objstore document, and a pair of environment variables for Loki. Write the endpoint as `HOST:PORT` with no scheme, because the objstore client rejects one. Set `insecure: true` for a store that serves plain HTTP. > [!WARNING] > The chart's validators grade credential handling and will tell you when a cloud backend has neither an identity annotation nor inline credentials — at which point the component falls back to ambient node credentials, which usually means it silently works in one cluster and fails in another. Read the render output rather than assuming silence. ## Customizing your Helm Installation The `materialize-monitoring` Helm chart is designed to be highly customizable, so you can easily integrate with your existing observability infrastructure. Typically, you would want to create a values.yaml file that has your specific configurations. You can start fresh, or start from the profiles in the chart's `profiles/` directory. > [!WARNING] > Be aware that when merging examples together that you do not have > multiple of the same key on the same level since they do not automatically merge. > YAML is whitespace sensitive. You must specify `-f YOUR_VALUES.yaml` in your `helm install`/`helm upgrade` command to apply these customizations. These are automatically overlaid on top of the default values of the chart, so you only need to specify the values that are different from the default. > [!INFO] > This documentation may refer to values in dotted notation (e.g., `component.subcomponent.key=value`) > which corresponds roughly to this YAML structure: > > ```yaml > component: > subcomponent: > key: value > ``` ### Get the profiles {#get-the-profiles} The profiles ship inside the chart package, so you don't need to download them one at a time. Pull the chart and unpack it: ```bash helm pull oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring --version VERSION --untar ``` Replace `VERSION` with the chart version you install. Every profile is then at `materialize-monitoring/profiles/`, and you can pass each one with `-f`. Pull the same version you install, because a profile can name values that exist only in that version. ### Choosing which components run, via tags Which subcharts install is driven by `tags`, and the semantics are **OR**: a component runs if *any* tag covering it is true. There is no need to disable a group before enabling one member. | Tag | Covers | |---|---| | `tags.default` | everything below except metrics-server. `true` by default | | `tags.bundled-backends` | Loki, Thanos, Alertmanager | | `tags.managed-grafana` | Grafana and grafana-operator | | `tags.pipeline` | both Alloy roles (agent + gateway) | | `tags.cluster-metrics` | kube-state-metrics, node-exporter, and metrics-server. metrics-server installs only through this tag or `tags.metrics-server`, because most clusters already run one | Per-chart overrides — `tags.loki`, `tags.thanos`, `tags.alloy-agent`, `tags.alloy-gateway`, `tags.grafana-standalone`, `tags.grafana-operator`, `tags.alertmanager`, `tags.kube-state-metrics`, `tags.metrics-server` — are OR'd on top. So a Loki-only install is `tags.default: false` plus `tags.loki: true`, and adding Thanos to a default install needs nothing at all. ### Disabling a Component Each subchart also has an `enabled` circuit breaker that takes **precedence over every tag**. Setting `loki.enabled: false` turns Loki off even with `tags.default: true`, which is the reliable way to subtract one component from an otherwise default install. ### Split the stack across releases {#split-releases} You can install the stack as several releases of the chart, so that each part has its own owner or upgrade cadence. For example, one release runs the pipeline, a second runs the backends, and a third runs Grafana. Each release then addresses Services that another release runs, such as the gateway writing to Thanos Receive. The render fails on an address that names a component its release doesn't run, unless `otherReleases` says that another release runs it. To split the stack, configure the releases as follows: 1. Pass every release the same values files. Both ends of a hop read some settings, such as the gateway's TLS, and each release renders and checks its end from its own values. 2. In each release, set `tags` to the components that release runs. 3. In each release, set `otherReleases.COMPONENT: true` for each component that another release runs. Replace `COMPONENT` with `alloy-agent`, `alloy-gateway`, `loki`, `thanos`, or `alertmanager`. 4. Keep each resource that belongs to no component in one release, and turn it off in the others. The following table lists the resources that belong to no component, the release that keeps them, and the values that you set to `false` in every other release: | Resources | Keep them in | Set to `false` elsewhere | |---|---|---| | PriorityClasses | Any one release | `priorityClasses.create` | | Grafana instance, dashboard folders, and data sources | The release that runs Grafana | `connections.grafana.enabled`, `connections.datasources.enabled`, and `dashboards.config.grafana.enabled` | | Materialize, Materialize operator, and CNI PodMonitors | The release that runs the gateway | `serviceMonitor.enabled` under each of `materialize.environmentd`, `materialize.environmentdSQL`, `materialize.clusterd`, `materialize.balancerd`, `materialize.console`, and `materialize-operator`, and `networking.cni.*.enabled` | Install the release that keeps the PriorityClasses first. The other releases warn that the classes have to exist already, and the API server rejects their pods until they do. The release that runs the gateway keeps the PodMonitors because it installs the PromQL alert rules, and it chooses which rules apply from the scrapes and components it runs itself. For the same reason, run kube-state-metrics, node-exporter, and Loki in that release to keep their alert rules. The log-derived alert rules install only in a release that runs both the gateway and Loki. For example, the following values split the stack into a pipeline release and a backends release that also runs Grafana. Each release also takes the shared values files. ```yaml # pipeline.values.yaml tags: default: false pipeline: true kube-state-metrics: true node-exporter: true otherReleases: loki: true thanos: true alertmanager: true connections: grafana: enabled: false datasources: enabled: false dashboards: config: grafana: enabled: false ``` ```yaml # backends.values.yaml tags: default: false bundled-backends: true managed-grafana: true otherReleases: alloy-agent: true alloy-gateway: true priorityClasses: create: false materialize: environmentd: serviceMonitor: enabled: false environmentdSQL: serviceMonitor: enabled: false clusterd: serviceMonitor: enabled: false balancerd: serviceMonitor: enabled: false console: serviceMonitor: enabled: false materialize-operator: serviceMonitor: enabled: false networking: cni: aws-vpc: enabled: false cilium: enabled: false ``` This layout runs Loki apart from the gateway, so Loki's own alert rules and the log-derived alert rules don't install. A default address names the component's namespace from the values, such as `thanos.namespaceOverride`. Because every release takes the same values files, each one resolves a component to the same namespace. ### Sizing The chart's defaults target a **medium** deployment, so the sizing profiles are deltas in either direction rather than a full configuration: Each backend has its own pair, because they size off different axes — Loki off log throughput, Thanos off active series and object count. Pick one from each row you deploy; they compose. | Profile | Use | |---|---| | `profiles/loki-small.values.yaml` | dev, or a constrained node pool | | `profiles/loki-large.values.yaml` | high-volume logging | | `profiles/thanos-small.values.yaml` | dev, or a constrained node pool | | `profiles/thanos-large.values.yaml` | high-cardinality metrics; also enables the Thanos query-frontend and repoints the datasource at it | | *(none)* | medium — the chart defaults, for either backend | | `profiles/loki-test.values.yaml` | CI only: SingleBinary Loki on local filesystem, no object storage | | `profiles/kind.values.yaml` | CI only: shrinks **every** workload to fit one `kind` node. Sizing only — see below | See Production Best Practices for the envelope each tier assumes: [logging throughput](../../operating/production-best-practices/loki/#sizing-the-logging-backend) for Loki, [active series and collections](../../operating/production-best-practices/thanos/#sizing-the-metrics-backend) for Thanos, including how to pick a tier from cluster inventory before you have any metrics to measure. #### The `kind` profile is sizing only, and composes last {#kind-profile} The chart defaults request roughly 33Gi of memory and 9 CPU in total, which no standard CI runner will schedule — pods sit Pending and it reads like a chart bug rather than a capacity one. `profiles/kind.values.yaml` cuts that to under 4Gi and about 1.6 CPU. It sets **container requests and limits, volume sizes, and cache allocations, and nothing else**: no feature toggles, no `tags`, and deliberately **no replica counts**. Compose it last so its sizes win: ```bash helm install mzmon . -f profiles/loki-test.values.yaml -f profiles/kind-tier1.values.yaml -f profiles/kind.values.yaml ``` > [!INFO] > **Why no replica counts.** `loki-test` *disables* Loki's distributed components by setting `replicas: 0` on them, so a replica count in a sizing overlay would switch them back on — and Loki then refuses to render at all, with `more than zero replicas configured for both the monolithic and distributed targets`. Separately, Loki's ingesters and Thanos Receive both sit at the replication-factor floor of 3, and the ring behaviour they provide is much of what an E2E run exists to prove. So this profile shrinks each pod and leaves the number of them to whoever owns the topology: three tiny pods, not one big one. > > Two sizing traps it has to work around, both of which fail as an OOM rather than as a small install if you only lower the request: Thanos Store Gateway holds a 2GB chunk pool and a 250MB index cache by default, and memcached derives its pod request from `allocatedMemory`. Both are shrunk explicitly. ### Other shape overlays | Profile | Use | |---|---| | `profiles/existing-grafana.values.yaml` | point at a Grafana you already run instead of installing one | | `profiles/grafana-postgres.values.yaml` | Grafana state in Postgres rather than SQLite — the production shape | | `profiles/grafana-pvc.values.yaml` | Grafana state on a PersistentVolume — durable, but still one replica. For a Helm-only install with no database | | `profiles/grafana-ingress.values.yaml` | make Grafana reachable over an Ingress, internal by default and TLS-terminated. Pair it with one of the two above | | `profiles/grafana-https.values.yaml` | have Grafana serve HTTPS itself, from the certificate `certificates.external` issues, for a Grafana behind an L4 `LoadBalancer` Service. See [Terminating TLS](../../dashboards/grafana/architecture/#terminating-tls) | | `profiles/scheduling.values.yaml` | node selector, tolerations, and priority-class names, fanned out to every subchart. Edit the four anchors at the top; see below | | `profiles/storage-class.values.yaml` | one StorageClass, aimed at the four workloads that claim a volume. Edit the single anchor at the top | | `profiles/no-storage-class.values.yaml` | for clusters with **no StorageClass**. Puts the same four workloads, and Grafana, on `emptyDir` and requests node disk for them instead. Grafana then keeps UI changes only until it restarts. Compose it last. On an existing install, delete the four StatefulSets first; see [StatefulSet fields that cannot be patched](../../operating/upgrading/#statefulset-fields-that-cannot-be-patched) | | `profiles/no-zone-spread.values.yaml` | for clusters with **no zone labels, or one zone**. Without it Thanos Receive and Loki's ingesters stay Pending; see [Production Best Practices](../../operating/production-best-practices/stack-wide/#fewer-than-two-zones) | | `profiles/s3-worm.values.yaml` | archive a copy of the logs to an S3 bucket with Object Lock, for compliance retention. The bucket and its lock are yours to create; the header has the steps, and see [Log compliance](../../operating/production-best-practices/log-compliance/) | | `profiles/s3-compatible.values.yaml` | Ceph RGW, MinIO, or another S3-compatible store with static keys, for both backends. The keys come from two Secrets you create; the header has the commands | | `profiles/adhoc.values.yaml` | the whole stack in one file: one replica of everything, no volumes, no NetworkPolicies. For troubleshooting, not production; see [Ad-hoc install](../adhoc/) | | `profiles/split-namespace.values.yaml` | one namespace per subchart. Changes every workload-identity subject; see [Namespace layout](../../operating/production-best-practices/stack-wide/#namespace-layout) | | `profiles/noop.values.yaml` | a release that creates no resources. Install it first, then turn components on over it one tag at a time | | `profiles/otel-metrics-fanout.values.yaml` | additional metric destinations (GCM, Datadog) with per-destination importance tiers | | `profiles/otlp-metrics-honeycomb.values.yaml` | a generic OTLP metrics backend | ### Hardened images and private registries {#registry-profiles} `profiles/registry/` repoints the stack at a hardened-image vendor and wires up the pull secret that vendor's registry needs. | Profile | Use | |---|---| | `profiles/registry/pull-secret.values.yaml` | name a `dockerconfigjson` Secret on every workload. Compose it first; it sets no registries, so it works for a mirror as well as a vendor | | `profiles/registry/mirror.values.yaml` | a private mirror of the upstream images — same binaries, different host | | `profiles/registry/chainguard.values.yaml` | Chainguard Images — `cgr.dev//` | | `profiles/registry/docker-hardened-images.values.yaml` | Docker Hardened Images — `docker.io//dhi-` | Compose the pull secret first, then one of the others: ```bash helm install mzmon . -f profiles/registry/pull-secret.values.yaml -f profiles/registry/mirror.values.yaml ``` #### Mirroring {#mirror-profile} A mirror changes where each image is pulled from and nothing about the image itself — same binary, same digest, same tag — so it is the least disruptive option here and the one most often mandated rather than chosen. It is also the only one that can carry the Alloy image, since that is a Materialize build rather than something a hardened-image vendor publishes. Check first whether you need chart values at all. Mirroring happens at two layers, and only one is a chart concern: - **At the node** — containerd's `registry.mirrors` / `hosts.toml`, or an `ImageDigestMirrorSet` on OpenShift — redirects pulls transparently while manifests keep naming the upstream registry. If you have this, change nothing. Rewriting references in values bypasses the indirection you set up and leaves the same mapping maintained in two places. - **In the manifest** — what `mirror.values.yaml` does. Needed when you cannot reach node configuration, which covers most managed control planes, or when policy requires the mirror to be visible in the manifest rather than inferred from cluster state. The profile assumes one project per upstream registry, the Harbor proxy-cache and Artifactory remote-repository shape, and this stack pulls from four: `docker.io`, `quay.io`, `registry.k8s.io`, and `ghcr.io`. Four anchors at the top carry those; if your mirror is flat and path-preserving instead, set all four to the same value. Four images spell their registry inside `image.repository` rather than taking a separate `registry` key, so anchors cannot reach them and they are written out in full — they are marked `HAND-EDIT` at the line, and missing one is the likely way to get this half-applied. The pull-secret overlay is a separate file because the subcharts do not agree on which global to read. Alloy reads `global.image.pullSecrets`; Thanos, Grafana, kube-state-metrics and node-exporter read `global.imagePullSecrets`; and Loki, grafana-operator, Alertmanager and metrics-server read neither and need their own key. Setting only `global.imagePullSecrets` leaves Loki — the largest pod count in the release — pulling anonymously, which surfaces as `ImagePullBackOff` on the ingesters long after `helm install` reports success. > [!WARNING] > `global.imageRegistry` is not the one-line version of the mirror profile. > It misses Alloy (which reads `global.image.registry`) and Alertmanager and metrics-server (which carry the host inside `image.repository` and have no `registry` key) — each of which then keeps pulling from the internet rather than erroring. > In the Loki chart it also outranks every per-component `image.registry`, so setting it alongside a profile silently overrides everything that profile chose. Each vendor profile's header lists what it deliberately leaves on an upstream registry. Common to both: the Alloy image is a Materialize build carrying the custom components the pipelines are written against, so it is a rebuild rather than a retag — which is why a mirror can carry it and a hardened rebuild cannot. The three Jobs this chart renders itself — the `pre-delete` cleanup hook and the two Alloy pre-install validators — read both globals directly, so the pull-secret overlay covers them with nothing extra to set. Confirm the cleanup hook's image is pullable before you rely on it: it runs at `pre-delete`, so an image it cannot pull hangs `helm uninstall` on the hook rather than failing an install you can retry. > [!WARNING] > Hardened images run as their own non-root UID, which is generally not the UID the upstream chart writes into `securityContext` — Grafana's chart assumes 472, Loki's assumes 10001. > A mismatch renders and schedules cleanly and then crash-loops on a volume the container cannot write. > Check the UID before pointing one of these at a cluster that holds data. > > These images also ship no package manager, so `grafana.plugins` — which installs at container start — silently gets you a Grafana without those plugins. Bake them into a derived image instead. #### Why there is no Bitnami profile {#no-bitnami-profile} Bitnami Secure Images publishes hardened builds of nearly everything in this stack, so its absence here is deliberate rather than an oversight. The two profiles above are retags: Chainguard and DHI rebuild the upstream image and keep its entrypoint, arguments, and filesystem layout, so a values change is the whole change. Bitnami's images are built for *Bitnami's own charts* — Bitnami entrypoint scripts, `/opt/bitnami` paths, UID 1001 — while the subcharts here are the upstream Grafana, Thanos, and prometheus-community ones, which pass their own arguments and mount config at upstream paths. Reconciling that is per-component work no values file can express, which is the part a profile could not have finished for you. Three concrete things sit on top of that, if you are weighing the port anyway: - Bitnami publishes no `loki-canary`, and the canary is **on by default** — so an otherwise-complete swap is broken until you also set `loki.lokiCanary.enabled: false`. There is likewise no `k8s-sidecar`, `prometheus-config-reloader`, or `access-log-exporter`, though all three are disabled by default here. - Bitnami tags carry a distro and revision suffix (`3.7.6-debian-12-r0`) rather than plain semver. The `prometheus-node-exporter` chart feeds its tag to `semverCompare`, so that suffix **fails the render outright** with `invalid semantic version` until you also set `node-exporter.version` to the upstream version the tag was built from. - The catalog is subscription-gated. The free `docker.io/bitnamisecure` namespace carries a small sampler that includes none of these applications, so there is no anonymous tag to pin against. ### Disabling a Component If you want further control of the managed components, you can selectively disable components in the `materialize-monitoring` Helm chart by setting the `enabled` field for that component to `false` in your `YOUR_VALUES.yaml` file or via `--set` in your `helm install`/`helm upgrade` command. #### Scheduling: where each subchart wants it {#scheduling-profile} The subcharts disagree about where scheduling settings go, which is the whole reason this is a profile rather than three `--set` flags. `thanos.global` covers everything Thanos runs; `loki.defaults` covers everything Loki runs *except* three components that render from their own templates; Alloy puts it under `controller`; the rest take it at the top level. A selector written to the wrong one of those renders perfectly and does nothing. **Edit the four anchors at the top of the file and nothing else** — every fan-out site aliases them. The two class names match the chart's defaults, so they are a no-op until you change them; aliasing both the created classes and all eleven references off one anchor is what makes renaming them a two-line edit that cannot half-apply. > [!WARNING] > **A nodeSelector and a toleration are not two spellings of the same idea, and per-node collectors need them treated differently.** > > A **nodeSelector narrows** where a pod may run. On a DaemonSet whose job is to observe every node, that is a silent blind spot rather than a placement preference — an `alloy-agent` constrained to a workload pool simply stops collecting logs from everywhere else, and no dashboard shows a hole where the nodes should be. **Tolerations widen** placement, which is exactly what a DaemonSet wants so it can reach tainted, spot, and system pools. > > So `alloy-agent` receives tolerations and never a selector. `node-exporter` receives neither: its chart default already tolerates *every* `NoSchedule` taint, and a toleration list would **replace** that rather than extend it, because Helm merges maps but overwrites lists. `metrics-server` takes a selector and tolerations but deliberately no priority class — it stays on the upstream `system-cluster-critical`, because it backs the metrics API that HPAs and the Materialize Console read, which makes it cluster plumbing rather than monitoring. **Terraform users need neither file.** The module's `node_selector`, `tolerations`, and `storage_class` variables carry the same maps and take the values directly. The two copies are kept honest by `make terraform-render`, which plans each example, renders the chart against the values it composed, and asserts that every pod template carries the selector and every `volumeClaimTemplate` carries the class — so a subchart that renames a key fails in this repository rather than in an apply. ## Initial Installation CRDs first, then the chart with `--skip-crds`. Both releases go in the same namespace; the CRDs are cluster-scoped, so that namespace only holds the release metadata. ```bash helm install mzmon-crds oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-crds \ --namespace monitoring --create-namespace \ --version X.Y.Z --wait ``` ```bash helm install mzmon oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring \ --namespace monitoring --skip-crds \ --version X.Y.Z \ -f my-values.yaml \ --timeout 15m ``` The dashboards ship in charts of their own, one per Grafana folder, and the main chart doesn't install them. Without these releases, Grafana comes up with its data sources and folders but no dashboards: ```bash helm install mzmon-dashboards oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-dashboards \ --namespace monitoring \ --version X.Y.Z helm install mzmon-dashboards-infra oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-dashboards-infra \ --namespace monitoring \ --version X.Y.Z helm install mzmon-dashboards-meta oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-dashboards-meta \ --namespace monitoring \ --version X.Y.Z ``` The first carries the Materialize environment dashboards, the second the platform underneath it, and the third the monitoring stack itself. Install the ones you want. Their defaults match a `materialize-monitoring` release named `mzmon`. For a different release name, see [Importing dashboards via Grafana Operator](../../dashboards/grafana/grafana-operator/#importing-dashboards-via-grafana-operator). Pin `--version` on each. The CRDs chart, the main chart, and the dashboards charts have separate lifecycles, so they don't share a version. The dashboards charts share one version among themselves. For the pairs that are tested together, see [Compatibility](../../reference/compatibility/). A few flags earn their place: - **`--timeout 15m`.** Helm's default is 5 minutes, and a first install brings up Loki, Thanos, Grafana, Alertmanager, and both Alloy roles together. If it still times out, **do not raise it again** — a timeout almost always means a pod is broken rather than slow. See [Troubleshooting](../../operating/o11y-troubleshooting/#start-here-a-timeout-is-not-a-duration-problem). - **`--wait` on the CRDs release only.** The main chart runs pre-install validation Jobs; if you add `--wait` there, add `--wait-for-jobs` too, or their verdict is never observed and a bad config rolls anyway. - **Not `--atomic`** on a first install. A rollback destroys the evidence of which component failed, and this stack has enough moving parts that the diagnostic is usually worth more than the cleanup. ### Verify it came up The chart's validators print to `NOTES.txt`, so read the install output rather than assuming silence means success — warnings there are the earliest signal that a backend key or a quorum setting is wrong. ```bash kubectl --namespace monitoring get pods kubectl --namespace monitoring get grafana,grafanadatasource,grafanamanifest ``` Grafana is `ClusterIP`, so reach it with a port-forward: ```bash kubectl --namespace monitoring port-forward svc/grafana 3000:80 ``` The admin credentials are in the `grafana` Secret (keys `admin-user` and `admin-password`). > [!INFO] > The chart generates that password and **reuses it across `helm upgrade`**, because the Grafana chart looks the existing Secret up before generating. > > That lookup returns nothing during `helm template` and `--dry-run`, so any render-only pipeline — most [GitOps](../gitops/) setups — regenerates the password on **every sync**. Supply your own Secret and set `grafana.grafana.admin.existingSecret` there. The Terraform module does exactly this, for exactly this reason. ### Override a validation check {#footgun-overrides} The chart checks your values every time it renders. An error fails the render with `Validation failed:`, and a warning prints in the install notes. Fix the value when you can, because each check catches a configuration that installs cleanly and then fails quietly. If a check fails because another release of this chart runs the component, set `otherReleases` instead. For more information, see [Split the stack across releases](#split-releases). When a check is wrong about your install, `footgunOverrides` changes how the chart reports it. Each entry is a regular expression that matches anywhere in a message: `downgradeErrors` : An error that matches is reported as a warning instead, and the render continues. An expression that matches no error adds a warning of its own. `silenceWarnings` : A warning that matches is dropped. A downgraded error is never dropped. For example, the chart treats any in-cluster URL that contains `thanos-receive` as its own Thanos. If a different chart installed your Thanos with a Service of that name, the following values let the release render: ```yaml footgunOverrides: downgradeErrors: - 'pipeline\.destinations\.thanos\.url points at the bundled Thanos' ``` The overrides can't change a required value, a template that fails while it renders, or a subchart's own checks. With the Terraform module, set them in `additional_values`. For the full rules, see [Footgun overrides](../../reference/helm/materialize-monitoring-values/#footgun-overrides). ## After changing values **Restart Alloy after any pipeline or filter change.** This is the one place the chart cannot own its own rollout, and it is the reverse of what a chart normally guarantees: a `helm upgrade` that reports success can leave both Alloy roles serving the *previous* configuration indefinitely. ```bash kubectl --namespace monitoring rollout restart deployment/alloy-gateway daemonset/alloy-agent ``` The mechanism, and why a reload cannot substitute for a restart, is in [Production Best Practices](../../operating/production-best-practices/alloy/#restart-after-a-config-change). The short version: the pipeline's `-env` ConfigMaps are consumed with `envFrom`, and environment variables are fixed at container start, so neither Alloy's `/-/reload` nor a config-reloader sidecar can pick up a filter change. ## Going to production [Production Best Practices](../../operating/production-best-practices/checklist/) is the deployment checklist, ordered by what goes wrong if you skip an item and tagged with who owns it: the chart, you as the chart consumer, or you as the operator. On the Helm path every `[consumer]` item is yours, including the buckets and workload-identity bindings that the [Terraform modules](../terraform/) would otherwise create. ## Upgrading and uninstalling - [Upgrading](../../operating/upgrading/) — ingester rollouts are ordered and readiness-gated, so budget roughly a minute per ingester and raise your client's timeout accordingly. - [Uninstalling](../../operating/uninstalling/) — a `pre-delete` hook now clears the Grafana custom resources while grafana-operator is still running, so an ordinary `helm uninstall` no longer deadlocks on finalizers with no remover. Read this before tearing down a release **installed before that hook existed**, or one you plan to remove with `--no-hooks`. ## Reference - [Chart values](../../reference/helm/materialize-monitoring-values/) — every value, generated from `values.yaml` - [CRDs](../../reference/crds/) — what the CRDs chart installs - [Compatibility](../../reference/compatibility/) — chart, CRDs chart, and Terraform module versions -------------------------------------------------------------------------------- title: "Ad-hoc install" url: https://materializeinc.github.io/materialize-monitoring/getting-started/adhoc/index.md -------------------------------------------------------------------------------- # Install an ad-hoc stack for troubleshooting This page is for you if you need a Grafana with metrics in it, on a cluster you don't run day to day, as soon as possible. Typical cases are a day-1 install, an on-premises cluster you haven't seen before, or a node pool that Materialize already fills. The install uses `profiles/adhoc.values.yaml`, one file that covers the whole stack: - Every component runs one replica, with a replication factor of 1. - Nothing claims a PersistentVolume, so the cluster doesn't need a StorageClass. - Loki runs as one process on its own filesystem, so it needs no bucket. - Topology spread constraints are removed, so pods schedule on nodes with no zone labels. - NetworkPolicies are off. Losing a pod loses that pod's data, so don't run this profile in production. Grafana keeps users, API keys, and anything else changed in its UI only until it restarts. The dashboards, data sources, and folders the charts install come back on their own. For a lasting install, see [Helm](../helm/). ## Before you begin You need the following: - `helm` and `kubectl`, with access to the cluster. - An object store bucket for Thanos. Thanos has no local-disk mode that works across its components, so even an ad-hoc install needs one. S3, an S3-compatible store such as Ceph RGW or MinIO, GCS, and Azure Blob Storage all work. - An address you can reach Grafana on, such as a node IP. ## Install the stack 1. Pull the chart, which includes every profile: ```bash helm pull oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring --version VERSION --untar ``` Replace `VERSION` with the chart version you install. 1. Create the Secret that Thanos reads its bucket from. For S3 or an S3-compatible store: ```bash kubectl create namespace monitoring kubectl create secret generic thanos-objstore-config --namespace monitoring \ --from-file=objstore.yml=/dev/stdin <<'EOF' type: S3 config: bucket: BUCKET endpoint: HOST:PORT access_key: ACCESS_KEY secret_key: SECRET_KEY insecure: true EOF ``` Replace the following: - `BUCKET`: the bucket for Thanos blocks. - `HOST:PORT`: the store's endpoint, with no scheme. - `ACCESS_KEY` and `SECRET_KEY`: the store's credentials. Set `insecure: false` when the store serves HTTPS. GCS and Azure take the same Secret with their own `type` and `config`. 1. Install the CRDs: ```bash helm install mzmon-crds oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-crds \ --namespace monitoring --version VERSION --wait ``` 1. Install the chart with the profile: ```bash helm install mzmon oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring \ --namespace monitoring --skip-crds --version VERSION \ -f materialize-monitoring/profiles/adhoc.values.yaml \ --timeout 15m ``` The install output warns about every production setting the profile turns off. Read past those for anything else. 1. Install the dashboards, which ship in charts of their own, one per Grafana folder: ```bash helm install mzmon-dashboards oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-dashboards \ --namespace monitoring --version VERSION helm install mzmon-dashboards-infra oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-dashboards-infra \ --namespace monitoring --version VERSION helm install mzmon-dashboards-meta oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-dashboards-meta \ --namespace monitoring --version VERSION ``` The dashboards charts share one version, but the other charts each have their own, so replace `VERSION` in each command. Skip any dashboards chart you don't want: `-infra` carries the platform dashboards, and `-meta` carries the ones that watch the monitoring stack itself. 1. Check that the pods are running and that the dashboards reached Grafana: ```bash kubectl get pods --namespace monitoring kubectl get grafana,grafanamanifest --namespace monitoring ``` ## Reach Grafana A port-forward works, but it drops connections when a dashboard opens many requests at once. On a cluster whose nodes you can reach, a NodePort Service is steadier. `kubectl expose` copies the selector and ports from the existing Service: ```bash kubectl expose service grafana --namespace monitoring --name grafana-nodeport --type NodePort kubectl get service grafana-nodeport --namespace monitoring -o jsonpath='{.spec.ports[0].nodePort}' ``` Open `http://NODE_IP:NODE_PORT`. Replace `NODE_IP` with the address of any node, and `NODE_PORT` with the port the second command prints. The admin password is in the `grafana` Secret: ```bash kubectl get secret grafana --namespace monitoring -o jsonpath='{.data.admin-password}' | base64 -d ``` Delete the `grafana-nodeport` Service when you're done; Helm doesn't manage it. ## Adjust for the cluster The profile marks each setting you're likely to change with `SWITCH`. Edit your copy of `adhoc.values.yaml`, or pass the key with `--set`: NetworkPolicies : Each key that renders one is listed, so you can turn a single component's policy back on. If you turn them on, see [grafana-operator times out reaching Grafana](../../operating/o11y-troubleshooting/#grafana-operator-timeout) for clusters that run NodeLocal DNSCache. DNS Service name : Loki's gateway resolves through a Service named `kube-dns`. Set `loki.global.dnsService` when the cluster names it something else. For more information, see [`host not found in resolver`](../../operating/o11y-troubleshooting/#host-not-found-in-resolver). metrics-server : Off, as at the chart default, because most clusters already serve `metrics.k8s.io`. Turn it on when `kubectl get apiservice v1beta1.metrics.k8s.io` finds nothing. Loki : Turn it off when the cluster can't fit one more pod. Also turn off the Alloy gateway's Loki destination, which the profile lists beside it. To keep logs in object storage, add `-f materialize-monitoring/profiles/s3-compatible.values.yaml` after `adhoc`. Its header has the commands for the two Secrets it reads. ## Clean up Uninstall the releases in the reverse order you installed them: ```bash helm uninstall mzmon-dashboards-meta --namespace monitoring helm uninstall mzmon-dashboards-infra --namespace monitoring helm uninstall mzmon-dashboards --namespace monitoring helm uninstall mzmon --namespace monitoring helm uninstall mzmon-crds --namespace monitoring ``` The following outlive the releases: - The CRDs, which the CRDs release leaves in place. Other installs in the cluster might use them. - The Secrets you created, and any NodePort Service. - The Thanos blocks in the bucket. For when and how to remove the CRDs, see [Uninstalling](../../operating/uninstalling/). -------------------------------------------------------------------------------- title: "Architecture" url: https://materializeinc.github.io/materialize-monitoring/metrics/architecture/index.md -------------------------------------------------------------------------------- # Metrics Architecture This page describes how metrics move through `materialize-monitoring`. In the bundled stack, every metric enters through the [`alloy-gateway`](../../o11y-glossary/#alloy), which scrapes or receives it, processes it, and writes it to one or more destinations. The default destination is the bundled [Thanos](../../o11y-glossary/#thanos). Thanos keeps recent data in Receive, keeps everything older as blocks in object storage, and answers [PromQL](../../o11y-glossary/#promql) for Grafana and for the Thanos Ruler. The other pages in this section cover each stage in depth: | Page | Covers | |---|---| | [Collecting](../collecting/overview/) | The five ways metrics reach the gateway, and its ingress ports | | [Scraping](../scraping/) | Monitor discovery, the kubelet scrape, and scrape configurations for a separately run Prometheus | | [Storing](../storing/) | Destinations, importance tiers, the denylist, object storage, retention and downsampling | | [Querying](../querying/) | PromQL through Grafana and the Thanos Query API | | [Alerting > Configuring](../../alerting/configuring/) | The Thanos Ruler, and how `PrometheusRule` resources reach it | The production checklist for this backend is [Production Best Practices > Metrics (Thanos)](../../operating/production-best-practices/thanos/). The gateway's component-level pipeline is in the [metrics pipeline reference](../../reference/development/pipelines/metrics/). ## End-to-end dataflow ```mermaid flowchart TB src["Sources
monitor targets · kubelets
cloud APIs · pushers"] graf["Grafana"] ruler["Thanos Ruler"] gw["alloy-gateway"] query["Thanos Query"] am["Alertmanager"] other["Other destinations
AMP · OTLP backends"] recv["Thanos Receive
RF 2"] sg["Thanos Store Gateway"] bucket[("Object storage")] comp["Thanos Compactor"] src --> gw ruler -->|"rule results"| gw ruler -->|"PromQL"| query ruler -->|"alerts"| am graf -->|"PromQL"| query gw -.->|"optional"| other gw -->|"remote-write"| recv query -->|"recent"| recv query -->|"historical"| sg recv -->|"2h blocks"| bucket sg -->|"read blocks"| bucket bucket <-->|"rewrite blocks"| comp ``` Writes flow downward. The gateway [collects](#collection) from every source, runs the shared [processing pipeline](#gateway-pipeline), and writes each destination its own filtered copy. [Thanos Receive](#thanos-receive) replicates each write to three pods, serves the recent window, and uploads a block to object storage every two hours. The [Compactor](#block-lifecycle) merges, deduplicates, downsamples, and expires those blocks in place. Reads flow upward from Grafana into [Thanos Query](#read-path), which merges recent data from Receive with historical data from the Store Gateway. The [Thanos Ruler](#rule-evaluation) queries the same path, sends alerts to Alertmanager, and writes its results back into the gateway, so rule output takes the same route as every other metric. ## Collection {#collection} All metric collection runs in the `alloy-gateway` Deployment. The `alloy-agent` DaemonSet collects logs only. Container metrics come from the kubelet, which the gateway scrapes directly, so no per-node metrics agent is needed. | Source | Reaches the gateway by | Default | |---|---|---| | `PodMonitor` and `ServiceMonitor` targets | Scrape, discovered from every monitor in the cluster | On. The chart ships monitors for Materialize, kube-state-metrics, node-exporter and each stack component | | Kubelet `/metrics/cadvisor` | Scrape of every node's kubelet | On, every 60s | | kube-proxy | Scrape, discovered by a server-side pod selector | On where kube-proxy runs | | Cloud provider monitoring APIs | Pull through an in-process exporter | Off | | Prometheus remote-write | Push to `:9090` | Listener on | | OTLP | Push to `:4317` (gRPC) or `:4318` (HTTP) | Listener on | | Thanos Ruler and Loki Ruler | Remote-write push to `:9090` | On | The gateway replicas form an [Alloy cluster](https://grafana.com/docs/alloy/latest/get-started/clustering/). Every scrape and pull is clustered, so each target has one owner among the replicas. Adding a replica spreads scrape load without duplicating samples. A cloud provider's API is therefore called once per interval, however many replicas run. *See more:* [Collecting](../collecting/overview/), [Scraping](../scraping/), and [Cloud Provider Metrics](../collecting/cloud-provider-metrics/). ### Inside the gateway {#gateway-pipeline} ```mermaid flowchart TB scrape["Scrapes and pulls
monitors · kubelet · kube-proxy · providers"] rw["prometheus.receive_http :9090
fills job / instance on pushed samples"] otlpin["otelcol.receiver.otlp
:4317 / :4318"] bridge["inputBridge
Prometheus → OTLP"] subgraph proc["Shared processing"] transform["inputMetricProcessor"] deny["inputMetricDeny
denyMetrics"] limit["memory_limiter
refuses at 85%"] batch["batch"] transform --> deny --> limit --> batch end egress{"egress seam"} subgraph promdest["Per remote-write destination"] out["outputBridge
OTLP → Prometheus"] tierp["tier filter
minMetricImportance"] wal["prometheus.remote_write
own WAL · cluster external label"] out --> tierp --> wal end subgraph oteldest["Per OTLP exporter"] tiero["tier filter
minMetricImportance"] exp["otlp · googlecloud · datadog exporter"] tiero --> exp end scrape --> bridge rw --> bridge bridge --> transform otlpin --> transform batch --> egress egress --> out egress -.->|"when enabled"| tiero ``` Prometheus input is bridged into OTLP at the start, processed by `otelcol.processor.*` components, and bridged back for remote-write. Every scrape keeps the target's metric type, so a counter or histogram arrives typed at an OTLP destination. The memory limiter refuses new data at 75% of its limit, and the refusal propagates back to the receivers as backpressure. The stream forks after the shared stages. Each destination has its own importance-tier filter **upstream** of its writer. A remote-write destination's write-ahead log therefore holds only the series that destination accepts, and a stuck destination backs up its own WAL and no other. Each remote-write destination also stamps a `cluster` external label. *See more:* [Storing > The remote-write destinations](../storing/#the-remote-write-destinations), [Controlling what each destination stores](../storing/#controlling-what-each-destination-stores), and the [metrics pipeline reference](../../reference/development/pipelines/metrics/). ## Thanos write path {#write-path} ### Thanos Receive {#thanos-receive} ```mermaid flowchart TB gw["alloy-gateway
thanos destination"] svc["thanos-receive Service :10908"] router["any Receive pod
routes the request"] ring(("ketama hashring
2 owners per series")) own0["receive-0
head + WAL"] own1["receive-1
head + WAL"] own2["receive-2
head + WAL"] bucket[("Object storage
one 2h block per pod
receive_replica=<pod>")] gw -->|"remote-write"| svc --> router --> ring ring --> own0 ring -->|"gRPC :10901"| own1 ring -->|"gRPC :10901"| own2 own0 --> bucket own1 --> bucket own2 --> bucket ``` Receive runs in `standalone` mode, where every pod both routes and ingests. The pod that takes a request hashes each series onto an auto-generated ketama hashring and forwards it to the pods that own it. With replication factor 2, each series has two owners, and a write succeeds after one of them accepts it. At the default 3 replicas, each series lives on 2 of the 3 pods. The `thanos-large` profile runs 6, and each series then lives on 2 of the 6. Each pod keeps its head and WAL on an `emptyDir`, cuts a block every two hours, and uploads it to object storage. Local retention is 6h, which serves the recent window to queries until the Store Gateway has the uploaded blocks. Every pod uploads its own copy of what it holds, labelled with its own `receive_replica`. The [Compactor](#block-lifecycle) later merges those copies into one. *See more:* [Production Best Practices > Receive](../../operating/production-best-practices/thanos/#receive-replication) for the quorum table and zone spread, and [Storage: ephemeral by default](../../operating/production-best-practices/thanos/#thanos-ephemeral-storage) for why Receive has no volume. ## Object storage and the block lifecycle {#block-lifecycle} ```mermaid flowchart TB b0["2h block
receive-0"] b1["2h block
receive-1"] b2["2h block
receive-2"] merge["deduplicate on receive_replica
vertical compaction"] levels["compact 8h → 2d → 14d"] ds["downsample
5m from ≥ 40h · 1h from ≥ 10d"] ret["retention
raw 30d · 5m 90d · 1h 365d"] del["marked for deletion
removed after 48h"] b0 --> merge b1 --> merge b2 --> merge merge --> levels --> ds levels -->|"replaced sources"| del ds --> ret --> del ``` Object storage is the only durable copy of metric data. Every other Thanos component either writes blocks into it or reads them out. The Compactor is a **singleton**, because two compactors against one bucket corrupt it. It works only on blocks older than its 30m consistency delay, so it never reads a block whose upload is still in progress. | Stage | What happens | |---|---| | Deduplicate | Blocks are grouped by their external labels with `receive_replica` removed, so each replica's copy of a window lands in one group. Vertical compaction merges them into a single block and drops identical samples | | Compact | Blocks are merged into longer ranges, 2h into 8h, 8h into 2d, and 2d into 14d | | Downsample | A raw block spanning 40h or more gets a 5m copy. A 5m block spanning 10d or more gets a 1h copy | | Retain | Each resolution expires on its own horizon, by default raw 30d, 5m 90d and 1h 365d | | Delete | A block that a compaction replaced, or that retention expired, is marked for deletion and removed 48h later. The delay lets the Store Gateway load the replacement before the source disappears | Vertical compaction also absorbs the few seconds of overlap Receive writes when a pod restarts. Without it, the first overlap halts the Compactor, and a halted Compactor runs none of the stages above. The Store Gateway reads blocks directly from the bucket and keeps an index-header cache on its own volume. *See more:* [Storing > Retention and downsampling](../storing/#retention-and-downsampling), [Production Best Practices > Retention & compaction](../../operating/production-best-practices/thanos/#thanos-retention-compaction), and [Thanos Compactor halted](../../operating/runbooks/thanos/#thanos-compactor-halted). ## Read path {#read-path} ```mermaid flowchart TB graf["Grafana
Thanos datasource"] qfe["Thanos Query Frontend
splitting · result cache"] query["Thanos Query
deduplicates on
receive_replica · ruler_replica · prometheus_replica"] receive["Thanos Receive
head + last 6h of blocks"] store["Thanos Store Gateway
uploaded blocks"] ruler["Thanos Ruler"] bucket[("Object storage")] graf -->|"PromQL :9090"| query graf -.->|"thanos-large: PromQL"| qfe qfe -.-> query query -->|"StoreAPI gRPC"| receive query -->|"StoreAPI gRPC"| store query -->|"rules · alerts gRPC"| ruler store --> bucket ruler -->|"PromQL :9090"| query ``` Thanos Query presents a Prometheus-compatible API and holds no data of its own. It discovers Receive, the Store Gateway and the Ruler through DNS SRV lookups, and fans each query out over gRPC. A query that spans recent and historical data reads both, and Query merges the overlap. It then deduplicates on the replica labels. Each Receive replica's copy of a series, each Ruler replica's copy of a rule result, and a deduplicated block from the bucket all collapse to one series. Grafana's `Thanos` datasource points at Thanos Query. The `thanos-large` profile enables the Query Frontend and moves the datasource to it, while the Ruler stays on Thanos Query, so rules never evaluate against a cached result. *See more:* [Querying](../querying/) and [Alerting > Configuring](../../alerting/configuring/#the-rulers-do-not-follow-the-query-frontend). ## Rule evaluation {#rule-evaluation} ```mermaid flowchart TB pr["PrometheusRule resources
flavor ≠ logql"] sidecar["import sidecar
lists every 60s · reloads"] ruler["Thanos Ruler
2 replicas · stateless"] query["Thanos Query"] am["Alertmanager
every replica"] gw["alloy-gateway :9090"] receive["Thanos Receive"] pr --> sidecar --> ruler ruler -->|"PromQL"| query ruler -->|"alerts"| am ruler -->|"ALERTS · recording results"| gw gw --> receive ``` The Thanos Ruler is stateless. It evaluates by querying Thanos Query, keeps no TSDB of its own, and remote-writes every result to the gateway. Rule output therefore reaches every destination the gateway writes to, and not only the bundled Thanos. Both replicas evaluate every rule. Each stamps its own `ruler_replica`, Thanos Query deduplicates on that label, and Alertmanager deduplicates the notifications. The gateway fills in `job="thanos-ruler"` and a constant `instance` on the Ruler's samples. The two replicas' series then differ only in `ruler_replica`, which is the label Query deduplicates on. The Loki Ruler's recording rules arrive the same way, as remote-write to the gateway. *See more:* [Alerting > Configuring](../../alerting/configuring/) for rule delivery and stateless mode, and [Alert Architecture](../../alerting/architecture/) for what Alertmanager does next. ## Destinations beyond the bundled Thanos {#other-destinations} The gateway writes to any number of destinations at once, each filtered to its own importance tier. The bundled Thanos is the default entry in that map, and it can be removed. | Profile | Shape | |---|---| | `aws-amp-fanout` | Full fidelity in the bundled Thanos, and only essential series in Amazon Managed Prometheus | | `otlp-metrics-honeycomb` | Gateway metrics to a generic OTLP backend | | `otel-metrics-fanout` | Several OTLP backends at once, each on its own tier | A deployment that already runs its own Prometheus can skip the gateway entirely and scrape Materialize with the configurations on [Scraping](../scraping/). *See more:* [Storing > Other Metric Storage Backends](../storing/#other-metric-storage-backends). ## See more - [Thanos components](https://thanos.io/tip/components/receive.md/) (official): Receive, Store, Compact, Query, Query Frontend and Rule. - [Thanos Receive hashrings and replication](https://thanos.io/tip/proposals-done/201812-thanos-remote-receive.md/) (official). - [Thanos Compactor](https://thanos.io/tip/components/compact.md/) (official): compaction levels, vertical compaction and downsampling. - [Alloy clustering](https://grafana.com/docs/alloy/latest/get-started/clustering/) (official). - [Metrics pipeline reference](../../reference/development/pipelines/metrics/): the gateway's pipeline, component by component. - [Observability glossary](../../o11y-glossary/): definitions for the vocabulary used on this page. -------------------------------------------------------------------------------- title: "Overview" url: https://materializeinc.github.io/materialize-monitoring/metrics/collecting/overview/index.md -------------------------------------------------------------------------------- # Collecting Metrics Metrics reach `materialize-monitoring` by the paths in the following table, and most deployments use more than one. | Path | Use it when | Direction | |---|---|---| | [Prometheus Operator](../prometheus-operator/) | An application in the cluster ships — or can be given — a `ServiceMonitor` or `PodMonitor` | The gateway **pulls** | | [Prometheus Remote Write](../prometheus-remote-write/) | You already run Prometheus and want it to ship into this stack | Something **pushes** | | [OpenTelemetry Endpoint](../opentelemetry-endpoint/) | You run an OpenTelemetry Collector, or an application that emits OTLP directly | Something **pushes** | | [Prometheus Scraper](../prometheus-scraper/) | You run your own Prometheus and this stack is not in the path at all | Your Prometheus **pulls** | | [Cloud Provider Metrics](../cloud-provider-metrics/) | The metadata database or the buckets are managed by AWS, Google Cloud or Azure, and their CPU, storage or burst headroom is wanted beside the rest | The gateway **pulls** from the provider's API | | [PostgreSQL exporter](../postgresql-exporter/) | The metadata database or Grafana's database is PostgreSQL, and what it reports about itself is wanted: connections, transaction-ID age, vacuum, and the size of every database on the instance | The gateway **pulls** from an exporter the chart runs | Everything but the Prometheus Scraper lands in the same place: the `alloy-gateway` pipeline, which processes and enriches everything uniformly before writing it to [storage](../../storing/). How it got in makes no difference downstream. ## The gateway's ingress ports Three listeners, on the `alloy-gateway` Service: | Port | Component | Accepts | |---|---|---| | `4317` | `otelcol.receiver.otlp` | OTLP over gRPC — **metrics and logs** | | `4318` | `otelcol.receiver.otlp` | OTLP over HTTP — **metrics and logs** | | `9090` | `prometheus.receive_http` | Prometheus remote-write | (A fourth, `3100`, takes Loki-format log pushes — see [Logs & Events > Collecting](../../../logs-and-events/collecting/).) > [!WARNING] > **None of these ports authenticate their callers**, and they are reachable from the whole cluster by default, because usually something in it legitimately pushes telemetry. > NetworkPolicy is the only access control here. Narrow the ingest ports if nothing in your cluster needs them — see [Securing](../../../operating/securing/#the-cluster--the-stack). ## Serving TLS on them The listeners are split across the two pipeline trees, and **securing one does not secure the other**: | Listener | Governed by | |---|---| | `loki.source.api` (`3100`), `otelcol.receiver.otlp` (`4317`/`4318`) | `pipeline.logging.gateway.server.tls` | | `prometheus.receive_http` (`9090`) | `pipeline.metrics.gateway.server.tls` | The OTLP listeners carry metrics as well as logs but are governed by the *logging* tree, because the listener belongs to that tree even when the payload does not. Set both if you want every ingress port on TLS. See [Securing](../../../operating/securing/#certificates). -------------------------------------------------------------------------------- title: "Prometheus Remote Write" url: https://materializeinc.github.io/materialize-monitoring/metrics/collecting/prometheus-remote-write/index.md -------------------------------------------------------------------------------- # Prometheus Remote Write If you already run Prometheus, the simplest way to get its metrics into this stack is to have it remote-write to the gateway. Nothing needs to change about how that Prometheus discovers or scrapes its targets. The gateway runs a [`prometheus.receive_http`](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.receive_http/) listener on port **9090**, serving the standard remote-write path: ``` http://alloy-gateway..svc:9090/api/v1/metrics/write ``` ## Configuring the sender In `prometheus.yml`: ```yaml remote_write: - url: http://alloy-gateway.monitoring.svc:9090/api/v1/metrics/write ``` Any remote-write client works — a Prometheus server, Prometheus Agent, Grafana Alloy, or an OpenTelemetry Collector with the `prometheusremotewrite` exporter. ## What happens after it arrives Received series are converted to OTLP by an internal bridge and join everything the gateway collects itself, so they get the same processing: enrichment, the [importance tiers](../../storing/#the-importance-tiers) that decide which destinations each series reaches, and the egress filter. That means remote-written metrics are subject to the same [denylist](../../storing/#the-denylist) and tiering as scraped ones. If a series you push is not appearing at a destination, check those before the transport. ## Securing it The listener does not authenticate callers. Access control is [NetworkPolicy](../../../operating/securing/#the-cluster--the-stack), and TLS on this port is governed by `pipeline.metrics.gateway.server.tls` — a **different** value from the OTLP and Loki listeners. See [the ingress overview](../overview/#serving-tls-on-them). ## See also - [Overview](../overview/) — the four collection paths, and the ports they use. - [Storing](../../storing/) — where the metrics go once they are in. - [OpenTelemetry Endpoint](../opentelemetry-endpoint/) — the OTLP equivalent of this page. -------------------------------------------------------------------------------- title: "Scraping" url: https://materializeinc.github.io/materialize-monitoring/metrics/scraping/index.md -------------------------------------------------------------------------------- # Scraping Metrics with `materialize-monitoring` By default, `materialize-monitoring` is configured to scrape metrics from any ServiceMonitor or PodMonitor resources in the cluster. This allows you to easily add new metrics to your monitoring stack by simply creating a new ServiceMonitor resources. ## Scrape Architecture with Grafana Alloy `materialize-monitoring` runs Grafana Alloy with `prometheus.operator` components on `alloy-gateway` instances (Deployment) which read ServiceMonitors and PodMonitors in order to determine what targets to scrape. `alloy-gateway` runs in [clustering mode](https://grafana.com/docs/alloy/latest/get-started/clustering/) by default, which means that scraping is distributed across all replicas of `alloy-gateway` and the scrape state is shared between them. For what each monitor kind is and how they differ, see [Custom Resource Definitions](../../reference/crds/#servicemonitor-vs-podmonitor). ### Container metrics from the kubelet {#kubelet} One target is not discovered from a monitor resource: `alloy-gateway` scrapes `/metrics/cadvisor` on **every kubelet** directly, on by default. The kubelet already computes these statistics, so running an in-process cAdvisor on each agent would compute them twice — measured at roughly 750Mi per agent against a 200Mi logs-only envelope. Scraping the kubelet removes that cost and puts the remaining scrape cost on the gateway, where it is shared across replicas rather than reserved on every node. Coverage does not suffer for it: a GKE kubelet serves 69 distinct `container_*` metrics against the 70 an in-process cAdvisor produced, and every `container_*` metric this chart's queries reference is present. | Value | Default | Notes | |---|---|---| | `pipeline.metrics.kubelet.scrapeInterval` | `60s` | The dominant cost lever — roughly 6.7k series per node per scrape | | `pipeline.metrics.kubelet.tlsInsecureSkipVerify` | `false` | On GKE and EKS the kubelet certificate verifies against the in-cluster CA, which the chart passes as `ca_file` | > [!WARNING] > A distribution that signs kubelet certificates with a CA the pods do not trust **fails this scrape quietly** — container metrics simply stop, and nothing errors at install. > `kind` is the known case. Check `up{job="cadvisor"}` when bringing up a new distribution, and set `tlsInsecureSkipVerify: true` only where you must. ## Manually Configured Scraping If you are not using default `materialize-monitoring` setup, you can use the following scrape configuration files as a starting point for your own Prometheus setup. ### Authenticating the SQL metrics endpoint The `materialize-sql` scrapers collect SQL-derived metrics from the environmentd `/metrics/mz_compute`, `/metrics/mz_frontier`, `/metrics/mz_storage`, and `/metrics/mz_usage` endpoints. Scrape it as the built-in `mz_support` role. The **Classic** and **Google Cloud Managed Prometheus** configs carry `username: mz_support` inline, so they need no extra setup. Prometheus Operator `basicAuth` can only reference a Kubernetes Secret — it has no inline fields — so the `materialize-sql` `PodMonitor` reads its credentials from a Secret named `materialize-sql-monitor`. It needs a `username` key (`mz_support`) and an **empty** `password` key. The empty password is not a placeholder to fill in later. **This endpoint does not validate passwords**: the Basic-auth username selects the role the underlying queries run as — which scopes what those queries can see — and the password field is never read. It has to be present at all only because alloy's `prometheus.operator.*` scrapeconfig generation rejects an absent password reference with `resource name may not be empty`. An empty value produces the same `mz_support:` Basic header the username-only configs already send. See [Securing](../../operating/securing/#materialize-metrics-endpoint) for what that means for network access to the port. Create it in the namespace the scrapers run in (for example, `materialize`): ```bash kubectl create secret generic materialize-sql-monitor \ --namespace materialize \ --from-literal=username=mz_support \ --from-literal=password= ``` ### Telling the four SQL endpoints apart The four endpoints share one port, so the Prometheus Operator `PodMonitor` gives all four the same `job`, `instance`, `pod` and `endpoint`. Each endpoint therefore copies its path into a `metrics_path` label, as kube-prometheus-stack does for the kubelet's several paths. Without it, the four endpoints' `up` and `scrape_*` series are one series written four times per scrape interval. A copy of the `PodMonitor` that replaces its endpoints keeps that relabeling on each of them. The Classic and Google Cloud Managed Prometheus configs scrape each path as its own job, so their `job` label already differs. ### Canonical label names {#canonical-labels} Materialize spells some ids and names more than one way. A cluster id is `instance_id` on most environmentd metrics, `compute_cluster_id` on the SQL endpoints, and `cluster_environmentd_materialize_cloud_cluster_id` on clusterd metrics. Every scrape config here copies each of those labels into one canonical name, such as `cluster_id`, on every Materialize monitor, and keeps the original beside it. A copy is made only where the canonical label is absent, so a value Materialize emits under the canonical name is never overwritten. A copy adds no series. Each canonical label is copied from the following labels, as [`labels.yaml`](https://github.com/MaterializeInc/materialize-monitoring/blob/main/packages/prometheus-scrapers/labels.yaml) lists them: {{< list-label-aliases >}} Write new queries against the canonical names, because the labels they're copied from are deprecated. A copy of a `PodMonitor` that replaces its endpoints keeps their `metricRelabelings`, or its series lack the canonical labels. clusterd's `up` and `scrape_*` series carry only the long-form ids, because metric relabeling doesn't apply to them. ### Which Prometheus Distribution Am I Using? An easy way to check if you are using Prometheus Operator is to see if you have `PodMonitor` or `ServiceMonitor` CRDs in your cluster: ```bash # Get the CRD resource directly: kubectl get crd podmonitors.monitoring.coreos.com servicemonitors.monitoring.coreos.com # This may fail with NotFound if you do not have Prometheus Operator. # If you get a permission error, try the api-resources command instead: kubectl api-resources | grep monitoring.coreos.com # This is empty if you do not have Prometheus Operator ``` > [!INFO] > `monitoring.coreos.com` is the group for Prometheus Operator CRDs. Alternatively, you can check this table of common Prometheus Distributions if you remember how you installed it: {{< details "List of Common Prometheus Distributions" >}} | Distribution | Monitor Format | Install Methods | Notes | |--------------|----------------|-----------------|-------| | [materialize-monitoring](https://materializeinc.github.io/materialize-monitoring/) | [Prometheus Operator](#prometheus-operator) | Helm, Terraform | The default monitoring stack for Materialize. Uses Grafana Alloy with `prometheus.operator` components. | | [kube-prometheus-stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) | [Prometheus Operator](#prometheus-operator) | Helm | prometheus-community Helm Chart of Prometheus Operator. | | [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) | [Prometheus Operator](#prometheus-operator) | Helm | Helm distribution of Prometheus Operator. | | [prometheus-operator](https://prometheus-operator.dev/) ([Github](https://github.com/prometheus-operator/prometheus-operator)) | [Prometheus Operator](#prometheus-operator) | Manual | The upstream distribution of Prometheus Operator | | [Bitnami kube-prometheus](https://github.com/bitnami/charts/tree/main/bitnami/kube-prometheus) | [Prometheus Operator](#prometheus-operator) | Helm | Bitnami's Helm distribution of Prometheus Operator. | | [Bitnami Prometheus chart](https://github.com/bitnami/charts/tree/main/bitnami/prometheus) | [Classic](#classic) | Helm | Bitnami's Helm distribution of Prometheus without Operator. | | [k8s-monitoring-helm](https://grafana.com/docs/grafana-cloud/monitor-infrastructure/kubernetes-monitoring/configuration/helm-chart-config/) ([Github](https://github.com/grafana/k8s-monitoring-helm/tree/main/charts/k8s-monitoring)) | [Prometheus Operator](#prometheus-operator) | Helm | Grafana's Kubernetes monitoring Helm chart. | | [prometheus.io](https://prometheus.io/docs/prometheus/latest/installation/) | [Classic](#classic) | Source, Binary, Docker | Download of binary prometheus (brew, apt, et al.) or Docker image | | [thanos-community helm](https://github.com/thanos-community/helm-charts/tree/master/charts/thanos) | [Prometheus Operator](#prometheus-operator) | Helm | Only if kube-prometheus-stack.enabled=true, otherwise refer to another technology. | | [Amazon Managed Prometheus (AMP)](https://aws.amazon.com/prometheus/) | [Classic](#classic) | AWS | Amazon's managed service for Prometheus. Not enabled by default. | | [Google Cloud Managed Service for Prometheus (GMP)](https://cloud.google.com/stackdriver/docs/managed-prometheus) | [Google Cloud Monitoring](#gmp) | Google Cloud (GCP/GKE) | Google's managed service for Prometheus. Enabled by default with GKE. | | [Grafana Alloy with prometheus.operator](https://grafana.com/docs/alloy/latest/reference/components/prometheus/prometheus.operator.servicemonitors/) | [Prometheus Operator](#prometheus-operator) | Grafana Alloy | Grafana Alloy's implementation of Prometheus Operator.| The following distributions are known to not work at this time: * VictoriaMetrics * Grafana Mimir * Cortex {{< /details >}} > [!INFO] > If your metrics backend is an OpenTelemetry database, these scrape configurations are not the path you want. > The Alloy gateway forwards over OTLP natively today — see [Metrics > Storing](../storing/#otlp) for a generic OTLP backend such as Honeycomb, or [Google Cloud Monitoring](../storing/#gcm) and [Datadog](../storing/#datadog) for those exporters. > The `otlp-metrics-honeycomb` and `otel-metrics-fanout` profiles assemble both shapes. ### Prometheus Operator Scrape Downloads {#prometheus-operator} These individual files can be used with a manual `prometheus-operator` (including `kube-prometheus-stack` and `kube-prometheus`) setup or a less-common manual Grafana Alloy `prometheus.operator` setup. {{< scrapers pattern="prometheus-scrapers/prometheus-operator/*.yaml" >}} #### Installing Prometheus Operator Scrape Configurations The above files are meant to be used as manifests that can be passed to `kubectl apply` directly. The namespace isn't generally too important, but you may elect to put them alongside your materialize-operator resource. If your `materialize-operator` is in the `materialize` namespace, you can download each into a directory and apply like: ```bash kubectl apply -f scrapers/ -n materialize ``` ### Classic ScrapeConfig Downloads {#classic} These are classic scrape_configs for non-operator Prometheus setups. These are placed into your Prometheus configuration (prometheus.yml) as a single scrape_config. > [!WARNING] > These are provided as best-effort convenience. Prometheus Operator > Monitors are the preferred implementation. {{< scrapers pattern="prometheus-scrapers/classic/*.yaml" >}} See [Prometheus Configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/) for information on how to configure scrape_configs in your Prometheus setup. ### Google Cloud Managed Service for Prometheus PodMonitoring {#gmp} These are PodMonitoring resources specifically for [Google Cloud Managed Service for Prometheus (GMP)](https://docs.cloud.google.com/stackdriver/docs/managed-prometheus). > [!WARNING] > These are provided as best-effort convenience. {{< scrapers pattern="prometheus-scrapers/gmp/*.yaml" >}} > [!INFO] > GMP collects a subset of cAdvisor metrics by default, but the chart does not depend on it. > `alloy-gateway` scrapes each kubelet's `/metrics/cadvisor` endpoint itself, which yields a fuller set than GMP's default collection — see [Container metrics from the kubelet](#kubelet). -------------------------------------------------------------------------------- title: "OpenTelemetry Endpoint" url: https://materializeinc.github.io/materialize-monitoring/metrics/collecting/opentelemetry-endpoint/index.md -------------------------------------------------------------------------------- # OpenTelemetry Endpoint The gateway accepts OTLP directly, so an OpenTelemetry Collector — or any application that emits OTLP — can send telemetry into this stack without a Prometheus in the path. An [`otelcol.receiver.otlp`](https://grafana.com/docs/alloy/latest/reference/components/otelcol/otelcol.receiver.otlp/) component listens on two ports of the `alloy-gateway` Service: | Endpoint | Protocol | |---|---| | `alloy-gateway..svc:4317` | OTLP over gRPC | | `alloy-gateway..svc:4318` | OTLP over HTTP | **Both metrics and logs are accepted on both ports.** The receiver routes them onward by signal: metrics into the metrics processing pipeline, logs across a bridge into the logging pipeline. A collector sending both needs one exporter, not two. Traces are not accepted — this stack does not carry them. ## Sending from an OpenTelemetry Collector ```yaml exporters: otlp: endpoint: alloy-gateway.monitoring.svc:4317 tls: insecure: true # see Securing, below service: pipelines: metrics: exporters: [otlp] logs: exporters: [otlp] ``` For the HTTP port use the `otlphttp` exporter with `endpoint: http://alloy-gateway.monitoring.svc:4318`. ## What happens after it arrives OTLP metrics join the gateway's own collection at the processing choke point, so they are treated identically to scraped series from there on: enrichment, the [importance tiers](../../storing/#the-importance-tiers) that decide which destinations each series reaches, and the egress filter. This is worth knowing in both directions — the same gateway that *receives* OTLP can also *forward* over OTLP to an external backend. Ingesting from a collector and fanning out to [Honeycomb](../../storing/#otlp), [Google Cloud Monitoring](../../storing/#gcm), or [Datadog](../../storing/#datadog) are independent choices, and a gateway can do both at once. ## Securing it The listeners do not authenticate callers. Access control is [NetworkPolicy](../../../operating/securing/#the-cluster--the-stack). TLS on both OTLP ports is governed by `pipeline.logging.gateway.server.tls` — the **logging** tree, even though these ports also carry metrics, because the listener belongs to that tree regardless of payload. See [the ingress overview](../overview/#serving-tls-on-them). ## See also - [Overview](../overview/) — the four collection paths, and the ports they use. - [Prometheus Remote Write](../prometheus-remote-write/) — the same idea over the Prometheus protocol. - [Storing](../../storing/#other-metric-storage-backends) — forwarding *out* over OTLP. -------------------------------------------------------------------------------- title: "Storing" url: https://materializeinc.github.io/materialize-monitoring/metrics/storing/index.md -------------------------------------------------------------------------------- # Storing Metrics `alloy-gateway` forwards the metrics it collects to one or more storage backends. Out of the box that is an in-cluster **Thanos Receive**, but you can send metrics to any number of Prometheus remote-write stores (Thanos, Mimir, Amazon Managed Prometheus, Grafana Cloud, …), to an **OpenTelemetry (OTLP)** endpoint, or to several of both at once — each enabled destination receives its own copy, filtered to its own [importance tier](#the-importance-tiers). This page walks through the three decisions that setup involves: - **Where** metrics go — the [remote-write destinations](#the-remote-write-destinations) and the [other backends](#other-metric-storage-backends) (generic OTLP, Google Cloud Monitoring, Datadog). - **How** the gateway authenticates — [authentication](#authentication) and the [gateway Secret](#supplying-credentials-the-gateway-secret) that holds the credentials. - **What** each backend stores — the [importance tiers and denylist](#controlling-what-each-destination-stores) that decide which metrics reach each destination. If you keep the bundled Thanos, it persists blocks to object storage — see [Thanos object storage](#thanos-object-storage) for the S3/GCS/Azure setup. ## The remote-write destinations {#the-remote-write-destinations} A Prometheus remote-write store is an entry of type `prometheus` in `pipeline.destinations`, the map of everything the gateway writes to. The chart ships one such entry, `thanos`, pointing at the bundled in-cluster Thanos Receive; adding an entry adds a destination. A `prometheus` entry takes the following values: | Value | Purpose | |---|---| | `type` | `prometheus`. Required on an entry you add; `thanos` already has it. | | `enabled` | Write to this destination (default `true`). | | `url` | Remote-write endpoint. Required, except on `thanos`, which defaults to the in-cluster Receive. | | `metrics.minMetricImportance` | Which [tier](#the-importance-tiers) this destination receives (default `all`). | | `authType` | `none` (default), `basicAuth`, `bearer`, or `sigv4`. | | `sigv4` | AWS SigV4 signing config, for [Amazon Managed Prometheus](#amazon-managed-prometheus-sigv4--irsa). | | `tls` | Client TLS for this hop. | | `externalLabels` | Extra labels stamped on every sample this destination receives. | For every field, see [`pipeline.destinations` in the values reference](../../reference/helm/materialize-monitoring-values/#destinations). Point the bundled destination at an external store instead: ```yaml pipeline: destinations: thanos: url: https:///api/v1/write ``` Or write to two backends at once, each on its own tier: ```yaml pipeline: destinations: # Everything, to storage you already pay for. thanos: metrics: minMetricImportance: all # Alerting metrics only, to a backend that bills per sample. amp: type: prometheus url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-.../api/v1/remote_write metrics: minMetricImportance: essential authType: sigv4 sigv4: region: us-east-1 ``` The name is more than a label in `values`. It becomes the **Alloy component label**, so it appears as `prometheus.remote_write.` in the gateway's own metrics and in its UI. It's also the fragment the [credential environment variables](#credential-environment-variables) are derived from. It must match `[a-zA-Z_][a-zA-Z0-9_]*`, with no dashes or dots, and `egress` and `destination` are reserved. Both rules fail at render rather than at gateway startup. ### One component per destination, and why it matters {#one-component-per-destination} Each destination gets its own `prometheus.remote_write` component with its own write-ahead log, rather than sharing one component with several `endpoint` blocks. That costs a WAL per destination and buys two things: - **Tiers that are real.** The importance filter sits *upstream* of the component, so a destination on `essential` writes a WAL holding only essential series. Sharing one component would mean filtering with `write_relabel_config` on the way *out* of the WAL — every destination paying full-firehose disk regardless of the tier it asked for. - **Independent failure domains.** A backend that stops accepting writes backs up its own WAL and nothing else. On a shared component a stuck endpoint holds back WAL truncation for every other endpoint too, so one unreachable SaaS backend would grow the disk the in-cluster path depends on. Budget disk accordingly: the gateway's WAL volume has to hold the sum of the enabled destinations' buffers, not one. The `essential` tier is a small fraction of `all`, so a filtered second destination costs far less than a doubling. ### External labels Every sample carries a `cluster` label identifying its source cluster, set from the chart's `clusterName` (default `default`). Set it per install so series from different clusters stay distinct once they land in a shared backend: ```yaml clusterName: prod-us-east-1 ``` The Terraform module sets it from its `cluster_name` input. The same value labels log lines and alerts. `cluster` is applied to every destination. Add labels for one destination only with its `externalLabels` map, which is useful where one backend needs a tenant or account key the others do not: ```yaml pipeline: destinations: amp: externalLabels: account: "012345678901" ``` Setting `cluster` in `externalLabels` replaces the environment-derived one for that destination. ### Upgrading from `prometheusRemoteWrite` {#upgrading-from-prometheusremotewrite} Remote-write destinations used to be configured under `pipeline.metrics.gateway.destination.prometheusRemoteWrite`, a map of the same shape. That map is deprecated. Each entry you still set there applies as a `prometheus` entry of `pipeline.destinations`, and the render warns. To move an entry, apply the following changes: - Put it under `pipeline.destinations`, with `type: prometheus`. - Move `minMetricImportance` and `unfilteredMetricsEnv` under `metrics`. Every other field keeps its name. - Check the credential variable names. While an entry stays in the old map, it keeps the credential names that map derived, such as `GATEWAY_PROMETHEUS_DEST_AMP_PASSWORD`. That includes a retune of `thanos`. Once moved, it derives `GATEWAY_AMP_DEST_PASSWORD`. If a Secret already uses the old names, keep them with the `*Env` fields, such as `basicAuth.passwordEnv`. The URL and allowlist variables are written by the chart itself, so their new names need nothing from you. The following example moves an entry: ```yaml pipeline: destinations: amp: # was: prometheusRemoteWrite.amp type: prometheus url: https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-.../api/v1/remote_write metrics: minMetricImportance: essential # was: minMetricImportance authType: sigv4 sigv4: region: us-east-1 ``` An entry named after a destination of another type, such as `loki`, fails at render rather than turning that destination into a remote write. The map's older shape, with `url`, `authType`, and the rest directly under `prometheusRemoteWrite` rather than under a name, **fails at render** with a message naming the keys to move. Helm merges maps, so a leftover `prometheusRemoteWrite.url` would otherwise apply to nothing, and the default endpoint would keep receiving with no error. Move those keys onto `pipeline.destinations.thanos` to keep the bundled Receive. ## Authentication Credentials are supplied through environment variables (not inline in values), so they can be sourced from a mounted Secret. Set `authType` and fill in the matching block: - **`none`** — no auth (the in-cluster Thanos default). - **`basicAuth`** — username / password from env. - **`bearer`** — bearer token from env. - **`sigv4`** — AWS SigV4 signing, derived from IRSA (see [Amazon Managed Prometheus](#amazon-managed-prometheus-sigv4--irsa) below). An `otlp` destination in `pipeline.destinations` takes `none`, `basicAuth`, `bearer`, `headers`, or `sigv4`; see [Generic OTLP](#otlp). A `googleCloud` destination uses the gateway's Google identity, and a `datadog` destination its API key. Whichever destination and method you choose, the gateway reads the actual secret from an environment variable at runtime (`sys.env(...)`). There are two ways to populate that variable: - **Inline in `values`** — the Prometheus remote-write and Loki destinations accept the credential directly (`basicAuth.password`, `bearer.token`, …). When set, the value is baked into the gateway ConfigMap in plaintext, so this is convenient for a quick start but weaker for production. - **From a Secret** — leave the inline field blank and provide the same env var through a Secret. This is the preferred path for every destination. Both are described next. ### Supplying credentials (the gateway Secret) The gateway loads its environment from two objects that share one name — `mzmon-alloy-gateway-env` (that is `-alloy-gateway-env`, `mzmon` by default) — in the release namespace: - a **ConfigMap** the chart generates — non-secret env such as the per-destination allowlists and tenant map, plus any credentials you chose to set inline in `values`; - a **Secret** *you* create — the credential values in the table below, kept out of the rendered manifests. The chart does **not** create the Secret; it is mounted `optional: true`, so you populate it out-of-band with only the keys your enabled destinations need. For example, a bearer-authenticated `otlp` destination named `collector` and a `datadog` destination named `datadog` need two keys: ```bash kubectl create secret generic mzmon-alloy-gateway-env \ --namespace monitoring \ --from-literal=GATEWAY_COLLECTOR_DEST_BEARER_TOKEN='' \ --from-literal=GATEWAY_DATADOG_DEST_API_KEY='' ``` The Secret's keys are injected as environment variables next to the ConfigMap's, so the key names **are** the env-var names in the table below. > [!WARNING] > The Secret name must match the release: with the default `fullnameOverride: mzmon` it is `mzmon-alloy-gateway-env`, and it must live in the namespace the gateway runs in (`monitoring` above). > Because the mount is optional, a mismatched name or namespace is silently ignored — the destination then authenticates with empty credentials instead of failing loudly. > [!INFO] > `kubectl create secret` is fine for a first install, but in production source the Secret from Sealed Secrets, External Secrets, or SOPS rather than committing raw credentials. ### Credential environment variables Populate only the rows for the destinations and auth methods you enable: `` below is the destination's key in `pipeline.destinations`, uppercased with anything outside `A-Z0-9` folded to `_`. A `prometheus` destination named `amp` reads `GATEWAY_AMP_DEST_PASSWORD`, and a `datadog` destination named `datadog` reads `GATEWAY_DATADOG_DEST_API_KEY`. Every row is per destination, so two destinations on `basicAuth` need two pairs of keys. | Destination · method | `values` block | Secret keys (env vars) | |---|---|---| | `prometheus` or `otlp` · `basicAuth` | `pipeline.destinations..basicAuth` | `GATEWAY__DEST_USERNAME`, `GATEWAY__DEST_PASSWORD` | | `prometheus` or `otlp` · `bearer` | `pipeline.destinations..bearer` | `GATEWAY__DEST_BEARER_TOKEN` | | `otlp` · `headers` | `pipeline.destinations..headers` | whatever each header's `valueEnv` names; you choose | | `datadog` | `pipeline.destinations..apiKey` | `GATEWAY__DEST_API_KEY` | | `prometheus` or `otlp` · `sigv4` | `pipeline.destinations..sigv4` | none; uses the pod's IRSA identity | | Deprecated `prometheusRemoteWrite` | `basicAuth`/`bearer` | `GATEWAY_PROMETHEUS_DEST__USERNAME`, `…_PASSWORD`, `…_BEARER_TOKEN` | | Deprecated `otel` block | `…otel.auth.*`, `…otel.datadogExporter` | `GATEWAY_OTEL_DEST_USERNAME`, `…_PASSWORD`, `…_BEARER_TOKEN`, `…_DATADOG_API_KEY` | The deprecated `prometheusRemoteWrite` row covers every entry still set there, a retune of `thanos` included. SigV4 and the cloud-native exporters (GCM via Workload Identity) carry no secret keys — they authenticate with the gateway pod's ambient cloud identity, so those rows stay out of the Secret entirely. Each destination can also name its own variables, such as `basicAuth.usernameEnv`, `bearer.tokenEnv`, or `apiKey.keyEnv`, where the derived name is inconvenient or a Secret already uses another. The Terraform module sets them explicitly for this reason; see [Through Terraform](#through-terraform). ## Controlling what each destination stores Not every backend should receive every metric. `alloy-gateway` gives you two independent controls: a global **denylist** that drops metrics before they reach any destination, and a per-destination **importance filter** that keeps only metrics at or above a chosen tier. ### The importance tiers Every metric in the registry is classified by *importance* — how likely you are to want it — independent of which backend you use. The levels, from most to least important, are **essential**, **recommended**, **extended**, and **diagnostic**. Two narrower tiers sit above them, and only series the chart's own rules write belong to them. **heartbeat** is the call-home heartbeat, a fixed record of what the install runs and whether its monitoring stack works. **alerts** is `ALERTS`, the state of every pending or firing alert. One more value, **all**, is a firehose meaning "everything scraped, including metrics the registry has not classified." The tier definitions and the full membership of each tier live in [the metric list](../../reference/list-metrics/). Each destination picks a floor with `minMetricImportance`. The filter is cumulative — a floor keeps that tier **and every tier more important than it**: | `minMetricImportance` | Metrics kept | |---|---| | `heartbeat` | the heartbeat series | | `alerts` | the heartbeat series + `ALERTS` | | `essential` | the heartbeat series + `ALERTS` + essential | | `recommended` | essential + recommended | | `extended` | essential + recommended + extended | | `diagnostic` | essential + recommended + extended + diagnostic | | `all` | everything scraped, classified or not (`.*`) | The defaults lean permissive for cheap local storage and frugal for metered SaaS backends: | Destination | Default `minMetricImportance` | |---|---| | `pipeline.destinations.thanos` (bundled Thanos) | `all` | | `pipeline.destinations` entry of type `prometheus` | `all` | | `pipeline.destinations` entry of type `otlp` | `all` | | `pipeline.destinations` entry of type `googleCloud` | `recommended` | | `pipeline.destinations` entry of type `datadog` | `recommended` | Set it per destination: ```yaml pipeline: destinations: datadog: type: datadog metrics: minMetricImportance: essential # ship only alerting metrics to Datadog ``` The series the chart's recording rules write, such as the `ext:*` series for the metadata database, have tiers too. A destination on `recommended` receives them without the metrics they're recorded from. The Thanos ruler reads those metrics from the bundled Thanos, so they have to reach the `thanos` destination. The render warns when the `thanos` destination's tier drops a metric that an installed recording rule reads. For more information, see [Recorded series](../../alerting/configuring/#recorded-series). The heartbeat and `ALERTS` reach a destination the same way: the ruler writes them through the gateway, so a destination at `alerts` forwards alert state without any other metric. For what the heartbeat carries, see [What the heartbeat sends](../../operating/call-home/#heartbeat). For a worked example that keeps full fidelity in Thanos while sending a smaller, cheaper slice to Google Cloud Monitoring and Datadog, see the annotated [`otel-metrics-fanout.values.yaml`](https://github.com/MaterializeInc/materialize-monitoring/blob/main/charts/materialize-monitoring/profiles/otel-metrics-fanout.values.yaml) profile. The remote-write equivalent — Thanos on `all`, Amazon Managed Prometheus on `essential` — is [`aws-amp-fanout.values.yaml`](https://github.com/MaterializeInc/materialize-monitoring/blob/main/charts/materialize-monitoring/profiles/aws-amp-fanout.values.yaml). On the remote-write side the tier does more than filter, because the filter runs before the write-ahead log: a destination on `essential` also buffers only essential series to disk. See [One component per destination](#one-component-per-destination). ### How the allowlist is built Tier membership is generated from the query registry into `charts/materialize-monitoring/pre-rendered/metrics/metric-tiers.yaml` (via `mz-monitoring-build gen-metric-tiers`, or `make metric-tiers`). At render time the chart reads that file, unions the tiers at or above each destination's `minMetricImportance`, and hands the gateway the result as an allowlist regex — one environment variable per destination, such as `GATEWAY_AMP_DEST_METRICS_ALLOWLIST` for a destination named `amp`. `all` skips the file entirely and uses `.*`. Do not edit `metric-tiers.yaml` by hand: reclassify metrics in the registry and regenerate. ### The denylist `denyMetrics` drops metrics for **every** destination, at the gateway input, before the per-destination fan-out: ```yaml pipeline: metrics: gateway: denyMetrics: - some_noisy_metric - another_expensive_.* # entries are regex fragments, OR-joined ``` Reach for the denylist to shed a metric everywhere (cost, cardinality, noise); reach for `minMetricImportance` to tune what an *individual* backend receives. Entries in both the denylist and the tiers name Prometheus series, as Thanos shows them: `foo_total`, `foo_bucket`, `foo_count`. Inside the gateway a typed histogram is one metric named for its family, and the match is made per series all the same. | Entry names | Effect on a histogram `foo` | |---|---| | `foo_bucket` on the denylist | `foo_count` and `foo_sum` still reach every destination; the buckets do not | | `foo_count` in a tier, `foo_bucket` in a lower one | A destination on the higher tier receives `foo_count` alone | | `foo_bucket` in a tier | A destination on that tier receives the whole histogram | ### Through Terraform The `materialize-monitoring` module exposes the OTLP-family destinations directly, each taking the same `min_importance` tier documented above: ```hcl google_cloud_metrics = { min_importance = "recommended" } datadog_metrics = { site = "datadoghq.com", min_importance = "essential" } datadog_api_key = var.datadog_api_key otlp_metrics = { url = "api.honeycomb.io" min_importance = "recommended" auth_headers = { "x-honeycomb-dataset" = "mzmon" } } otlp_auth_header_secrets = { "x-honeycomb-team" = var.honeycomb_api_key } ``` Remote-write destinations go through `prometheus_remote_write`, keyed by name; each key becomes a `pipeline.destinations` entry of type `prometheus`. A key of `thanos` **retunes** the bundled destination rather than adding a second one, so dropping local fidelity and adding AMP is one input: ```hcl prometheus_remote_write = { thanos = { min_importance = "all" } amp = { url = "https://aps-workspaces.us-east-1.amazonaws.com/workspaces/ws-.../api/v1/remote_write" min_importance = "essential" auth_type = "sigv4" sigv4_region = "us-east-1" } } # SigV4 needs no credentials, only an identity to sign as. gateway_service_account_annotations = { "eks.amazonaws.com/role-arn" = aws_iam_role.amp_writer.arn } ``` `basicAuth` and `bearer` destinations take their credentials from `prometheus_remote_write_credentials`, keyed by the same destination name. The module derives the environment variable names from that name, as `GATEWAY_PROMETHEUS_DEST__PASSWORD` and so on, and writes them into the chart's values, so the Secret it creates and the config the gateway reads cannot disagree. Credentials do **not** go through the Helm values — the module puts them in the gateway Secret instead, and rolls the gateway when one changes. See the [module README](https://github.com/MaterializeInc/materialize-monitoring/blob/main/terraform/modules/materialize-monitoring/README.md#metric-destinations) for the environment variables each input becomes. The per-cloud wrappers in `materialize-terraform-self-managed` surface Google Cloud Monitoring under flatter names (`enable_google_cloud_metrics`, `google_cloud_metrics_min_importance`), because it is the one destination needing cloud resources the chart cannot create — a service account and the Workload Identity binding it authenticates with. Anything not modelled here is still reachable through `additional_values`. See [Getting Started > Terraform](../../getting-started/terraform/#extra-metrics-destinations). ### Operational notes > [!NOTE] > **The filter fails open.** > If a destination's allowlist environment variable is empty or unset, the gateway falls back to `.*`. > A misconfiguration therefore ships *everything* to that backend rather than nothing — safe for visibility, but watch cost on metered backends. > [!WARNING] > **The gateway shards scrape targets across replicas.** > Scraping runs with clustering enabled, so targets are distributed over the gateway pods. > During a partial rollout a metric can look "missing" simply because its target is being scraped by a pod that has not yet picked up the new config — roll out **all** gateway replicas before concluding a metric is filtered out. > [!INFO] > **Backend schema browsers are historical.** > A metric appearing in a backend's schema or column list (Honeycomb, Datadog, …) is not proof it is arriving *now* — those views are cumulative and can show columns from before a filter change. > Query for recent samples to confirm what is currently flowing. ## Thanos object storage When you run the bundled **Thanos** (the default destination), it persists metric blocks to object storage — the same durability model as Loki's chunks. The supported backends are **S3-compatible** storage (AWS S3, MinIO, Ceph, R2, …), **Google Cloud Storage**, and **Azure Blob Storage**. See the [Thanos storage reference](https://thanos.io/tip/thanos/storage.md/) for the full config schema. ### The objstore Secret Thanos reads its object-store config from a Kubernetes Secret. The chart does **not** create it by default (`thanos.global.objstore.createSecret: false`), so you supply it: | Value | Default | Purpose | |---|---|---| | `thanos.global.objstore.secretName` | `thanos-objstore-config` | Secret holding the object-store config. | | `thanos.global.objstore.secretKey` | `objstore.yml` | Key within that Secret. | The Secret holds a Thanos `objstore.yml` — a `type:` plus a provider `config:`. Create it in the namespace Thanos runs in: ```bash kubectl create secret generic thanos-objstore-config \ --namespace monitoring \ --from-file=objstore.yml=./objstore.yml ``` > [!INFO] > Prefer cloud **workload identity** (IRSA on AWS, Workload Identity on GKE, Azure Workload ID) over long-lived keys in `objstore.yml`. > Omit the credential fields from the config and annotate the Thanos ServiceAccount instead — no static secrets in the cluster. ### Granting object-storage access (workload identity) Annotate the Thanos ServiceAccount through `thanos.global.serviceAccount.annotations` (shared by receive, store gateway, and compactor), and leave the credential fields out of `objstore.yml` so the SDK uses the ambient identity. The Thanos ServiceAccount is `thanos-thanos` (a deterministic `fullnameOverride`), in the release namespace (recommended `monitoring`); the split-namespace profile places it in a dedicated `thanos` namespace instead. Scope the binding to that exact namespace/ServiceAccount. {{< tabs >}} {{% tab "AWS · EKS (IRSA)" %}} **IRSA** (IAM Roles for Service Accounts). Chain: the Thanos ServiceAccount is annotated with a role ARN → EKS projects an OIDC token → the SDK calls **STS `AssumeRoleWithWebIdentity`** → temporary credentials → **S3**. Requires the cluster's **OIDC provider** registered in IAM (one-time). A ready-made starting point lives at `charts/materialize-monitoring/profiles/aws-example.values.yaml`. *Trust policy* — scope `:sub` to the **Thanos namespace and ServiceAccount**, not another workload's: ```json { "Effect": "Allow", "Principal": { "Federated": "arn:aws:iam:::oidc-provider/oidc.eks..amazonaws.com/id/" }, "Action": "sts:AssumeRoleWithWebIdentity", "Condition": { "StringEquals": { "oidc.eks..amazonaws.com/id/:aud": "sts.amazonaws.com", "oidc.eks..amazonaws.com/id/:sub": "system:serviceaccount:monitoring:thanos-thanos" }} } ``` > [!INFO] > The default assumes the release is installed into `monitoring`. > Under [split namespaces](../../operating/production-best-practices/stack-wide/#namespace-layout), the `:sub` is `system:serviceaccount:thanos:thanos-thanos` instead. *Permissions policy* — least-privilege to the single bucket. `DeleteObject` is required (the compactor rewrites and deletes blocks during compaction/downsampling): ```json { "Statement": [ { "Effect": "Allow", "Action": ["s3:ListBucket", "s3:GetBucketLocation"], "Resource": "arn:aws:s3:::" }, { "Effect": "Allow", "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"], "Resource": "arn:aws:s3:::/*" } ] } ``` *ServiceAccount* — annotate via chart values; the EKS webhook injects `AWS_ROLE_ARN` / `AWS_WEB_IDENTITY_TOKEN_FILE`: ```yaml thanos: global: serviceAccount: annotations: eks.amazonaws.com/role-arn: arn:aws:iam:::role/ ``` *`objstore.yml`* — no `access_key`/`secret_key`, so the default chain uses the IRSA token: ```yaml type: S3 config: bucket: endpoint: s3..amazonaws.com region: ``` {{% /tab %}} {{% tab "GCP · GKE (Workload Identity)" %}} **GKE Workload Identity.** Chain: the Thanos ServiceAccount is annotated with a Google service account (GSA) → GKE exchanges the pod token for that GSA's credentials → **GCS**. Requires Workload Identity enabled on the cluster and node pool. Below, `` is the GSA; `[/thanos-thanos]` is the Kubernetes ServiceAccount (KSA). 1. Grant the GSA object access on the bucket (`roles/storage.objectAdmin`). 2. Bind the GSA's IAM policy so the Thanos KSA may impersonate it — the KSA **must match Thanos's namespace/ServiceAccount**: ```bash gcloud iam service-accounts add-iam-policy-binding @.iam.gserviceaccount.com \ --role="roles/iam.workloadIdentityUser" \ --member="serviceAccount:.svc.id.goog[monitoring/thanos-thanos]" ``` > [!INFO] > The default assumes the release is installed into `monitoring`. > Under [split namespaces](../../operating/production-best-practices/stack-wide/#namespace-layout), use `--member="serviceAccount:.svc.id.goog[thanos/thanos-thanos]"` here. 3. Annotate the ServiceAccount: ```yaml thanos: global: serviceAccount: annotations: iam.gke.io/gcp-service-account: @.iam.gserviceaccount.com ``` *`objstore.yml`* — no `service_account` key, so ambient Workload Identity credentials are used: ```yaml type: GCS config: bucket: ``` {{% /tab %}} {{% tab "Azure · AKS (Workload ID)" %}} **Microsoft Entra Workload ID.** Chain: the Thanos ServiceAccount is annotated with a managed-identity client ID → AKS projects a token → exchanged with Entra for the identity's credentials → **Azure Blob**. Requires the OIDC issuer + workload identity enabled on the cluster. 1. Grant the user-assigned managed identity **`Storage Blob Data Contributor`** on the storage account (or container scope). 2. Create a **federated identity credential** on that identity — subject **must match Thanos's namespace/ServiceAccount** (`system:serviceaccount:monitoring:thanos-thanos`), audience `api://AzureADTokenExchange`. > [!INFO] > The default assumes the release is installed into `monitoring`. > Under [split namespaces](../../operating/production-best-practices/stack-wide/#namespace-layout), the subject is `system:serviceaccount:thanos:thanos-thanos` instead. 3. Annotate the ServiceAccount and label the pods so the webhook injects the token: ```yaml thanos: global: serviceAccount: annotations: azure.workload.identity/client-id: ``` *`objstore.yml`* — see the [Thanos Azure config](https://thanos.io/tip/thanos/storage.md/#azure) for the exact keys (`storage_account`, `container`); omit the shared key so the workload identity is used. {{% /tab %}} {{< /tabs >}} > [!INFO] > The token exchange and the object store are both 443 hops to your cloud's identity and storage endpoints. If a Thanos NetworkPolicy is enabled you must allow that egress, or the credential fetch hangs the component at startup. > [!NOTE] > **Verifying.** Split the two failure modes: a `403`/AccessDenied during the **token exchange** (`AssumeRoleWithWebIdentity` or the GCP/Azure equivalent) is a **binding/trust-scope** problem — usually a namespace/ServiceAccount subject mismatch; an authorization error on the **bucket operation itself**, after the exchange succeeds, is a **permissions** problem on the bucket. These are the same mechanics as [Loki's object store](../../logs-and-events/storing/#granting-object-storage-access-workload-identity). ### Retention and downsampling The Thanos **Compactor** compacts raw blocks and produces downsampled resolutions, each with independent retention (`thanos.compactor.retention`): | Resolution | Default retention | |---|---| | raw | `30d` | | 5m | `90d` | | 1h | `365d` | Downsampling keeps long-range queries cheap: a year-wide query reads 1h blocks, not raw samples. Tune these to trade storage cost against how far back high-resolution data stays available. Each Receive replica uploads its own copy of every block it holds, labelled with `receive_replica`. The Compactor deduplicates on that label (`--deduplication.replica-label=receive_replica`), so compacted blocks hold one copy of each series rather than one per replica. The same flag enables vertical compaction, which merges the short overlaps Receive writes across a restart instead of halting on them. See [Retention & compaction](../../operating/production-best-practices/thanos/#thanos-retention-compaction) for the upgrade caveats. ### Components The bundled Thanos runs as a small set of roles over the shared bucket: - **Receive** — the remote-write endpoint `alloy-gateway` writes to; buffers recent data and uploads TSDB blocks to object storage. - **Store Gateway** — serves historical blocks *from* object storage for queries. - **Compactor** — a **singleton** that compacts and downsamples blocks in the bucket (owns retention). - **Query** — federates recent data (Receive) and historical data (Store Gateway) behind one PromQL endpoint. The **Ruler** is on by default and evaluates PromQL rules. `queryFrontend` is off by default and enabled by the `thanos-large` profile. See [Metrics Architecture](../architecture/) for how the roles connect. ## Other Metric Storage Backends Two families of backend sit alongside the bundled Thanos, and both fan out *in addition to* it rather than replacing it. **More remote-write stores** — Amazon Managed Prometheus, Mimir, Grafana Cloud, another Thanos — are added as further [`prometheus` entries](#the-remote-write-destinations). Nothing on this page is an either/or with the in-cluster Thanos: keeping full fidelity locally while shipping a filtered slice to a metered backend is the shape the map exists for. **OpenTelemetry (OTLP) backends** are entries in `pipeline.destinations`, the map that also holds the gateway's log destinations. Each entry is one connection with its own credentials, metric tier, and queue, and it can carry logs as well as metrics. The following types send metrics: `otlp` : Any OTLP endpoint, such as Honeycomb, Grafana Cloud, or your own OpenTelemetry Collector. `googleCloud` : Google Cloud Managed Service for Prometheus, through the Telemetry API. `datadog` : Datadog, through its own exporter. A metrics-carrying entry gets its own tier filter, set by `metrics.minMetricImportance`, so each backend receives only the tiers you choose. For the full list of fields, see [`pipeline.destinations` in the values reference](../../reference/helm/materialize-monitoring-values/#destinations). > [!NOTE] > The exporters under `pipeline.metrics.gateway.destination.otel` are deprecated. > They still render as before, and the render warns while any is enabled. > See [Move off the deprecated `otel` block](#otel-migration). ### Generic OTLP (Honeycomb, Grafana Cloud, collectors) {#otlp} An `otlp` entry sends to any OTLP-compatible endpoint: ```yaml pipeline: destinations: collector: type: otlp url: collector.example.com:4317 protocol: grpc tls: enabled: true metrics: minMetricImportance: recommended ``` Over gRPC, `url` is a `host:port`. Over `protocol: http`, the default, `url` is a base URL such as `https://otlp.example.com`, and the exporter appends `/v1/metrics`. Turn `tls.enabled` on for any endpoint that serves TLS; with it off, the gRPC exporter connects in plaintext. Set `authType` to authenticate: | `authType` | Credentials | |---|---| | `basicAuth` | `basicAuth.username` and `basicAuth.password`, or the variables `basicAuth.usernameEnv` and `basicAuth.passwordEnv` name | | `bearer` | `bearer.token`, or the variable `bearer.tokenEnv` names | | `headers` | `headers`, a list of request headers | | `sigv4` | The gateway's AWS identity, with `sigv4.region`, `sigv4.service`, and an optional `sigv4.roleArn` to assume | The credential values come from the gateway Secret. For the variable names, see [Supplying credentials](#supplying-credentials-the-gateway-secret). #### API-key headers {#otlp-headers} Several OTLP vendors authenticate with a custom request header rather than a bearer token, such as Honeycomb's `x-honeycomb-team`. `authType: headers` covers that case: ```yaml pipeline: destinations: honeycomb: type: otlp url: api.honeycomb.io:443 protocol: grpc tls: enabled: true metrics: {} authType: headers headers: - key: x-honeycomb-team valueEnv: GATEWAY_HONEYCOMB_API_KEY - key: x-honeycomb-dataset value: mzmon ``` Each header sets exactly one of `value` or `valueEnv`. A `value` renders into the gateway's pipeline ConfigMap in plaintext, so keep it for non-secret routing headers such as a dataset or tenant name. A `valueEnv` names an environment variable the gateway reads at startup, which is where a credential belongs. The variable name is yours to pick, so name it after the backend. The chart checks the shape at render time. An empty header list, a header with no `key`, a header that sets both `value` and `valueEnv` or neither, and a `valueEnv` that no `extraEnv` or `envFrom` source could supply all fail the install. For a ready-made starting point, copy the annotated [`otlp-metrics-honeycomb.values.yaml`](https://github.com/MaterializeInc/materialize-monitoring/blob/main/charts/materialize-monitoring/profiles/otlp-metrics-honeycomb.values.yaml) profile. ### Google Cloud Monitoring (GCM) {#gcm} A `googleCloud` entry writes OTLP to Google's [Telemetry API](https://docs.cloud.google.com/stackdriver/docs/otlp/overview) (`telemetry.googleapis.com`). The Telemetry API stores the series in Cloud Monitoring as Google Cloud Managed Service for Prometheus metrics, under `prometheus.googleapis.com/`, where PromQL reads them. ```yaml pipeline: destinations: googleCloud: type: googleCloud metrics: minMetricImportance: recommended ``` The project needs the `telemetry.googleapis.com` API enabled. Authentication uses **Workload Identity**: the `alloy-gateway` ServiceAccount is annotated with a Google service account, and the gateway takes that ambient identity with no credentials in the config. The token exchange works the same way as for [Thanos on GCS](#granting-object-storage-access-workload-identity), but on the gateway's ServiceAccount rather than Thanos's. That identity needs `roles/monitoring.metricWriter` on the project. #### What it costs Samples ingested into `prometheus.googleapis.com/` are billed per sample. The `workload.googleapis.com/` custom metrics it wrote before v0.31.0 are billed per byte, at 8 bytes a point. | Domain | Billed by | List price | Measured at `recommended` | |---|---|---|---| | `workload.googleapis.com/` (`otelcol.exporter.googlecloud` before v0.31.0) | MiB ingested | $0.26 per MiB, about $2 per million points | 307 MiB a day across 212 metric types, about $2,400 a month | | `prometheus.googleapis.com/` (OTLP to the Telemetry API) | Samples ingested | $0.06 per million samples | The same points as samples, about $75 a month | A histogram point is billed as two samples plus one per non-empty bucket, so a histogram costs no more than its buckets did as separate points. An untyped series is written twice, as `/unknown` and `/unknown:counter`, and each scrape target adds a `target_info` series. Both are small next to the total. #### What arrives The metric type carries the kind of point: `prometheus.googleapis.com//gauge`, `/counter`, `/histogram` or `/summary`. PromQL takes the bare name, as it does against Thanos, and a histogram answers to `_bucket`, `_count` and `_sum`. The kind comes from each target's `# TYPE` line, because every gateway scrape sets `honor_metadata`. Pushed remote-write carries no types, so it arrives untyped. Cloud Monitoring files every series under a `prometheus_target` resource. | Label | Value | |---|---| | `project_id` | The project the gateway runs in, or the destination's `project` | | `location` | The GKE cluster's region or zone, or the destination's `location` | | `cluster` | The series' own `cluster` label, or else `CLUSTER_NAME`, which is the `cluster` every other destination carries | | `namespace` | The series' own `namespace` label, as in Thanos | | `job`, `instance` | The scrape target's | A series label named `location`, `project_id`, `job` or `instance` keeps its value as `exported_