# materialize-monitoring values # `materialize-monitoring` Helm Reference Unified observability stack for Materialize: Alloy-based pipeline, optional bundled backends (Loki / Thanos / Grafana / Alertmanager), and the dashboards / alerts / rules that consume the Materialize metrics surface. ## Source Code * ## Requirements Kubernetes: `>=1.27.0-0` ### Storage Requirements Using Thanos or Loki (both enabled by default) requires object storage (such as an AWS S3 bucket) for long-term storage. In development, Thanos may use a PVC for less-reliable storage. Loki however will not run without object storage. You may consider Garage or RustFS or MinIO for manually provisioned object storage. ## Included Subcharts | Repository | Name | Version | |------------|------|---------| | https://grafana.github.io/helm-charts | alloy(alloy-agent) | 1.12.1 | | https://grafana.github.io/helm-charts | alloy(alloy-gateway) | 1.12.1 | | https://kubernetes-sigs.github.io/metrics-server | metrics-server | 3.14.0 | | [oci://ghcr.io/grafana-community/helm-charts](https://github.com/grafana-community/helm-charts) | grafana | 12.11.1 | | [oci://ghcr.io/grafana-community/helm-charts](https://github.com/grafana-community/helm-charts) | loki | 18.11.0 | | [oci://ghcr.io/grafana/helm-charts](https://github.com/grafana/helm-charts) | grafana-operator | 5.24.0 | | [oci://ghcr.io/prometheus-community/charts](https://github.com/prometheus-community/helm-charts) | alertmanager | 1.42.0 | | [oci://ghcr.io/prometheus-community/charts](https://github.com/prometheus-community/helm-charts) | kube-state-metrics | 8.4.0 | | [oci://ghcr.io/prometheus-community/charts](https://github.com/prometheus-community/helm-charts) | prometheus-node-exporter(node-exporter) | 4.56.1 | | [oci://ghcr.io/thanos-community/helm-charts](https://github.com/thanos-community/helm-charts) | thanos | 0.30.0 | ## Values {#values} ### Globals Values that are passed to all subcharts

Key	Type	Default	Description
global.imageRegistry	string	`""`	Override the registry for all images in this chart and its subcharts. Leave empty to use upstream defaults.
global.imagePullSecrets	list	[]	Image pull secrets for this chart's own workloads and for the subcharts that read a global. Not all of them do. Reaches this chart's own workloads, plus Thanos, Grafana, kube-state-metrics and node-exporter. It does not reach every subchart on its own, because they do not agree on where to read it from: * Alloy reads `global.image.pullSecrets` and never this key. This chart honors both spellings for the workloads it renders itself, but the Alloy subchart only sees its own. * Loki, grafana-operator, Alertmanager and metrics-server read no global at all and need their own `.imagePullSecrets`. `profiles/registry/pull-secret.values.yaml` sets every one of those paths from a single Secret name; prefer it over setting this key alone, which leaves Loki — the largest pod count in the release — pulling anonymously.
global.clusterDomain	string	`"cluster.local"`	The cluster's DNS domain. `cluster.local` is a default, not a fact — clusters get built with `--cluster-domain=cluster.internal` or a site-specific domain often enough that hardcoding it is a reliable way to ship a feature that works everywhere we test and nowhere a customer runs. It lives under `global` because two subcharts already read `global.clusterDomain` and build real addresses from it — Loki's memberlist join address and canary URL, and Thanos's in-cluster endpoints. Helm propagates `global` into every subchart, so setting it here is what makes one value mean one thing rather than three keys that can disagree. One subchart is not covered by that propagation. `metrics-server` reads its own `metrics-server.tls.clusterDomain`, which is set to the same default below; a render-time check warns when the two disagree, because the failure is a certificate whose SANs name a domain the cluster does not use. Load-bearing for `certificates`: the SAN ladder ends in `$svc.$ns.svc.$clusterDomain`, so a wrong value here produces certificates that fail verification against the endpoints this chart itself dials.

### Main chart configuration Configuration for the main chart

Key	Type	Default	Description
nameOverride	string	`""`	Standard Helm name override. Note that in umbrella charts, fullname may be shorted. See fullnameOverride.
fullnameOverride	string	`"mzmon"`	Standard Helm fullname override. Note that in umbrella charts, this may be shortened to avoid long prefixes. (and have more stable names in resources)
namespaceOverride	string	`""`	Namespace override for default workloads.

### Subchart enablement (Helm tags) Group toggles for subchart enablement. Subchart enablement has two layers, in precedence order: 1. **Per-chart circuit breaker (`.enabled`).** Each subchart dependency carries a `condition: .enabled` in `Chart.yaml`. The corresponding `enabled:` key is left commented-out under each subchart block in `values.yaml`; uncommenting it force-includes (`true`) or force-excludes (`false`) that chart regardless of any tag state. Helm evaluates `condition:` first and only falls through to tags when the path is absent from values, so this is a hard override. 2. **Helm tags (the `tags:` block).** When no `.enabled` is set, enablement is decided by tags. Each dependency in `Chart.yaml` carries the master `default` tag, a *group tag* (e.g. `bundled-backends`), and a *per-chart tag* (e.g. `loki`); a dependency is included if any of its tags evaluates true (OR semantics). To opt one chart out of an otherwise-enabled group, set the group tag false and flip the per-chart tags individually — or use the circuit breaker. Group ↔ chart mapping: | Group tag | Charts | | ------------------ | --------------------------------------------------- | | `default` | `pipeline` + `bundled-backends` + `managed-grafana` | | | groups + `kube-state-metrics` (recommended stack) | | `pipeline` | `alloy-agent`, `alloy-gateway` | | `bundled-backends` | `loki`, `thanos`, `alertmanager` | | `managed-grafana` | `grafana`, `grafana-operator` | | `cluster-metrics` | `kube-state-metrics`, `node-exporter`, | | | `metrics-server` | | `crds` | `prometheus-operator-crds`, `grafana-operator-crds` | | | (in the sibling `materialize-monitoring-crds` chart)| `default` is the only group on by default and enables the full recommended stack. `--set tags.default=false` turns everything off, so you can enable a single group (`bundled-backends`, `managed-grafana`, `pipeline`, `cluster-metrics`) or individual charts on top. Profile preset values files under `profiles/` flip these appropriately. (`kube-state-metrics` and `node-exporter` are in `default`, but `metrics-server` is not — most clusters already run metrics-server; enable it via `tags.cluster-metrics` or `tags.metrics-server` if yours doesn't. A cluster that already runs its own node-exporter should turn ours off with the `node-exporter.enabled: false` circuit breaker — setting `tags.node-exporter: false` does nothing while `tags.default` is true, because tags are OR'd.)

Key	Type	Default	Description
tags.default	bool	`true`	Enable all recommended defaults. You can `--set tags.default=false` to disable all services and explicitly enable others.
tags.pipeline	bool	`false`	Enable both Alloy releases (agent + gateway).
tags.bundled-backends	bool	`false`	Enable Loki, Thanos, and Alertmanager as a group. (Grafana is in the `managed-grafana` group.)
tags.cluster-metrics	bool	`false`	Enable kube-state-metrics and metrics-server as a group.
tags.alloy-agent	bool	`false`	Per-chart override: enable just the Alloy agent. OR'd with `tags.default` / `tags.pipeline`.
tags.alloy-gateway	bool	`false`	Per-chart override: enable just the Alloy gateway. OR'd with `tags.default` / `tags.pipeline`.
tags.loki	bool	`false`	Per-chart override: enable just Loki. OR'd with `tags.default` / `tags.bundled-backends`.
tags.thanos	bool	`false`	Per-chart override: enable just Thanos. OR'd with `tags.default` / `tags.bundled-backends`.
tags.grafana-standalone	bool	`false`	Per-chart override: enable just Grafana standalone. OR'd with `tags.default` / `tags.managed-grafana`.
tags.grafana-operator	bool	`false`	Per-chart override: enable just Grafana operator. OR'd with `tags.default` / `tags.managed-grafana`.
tags.alertmanager	bool	`false`	Per-chart override: enable just Alertmanager. OR'd with `tags.default` / `tags.bundled-backends`.
tags.kube-state-metrics	bool	`false`	Per-chart override: enable just kube-state-metrics. OR'd with `tags.default` / `tags.cluster-metrics`.
tags.node-exporter	bool	`false`	Per-chart override: enable just node-exporter. OR'd with `tags.default` / `tags.cluster-metrics`.
tags.metrics-server	bool	`false`	Per-chart override: enable just metrics-server. OR'd with `tags.cluster-metrics`.

### Priority classes Scheduling priority shared across the stack's subcharts. Two PriorityClasses, created here and referenced by name from the subchart blocks further down. The split is about **what losing a pod costs you**: | Class | Used by | Losing a pod means | | --------------------- | ------------------------------------------------ | ------------------------------------------------------- | | `monitoring-critical` | `alloy-agent`, `node-exporter`, `alloy-gateway` | a blind spot with no replica to cover it | | `monitoring-scalable` | `loki`, `thanos`, `grafana`, `alertmanager`, KSM | reduced capacity a surviving replica or a retry absorbs | The per-node collectors are singletons *per node*: when the agent or node-exporter is evicted from a node, nothing else reports that node, and the gap is permanent (there is no backfill). The backends are replicated, buffered, or both — Alloy retries writes, so a Loki ingester that is evicted costs latency rather than data. The gateway is graded critical despite being a Deployment because it is the single egress choke point for every signal. Both classes set **`preemptionPolicy: Never`**. That is the load-bearing choice: priority still decides scheduling-queue order and, more importantly, which pod the kubelet evicts first under node pressure — but monitoring will never evict a Materialize pod to make room for itself. Monitoring that takes down the thing it monitors is worse than monitoring that is late. Both sit well below `system-cluster-critical` (2000000000) and `system-node-critical` (2000001000), so cluster plumbing still outranks us. **PriorityClasses are cluster-scoped.** Two releases of this chart in one cluster will fight over these objects. Set `create: false` on all but one, or rename them per release — and if you rename, update the `priorityClassName` values in the subchart blocks to match. A `priorityClassName` naming a class that does not exist does not degrade: the API server *rejects* the pod. The chart warns at render time when it can tell that has happened.

Key	Type	Default	Description
priorityClasses.create	bool	`true`	Create the PriorityClass objects. Set false when they are managed elsewhere (another release, or cluster-wide by a platform team); the `priorityClassName` references below still apply, so the classes must already exist.
priorityClasses.critical	object	{ "description": "Materialize monitoring components whose loss creates a blind spot with no replica to cover it. Never preempts other workloads.", "name": "monitoring-critical", "preemptionPolicy": "Never", "value": 1000000 }	Priority for components whose loss creates an unrecoverable blind spot — the per-node collectors and the egress gateway.
priorityClasses.scalable	object	{ "description": "Materialize monitoring backends that are replicated or write-buffered. Evicted before the critical tier; never preempts other workloads.", "name": "monitoring-scalable", "preemptionPolicy": "Never", "value": 1000 }	Priority for replicated or buffered components, where losing a pod costs capacity rather than visibility.

### Network policies NetworkPolicies for the components whose own charts ship none. Most of the stack polices itself, and each component is configured in its own block further down: | Component | Key | | -------------------- | -------------------------------- | | Alloy agent | `alloy-agent.networkPolicy` | | Alloy gateway | `alloy-gateway.networkPolicy` | | Loki | `loki.networkPolicy` | | Thanos | `thanos.global.networkPolicies` | | Grafana | `grafana.networkPolicy` | | kube-state-metrics | `kube-state-metrics.networkPolicy` | | node-exporter | `node-exporter.networkPolicy` | This block covers the three subcharts that render no `networkpolicy.yaml` at all — **Alertmanager**, **grafana-operator**, and **metrics-server** — so the stack does not leave three unpoliced pods in an otherwise policed namespace. The policies are rendered by this chart, from `templates/networkpolicies.yaml`. The shape mirrors Loki's, because Loki is the component whose policy set this repo has actually operated: allow the release to talk to itself, allow DNS, name the destinations that live outside the cluster, deny the rest. Each app gets one `Ingress`+`Egress` policy plus a separate DNS policy, for the same reason Loki splits them — a rule you have to disable should not take the rest of the policy with it. **A NetworkPolicy is a declaration, not always a control.** Four limits are worth knowing before treating any of this as a boundary: * It does nothing unless the CNI enforces it. `kindnet` (the default in `kind`, and therefore in the E2E tiers) ignores NetworkPolicy entirely; Cilium and Calico enforce it. * Most CNIs do not apply pod policy to host-networked pods. Nothing in this block is host-networked today, but `node-exporter.networkPolicy` carries the same caveat at greater length, and it is the same caveat. * Rules written with `podSelector` alone stop at the namespace boundary. Under `profiles/split-namespace.values.yaml` the components no longer share a namespace, so those rules match nothing — add `namespaceSelector` entries through the `extra` hooks below, or turn the affected policy off. * Nothing here allows the kubelet's liveness and readiness probes, which arrive from the node's own IP. Cilium and Calico both exempt host-to-pod traffic so probes keep working; a CNI that does not would fail every pod in the namespace, Loki's included, and that is the symptom to recognize. Egress is where a wrong rule hurts: it fails silently, minutes or hours later, somewhere other than the pod you changed. So egress is narrowed only where the destination set is knowable from the chart. Where it is not — the API server's address, kubelet ports on node IPs, an arbitrary notification provider — the rule is broad and says so, and `egress.extra` is where you narrow it against facts only your cluster has. Every component below takes the same eight keys, so they are described once here rather than three times over. Each component's own entry then says only what is particular to it — which ports, and why that egress is as wide as it is. | Key | Meaning | | --- | --- | | `enabled` | Render this policy. Unset follows `networkPolicies.enabled`. | | `ingress.ports` | Ports opened to the rest of the component's namespace. A bare number means TCP; `{port, protocol}` spells out anything else. | | `ingress.allowExternal` | Drop the source restriction entirely, so the ports above accept connections from any namespace and from outside the cluster. | | `ingress.extra` | Raw `NetworkPolicyIngressRule` entries, appended verbatim. The hook for a cross-namespace source. | | `egress.inNamespace` | Reach the other pods of this release, on any port. | | `egress.dns` | Reach cluster DNS. Rendered as a policy of its own. | | `egress.external.ports` / `.cidrs` | Ports and CIDRs outside the cluster. Ports with no CIDRs means those ports *anywhere*. | | `egress.extra` | Raw `NetworkPolicyEgressRule` entries, appended verbatim. |

Key	Type	Default	Description
networkPolicies.enabled	bool	`true`	Render the policies in this block. Off skips all three at once; the per-component `enabled` keys below override it either way.
networkPolicies.alertmanager	object	{ "egress": { "dns": true, "external": { "cidrs": [ "0.0.0.0/0" ], "ports": [ 443, 587, 465 ] }, "extra": [], "inNamespace": true }, "enabled": null, "ingress": { "allowExternal": false, "extra": [], "ports": [ 9093, { "port": 9094, "protocol": "TCP" }, { "port": 9094, "protocol": "UDP" } ] } }	Alertmanager: reachable from the release namespace, free to egress to notification providers. Ingress covers `9093` (the API — Loki's ruler, the Alloy gateway's scrape, and Grafana's alerting all arrive here) and `9094`, the gossip port a multi-replica Alertmanager uses to deduplicate notifications between peers. `9094` is listed twice, TCP and UDP, and both are required. Alertmanager's mesh is memberlist: it joins and pushes state over TCP and gossips over UDP, and the upstream chart's Service declares `clusterpeer-tcp` and `clusterpeer-udp` on the same port for exactly that reason. A TCP-only rule produces the worst version of this failure — the peers find each other, the cluster forms, and it never converges, so every notification goes out once per replica. Both are listed even at the default `replicaCount: 1`, where nothing dials either: the cost is an open port that no pod connects to, and the alternative is a cluster that silently sends every notification twice the day someone scales it up. Egress is deliberately wide. Alertmanager's job is to reach PagerDuty, Slack, an SMTP relay, or a webhook on someone's internal network, and this chart cannot know which — `443` and the SMTP ports are the defaults because they are what receivers use, not because they are a boundary.
networkPolicies.grafana-operator	object	{ "egress": { "dns": true, "external": { "cidrs": [ "0.0.0.0/0" ], "ports": [ 443, 6443 ] }, "extra": [], "inNamespace": true }, "enabled": null, "ingress": { "allowExternal": false, "extra": [], "ports": [ 9090 ] } }	grafana-operator: metrics scraped by the Alloy gateway, egress to the API server and to whichever Grafana it reconciles. Ingress is only the metrics port. `8888` (pprof) is deliberately left out — it is a debugging surface, and a policy that opens it by default defeats the point of having one. Egress carries `443`/`6443` to `0.0.0.0/0` because the operator is an API client first: it watches `Grafana`, `GrafanaDashboard` and `GrafanaDatasource` resources continuously, and a policy that cannot reach the API server leaves it running, healthy, and reconciling nothing. The API server's address is not something a chart can derive — in-cluster it is a Service ClusterIP that most CNIs evaluate against the real endpoint behind it, which on a managed control plane is outside the cluster network entirely. `connections.grafana.mode: external` puts the Grafana it writes to outside the cluster as well; `443` already covers the usual case, but a self-hosted Grafana on another port needs an `egress.extra` entry. The render warns when it can tell that applies.
networkPolicies.grafanaGossip	object	{ "enabled": null, "port": 9094 }	Grafana's unified-alerting gossip port, which the `grafana` subchart's own policy cannot open. A supplement, not a replacement: `grafana.networkPolicy` stays on and keeps doing what it does. That template emits exactly one ingress rule, on `service.targetPort`, with no way to add a second port through values — so a multi-replica Grafana running HA alerting has its `9094` gossip closed by the very policy that protects its UI. The failure is quiet in the worst way: the replicas start healthy, never find each other, and every Grafana-managed alert notifies once per replica. This renders the missing rule from the umbrella chart, which can. TCP and UDP, because memberlist uses both — TCP for the join and for state pushes, UDP for the gossip itself — and a TCP-only rule produces a cluster that forms and then does not converge. It renders only when `grafana.ini.unified_alerting.ha_peers` is set, which is what makes the replicas dial `9094` in the first place; `profiles/grafana-postgres.values.yaml` is the shipped shape that sets it. Nothing is created for a single-replica Grafana that never gossips.
networkPolicies.metrics-server	object	{ "egress": { "dns": true, "external": { "cidrs": [ "0.0.0.0/0" ], "ports": [ 10250, 443, 6443 ] }, "extra": [], "inNamespace": true }, "enabled": null, "ingress": { "allowExternal": true, "extra": [], "ports": [ 10250 ] } }	metrics-server: serves the metrics API to the API server, scrapes every kubelet. Both directions are broad, and both for the same reason: the peers are the control plane and the nodes, neither of which has an address this chart can select. `allowExternal` is true by default here, unlike the other two — the aggregation layer dials `10250` from the API server, which on a managed control plane is not a pod, not in the cluster's pod CIDR, and not reachable by any `podSelector`. A policy that closed that port would break every `kubectl top` and every HPA in the cluster, not just this release. What the policy still buys is the rest of the surface: no other port on the pod accepts a connection, and egress is limited to the kubelet port plus the API server rather than anywhere at all. Egress `10250` to `0.0.0.0/0` is the kubelet summary API on each node. Node IPs are not selectable either, and narrowing this to the node CIDR is a good idea wherever you know it — `egress.extra` with an `ipBlock`, and drop `10250` from `external.ports`.

### Certificates cert-manager `Certificate` resources for in-cluster TLS. **Off by default, and cert-manager is never a hard dependency.** With `enabled: false` this section renders nothing at all, which is what keeps a hardening feature from becoming a new prerequisite for an install that does not want it. When it is on, the chart renders one `Certificate` per component with the full SAN ladder for that component's Services. It does **not** turn TLS on anywhere: issuing a certificate and using it are separate switches, on purpose. Each hop moves through the phases in [Securing](https://materializeinc.github.io/materialize-monitoring/operating/securing/) under its own flag, once that component's renewal behaviour has been proven. **Gated on values, never on `.Capabilities`.** The obvious implementation is to probe the API server for `cert-manager.io/v1`, and it is wrong: the same chart would then render differently under `helm template`, the tier-0 Terraform render check, and an ArgoCD server-side diff than it does under a live install — which is exactly the class of bug the render tests exist to catch. A missing CRD with this flag on is a clear apply-time failure with a name in it. ## Two issuers, because they cannot be one | | Internal | External | | --- | --- | --- | | Names | `$svc`, `$svc.$ns`, `$svc.$ns.svc`, `$svc.$ns.svc.$clusterDomain`, `localhost` | the public DNS name the load balancer answers on | | Typical issuer | a self-signed root, or a private CA | ACME, or a private CA that signs your public names | | Who verifies it | the stack's own components | a browser | A public ACME issuer cannot sign `loki-distributor.monitoring.svc`, and a self-signed root means nothing to a browser, so collapsing these into one key would make one of the two unusable. They are separate `issuerRef`s for that reason and no other. **The external certificate is only needed for an L4 load balancer**, which passes TCP through and leaves TLS to terminate at the pod — so the material has to exist in the cluster. An L7 load balancer terminating with a cloud-managed certificate (ACM, Google Certificate Manager, Azure Key Vault) attaches it by ARN or resource ID and the key never enters the cluster; for that shape, leave `external` unset and pass the annotation through `grafana.service.annotations`.

Key	Type	Default	Description
certificates.enabled	bool	`false`	Render the cert-manager resources in this section.
certificates.internal	object	{ "issuerRef": { "group": "cert-manager.io", "kind": "ClusterIssuer", "name": "" }, "selfSigned": { "caSecretNamespace": "cert-manager", "commonName": "mzmon-internal-ca", "duration": "43800h", "enabled": false, "kind": "ClusterIssuer", "renewBefore": "2160h", "secretName": "mzmon-internal-ca" } }	The internal mesh: certificates the stack's components present to each other, carrying in-cluster DNS names. Supply an issuer with `issuerRef`, or let the chart bootstrap a self-signed root with `selfSigned.enabled`. The two are mutually exclusive and the render refuses both at once. Prefer an issuer scoped to this stack over the cluster's general-purpose one. None of the receiving components here implement per-client authorization — Loki, Thanos receive and Alloy's receivers can be told to require a certificate signed by a given CA, and none of them can be told that this identity may write and that one may not. The whole authorization decision is "is this signed by the CA we trust", so the size of the trust domain is the security property. Reusing a `ClusterIssuer` that signs for every workload in the cluster reduces mTLS to "has any certificate", which is a real improvement over "can reach the Service" and much narrower than it sounds.
certificates.internal.issuerRef	object	{ "group": "cert-manager.io", "kind": "ClusterIssuer", "name": "" }	An existing cert-manager issuer to sign internal certificates. `kind` is `Issuer` or `ClusterIssuer`. Leave `name` empty to use `selfSigned` instead.
certificates.internal.selfSigned	object	{ "caSecretNamespace": "cert-manager", "commonName": "mzmon-internal-ca", "duration": "43800h", "enabled": false, "kind": "ClusterIssuer", "renewBefore": "2160h", "secretName": "mzmon-internal-ca" }	Have the chart bootstrap a self-signed root and issue from it. Renders the three resources cert-manager needs for a private CA: a `selfSigned` issuer, a CA `Certificate` signed by it, and an issuer backed by that CA which every component certificate then references. Convenient rather than principled — a real deployment usually has a PKI already, and `issuerRef` above is the path for it. This exists so that trying the feature does not start with a cert-manager tutorial, and so the chart-only test tier has a root to issue from. `kind: ClusterIssuer` is the default deliberately. A namespaced `Issuer` can only sign for `Certificate` resources in its own namespace, and `profiles/split-namespace.values.yaml` puts components in several — so a namespaced issuer silently signs some components and leaves the rest stuck `Pending`. `Issuer` is correct and cheaper for the single-namespace default; the render warns when it sees the combination that does not work.
certificates.internal.selfSigned.commonName	string	`"mzmon-internal-ca"`	Subject common name on the generated root.
certificates.internal.selfSigned.duration	string	`"43800h"`	Lifetime of the root, not of the leaves. Long, because rotating a root means re-trusting it everywhere at once.
certificates.internal.selfSigned.secretName	string	`"mzmon-internal-ca"`	Secret the generated root's key pair lands in.
certificates.internal.selfSigned.caSecretNamespace	string	`"cert-manager"`	Namespace that Secret is created in. cert-manager's cluster resource namespace, not this release's. A `ClusterIssuer` of kind `ca` reads its Secret from wherever cert-manager was told its cluster resources live — `cert-manager` unless the install overrode `--cluster-resource-namespace` — regardless of where the `Certificate` that produced it was created. Render the CA into the release namespace and the issuer sits `False` with `secret not found` while the Secret it wants is one namespace away. Only read for `kind: ClusterIssuer`. A namespaced `Issuer` has the opposite rule and takes its own namespace; the chart handles both.
certificates.external	object	{ "dnsNames": [], "issuerRef": { "group": "cert-manager.io", "kind": "ClusterIssuer", "name": "" } }	The browser-facing name, for a Grafana behind an L4 load balancer. Only Grafana takes an external certificate today, because it is the only component in the stack meant for a human and the only one this chart helps expose. Set `dnsNames` to the hostname the load balancer answers on; it must match what users actually type, and what `grafana.ini.server.root_url` says. Leave `issuerRef.name` empty and nothing external is rendered, which is the right answer for an L7 load balancer holding a cloud-managed certificate.
certificates.external.dnsNames	list	[]	Public DNS names to put on the certificate.
certificates.duration	string	`"2160h"`	Lifetime of each component certificate, and how long before expiry cert-manager renews it. Short durations are the only mitigation available for the fact that nothing here can check revocation — `otelcol.receiver.otlp` has no CRL support, and in-cluster there is no load balancer to act as a checkpoint, so revoking means re-issuing the CA. They also raise the cost of a component that does not reload its certificate cleanly, which is why no hop turns on until its rotation behaviour is proven. 90 days with 30 days of headroom is a starting point rather than a researched one.
certificates.components	object	{ "alertmanager": { "enabled": null, "extraDnsNames": [], "extraIpAddresses": [], "secretName": "", "services": [ "mzmon-alertmanager" ] }, "alloy-agent": { "enabled": null, "extraDnsNames": [], "extraIpAddresses": [], "secretName": "", "services": [ "alloy-agent" ] }, "alloy-gateway": { "enabled": null, "extraDnsNames": [], "extraIpAddresses": [], "secretName": "", "services": [ "alloy-gateway", "alloy-gateway-cluster" ] }, "grafana": { "enabled": null, "extraDnsNames": [], "extraIpAddresses": [], "secretName": "", "services": [ "grafana" ] }, "loki": { "enabled": null, "extraDnsNames": [], "extraIpAddresses": [], "secretName": "", "services": [ "loki-distributor", "loki-query-frontend" ] }, "thanos": { "enabled": null, "extraDnsNames": [], "extraIpAddresses": [], "secretName": "", "services": [ "thanos-receive", "thanos-query" ] } }	Per-component certificates. One `Certificate` per component rather than one shared across the stack: the SANs differ, the namespaces differ under `split-namespace`, and a single key shared by every workload makes any one compromise total. `services` is the list the SAN ladder is built from, and it is a list rather than a single name because Loki and Thanos are several Services behind one trust boundary. The defaults name exactly the Services this chart's own URLs dial — a render check asserts that every in-cluster destination URL matches a SAN on the corresponding certificate, so a Service added to a destination and forgotten here fails the render rather than the handshake. Each entry takes: \| Key \| Meaning \| \| --- \| --- \| \| `enabled` \| Render this certificate. Unset follows `certificates.enabled`. \| \| `services` \| Service names the SAN ladder is built for. \| \| `extraDnsNames` \| Appended verbatim — an ingress host, a mesh name. \| \| `extraIpAddresses` \| Appended to the IP SANs, which already carry `127.0.0.1`. \| \| `secretName` \| Where the material lands. Defaults to `--tls`. \|

### Materialize Integration Materialize-specific configuration values.

Key	Type	Default	Description
materialize.namespaces	list	[]	Namespaces to scrape Materialize workloads from. Empty list means all namespaces the chart can read.
materialize.environmentd.serviceMonitor	object	{ "enabled": true, "metricEndpoints": [], "selector": {}, "targetLabels": [] }	ServiceMonitor/PodMonitor configuration for environmentd
materialize.environmentd.serviceMonitor.enabled	bool	`true`	Deploy a PodMonitor to collect Materialize metrics.
materialize.environmentd.serviceMonitor.selector	object	{}	Override for default selector
materialize.environmentd.serviceMonitor.targetLabels	list	[]	Override for default labels to copy from the pod into metrics
materialize.environmentd.serviceMonitor.metricEndpoints	list	[]	Override for default metric endpoints
materialize.environmentdSQL.serviceMonitor	object	{ "enabled": true, "metricEndpoints": [], "selector": {}, "targetLabels": [] }	ServiceMonitor/PodMonitor configuration for environmentd-sql
materialize.environmentdSQL.serviceMonitor.enabled	bool	`true`	Deploy a PodMonitor to collect Materialize metrics.
materialize.environmentdSQL.serviceMonitor.selector	object	{}	Override for default selector
materialize.environmentdSQL.serviceMonitor.targetLabels	list	[]	Override for default labels to copy from the pod into metrics
materialize.environmentdSQL.serviceMonitor.metricEndpoints	list	[]	Override for default metric endpoints
materialize.clusterd.serviceMonitor	object	{ "enabled": true, "metricEndpoints": [], "selector": {}, "targetLabels": [] }	ServiceMonitor/PodMonitor configuration for clusterd
materialize.clusterd.serviceMonitor.enabled	bool	`true`	Deploy a PodMonitor to collect Materialize metrics.
materialize.clusterd.serviceMonitor.selector	object	{}	Override for default selector
materialize.clusterd.serviceMonitor.targetLabels	list	[]	Override for default labels to copy from the pod into metrics
materialize.clusterd.serviceMonitor.metricEndpoints	list	[]	Override for default metric endpoints
materialize.deploymentMode	string	`"self-managed"`	Deployment mode normalization hint. One of: `self-managed`, `cloud`. Drives relabeling rules in the pipeline.
materialize-system.namespace	string	`"materialize-environment"`	The namespace materialize-system was deployed into
materialize-system.serviceMonitor	object	{ "enabled": true }	ServiceMonitor/PodMonitor configuration TODO: nothing uses this yet
materialize-operator.namespace	string	`"materialize"`	The namespace materialize-operator was deployed into
materialize-operator.serviceMonitor	object	{ "enabled": true, "metricEndpoints": [], "selector": {}, "targetLabels": [] }	ServiceMonitor/PodMonitor configuration
materialize-operator.serviceMonitor.selector	object	{}	Override for default selector
materialize-operator.serviceMonitor.targetLabels	list	[]	Override for default labels to copy from the pod into metrics
materialize-operator.serviceMonitor.metricEndpoints	list	[]	Override for default metric endpoints

### Pipeline configuration Pipeline configuration values that drive Alloy behavior and defaults. The Alloy pipeline is the flagship deliverable of this chart. Pipeline configuration is expressed in values, rendered into ConfigMaps under `templates/pipelines/`, and consumed by the Alloy agent and gateway release instances.

Key	Type	Default	Description
pipeline.preValidateJob.enabled	bool	`true`	Run a hook/dependency to validate rendered configs before rolling them out
pipeline.preValidateJob.annotations	object	{ "helm.sh/hook": "pre-install,pre-upgrade", "helm.sh/hook-delete-policy": "before-hook-creation" }	Job specific annotations This should be used to control when the job is executed. The default is as a helm hook, but setting any other annotation overrides this.
pipeline.preValidateJob.backoffLimit	int	`1`	Number of times to retry the job before failing the release.
pipeline.preValidateJob.podSecurityContext	object	{ "fsGroup": 473, "runAsGroup": 473, "runAsUser": 473 }	Security context for the pre-validate job pod. This is the hardened recommendation with the alloy user.
pipeline.preValidateJob.containerSecurityContext	object	{ "allowPrivilegeEscalation": false, "capabilities": { "drop": [ "ALL" ] }, "readOnlyRootFilesystem": true, "runAsGroup": 473, "runAsNonRoot": true, "runAsUser": 473 }	Security context for the pre-validate job container. This is the hardened recommendation with the alloy user.
pipeline.env	object	{}	Environment variables to set in agent and gateway pods. These support tpl rendering. Since these are injected into an envFrom, any env entries take precedence.
pipeline.env.CLUSTER_NAME	string	`"default"`	Name of the cluster to discriminate workloads from different sources.
pipeline.env.GATEWAY_LOG_LEVEL	string	`"info"`	Level for the gateway logs.

#### Log configuration Configuration for log behavior

Key	Type	Default	Description
pipeline.logging.agent.rateLimit	int	`5000`	Rate limit for alloy agent incoming pod logs. This is per agent.
pipeline.logging.agent.burst	int	`20000`	Burst limit for alloy agent incoming pod logs.
pipeline.logging.agent.destination.loki.url	string	`"http://alloy-gateway.{{ include \"mzmon.alloyGateway.namespace\" $ }}.svc:3100/loki/api/v1/push"`	alloy-gateway push endpoint URL.
pipeline.logging.agent.destination.loki.retries	object	{ "maxBackoffPeriod": "5m", "maxBackoffRetries": 10, "minBackoffPeriod": "1s", "retryOnHttp429": true }	Retry configuration.
pipeline.logging.agent.destination.loki.authType	string	`"none"`	Type of authentication to use with the alloy-gateway endpoint. Use none if no authentication is required.
pipeline.logging.agent.destination.loki.tls.enabled	bool	`false`	Whether to enable TLS for alloy-gateway dest.
pipeline.logging.agent.destination.loki.tls.verify	bool	`true`	Whether to verify the TLS certificate for the alloy-gateway dest.
pipeline.logging.agent.destination.loki.tls.ca	string	`""`	Certificate Authority (CA) PEM contents for TLS.
pipeline.logging.agent.destination.loki.tls.cert	string	`""`	Client certificate PEM contents for TLS.
pipeline.logging.agent.destination.loki.tls.key	string	`""`	Client private key PEM contents for TLS.
pipeline.logging.agent.destination.loki.tls.caFile	string	`""`	Paths to certificate material on disk, and the preferred carrier for anything cert-manager renews. The inline `ca`/`cert`/`key` above travel through environment variables, which are captured once at process start. cert-manager renews by rewriting the Secret in place, so an env-carried PEM works for exactly one certificate lifetime and then fails on every hop at once, months after the change that caused it and with no deploy nearby to blame. A mounted file does not have that problem: the kubelet refreshes Secret contents atomically, and Alloy's client paths pick up the new material on the next connection. Set these to paths under `/etc/mzmon/tls`, which is where the chart mounts the certificate it issues for this component. Inline PEM stays supported as the bring-your-own-PKI escape hatch.
pipeline.logging.agent.destination.loki.tls.serverName	string	`""`	Alternative SNI (Server Name Indication) to specify.
pipeline.logging.agent.destination.loki.tls.minVersion	string	`"TLS13"`	Minimum TLS version to allow. Use TLS12 if you need better compat. TLS11 and TLS10 are not recommended for production.
pipeline.logging.gateway.server	object	{ "tls": { "certFile": "", "clientAuth": "NoClientCert", "clientCAFile": "", "enabled": false, "keyFile": "", "minVersion": "TLS13", "reloadInterval": "1m" } }	Server-side TLS on this role's listeners. The other half of the `destination..tls` blocks: those configure Alloy as a client, this configures it as a server. A configured client against an unconfigured server is TLS off, which is why the two have to move together. The gateway has three listeners, not two, and they are split across the two pipeline trees the same way the listeners are: \| Listener \| Port \| Configured by \| \| --- \| --- \| --- \| \| `loki.source.api` \| 3100 \| `pipeline.logging.gateway.server.tls` \| \| `otelcol.receiver.otlp` \| 4317 / 4318 \| `pipeline.logging.gateway.server.tls` \| \| `prometheus.receive_http` \| 9090 \| `pipeline.metrics.gateway.server.tls` \| `receive_http` is the one to check last: it lives in the metrics tree and is easy to leave behind, and forgetting it leaves the metrics write path wide open behind a logs path that looks secured. A validator refuses that combination rather than letting it ship. `clientAuth` is exposed rather than implied because the middle state is what a safe rollout needs, and an operator who cannot name that state cannot perform one. Values are Go's `tls.ClientAuthType`: \| Value \| Meaning \| \| --- \| --- \| \| `NoClientCert` \| do not ask (phase 1) \| \| `RequestClientCert` \| ask, do not verify \| \| `RequireAnyClientCert` \| require, do not verify against a CA \| \| `VerifyClientCertIfGiven` \| verify what is presented, allow none (phase 2) \| \| `RequireAndVerifyClientCert` \| require and verify (phase 3) \| Unlike Loki's server, Alloy's listeners are not probed by the kubelet on the same port — the readiness probe is on `12345` — so phase 3 is* reachable here.
pipeline.logging.gateway.server.tls.enabled	bool	`false`	Serve TLS on this role's listeners.
pipeline.logging.gateway.server.tls.certFile	string	`""`	Certificate and key the listener presents. Files rather than inline PEM, because cert-manager renews by rewriting the Secret and Alloy re-reads the file on new connections.
pipeline.logging.gateway.server.tls.clientCAFile	string	`""`	CA used to verify client certificates. Required by every `clientAuth` value except `NoClientCert`.
pipeline.logging.gateway.server.tls.clientAuth	string	`"NoClientCert"`	How hard to insist on a client certificate. See the table above.
pipeline.logging.gateway.server.tls.minVersion	string	`"TLS13"`	Minimum TLS version the listener accepts.
pipeline.logging.gateway.server.tls.reloadInterval	string	`"1m"`	How often the OTLP listeners re-read their certificate files. `otelcol.receiver.otlp` only — the `loki.source.api` listener has no equivalent and re-reads on new connections instead. This is the best renewal behaviour of anything in the stack, so it is worth setting: without it a long-lived gRPC stream can outlive the certificate that established it.
pipeline.logging.gateway.destination.loki.enabled	bool	`true`	Enable writing to a loki destination. By default, we use the in-cluster loki
pipeline.logging.gateway.destination.loki.url	string	`"http://loki-distributor.{{ include \"mzmon.loki.namespace\" $ }}.svc:3100/loki/api/v1/push"`	Loki push endpoint URL.
pipeline.logging.gateway.destination.loki.retries	object	{ "maxBackoffPeriod": "5m", "maxBackoffRetries": 10, "minBackoffPeriod": "1s", "retryOnHttp429": true }	Retry configuration.
pipeline.logging.gateway.destination.loki.authType	string	`"none"`	Type of authentication to use with the loki endpoint. Use none if no authentication is required. Use basicAuth for username/password. Use bearer for bearer token. Use oauth2 for OAuth2 client credentials.
pipeline.logging.gateway.destination.loki.basicAuth	object	{ "password": "", "passwordEnv": "GATEWAY_LOKI_DEST_PASSWORD", "username": "", "usernameEnv": "GATEWAY_LOKI_DEST_USERNAME" }	Configuration for auth when using authType=basicAuth You will need to provide alloy-gateway.alloy.agent.extraEnv TODO: add a check for this
pipeline.logging.gateway.destination.loki.bearer	object	{ "token": "", "tokenEnv": "GATEWAY_LOKI_DEST_BEARER_TOKEN" }	Configuration for bearer token when using authType=bearer This is used for bearer type tokens.
pipeline.logging.gateway.destination.loki.oauth2	object	{ "clientId": "", "clientIdEnv": "GATEWAY_LOKI_DEST_OAUTH2_CLIENT_ID", "clientSecret": "", "clientSecretEnv": "GATEWAY_LOKI_DEST_OAUTH2_CLIENT_SECRET", "scopes": [], "tokenUrl": "", "tokenUrlEnv": "GATEWAY_LOKI_DEST_OAUTH2_TOKEN_URL" }	Configuration for OAuth2 when using authType=oauth2
pipeline.logging.gateway.destination.loki.tls.enabled	bool	`false`	Whether to enable TLS for the loki destination.
pipeline.logging.gateway.destination.loki.tls.verify	bool	`true`	Whether to verify the TLS certificate for the loki destination.
pipeline.logging.gateway.destination.loki.tls.ca	string	`""`	Certificate Authority (CA) PEM contents for TLS.
pipeline.logging.gateway.destination.loki.tls.cert	string	`""`	Client certificate PEM contents for TLS.
pipeline.logging.gateway.destination.loki.tls.key	string	`""`	Client private key PEM contents for TLS.
pipeline.logging.gateway.destination.loki.tls.caFile	string	`""`	Paths to certificate material on disk, and the preferred carrier for anything cert-manager renews. The inline `ca`/`cert`/`key` above travel through environment variables, which are captured once at process start. cert-manager renews by rewriting the Secret in place, so an env-carried PEM works for exactly one certificate lifetime and then fails on every hop at once, months after the change that caused it and with no deploy nearby to blame. A mounted file does not have that problem: the kubelet refreshes Secret contents atomically, and Alloy's client paths pick up the new material on the next connection. Set these to paths under `/etc/mzmon/tls`, which is where the chart mounts the certificate it issues for this component. Inline PEM stays supported as the bring-your-own-PKI escape hatch.
pipeline.logging.gateway.destination.loki.tls.serverName	string	`""`	Alternative SNI (Server Name Indication) to specify.
pipeline.logging.gateway.destination.loki.tls.minVersion	string	`"TLS13"`	Minimum TLS version to allow. Use TLS12 if you need better compat. TLS11 and TLS10 are not recommended for production.
pipeline.logging.gateway.destination.otel.enabled	bool	`false`	Enable writing to an OpenTelemetry/OTLP destination. By default, we do not use the OTLP destination. NOTE: This is configured in the pipeline.metrics.gateway.destination.otlp block, not the logging block.
pipeline.logging.tenancy.staticTenant	string	`"loki"`	Default tenant to write logs to. This is used when tenantMap values is set to `static`.
pipeline.logging.tenancy.tenantMap	object	{ "audit": "static", "default": "static", "environment": "static", "infra": "static" }	Type of tenancy used to write logs. Use static to write to the staticTenant value (recommended). Use byEnvironment to separate by the environment name. Use byNamespace to use the namespace of the source pod as the tenant. Use byLabel to use a label (tenantLabel) to specifically identify the tenant. Use none to disable tenancy (only use if loki does not have tenancy).

#### Metrics configuration Configuration for metrics behavior

Key	Type	Default	Description
pipeline.metrics.kubelet	h5	`{"scrapeInterval":"60s", "tlsInsecureSkipVerify":false}`	Node-local container metrics, scraped from each node's kubelet. The gateway scrapes `/metrics/cadvisor` on every kubelet rather than the agent running its own cAdvisor. The kubelet already computes these stats, so running our own meant computing them twice per node — measured at ~750Mi per agent against a 200Mi logs-only envelope. Scraping the kubelet removes the housekeeping cost entirely and puts the remaining scrape cost on the gateway, where it is shared across replicas rather than reserved on every node — the distinction that matters when bin packing. Coverage is not the tradeoff it was assumed to be: a GKE kubelet serves 69 distinct `container_` metrics against the 70 an in-process cAdvisor produced, including the descriptor / socket / thread family that otherwise needs the host PID namespace. Every `container_` metric this chart's queries reference is present. Needs `nodes` and `nodes/metrics` RBAC, which the Alloy subchart's default `clusterRules` already grant.
pipeline.metrics.kubelet.scrapeInterval	string	`"60s"`	Scrape interval for the kubelet cAdvisor endpoint. The dominant cost lever — roughly 6.7k series per node per scrape.
pipeline.metrics.kubelet.tlsInsecureSkipVerify	bool	`false`	Skip verification of the kubelet's serving certificate. Left false: on GKE the kubelet certificate verifies against the in-cluster CA, which the chart passes as `ca_file`. A distribution that signs kubelet certs with a CA the pod does not trust needs this true, and the failure is quiet — scrapes fail and container metrics stop rather than anything erroring at install. Check `up{job="cadvisor"}` when bringing up a new distribution.
pipeline.metrics.gateway.denyMetrics	list	[]	Denylist of metrics that are excluded from being exported These are \|'d in a regex pattern
pipeline.metrics.gateway.server	object	{ "tls": { "certFile": "", "clientAuth": "NoClientCert", "clientCAFile": "", "enabled": false, "keyFile": "", "minVersion": "TLS13", "reloadInterval": "1m" } }	Server-side TLS on this role's listeners. The other half of the `destination..tls` blocks: those configure Alloy as a client, this configures it as a server. A configured client against an unconfigured server is TLS off, which is why the two have to move together. The gateway has three listeners, not two, and they are split across the two pipeline trees the same way the listeners are: \| Listener \| Port \| Configured by \| \| --- \| --- \| --- \| \| `loki.source.api` \| 3100 \| `pipeline.logging.gateway.server.tls` \| \| `otelcol.receiver.otlp` \| 4317 / 4318 \| `pipeline.logging.gateway.server.tls` \| \| `prometheus.receive_http` \| 9090 \| `pipeline.metrics.gateway.server.tls` \| `receive_http` is the one to check last: it lives in the metrics tree and is easy to leave behind, and forgetting it leaves the metrics write path wide open behind a logs path that looks secured. A validator refuses that combination rather than letting it ship. `clientAuth` is exposed rather than implied because the middle state is what a safe rollout needs, and an operator who cannot name that state cannot perform one. Values are Go's `tls.ClientAuthType`: \| Value \| Meaning \| \| --- \| --- \| \| `NoClientCert` \| do not ask (phase 1) \| \| `RequestClientCert` \| ask, do not verify \| \| `RequireAnyClientCert` \| require, do not verify against a CA \| \| `VerifyClientCertIfGiven` \| verify what is presented, allow none (phase 2) \| \| `RequireAndVerifyClientCert` \| require and verify (phase 3) \| Unlike Loki's server, Alloy's listeners are not probed by the kubelet on the same port — the readiness probe is on `12345` — so phase 3 is* reachable here.
pipeline.metrics.gateway.server.tls.enabled	bool	`false`	Serve TLS on this role's listeners.
pipeline.metrics.gateway.server.tls.certFile	string	`""`	Certificate and key the listener presents. Files rather than inline PEM, because cert-manager renews by rewriting the Secret and Alloy re-reads the file on new connections.
pipeline.metrics.gateway.server.tls.clientCAFile	string	`""`	CA used to verify client certificates. Required by every `clientAuth` value except `NoClientCert`.
pipeline.metrics.gateway.server.tls.clientAuth	string	`"NoClientCert"`	How hard to insist on a client certificate. See the table above.
pipeline.metrics.gateway.server.tls.minVersion	string	`"TLS13"`	Minimum TLS version the listener accepts.
pipeline.metrics.gateway.server.tls.reloadInterval	string	`"1m"`	How often the OTLP listeners re-read their certificate files. `otelcol.receiver.otlp` only — the `loki.source.api` listener has no equivalent and re-reads on new connections instead. This is the best renewal behaviour of anything in the stack, so it is worth setting: without it a long-lived gRPC stream can outlive the certificate that established it.

##### Prometheus remote-write destinations. Prometheus remote-write destinations, as a map of name to destination. **A map, not a single destination.** Each key names one destination and becomes the label on that destination's Alloy components, so the name is visible in `component_id` on the gateway's own metrics and in its UI. The default map holds one entry, `thanos`, pointing at the bundled in-cluster Thanos Receive. Every destination gets its **own** `prometheus.remote_write` component rather than a second `endpoint` block on a shared one. That costs a WAL per destination and buys the two things a shared component cannot give: * **Its own `minMetricImportance`.** The tier filter is a `prometheus.relabel` *upstream* of the component, so a destination on `essential` writes a WAL holding only essential series. A shared component can only filter with `write_relabel_config`, which runs on the way *out* of the WAL — every destination would pay full firehose disk regardless of tier. This is the whole reason the tiers exist: Amazon Managed Prometheus bills per sample ingested and per series. * **Its own failure domain.** A destination that stops accepting writes backs up its own WAL and nothing else. On a shared component a stuck endpoint holds back WAL truncation for every other endpoint too. `external_labels` is per destination for the same reason, though the default (`cluster` from `CLUSTER_NAME`) is what every destination wants. **Names are Alloy component labels**, so they must match `[a-zA-Z_][a-zA-Z0-9_]*`, and `egress` is reserved — the fan-out seam already uses it. A name that breaks either rule fails at render. **Fields are optional.** Anything a destination omits falls back to the per-destination defaults below, so a second destination is usually three lines. Environment variable names are derived from the destination name when not given — `GATEWAY_PROM_DEST_AMP`, `GATEWAY_PROMETHEUS_DEST_AMP_PASSWORD`, `GATEWAY_UNFILTERED_PROM_METRICS_AMP` — and can be set explicitly where a caller (such as the Terraform module's `mzmon-alloy-gateway-env` Secret) wants to choose them. | Field | Default | Meaning | | --- | --- | --- | | `enabled` | `true` | Write to this destination. | | `url` | — | Remote-write endpoint. Required when enabled. | | `minMetricImportance` | `all` | Tier floor: `essential`, `recommended`, `extended`, `diagnostic`, `all`. | | `unfilteredMetricsEnv` | `GATEWAY_UNFILTERED_PROM_METRICS_` | Env var carrying the tier allowlist regex. | | `urlEnv` | `GATEWAY_PROM_DEST_` | Env var carrying the endpoint URL. | | `externalLabels` | `{cluster: CLUSTER_NAME}` | `external_labels` on the component. Values are Alloy expressions. | | `authType` | `none` | `none`, `basicAuth`, `bearer`, `oauth2`, or `sigv4`. | | `basicAuth` / `bearer` / `oauth2` / `sigv4` | see below | Credentials for the chosen `authType`. | | `tls` | off | Client TLS for this hop. | A worked two-destination example — full fidelity in Thanos, only the alerting metrics in AMP — is in [`profiles/aws-amp-fanout.values.yaml`](https://github.com/MaterializeInc/materialize-monitoring/blob/main/charts/materialize-monitoring/profiles/aws-amp-fanout.values.yaml). ##### OpenTelemetry/OTLP destinations. OpenTelemetry/OTLP destination configuration for metrics (and logging). This supports several components and allows further customization of endpoints. Multiple exporters can be enabled at once.

Key	Type	Default	Description
pipeline.metrics.gateway.destination.otel.enabled	bool	`false`	Enable writing metrics to an OpenTelemetry/OTLP destination. By default, we do not use the OTel destination. WARNING: if logging otel destination is enabled, this block will still be used for configuration! (just not for metrics)
pipeline.metrics.gateway.destination.otel.googleCloudExporter.enabled	bool	`false`	Enable writing to a Google Cloud Monitoring / Cloud Logging destination.
pipeline.metrics.gateway.destination.otel.googleCloudExporter.compression	string	`"gzip"`	Compression for logs/metrics Only gzip is supported for Google Cloud Monitoring / Logging.
pipeline.metrics.gateway.destination.otel.googleCloudExporter.minMetricImportance	string	`"recommended"`	Only export metrics with the specified importance level. Values are "essential", "recommended", "extended", "diagnostic", "all"
pipeline.metrics.gateway.destination.otel.googleCloudExporter.handlers	list	[ "otelcol.exporter.googlecloud.destination.input" ]	Handlers to use for the Google Cloud exporter.
pipeline.metrics.gateway.destination.otel.googleCloudExporter.config	string	An `otelcol.exporter.googlecloud.destination` definition.	Raw configuration for an otelcol.exporter.googlecloud block. The default config uses Workload Identity Federation (WIF) to authenticate to GCP.
pipeline.metrics.gateway.destination.otel.datadogExporter.enabled	bool	`false`	Enable writing to a Datadog destination.
pipeline.metrics.gateway.destination.otel.datadogExporter.compression	string	`"gzip"`	Compression for logs/metrics Only gzip is supported for Datadog.
pipeline.metrics.gateway.destination.otel.datadogExporter.minMetricImportance	string	`"recommended"`	Only export metrics with the specified importance level. Values are "essential", "recommended", "extended", "diagnostic", "all"
pipeline.metrics.gateway.destination.otel.datadogExporter.hostMetadata.enabled	bool	`false`	Whether to include host metadata in the Datadog exporter. FIXME: how do we support this in an Agent->Gateway architecture?
pipeline.metrics.gateway.destination.otel.datadogExporter.handlers	list	[ "otelcol.exporter.datadog.destination.input" ]	Handlers to use for the Datadog exporter.
pipeline.metrics.gateway.destination.otel.datadogExporter.config	string	An `otelcol.exporter.datadog.destination` definition.	Raw configuration for an otelcol.exporter.datadog block.
pipeline.metrics.gateway.destination.otel.otlpExporter.enabled	bool	`false`	Enable writing to an OpenTelemetry/OTLP destination. This is the generic fallback for other destinations. You can also use this if you need to set a custom destination block.
pipeline.metrics.gateway.destination.otel.otlpExporter.url	string	`""`	OTLP push endpoint URL. This does not need a protocol prefix (http:// or https://)
pipeline.metrics.gateway.destination.otel.otlpExporter.protocol	string	`"grpc"`	Protocol to use for OTLP. Use grpc for gRPC protocol. Use http for HTTP protocol.
pipeline.metrics.gateway.destination.otel.otlpExporter.compression	string	`"gzip"`	Compression to use Use gzip for better compatibility. Use snappy for better performance.
pipeline.metrics.gateway.destination.otel.otlpExporter.minMetricImportance	string	`"all"`	Only export metrics with the specified importance level. Values are "essential", "recommended", "extended", "diagnostic", "all"
pipeline.metrics.gateway.destination.otel.otlpExporter.handlers	list	[ "otelcol.exporter.otlp[http].destination.input" ]	Handlers to use for the OTLP exporter. Update this if your config was customized.
pipeline.metrics.gateway.destination.otel.otlpExporter.config	string	{{- $exporterType := ternary "otlp" "otlphttp" ( eq .Values.pipeline.metrics.gateway.destination.otel.otlpExporter.protocol "grpc" ) }} otelcol.exporter.{{ $exporterType }} "destination" { client { endpoint = {{ .Values.pipeline.metrics.gateway.destination.otel.otlpExporter.url \| required "destination.otlp.url must be set" \| quote }} compression = {{ .Values.pipeline.metrics.gateway.destination.otel.otlpExporter.compression \| quote }} {{- if ( include "mzmon.alloyGateway.otelDest.authEnabled" $ ) }} auth = {{ include "mzmon.alloyGateway.otelDest.authHandler" $ }} {{- end }} } }	Raw configuration for an otelcol.exporter.otlp block. Use this if you need to configure multiple destinations or use an alternative endpoint entirely.

##### OpenTelemetry Authentication Configuration Configuration for OpenTelemetry/OTLP destinations. This is only needed for destinations that require an auth handler (like otlpExporter).

Key	Type	Default	Description
pipeline.metrics.gateway.destination.otel.auth.authType	string	`"none"`	Type of authentication to use with the OpenTelemetry destination. Valid values are: `none`, `basic`, `bearer`, `headers`, `awsSigv4`, and `custom`.
pipeline.metrics.gateway.destination.otel.auth.basic.config	string	An `otelcol.auth.basic.oteldest` definition.	Raw configuration for an otelcol.auth.basic block. This uses the GATEWAY_OTEL_DEST_USERNAME/GATEWAY_OTEL_DEST_PASSWORD env vars.
pipeline.metrics.gateway.destination.otel.auth.bearer.config	string	An `otelcol.auth.bearer.oteldest` definition.	Raw configuration for an otelcol.auth.bearer block. This uses the GATEWAY_OTEL_DEST_BEARER_TOKEN env var.
pipeline.metrics.gateway.destination.otel.auth.headers.headers	list	[]	Headers attached to every request to the OpenTelemetry destination. For backends that authenticate with an API-key header rather than a bearer token — Honeycomb's `x-honeycomb-team`, for instance. Before this existed the only way to send one was `authType: custom` and a hand-written `otelcol.auth.headers` block, which no validator could check. Each entry names a header with `key` and sets exactly one of: - `value` — a literal. It renders into the gateway's pipeline ConfigMap in plaintext, so use it only for non-secret routing headers such as a dataset or tenant name. - `valueEnv` — the name of an environment variable, read with `sys.env()` when the gateway starts. Use it for anything secret and supply the variable through `alloy-gateway.alloy.envFrom` (a secretRef). The name is yours to choose; nothing else in the chart depends on it. Unlike the Loki and Prometheus destinations, there is no `value`/`valueEnv` pair per header where the value is copied into the env ConfigMap. A header is either a literal or a lookup, and routing a literal through the env ConfigMap would only move it from one plaintext ConfigMap into another.
pipeline.metrics.gateway.destination.otel.auth.headers.config	string	An `otelcol.auth.headers.oteldest` definition built from `headers`.	Raw configuration for an otelcol.auth.headers block.
pipeline.metrics.gateway.destination.otel.auth.awsSigv4.region	string	`""`	Override the region to sign requests for.
pipeline.metrics.gateway.destination.otel.auth.awsSigv4.roleArn	string	`""`	Override the role ARN to assume for signing requests.
pipeline.metrics.gateway.destination.otel.auth.awsSigv4.config	string	An `otelcol.auth.sigv4.oteldest` definition.	Raw configuration for an otelcol.auth.sigv4 block.
pipeline.metrics.gateway.destination.otel.auth.custom.handler	string	`"{{ fail \"Be sure to set this\" }}"`	Handler for a custom auth handler (Advanced escape hatch). This should point to the definition you used. Most auth types use `.handler` for their capsule export. If you need multiple handlers, you can skip this field and instead modify your otlpExporter.config to not use "mzmon.alloyGateway.otelDest.authHandler".
pipeline.metrics.gateway.destination.otel.auth.custom.config	string	// THIS IS AN EXAMPLE otelcol.auth.basic "oteldest" { client_auth { username = sys.env("GATEWAY_OTEL_DEST_USERNAME") password = sys.env("GATEWAY_OTEL_DEST_PASSWORD") } }	Raw configuration for a custom auth handler.

### Monitoring configurations Configuration for dashboards, rules, and alerts Underlying content is generated into `pre-rendered/` from the sources under `packages/` and embedded via `.Files.Get`.

Key	Type	Default	Description
dashboards.config.grafana.enabled	bool	`true`	Install the bundled Grafana dashboards. Requires the Grafana operator or a writable Grafana instance.
dashboards.config.grafana.mode	string	`"operator"`	Grafana deployment mode, either "standalone" (the bundled Grafana chart) or "operator" (a separate Grafana Operator instance).
dashboards.config.grafana.manifest.resyncPeriod	string	`"5m"`	Time to sync the dashboard from the manifest
dashboards.config.grafana.manifest.instanceSelector	object	{}	Non-default label selector for a Grafana-operator Grafana instance. Defaults to the labels on the `Grafana` instance this chart creates (see `connections.grafana.labels`), so the two cannot drift.
dashboards.config.grafana.manifest.allowCrossNamespaceImport	string	`inferred`	Allow dashboards to match a Grafana instance outside their own namespace. Left unset, this is inferred — it turns on only when the `Grafana` resource lands in a different namespace than the dashboards, as it does under the `split-namespace` profile. Set it explicitly when pointing `instanceSelector` at an instance this chart does not create. Note that the CRDs forbid turning this back off in place; the resource has to be recreated.
dashboards.config.grafana.manifest.apiTarget	string	`"dashboard.grafana.app/v2"`	Dashboard API Version (v2 or v2beta1)

##### Grafana folders. Grafana folders the dashboards are filed into. **A map, not a list.** Each key names one folder and becomes both the `GrafanaFolder` resource name and, unless `existingUid` overrides it, the folder **UID**: `-`, which is `mzmon-` at the chart's default `fullnameOverride`. **The keys are the contract with the dashboards.** A dashboard carries its placement as a `grafana.app/folder` annotation naming a folder — the names are `Folder` in `packages/mzmon-lib/src/grafana/folder.rs`, which spells `infra`, `materialize` and `meta-o11y` — and the chart rewrites that name to the UID below on the way to the operator, the same way it rewrites `apiVersion`. So the UID is free to move with the release or with `existingUid`, and the dashboards follow it without being re-rendered. What does *not* move freely is a key: rename one and the dashboards that name it have their annotation dropped and land at the root instead. Adopt a folder with `existingUid` rather than renaming, or re-render the dashboards against the new name. Only rendered when `mode` is `operator`; the standalone Grafana chart has no folder resource to create. | Field | Default | Meaning | | --- | --- | --- | | `create` | `true` | Render a `GrafanaFolder` for this entry. `false` still resolves dashboards to its UID, for a folder created out of band. | | `title` | — | Display name in Grafana. Unlike the key, this is free to change. | | `existingUid` | `""` | Adopt the folder with this UID instead of deriving one from the key. | | `parent.folderRef` | — | Nest under another key in this map. Refers to that entry's resource name, so it needs `create: true`. | | `parent.folderUID` | — | Nest under a folder UID this chart does not manage. Takes precedence over `folderRef`. |

Key	Type	Default	Description
dashboards.config.datadog.enabled	bool	`false`	Install the bundled Datadog dashboards. Requires Datadog API credentials configured out-of-band.
dashboards.selected	list	[ "env-", "infra-" ]	List of dashboard patterns to render

#### Rule configuration Configuration for rules

Key	Type	Default	Description
config.rules.prometheus.enabled	bool	`true`	Install the bundled Prometheus recording and alerting rules as PrometheusRule resources.
config.rules.loki.enabled	bool	`false`	Install the bundled Loki rules.
config.rules.thanos.enabled	bool	`false`	Install the bundled Thanos rules.

#### Alert configuration Configuration for alerts

Key	Type	Default	Description
config.alerts.enabled	bool	`true`	Install the bundled Alertmanager routing and templates.

#### Scraper configuration Configuration for scrapers

Key	Type	Default	Description
config.scrapers.enabled	bool	`true`	Install ServiceMonitors / Alloy scrape configs for Materialize and adjacent components.

#### Grafana connection configuration How to talk to a grafana instance

Key	Type	Default	Description
connections.grafana.mode	string	`"bundled"`	How this establishes its connection to Grafana. `bundled` (default) targets the Grafana deployed by the bundled `grafana` subchart; the URL and admin-credential Secret are derived from it. `external` targets a Grafana you already run — Grafana Cloud, a shared platform Grafana, another cluster — and requires `external.url` plus either `external.apiKey` or `external.adminUser` + `external.adminPassword`. `operator` hands the instance lifecycle to grafana-operator itself, which builds it from the operator's own defaults and is not yet production-ready.
connections.grafana.labels	object	{}	Additional labels applied to the Grafana instance, and to the `instanceSelector` of every Grafana resource this chart targets at it. Merged over a static `monitoring.materialize.cloud/grafana-instance: mzmon` label, which is what keeps the selector non-empty — grafana-operator reads an empty `matchLabels` as every instance, not none. Add to this to narrow the selector further, e.g. to scope per release when two `materialize-monitoring` releases share a cluster.
connections.grafana.allowPublicAccess	bool	`false`	Acknowledge that Grafana is deliberately reachable from outside the cluster's private network. Left false, the chart refuses to render a `LoadBalancer` (or `NodePort`) Service with no `grafana.service.loadBalancerSourceRanges`, and an Ingress with no source-range allowlist it can see. That mirrors the load-balancer convention the Terraform modules already enforce: internal by default, and public only against an explicit allowlist. Setting this true is the escape hatch for a deployment whose allowlist lives somewhere the chart cannot read — a security group, an egress firewall, a service mesh, an authenticating proxy in front. It suppresses the error, not the exposure: the render still warns, because a Grafana on the open internet is protected by nothing but its admin password until you configure an identity provider under `grafana.ini`.
connections.grafana.operator	object	{}	Spec passed through to the `Grafana` resource in `mode: operator`. Break-glass. In that mode grafana-operator owns the server lifecycle and builds it from its own stock defaults — unpinned image, no persistence, no admin secret, no resource requests — none of which this chart models. This key is the raw `GrafanaSpec` escape hatch for configuring it anyway: whatever you put here is emitted verbatim as the resource's `spec`. Nothing in it is validated or defaulted, and the production guidance on the `grafana` block above does not reach it. Prefer `mode: bundled`, where the `grafana` subchart owns the Deployment and all of that applies. ```yaml connections: grafana: mode: operator operator: spec: version: "13.0.2" deployment: spec: template: spec: containers: - name: grafana resources: requests: cpu: 100m memory: 256Mi config: database: type: postgres host: grafana-db.example.internal:5432 ``` Reference: https://grafana.github.io/grafana-operator/docs/api/#grafanaspec
connections.grafana.external.url	string	`""`	External grafana uri
connections.grafana.external.adminPassword	object	{}	Secret for Grafana admin password
connections.grafana.external.adminUser	object	{}	Secret for Grafana admin user
connections.grafana.external.apiKey	object	{}	Secret for Grafana API key

#### Datasource configuration Datasources provisioned into the Grafana instance Provisioned as `GrafanaDatasource` resources, pushed into the same instance the dashboards target. `url` values are rendered with `tpl`, so they may reference chart helpers.

Key	Type	Default	Description
connections.datasources.enabled	bool	`true`	Install the bundled datasources. Requires the Grafana operator.
connections.datasources.resyncPeriod	string	`"5m"`	How often the operator re-pushes each datasource.
connections.datasources.editable	bool	`false`	Whether the datasources can be edited in the Grafana UI. Edits are reverted on the next resync either way; this only hides the controls so the reversion is not a surprise.
connections.datasources.thanos.enabled	string	follows `thanos.enabled	Provision the Thanos datasource. Unset follows whether the bundled Thanos is enabled. Set it explicitly to point Grafana at metrics storage this chart does not deploy.
connections.datasources.thanos.name	string	`"Thanos"`	Datasource name, as shown in Grafana.
connections.datasources.thanos.uid	string	`"mzmon-thanos"`	Stable datasource UID.
connections.datasources.thanos.url	string	`"http://thanos-query.{{ include \"mzmon.thanos.namespace\" $ }}.svc:9090"`	Thanos Query endpoint. Rendered with `tpl`.
connections.datasources.thanos.isDefault	bool	`true`	Make this Grafana's default datasource. The bundled dashboards deliberately do not pin a datasource: their `${metricsDatasource}` variable resolves to whichever Prometheus-type datasource is default. With no default, every panel renders empty and Grafana reports no error. Only turn this off if something else in the instance is already the default Prometheus datasource.
connections.datasources.thanos.tls	object	{ "caPem": "", "caSecret": { "key": "ca.crt", "name": "" }, "clientCert": { "certKey": "tls.crt", "keyKey": "tls.key", "secretName": "" }, "enabled": null, "serverName": "" }	TLS for Grafana's connection to this backend. Grafana does not read certificate material from files. It stores it in its own database as `secureJsonData`, provisioned through the datasource — which is why this is the one hop in the stack that does not renew on its own. cert-manager rewriting the Secret changes nothing until the datasource is re-provisioned; grafana-operator does that every `connections.datasources.resyncPeriod`, so the material is refreshed on that cadence rather than on the certificate's. `caSecret` names a Secret in the Grafana instance's namespace holding the roots to trust. Under `split-namespace` that is `grafana`, not the release namespace, and the certificate machinery already issues `-grafana-tls` there — its `ca.crt` key is the internal CA, so the default below works as-is once `certificates.enabled` is on. Prefer this over `jsonData.tlsSkipVerify`. Skipping verification leaves the connection encrypted and unauthenticated, which on the read path means Grafana will happily talk to anything that answers on that address — and the failure mode of getting it wrong is a dashboard that renders, from the wrong source.
connections.datasources.thanos.tls.enabled	string	`follows the URL scheme`	Verify the backend's certificate against `caSecret`. Unset follows whether the datasource URL is `https://`, so moving the URL is enough and this does not become a second switch to forget.
connections.datasources.thanos.tls.caPem	string	`""`	The CA to trust, inline as PEM. A CA certificate is public material — it is the thing you hand out — so putting it in values is not the leak that an inline key would be. Takes precedence over `caSecret`, and needs no operator involvement at all, which makes it the right choice when the CA is already in hand at render time. It does not track a rotation: the PEM here is a copy, and re-issuing the CA means re-rendering. Prefer `caSecret` when the material lives in the cluster.
connections.datasources.thanos.tls.caSecret	object	{ "key": "ca.crt", "name": "" }	Secret holding the CA bundle, referenced rather than inlined. Rendered as a `valuesFrom` entry on the `GrafanaDatasource` plus a placeholder for the operator to substitute into. Survives a CA rotation, which `caPem` does not, so this is the better default whenever the CA already lives in the cluster — a cert-manager issuer's own Secret, say. The Secret must be in the Grafana instance's namespace, which is not necessarily this release's under `split-namespace`. `key` is load-bearing beyond naming the field: grafana-operator substitutes `${}` — the `secretKeyRef` key verbatim, dots and all — so the chart derives the placeholder from it. A mismatch there fails in the least visible way available: the CR applies, the operator logs nothing, `secureJsonFields` reports `tlsCACert: true`, and only at query time does Grafana say `failed to parse TLS CA PEM certificate`, because what it stored is the literal placeholder text.
connections.datasources.thanos.tls.clientCert	object	{ "certKey": "tls.crt", "keyKey": "tls.key", "secretName": "" }	Present a client certificate as well. Only useful against a backend that requires one; Loki cannot (its port is probed by the kubelet), so this is here for an external backend that does.
connections.datasources.thanos.tls.serverName	string	`""`	SNI to send, when it differs from the URL host.
connections.datasources.thanos.jsonData	object	{}	Extra `jsonData`, merged over the chart's defaults (`prometheusType: Thanos`, `httpMethod: POST`).
connections.datasources.thanos.secureJsonData	object	{}	Inline `secureJsonData`. Prefer `valuesFrom` for real secrets — this renders into the release manifest.
connections.datasources.thanos.valuesFrom	list	[]	Secret- or ConfigMap-sourced field injection, passed through to the `GrafanaDatasource`. This is the supported way to supply credentials. ```yaml valuesFrom: - targetPath: secureJsonData.basicAuthPassword valueFrom: secretKeyRef: name: thanos-basic-auth key: password ```
connections.datasources.loki.enabled	string	follows `loki.enabled	Provision the Loki datasource. Unset follows whether the bundled Loki is enabled.
connections.datasources.loki.name	string	`"Loki"`	Datasource name, as shown in Grafana.
connections.datasources.loki.uid	string	`"mzmon-loki"`	Stable datasource UID.
connections.datasources.loki.url	string	`"http://loki-query-frontend.{{ include \"mzmon.loki.namespace\" $ }}.svc:3100"`	Loki read endpoint. Rendered with `tpl`. The Loki gateway is disabled by default, so reads go to the query frontend directly (see `loki.gateway.enabled`).
connections.datasources.loki.tenant	string	follows `pipeline.logging.tenancy.staticTenant	Tenant to read as, sent in the `X-Scope-OrgID` header. The bundled Loki runs `auth_enabled: true`, so reads without this header fail with `no org id`. Unset follows the tenant the pipeline writes to. Set to `""` to send no header, which is only correct against a Loki with `auth_enabled: false`.
connections.datasources.loki.tls	object	{ "caPem": "", "caSecret": { "key": "ca.crt", "name": "" }, "clientCert": { "certKey": "tls.crt", "keyKey": "tls.key", "secretName": "" }, "enabled": null, "serverName": "" }	TLS for Grafana's connection to this backend. Grafana does not read certificate material from files. It stores it in its own database as `secureJsonData`, provisioned through the datasource — which is why this is the one hop in the stack that does not renew on its own. cert-manager rewriting the Secret changes nothing until the datasource is re-provisioned; grafana-operator does that every `connections.datasources.resyncPeriod`, so the material is refreshed on that cadence rather than on the certificate's. `caSecret` names a Secret in the Grafana instance's namespace holding the roots to trust. Under `split-namespace` that is `grafana`, not the release namespace, and the certificate machinery already issues `-grafana-tls` there — its `ca.crt` key is the internal CA, so the default below works as-is once `certificates.enabled` is on. Prefer this over `jsonData.tlsSkipVerify`. Skipping verification leaves the connection encrypted and unauthenticated, which on the read path means Grafana will happily talk to anything that answers on that address — and the failure mode of getting it wrong is a dashboard that renders, from the wrong source.
connections.datasources.loki.tls.enabled	string	`follows the URL scheme`	Verify the backend's certificate against `caSecret`. Unset follows whether the datasource URL is `https://`, so moving the URL is enough and this does not become a second switch to forget.
connections.datasources.loki.tls.caPem	string	`""`	The CA to trust, inline as PEM. A CA certificate is public material — it is the thing you hand out — so putting it in values is not the leak that an inline key would be. Takes precedence over `caSecret`, and needs no operator involvement at all, which makes it the right choice when the CA is already in hand at render time. It does not track a rotation: the PEM here is a copy, and re-issuing the CA means re-rendering. Prefer `caSecret` when the material lives in the cluster.
connections.datasources.loki.tls.caSecret	object	{ "key": "ca.crt", "name": "" }	Secret holding the CA bundle, referenced rather than inlined. Rendered as a `valuesFrom` entry on the `GrafanaDatasource` plus a placeholder for the operator to substitute into. Survives a CA rotation, which `caPem` does not, so this is the better default whenever the CA already lives in the cluster — a cert-manager issuer's own Secret, say. The Secret must be in the Grafana instance's namespace, which is not necessarily this release's under `split-namespace`. `key` is load-bearing beyond naming the field: grafana-operator substitutes `${}` — the `secretKeyRef` key verbatim, dots and all — so the chart derives the placeholder from it. A mismatch there fails in the least visible way available: the CR applies, the operator logs nothing, `secureJsonFields` reports `tlsCACert: true`, and only at query time does Grafana say `failed to parse TLS CA PEM certificate`, because what it stored is the literal placeholder text.
connections.datasources.loki.tls.clientCert	object	{ "certKey": "tls.crt", "keyKey": "tls.key", "secretName": "" }	Present a client certificate as well. Only useful against a backend that requires one; Loki cannot (its port is probed by the kubelet), so this is here for an external backend that does.
connections.datasources.loki.tls.serverName	string	`""`	SNI to send, when it differs from the URL host.
connections.datasources.loki.jsonData	object	{}	Extra `jsonData`, merged over the chart's defaults (the tenant header name, and `timeout`).
connections.datasources.loki.secureJsonData	object	{}	Inline `secureJsonData`. Prefer `valuesFrom` for real secrets — this renders into the release manifest.
connections.datasources.loki.valuesFrom	list	[]	Secret- or ConfigMap-sourced field injection, passed through to the `GrafanaDatasource`. This is the supported way to supply credentials.

#### Uninstall cleanup Deleting resources this chart cannot delete on its own. grafana-operator puts `operator.grafana.com/finalizer` on the custom resources it reconciles, and clears it only after removing the corresponding object from the Grafana instance. Helm's uninstall does not order that against the operator's own removal, so the ordinary teardown races: the operator Deployment goes away with everything else, nobody is left to process the finalizers, and the `GrafanaManifest` / `GrafanaDatasource` objects sit in `Terminating` forever. The namespace then will not delete either, and the next install adopts the leftovers. A `pre-delete` hook is the fix because of when it runs: before Helm removes anything, so the operator is still up and still watching. `kubectl delete` blocks until the objects are actually gone, which is the point — it returns only once the finalizers have been processed, and Helm proceeds from there.

Key	Type	Default	Description
cleanup.grafanaOperator.enabled	bool	`true`	Run the pre-delete cleanup hook. Turning this off restores the hang described above; the manual recovery is to delete the resources yourself before `helm uninstall`, or to clear the finalizers by hand afterwards.
cleanup.grafanaOperator.kinds	list	[ "grafanamanifests.grafana.integreatly.org", "grafanadatasources.grafana.integreatly.org" ]	Resource types to delete, as `.`. Fully qualified on purpose: a bare `grafanamanifests` resolves through discovery and can collide with another CRD of the same short name. Only kinds that actually carry the operator's finalizer belong here — the `Grafana` instance CR does not, so Helm removes it unaided. Extend this if you add your own operator resources (`GrafanaFolder`, `GrafanaAlertRuleGroup`, and so on) with the chart's instance label.
cleanup.grafanaOperator.timeout	string	`"2m"`	How long `kubectl delete` waits for the finalizers to clear. This bounds the hook rather than the uninstall: on expiry the objects are already marked for deletion, the hook fails, and Helm stops before removing the operator — which leaves the cluster recoverable instead of wedged.
cleanup.grafanaOperator.backoffLimit	int	`1`	Number of times to retry the job before failing the uninstall.
cleanup.grafanaOperator.activeDeadlineSeconds	int	`420`	Hard ceiling on the Job, including retries. Sized above `timeout` × (`backoffLimit` + 1) so the kubectl timeout is what normally fires; this only catches a pod that never gets that far.
cleanup.grafanaOperator.annotations	object	{ "helm.sh/hook": "pre-delete", "helm.sh/hook-delete-policy": "before-hook-creation,hook-succeeded", "helm.sh/hook-weight": "0" }	Job specific annotations. The default makes this a pre-delete hook; setting any annotation replaces that, which is how you would move it to `post-delete` or set an argocd sync wave.
cleanup.grafanaOperator.image	object	{ "pullPolicy": "IfNotPresent", "pullSecrets": [], "registry": "registry.k8s.io", "repository": "kubectl", "tag": "v1.35.6" }	Image for the cleanup job. Upstream's own kubectl build: distroless, multi-arch, and published beside Kubernetes itself, so it tracks patch releases without a third-party rebuild. Its entrypoint is `/bin/kubectl` and there is no shell in the image — the hook runs one argv and needs nothing else. Keep `tag` within one minor of your API server, per Kubernetes' version-skew policy.
cleanup.grafanaOperator.resources	object	{ "limits": { "cpu": "100m", "memory": "64Mi" }, "requests": { "cpu": "50m", "memory": "32Mi" } }	Resources for the cleanup job. It issues one API call and waits, so this is deliberately small.
cleanup.grafanaOperator.podSecurityContext	object	{ "runAsGroup": 65532, "runAsNonRoot": true, "runAsUser": 65532, "seccompProfile": { "type": "RuntimeDefault" } }	Security context for the cleanup job pod.
cleanup.grafanaOperator.containerSecurityContext	object	{ "allowPrivilegeEscalation": false, "capabilities": { "drop": [ "ALL" ] }, "readOnlyRootFilesystem": true, "runAsGroup": 65532, "runAsNonRoot": true, "runAsUser": 65532 }	Security context for the cleanup job container. The image ships `USER 0`; kubectl needs no privileges to call an API server, so it is dropped to nonroot here rather than trusted to the image.
cleanup.grafanaOperator.nodeSelector	object	{}	Node selector for the cleanup job.
cleanup.grafanaOperator.tolerations	list	[]	Tolerations for the cleanup job. An uninstall has to work even when the ordinary pools are tainted or full, so this is worth widening if scheduling it is ever the thing that fails.

### Bundled subchart configurations Configuration for bundled subcharts #### Alloy Agent Alloy collector instance running close to scrape targets. Pre-egress shaping happens here. Upstream reference: * https://github.com/grafana/alloy/tree/main/operations/helm/charts/alloy * https://github.com/grafana/alloy/blob/main/operations/helm/charts/alloy/values.yaml

Key	Type	Default	Description
alloy-agent.fullnameOverride	string	`"alloy-agent"`	Standard Helm full-name override. We use a static name for deterministic relations.
alloy-agent.namespaceOverride	string	`nil`	Namespace override.
alloy-agent.crds	object	{ "create": false }	Control for the PodLogs crd.
alloy-agent.global.podSecurityContext	object	{ "fsGroup": 473, "runAsGroup": 473, "runAsUser": 473 }	Security context for the alloy agent pods.
alloy-agent.alloy.stabilityLevel	string	`"generally-available"`	Stability level of alloy components.
alloy-agent.alloy.extraEnv	list	[ { "name": "GOMEMLIMIT", "value": "240MiB" } ]	Extra environment variables to pass to the alloy agent pod. `GOMEMLIMIT` at ~80% of the memory limit is the part worth keeping regardless of the number. Go's GC has no knowledge of a cgroup limit, so it grows the heap toward a ceiling the kernel enforces by killing the process. Telling the runtime about the ceiling turns an OOM-kill into GC pressure — the failure mode becomes "slower", not "dead". Keep it in step with the memory limit above.
alloy-agent.alloy.envFrom	list	[ { "configMapRef": { "name": "mzmon-alloy-agent-env" } }, { "secretRef": { "name": "mzmon-alloy-agent-env", "optional": true } } ]	Environment variable configmaps/secrets to pass to the alloy agent pod.
alloy-agent.alloy.mounts	object	{ "extra": [ { "mountPath": "/tmp", "name": "tmp" }, { "mountPath": "/run/log/journal", "name": "runlogjournal", "readOnly": true }, { "mountPath": "/etc/machine-id", "name": "machineid", "readOnly": true }, { "mountPath": "/etc/mzmon/tls", "name": "mzmon-tls", "readOnly": true } ], "varlog": true }	Volume mounts to expose to alloy agent. Three mounts feed `loki.source.journal`, and journal collection needs all three. Every way of getting it wrong fails the same silent way: the component starts, reports healthy, and reads nothing. Where the journal lives is host-specific, so both roots have to be covered. Observed: \| Host \| Journal \| Covered by \| \| --- \| --- \| --- \| \| Bottlerocket (EKS) \| `/var/log/journal//`, persistent \| `varlog` \| \| `kindest/node` \| `/run/log/journal//`, volatile \| `runlogjournal` \| Bottlerocket also has an empty `/run/log/journal`, and kind has no `/var/log/journal` at all — neither is a problem, but neither alone is enough either. Deliberately not mounting `/var/log/journal` directly: journald's default `Storage=auto` switches to persistent storage as soon as that directory exists, so a `hostPath` mount that creates it would change how the host journals. `varlog` already exposes it wherever it legitimately exists. `/etc/machine-id` is required, and its absence is silent. systemd stores the journal under `//`, and the container has no machine-id of its own — the Alloy image ships none. Without the host's, `loki.source.journal` starts, reports healthy, and reads zero lines. Measured on a Bottlerocket node, controlling for the restart that a mount change forces: with the mount, `loki_source_journal_target_lines_total` went 0 -> 1244; removed again, back to 0 on a freshly started pod. A kind node happens to work without it, which makes this a bad thing to test on kind alone — the two differ in which directory holds the journal (`/run/log/journal` there, `/var/log/journal` on Bottlerocket), and only the latter needs the ID to resolve. Do not remove this because kind stays green.
alloy-agent.alloy.mounts.extra[3]	object	{ "mountPath": "/etc/mzmon/tls", "name": "mzmon-tls", "readOnly": true }	Certificate material from `certificates`, when it is enabled. `optional: true` is what lets this be unconditional. The Secret does not exist until `certificates.enabled` is on and cert-manager has signed, and an optional secret volume that is missing mounts empty rather than blocking the pod — so the same values work before, during and after issuance. That holds only while nothing reads the material, which is why the mTLS profiles flip it to `optional: false`. Once a config names a path inside this mount, an empty mount is not a pod serving plaintext: Alloy fails its initial load and enters CrashLoopBackOff, backing off to five minutes, on an error naming a file rather than the Secret behind it. Required, the same race is a container the kubelet holds in `ContainerCreating` with a `FailedMount` event, started on the next volume re-sync. Do not set this to false here — with certificates off there is no Secret to wait for and the pod would never start. The kubelet refreshes the contents in place on renewal, which is why the `tls.*File` carriers are preferred over the inline PEMs.
alloy-agent.alloy.securityContext	object	{ "allowPrivilegeEscalation": false, "capabilities": { "drop": [ "ALL" ] }, "readOnlyRootFilesystem": true, "runAsGroup": 473, "runAsNonRoot": false, "runAsUser": 0 }	Security context for the alloy agent containers. The agent MUST run as root in order to be able to read container logs. No capabilities are added and none are needed: everything it reads is reachable by uid 0 under ordinary DAC.
alloy-agent.alloy.resources	object	{ "limits": { "cpu": "250m", "memory": "300Mi" }, "requests": { "cpu": "100m", "memory": "300Mi" } }	Resources for the alloy agent containers. Sized for the logs-only agent. This needed ~750Mi while an in-process cAdvisor lived here; that moved to the gateway, so the envelope is back to what the log path actually costs, with headroom over the ~200Mi it ran at before. The CPU limit stays above the request: the agent is bursty, and a limit equal to the request turns that into CFS throttling on the busiest nodes.
alloy-agent.controller.extraAnnotations	object	{}	Extra annotations to apply to the alloy agent pod. If you are using pulumi, be sure to add `config.kubernetes.io/depends-on: job/mzmon-validate-agent`
alloy-agent.controller.tolerations	list	[ { "effect": "NoSchedule", "operator": "Exists" } ]	Taints the agent tolerates out of the box: every `NoSchedule` one. Coverage is correctness for this workload, not a placement preference. A node the DaemonSet never lands on produces no logs and no error, and nothing shows a hole where a node should be — so the default is the same blanket rule `node-exporter` already ships, for the same reason. Enumerating keys instead would make coverage depend on this chart knowing your taints, which it does not: a node pool tainted with anything the list does not name goes silently uncollected. That is the failure this replaced. It is not always silent, either. A bootstrap gate — a `NoSchedule` taint applied to every node at boot and lifted once the DaemonSets are running, which is what `node.materialize.com/daemonsets-not-scheduled` is — deadlocks outright against an agent that does not tolerate it: the taint waits on a pod that is waiting on the taint, and the node never admits any workload. `NoExecute` is deliberately not tolerated here — a node being drained for a problem should shed the agent too. The two exceptions are not the chart's to make: the DaemonSet controller adds `node.kubernetes.io/not-ready` and `node.kubernetes.io/unreachable` (both `NoExecute`) to every DaemonSet pod unconditionally, so the agent keeps running on a node that goes unreachable whatever is written here. Add other `NoExecute` taints through the Terraform module's `tolerations`, which appends to this list. To narrow this — a node pool whose per-node budget genuinely cannot absorb the agent — replace the list rather than adding to it, and record which pools you gave up. Helm overwrites lists, so setting this key at all replaces the whole default.
alloy-agent.controller.priorityClassName	string	`"monitoring-critical"`	Scheduling priority. See the Priority classes section. Critical: this is a per-node singleton, so an eviction is a log gap on that node with no replica to cover it.
alloy-agent.networkPolicy	object	{ "egress": null, "enabled": true, "flavor": "kubernetes", "ingress": [ { "from": [ { "podSelector": { "matchLabels": { "app.kubernetes.io/name": "alloy-gateway" } } } ], "ports": [ { "port": 12345, "protocol": "TCP" } ] } ], "policyTypes": [ "Ingress" ] }	NetworkPolicy for the agent. Ingress is one port from one peer. `12345` is Alloy's HTTP server — it serves `/metrics` and the live debugging UI, which shows the running pipeline and samples of the data moving through it — and the only pod that needs to reach it is the gateway performing the `serviceMonitor` scrape. Egress is not restricted, and the `policyTypes` list says so rather than pairing an `Egress` type with an allow-everything rule. Two of the agent's three destinations are unaddressable from here: the API server backs `discovery.kubernetes`, and on a managed control plane it is not in the pod network at all. The third, the gateway, is in-namespace and would be easy — but a policy that allows only the parts we happen to know is a policy that breaks the first time someone adds a scrape to the agent pipeline, and does it silently. Narrow it deliberately, with `egress` and `policyTypes`, once you know what your agent actually collects. The default `from` selects by pod label within the release namespace. Running the gateway in a different namespace (see `profiles/split-namespace.values.yaml`) requires replacing this list with one that adds a `namespaceSelector`.
alloy-agent.networkPolicy.egress	string	`nil`	Null rather than `[]`: an empty list still merges as "no rules", but null is what the subchart documents for "do not render this direction", and it keeps `egress` out of a spec whose `policyTypes` omits `Egress`.

#### Alloy Gateway Alloy gateway instance. Cardinality reduction and backend-specific egress happen here. Upstream reference: * https://github.com/grafana/alloy/tree/main/operations/helm/charts/alloy * https://github.com/grafana/alloy/blob/main/operations/helm/charts/alloy/values.yaml

Key	Type	Default	Description
alloy-gateway.fullnameOverride	string	`"alloy-gateway"`	Standard Helm full-name override. We use a static name for deterministic relations.
alloy-gateway.namespaceOverride	string	`nil`	Namespace override.
alloy-gateway.crds	object	{ "create": false }	Control for the PodLogs crd.
alloy-gateway.global.podSecurityContext	object	{ "fsGroup": 473, "runAsGroup": 473, "runAsUser": 473 }	Security context for the alloy gateway pods.
alloy-gateway.alloy.stabilityLevel	string	`"generally-available"`	Stability level of alloy components.
alloy-gateway.alloy.extraEnv	list	[ { "name": "GOMEMLIMIT", "value": "600MiB" } ]	Extra environment variables to pass to the alloy gateway pod. `GOMEMLIMIT` at ~80% of the memory limit, for the same reason as the agent's. The gateway now carries the kubelet cAdvisor scrape, so its heap scales with node count — keep this in step with the limit. It is the ceiling the GC works against, so it also sets where the gateway idles. Leaving it near the old limit while raising `resources` would waste the new headroom; leaving it above the limit forfeits the whole point, since the runtime would only start collecting hard after the kubelet has already OOM-killed the pod.
alloy-gateway.alloy.envFrom	list	[ { "configMapRef": { "name": "mzmon-alloy-gateway-env" } }, { "secretRef": { "name": "mzmon-alloy-gateway-env", "optional": true } } ]	Environment variable configmaps/secrets to pass to the alloy gateway pod.
alloy-gateway.alloy.extraPorts	list	[ { "name": "loki", "port": 3100, "protocol": "TCP", "targetPort": 3100 }, { "name": "otlp-grpc", "port": 4317, "protocol": "TCP", "targetPort": 4317 }, { "name": "otlp-http", "port": 4318, "protocol": "TCP", "targetPort": 4318 }, { "name": "prom", "port": 9090, "protocol": "TCP", "targetPort": 9090 } ]	Ports to expose from alloy-gateway.
alloy-gateway.alloy.mounts	object	{ "extra": [ { "mountPath": "/tmp", "name": "tmp" }, { "mountPath": "/etc/mzmon/tls", "name": "mzmon-tls", "readOnly": true } ], "varlog": false }	Volume mounts to expose to alloy gateway.
alloy-gateway.alloy.mounts.extra[1]	object	{ "mountPath": "/etc/mzmon/tls", "name": "mzmon-tls", "readOnly": true }	Certificate material from `certificates`, when it is enabled. `optional: true` is what lets this be unconditional. The Secret does not exist until `certificates.enabled` is on and cert-manager has signed, and an optional secret volume that is missing mounts empty rather than blocking the pod — so the same values work before, during and after issuance. That holds only while nothing reads the material, which is why the mTLS profiles flip it to `optional: false`. Once a config names a path inside this mount, an empty mount is not a pod serving plaintext: Alloy fails its initial load and enters CrashLoopBackOff, backing off to five minutes, on an error naming a file rather than the Secret behind it. Required, the same race is a container the kubelet holds in `ContainerCreating` with a `FailedMount` event, started on the next volume re-sync. Do not set this to false here — with certificates off there is no Secret to wait for and the pod would never start. The kubelet refreshes the contents in place on renewal, which is why the `tls.*File` carriers are preferred over the inline PEMs.
alloy-gateway.alloy.securityContext	object	{ "allowPrivilegeEscalation": false, "capabilities": { "drop": [ "ALL" ] }, "readOnlyRootFilesystem": true, "runAsGroup": 473, "runAsNonRoot": true, "runAsUser": 473 }	Security context for the alloy gateway containers.
alloy-gateway.alloy.resources	object	{ "limits": { "cpu": "500m", "memory": "768Mi" }, "requests": { "cpu": "500m", "memory": "768Mi" } }	Resources for the alloy gateway containers. Memory is the gateway's binding constraint and the only axis that actually relieves it — see the `targetMemoryUtilizationPercentage` note below for why adding replicas does not. Sized for the floor a CPU-scaled gateway settles at: at `minReplicas` each pod carries the whole scrape fan-out rather than a shard of it, so the per-pod working set is higher than it looks at a scaled-out replica count. Raise this, and `GOMEMLIMIT` with it, as node count grows.
alloy-gateway.controller.extraAnnotations	object	{}	Extra annotations to apply to the alloy gateway pod. If you are using pulumi, be sure to add `config.kubernetes.io/depends-on: job/mzmon-validate-gateway`
alloy-gateway.controller.priorityClassName	string	`"monitoring-critical"`	Scheduling priority. See the Priority classes section. Critical despite being a Deployment: every signal in the stack leaves through here, so losing it stops logs and metrics at once.
alloy-gateway.controller.autoscaling.horizontal.targetMemoryUtilizationPercentage	int	`0`	Memory scaling is deliberately OFF (`0` is the subchart's disable value; it renders the metric away rather than setting it to zero). Not a tuning choice — memory is the wrong signal for this component, because scaling out does not relieve it. The gateway's footprint is dominated by fixed per-process cost, not by per-replica load: measured on a 7-node cluster, going from 3 replicas to 6 moved per-pod memory from 370Mi to 341Mi while total consumption went from 1.1Gi to 2.0Gi. Each new replica adds a whole baseline to save a few Mi on its peers, so a memory-driven scale-out makes cluster memory pressure worse. It also cannot stabilize. Idle sat at ~62% of the request, so a 60% target was below the floor: the HPA scaled up, the metric did not move, and it flapped against maxReplicas indefinitely. No target value fixes that, because the control loop is open — the action does not change the measurement. Relieve gateway memory vertically instead: raise `resources` and keep `GOMEMLIMIT` in step (see both, above). That is also what the node-count scaling in the `GOMEMLIMIT` note means in practice.
alloy-gateway.serviceAccount.create	bool	`true`	Create a service account for alloy-gateway.
alloy-gateway.serviceAccount.annotations	object	{}	Extra annotations to set on the alloy-gateway service account. Use `eks.amazonaws.com/role-arn` to set up IRSA. Use `iam.gke.io/gcp-service-account` to set up Workload Identity Federation.
alloy-gateway.networkPolicy	object	{ "egress": null, "enabled": true, "flavor": "kubernetes", "ingress": [ { "from": [ { "namespaceSelector": {} } ], "ports": [ { "port": 3100, "protocol": "TCP" }, { "port": 4317, "protocol": "TCP" }, { "port": 4318, "protocol": "TCP" }, { "port": 9090, "protocol": "TCP" } ] }, { "from": [ { "podSelector": { "matchLabels": { "app.kubernetes.io/name": "alloy-gateway" } } } ], "ports": [ { "port": 12345, "protocol": "TCP" } ] } ], "policyTypes": [ "Ingress" ] }	NetworkPolicy for the gateway. Two ingress rules, because the gateway's ports fall into two groups with very different audiences. The ingest ports — `3100` (Loki push), `4317`/`4318` (OTLP), `9090` (Prometheus remote write) — are open to every pod in the cluster. That is deliberate and it is not laziness: these are the endpoints the stack exists to offer. The agent pushes logs to `3100`, and anything in the cluster with telemetry to send — a Materialize workload emitting OTLP, an application remote-writing metrics — is a legitimate client this chart cannot enumerate. `namespaceSelector: {}` still means in-cluster: a sender outside the pod network is denied, and the gateway is not meant to be internet-facing. `12345` is not an ingest port and is closed to everything but the gateway itself. It carries the clustering gossip between gateway replicas (`alloy.clustering.enabled`) and serves `/metrics` and the debugging UI, which exposes the pipeline and samples of the telemetry passing through it. The gateway scrapes its own `serviceMonitor`, so its own pods are the only peer. Egress is not restricted, and cannot usefully be. The gateway scrapes the kubelet cAdvisor endpoint on every node — port `10250` on a node IP, which no `podSelector` matches — and it discovers `ServiceMonitor` and `PodMonitor` targets cluster-wide, so its destination set is every pod and every port that anyone in the cluster has ever pointed a monitor at. On top of that it is the stack's egress point for external backends: Amazon Managed Prometheus, Honeycomb, a Datadog or OTLP destination under `connections`. Enumerating that is not a policy, it is a copy of the cluster's inventory that goes stale.
alloy-gateway.networkPolicy.egress	string	`nil`	See the note above: unrestricted, deliberately.

#### Loki Bundled Loki backend for logs. Upstream reference: * https://github.com/grafana-community/helm-charts/tree/main/charts/loki * https://github.com/grafana-community/helm-charts/blob/main/charts/loki/values.yaml

Key	Type	Default	Description
loki.fullnameOverride	string	`"loki"`	Standard Helm full-name override. We use a static name for deterministic relations.
loki.namespaceOverride	string	`nil`	Namespace override.
loki.global.priorityClassName	string	`"monitoring-scalable"`	Scheduling priority for Loki. See the Priority classes section. `global` covers every component that renders through the chart's `_pod.tpl` — but not the two memcached StatefulSets, which read only their own component key. They are set explicitly further down. This is the same asymmetry `terraform/modules/materialize-monitoring/scheduling.tf` calls out for nodeSelector and tolerations; rendering is what catches it.
loki.deploymentMode	string	`"Distributed"`	How loki is deployed. We prefer to run in Distributed/Microservices mode.
loki.networkPolicy.enabled	bool	`true`	Whether to enable a network policy for loki In production, this is recommended to be enabled.
loki.networkPolicy.metrics	object	{ "namespaceSelector": {} }	Selector for incoming traffic to metric endpoints. This must be configured manually (usually set to `kubernetes.io/metadata.name: monitoring`).
loki.networkPolicy.ingress	object	{ "namespaceSelector": {} }	Selector for incoming traffic to the read/write endpoints. This must be configured manually (usually set to `kubernetes.io/metadata.name: monitoring`).
loki.networkPolicy.externalStorage	object	{ "cidrs": [ "0.0.0.0/0" ], "ports": [ 443, 80 ] }	Outgoing traffic from loki to the object store. This is usually unrestricted, even in many production settings. Adjust if you have a VPCEndpoint in front of S3/STS or are using a non-standard port.
loki.loki.storage.bucketNames	object	{ "chunks": "\u003cREPLACE-ME\u003e", "ruler": "\u003cREPLACE-ME\u003e" }	Bucket names for object storage. These are required to be populated.
loki.loki.storage.use_thanos_objstore	bool	`true`	Use the thanos object store client
loki.loki.storage.object_store	object	{ "s3": { "endpoint": "s3.amazonaws.com" }, "type": "s3" }	Object storage configuration. Modify as needed. Only the block named by `type` is read; it is handed to Loki verbatim, with `bucket_name` filled in from `bucketNames` above.
loki.loki.schemaConfig	object	{ "configs": [ { "from": "2024-01-01", "index": { "period": "24h", "prefix": "loki_index_" }, "object_store": "s3", "schema": "v13", "store": "tsdb" } ] }	Schema configuration for the loki TSDB. This is append-only and MUST be copied into projects and mutated on upgrades (if v13 stops being in use).
loki.loki.commonConfig	object	{ "replication_factor": 3 }	Cluster-wide replication factor. Three is our HA floor and the reason at least three ingesters are required. (This matches the upstream default; surfaced here because it is load-bearing.)
loki.loki.limits_config	object	{ "ingestion_burst_size_mb": 20, "ingestion_rate_mb": 10, "max_global_streams_per_user": 10000, "reject_old_samples": true, "reject_old_samples_max_age": "168h", "retention_period": "30d", "volume_enabled": true }	Per-tenant limits, sized for a medium install. These are protective caps (per environment), not the expected volume — see Operating > Production Best Practices for the throughput envelope.
loki.loki.limits_config.retention_period	string	`"30d"`	Default retention before the compactor deletes logs. Upstream defaults to infinite retention; we set a real bound.
loki.loki.limits_config.ingestion_rate_mb	int	`10`	Per-tenant ingestion rate / burst, in MB. Per environment, not the fleet aggregate.
loki.loki.limits_config.max_global_streams_per_user	int	`10000`	Active-stream ceiling per tenant; a cardinality guard.
loki.loki.limits_config.reject_old_samples	bool	`true`	Reject writes too far in the past.
loki.loki.limits_config.volume_enabled	bool	`true`	Enable the log-volume endpoints Grafana's logs drilldown relies on.
loki.loki.compactor	object	{ "delete_request_store": "s3", "retention_delete_delay": "8h", "retention_enabled": true }	Compactor configuration (distinct from the compactor deployment below). Retention is enforced by the compactor and is OFF in upstream defaults.
loki.loki.compactor.delete_request_store	string	`"s3"`	Where delete requests are stored; matches the object-store backend.
loki.loki.compactor.retention_delete_delay	string	`"8h"`	Grace period before retention/deletes actually remove data.
loki.gateway.enabled	bool	`false`	Disable gateway by default. We recommend using alloy-gateway for loki writes. Use the query-frontend for loki reads.

##### Loki Microservice Configuration Configuration for each loki microservice. https://grafana.com/docs/loki/latest/get-started/components/

Key	Type	Default	Description
loki.distributor.enabled	bool	`true`	Enable the distributor microservice. Distributor is required. The Distributor is the stateless front door for writes. It validates incoming streams against per-tenant limits, enforces rate limits, and normalizes labels, then splits the batch into individual streams and forwards each to the owning ingesters.
loki.distributor.replicas	string	`nil`	Number of replicas for the distributor microservice. If autoscaling is enabled, this should be set to null.
loki.distributor.kind	string	`"Deployment"`	Type of workload for the distributor.
loki.distributor.autoscaling	object	{ "enabled": true, "maxReplicas": 4, "minReplicas": 2, "targetCPUUtilizationPercentage": 60, "targetMemoryUtilizationPercentage": 75 }	Configuration for autoscaling of distributor For production, this is recommended to be enabled. We provide opinionated defaults.
loki.distributor.service.type	string	`"ClusterIP"`	Service type for the distributor microservice. Without a gateway, this is the exposed write component.
loki.distributor.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	Configuration for pod disruption budget for the distributor microservice.
loki.distributor.resources	object	{ "requests": { "cpu": "150m", "memory": "256Mi" } }	Resource requests (medium install). Tune per profile.
loki.ingester.enabled	bool	`true`	Enable the ingester microservice. Ingester is required. The Ingester is the stateful heart of the write path, and it also serves the most recent reads. It buffers incoming entries into per-stream in-memory chunks, compresses them, and periodically flushes those chunks and their index to object storage.
loki.ingester.replicas	int	`3`	Number of ingester replicas. Three is the floor for replication_factor 3. Scale UP on memory / stream-cardinality (so streams shard across the ring), not on bytes — with replicas == replication_factor every ingester holds every stream.
loki.ingester.kind	string	`"StatefulSet"`	Ingesters are stateful.
loki.ingester.persistence.enabled	bool	`false`	Run ingesters ephemerally: no PVC, node-local emptyDir for the WAL and not-yet-flushed chunks. Durability comes from replication_factor 3, not from disk — a rescheduled ingester starts fresh and the ring backfills from its peers. This avoids EBS zonal pinning and the slow volume detach/attach that PVCs incur during node replacement.
loki.ingester.terminationGracePeriodSeconds	int	`60`	Termination grace period for ingesters. flush-on-shutdown is best-effort; if the node force-kills sooner, the other replicas still flush their copies, so do not depend on a long grace period for durability. Kept modest to survive enterprise force-kill windows.
loki.ingester.affinity	object	{ "podAntiAffinity": { "requiredDuringSchedulingIgnoredDuringExecution": null } }	Pod affinity for ingesters. Drop the chart's default hard per-host anti-affinity so host spread can be soft (see topologySpreadConstraints); zone spread stays hard. We null the nested list rather than the whole `affinity` map: `affinity: {}` is a no-op against the subchart default, and `affinity: null` clears it but makes helm-unittest log a noisy "cannot overwrite table" warning — nulling the list avoids both.
loki.ingester.topologySpreadConstraints	list	[ { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "ingester", "app.kubernetes.io/instance": "{{ .Release.Name }}", "app.kubernetes.io/name": "{{ include \"loki.name\" . }}" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "minDomains": 2, "nodeTaintsPolicy": "Honor", "topologyKey": "topology.kubernetes.io/zone", "whenUnsatisfiable": "DoNotSchedule" }, { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "ingester", "app.kubernetes.io/instance": "{{ .Release.Name }}", "app.kubernetes.io/name": "{{ include \"loki.name\" . }}" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "topologyKey": "kubernetes.io/hostname", "whenUnsatisfiable": "ScheduleAnyway" } ]	Topology spread for ingesters: hard across zones, soft across hosts. A pod that cannot satisfy the hard zone rule goes Pending, which is the signal Karpenter (or the cluster-autoscaler) uses to add a node in the deficient zone — soft rules cannot summon capacity that way. Host spread is soft so pods still schedule when nodes are momentarily scarce.
loki.ingester.zoneAwareReplication.enabled	bool	`false`	Enable zone-aware replication for the ingester microservice. Not needed at our size: topologySpreadConstraints give the AZ spread without the complexity.
loki.ingester.podDisruptionBudget	object	{ "enabled": true, "maxUnavailable": 1 }	PDB for the ingester. Protect ingest quorum across rollouts and node drains. Do not set maxUnavailable >= 2 to avoid potential quorum loss.
loki.ingester.resources	object	{ "requests": { "cpu": "500m", "memory": "1Gi" } }	Resource requests (medium install). The memory limit is intentionally left unset: an OOM-kill drops in-memory / WAL-buffered logs, so we alert on usage rather than cap hard.
loki.querier.enabled	bool	`true`	Enable the querier microservice. Querier is required. It executes LogQL, reading recent data from ingesters and historical data from object storage.
loki.querier.replicas	string	`nil`	Replicas are managed by autoscaling; leave null.
loki.querier.autoscaling	object	{ "enabled": true, "maxReplicas": 4, "minReplicas": 2, "targetCPUUtilizationPercentage": 60, "targetMemoryUtilizationPercentage": 75 }	Autoscale the stateless read workers with query load.
loki.querier.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for querier.
loki.querier.resources	object	{ "requests": { "cpu": "250m", "memory": "512Mi" } }	Resource requests. This assumes a medium install by default.
loki.queryFrontend.enabled	bool	`true`	Enable the query-frontend microservice. Query-frontend is required. It queues and splits queries and caches results.
loki.queryFrontend.replicas	string	`nil`	Static replicas for query-frontend when autoscaling is not enabled.
loki.queryFrontend.kind	string	`"Deployment"`	Kind of workload for query-frontend. Deployments are recommended for stateless components.
loki.queryFrontend.autoscaling	object	{ "enabled": true, "maxReplicas": 4, "minReplicas": 2, "targetCPUUtilizationPercentage": 60, "targetMemoryUtilizationPercentage": 75 }	Configure autoscaling for query-frontend.
loki.queryFrontend.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for query-frontend.
loki.queryFrontend.resources	object	{ "requests": { "cpu": "100m", "memory": "256Mi" } }	Resource requests for query-frontend.
loki.queryScheduler.enabled	bool	`true`	Enable the query-scheduler microservice. Recommended for scaled deployments: it decouples the read queue from the query-frontend so the two scale independently. Two replicas for availability.
loki.queryScheduler.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for query-scheduler.
loki.queryScheduler.resources	object	{ "requests": { "cpu": "100m", "memory": "256Mi" } }	Resources for the query-scheduler microservice.
loki.compactor.enabled	bool	`true`	Enable the compactor. Compactor is required and MUST be a singleton: it compacts the index and enforces retention against shared object storage.
loki.compactor.replicas	int	`1`	Compactor replicas. This MUST be one, since it runs as a singleton.
loki.compactor.kind	string	`"StatefulSet"`	Kind of workload for the compactor. This must be a statefulset.
loki.compactor.persistence.enabled	bool	`false`	Run the compactor ephemerally (no PVC, node-local emptyDir). Its local dir is just a working copy of the object-store index, and compaction is idempotent and off the critical path — losing it only restarts the current cycle. Ephemeral also lets the singleton float freely between zones instead of being pinned by a PVC. Our production index is ~30MB, so re-downloading each cycle is negligible.
loki.compactor.resources	object	{ "requests": { "cpu": "250m", "memory": "512Mi" } }	Resources for the compactor.
loki.indexGateway.enabled	bool	`true`	Enable the index-gateway. Index-gateway is required: it serves index lookups so queriers do not each download the whole index from object storage.
loki.indexGateway.persistence	object	{ "enabled": false }	Run the index-gateway ephemerally (no PVC, node-local emptyDir). The local index is a read-through cache of object storage, not authoritative — a fresh pod just re-downloads what it queries. This keeps the read path off EBS zonal pinning; the only cost is a little cold-start warmup and a burst of index reads after a reschedule.
loki.indexGateway.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for index-gateway.
loki.indexGateway.resources	object	{ "requests": { "cpu": "200m", "memory": "512Mi" } }	Resources for the index-gateway.
loki.ruler.enabled	bool	`true`	Enable the ruler. The ruler evaluates LogQL alerting and recording rules. Recording-rule samples are remote-written back through alloy-gateway to the metric store.
loki.ruler.persistence.enabled	bool	`true`	Keep a PVC for the ruler (unlike the other components, which are ephemeral). Rule definitions come from object storage, but the ruler's remote-write WAL buffers recording-rule samples when the metric store is unreachable — genuinely useful durability in the run-up to an incident, exactly when you don't want to drop derived signals.
loki.ruler.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for the ruler.
loki.ruler.resources	object	{ "requests": { "cpu": "100m", "memory": "256Mi" } }	Resources for the ruler.
loki.chunksCache	h5	`{"allocatedMemory":2048, "priorityClassName":"monitoring-scalable"}`	Chunk cache (memcached). Default allocation is sized for very large installs; we shrink it to match our volumes. The results cache keeps its upstream default. `priorityClassName` is repeated on both caches because the memcached StatefulSet template reads its component key only — `loki.global` does not reach it.
loki.resultsCache	h5	`{"priorityClassName":"monitoring-scalable"}`	Query results cache (memcached).
loki.monitoring.serviceMonitor.enabled	bool	`true`	Enable a ServiceMonitor for the loki microservices.
loki.lokiCanary	h5	`{"enabled":true, "kind":"Deployment", "lokiurl":"loki-query-frontend:3100", "priorityClassName":"monitoring-scalable", "push":false}`	End-to-end write→read canary for meta-monitoring. On by default upstream; surfaced here because self-monitoring the log store is a first-class requirement for us.
loki.lokiCanary.priorityClassName	string	`"monitoring-scalable"`	Scheduling priority. See the Priority classes section. Set explicitly because the canary reaches neither of the two keys that cover the rest of Loki: it renders from its own template rather than `_pod.tpl`, so `loki.global.priorityClassName` does not reach it, and `loki.defaults` does not either. Left unset it runs at priority 0 — below ordinary workloads — so the first node under pressure evicts the end-to-end write→read check, which is exactly the signal you want during the incident that caused the pressure.

#### Thanos Bundled Thanos backend for long-term metrics. Upstream reference: * https://github.com/thanos-community/helm-charts/tree/master/charts/thanos * https://github.com/thanos-community/helm-charts/blob/master/charts/thanos/values.yaml

Key	Type	Default	Description
thanos.fullnameOverride	string	`"thanos"`	Standard Helm full-name override. We use a static name for deterministic relations.
thanos.namespaceOverride	string	`nil`	Namespace override.
thanos.global.priorityClassName	string	`"monitoring-scalable"`	Scheduling priority for every Thanos pod. See the Priority classes section.
thanos.global.pdb	object	{ "enabled": true, "maxUnavailable": 1 }	PodDisruptionBudgets for every Thanos component. Upstream ships these off. One switch turns them on for all components, and the per-component `pdb.enabled` cannot opt back out — the subchart templates test `or .pdb.enabled global.pdb.enabled`. `maxUnavailable` rather than `minAvailable`, deliberately: * It scales with replica count instead of pinning an absolute floor. * On the single-replica Compactor, `minAvailable: 1` would block every voluntary eviction and hang node drains indefinitely. `maxUnavailable: 1` permits the eviction, so the singleton is a harmless no-op rather than a drain deadlock. * On Receive it must stay within what write quorum tolerates (`replicationFactor - ((replicationFactor / 2) + 1)`), which is 1 at the replication factor of 3 set below. A validator enforces this. This matches the Loki convention (ingester `maxUnavailable: 1`).
thanos.global.networkPolicies	bool	`true`	NetworkPolicies for every enabled Thanos component. One switch, no per-component knobs and no selectors — the subchart renders a fixed policy per component and exposes nothing to tune. What each one says is: ingress on that component's own service ports, from anywhere; egress unrestricted. So this is the weakest policy in the stack, and worth being clear about what it does and does not buy. It does not restrict who may reach Query, Receive or Store Gateway — any pod in the cluster still can. It does close every port that is not a declared service port, which is the surface an unrelated workload would otherwise find on these pods. It is on because it costs nothing to be right about. The rules cannot break a working install: Receive's `grpc`, `http` and `remote-write` ports are all in its own ingress list, so hashring replication between Receive pods and the gateway's writes both keep working, and unrestricted egress leaves the object store reachable. Narrowing the ingress sources needs an upstream change, or your own policy layered on top.
thanos.query	object	{ "autoscaling": { "enabled": true, "maxReplicas": 5, "minReplicas": 2, "targetCPUUtilizationPercentage": 80 }, "enabled": true, "resources": { "requests": { "cpu": "500m", "memory": "1Gi" } }, "topologySpreadConstraints": [ { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "query", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "pod-template-hash" ], "maxSkew": 1, "topologyKey": "topology.kubernetes.io/zone", "whenUnsatisfiable": "ScheduleAnyway" }, { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "query", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "pod-template-hash" ], "maxSkew": 1, "topologyKey": "kubernetes.io/hostname", "whenUnsatisfiable": "ScheduleAnyway" } ] }	Thanos Query configuration. Query provides a PromQL query endpoint.
thanos.query.autoscaling	object	{ "enabled": true, "maxReplicas": 5, "minReplicas": 2, "targetCPUUtilizationPercentage": 80 }	Horizontal autoscaling for Query. Query is a stateless PromQL fan-out, so it is the natural place to autoscale: no local state, no ring membership, no PVC.
thanos.query.topologySpreadConstraints	list	[ { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "query", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "pod-template-hash" ], "maxSkew": 1, "topologyKey": "topology.kubernetes.io/zone", "whenUnsatisfiable": "ScheduleAnyway" }, { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "query", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "pod-template-hash" ], "maxSkew": 1, "topologyKey": "kubernetes.io/hostname", "whenUnsatisfiable": "ScheduleAnyway" } ]	Topology spread for Query: soft on both axes. Stateless and autoscaled, so an unbalanced placement costs query capacity rather than correctness — soft is sufficient, and it keeps an HPA scale-up from stalling on a hard rule when the newest zone has no room yet. `pod-template-hash`, not `controller-revision-hash`: Query is a Deployment, and that is the label its ReplicaSets carry. Using the StatefulSet label here would match nothing and quietly disable the same-revision guard.
thanos.query.resources	object	{ "requests": { "cpu": "500m", "memory": "1Gi" } }	Resource requests for Thanos Query. Medium sizing; see the `thanos-small` / `thanos-large` profiles for the other two columns. A CPU request is load-bearing here beyond scheduling: `targetCPUUtilization` is a percentage of the request, so without one the HPA above has no denominator and never scales. No memory limit — a fan-out query's peak scales with the series it touches, and an OOM-kill mid-query is a worse failure than a slow one.
thanos.receive.enabled	bool	`true`	Enable Thanos receiver. Receive provides a Prometheus remote_write-compatible endpoint.
thanos.receive.mode	string	`"standalone"`	Whether to split receive distributors from ingesters. mode=split is not very stable in the helm chart
thanos.receive.replicaCount	int	`3`	Number of receive replicas.
thanos.receive.tsdb	object	{ "retention": "6h" }	How long Receive keeps blocks on local disk before they age out. 6h, overriding the subchart's 24h, because local retention is what sets Receive's disk requirement — and on an `emptyDir` that disk is the node's, a far scarcer and much smaller pool than a provisioned volume was. This is a recent-query cache, not a durability window: blocks ship to object storage every 2h and the Store Gateway serves everything older, so all that shortens here is how far back a query is answered from local TSDB rather than from the bucket. 6h covers the common dashboard ranges and leaves 4h of margin after a block closes, uploads, and the Store Gateway syncs it. Raising it back is a real cost rather than just a number. At the medium envelope (1.5M series) 24h is ~7.3Gi per pod, and a GKE node with a 47Gi boot disk offers only ~18.8Gi of allocatable ephemeral storage — one pod would claim 40% of the node's budget, and Loki's ingesters draw on the same pool.
thanos.receive.extraArgs	list	[ "--receive.replication-factor=3" ]	Extra CLI arguments for Receive. This is a list, and Helm overwrites lists rather than merging them. Anything that sets `receive.extraArgs` replaces this entry wholesale, so the replication factor must be restated or it silently falls back to Thanos's default of 1 — write quorum 1, no losses tolerated. The shipped profiles restate it; so must yours. See Production Best Practices > Metrics (Thanos) for the quorum table.
thanos.receive.vpa	object	{ "enabled": false }	VerticalPodAutoscaler for Receive. Off, overriding the subchart's `true`. Three reasons, in order: 1. It rewrites the requests below, so a sizing profile would not survive contact with it. 2. `updateMode: Auto` evicts pods to apply new requests, and Receive holds up to `tsdb.retention` (6h) of blocks on an `emptyDir` whose only redundancy is the replication factor — an eviction discards that pod's copy of the window. 3. The subchart's VPA template is CRD-gated, so leaving it on makes the stack behave differently on clusters with the VPA CRD installed than on those without — silently, in both directions. If you want its recommendations without its actions, set `updateMode: "Off"` rather than re-enabling Auto.
thanos.receive.resources	object	{ "limits": { "ephemeral-storage": "6Gi" }, "requests": { "cpu": "500m", "ephemeral-storage": "4Gi", "memory": "4Gi" } }	Resource requests for Receive. Medium sizing. Memory is the cell to get right: budget ~3 KB per active series held, and note that at `replicaCount == replicationFactor` every pod holds every series — sharding begins only above RF. No memory limit, deliberately, and for a sharper reason than elsewhere in this chart: an OOM-kill drops the un-uploaded block window, and unlike Loki's ingesters that window has nothing protecting it but the replication factor. Alert on usage instead. `ephemeral-storage` is declared because the TSDB lives on an `emptyDir` (see `persistence`). The request is what the scheduler places against, so a node without room refuses the pod instead of filling up silently. The limit carries deliberate headroom over the request: the kubelet's response to exceeding it is eviction, and evicting Receive destroys exactly the un-uploaded window this sizing exists to protect. *Size these against a node's allocatable* ephemeral storage, not its disk size.** The two are nowhere near each other: GKE reserves most of the boot disk for the image filesystem, so a 47Gi disk offers roughly 18.8Gi allocatable. A request above that cannot schedule anywhere, and the cluster-autoscaler will not rescue it — no node of that shape would fit either, so the pod sits Pending indefinitely on `Insufficient ephemeral-storage`. Check before raising these: kubectl get nodes -o custom-columns='NODE:.metadata.name,EPH:.status.allocatable.ephemeral-storage' 4Gi holds 6h of the medium envelope (~1.8Gi) with room for the WAL and compaction churn, and leaves the rest of the node for everything sharing that pool — including Loki's ingesters, which are also `emptyDir`-backed.
thanos.receive.topologySpreadConstraints	list	[ { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "receive", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "minDomains": 2, "nodeTaintsPolicy": "Honor", "topologyKey": "topology.kubernetes.io/zone", "whenUnsatisfiable": "DoNotSchedule" }, { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "receive", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "topologyKey": "kubernetes.io/hostname", "whenUnsatisfiable": "ScheduleAnyway" } ]	Topology spread for Receive: hard across zones, soft across hosts. This is the constraint that makes `--receive.replication-factor=3` mean what the quorum table says. Without it the scheduler is free to place all three replicas in one zone, and RF 3 buys nothing against the failure it exists for — losing a zone takes every copy and, since write quorum is 2, takes writes with it. Hard (`DoNotSchedule`) on zones, deliberately. A pod that cannot satisfy it goes Pending, which is the signal Karpenter or the cluster-autoscaler uses to add a node in the deficient zone — a soft rule cannot summon capacity that way, it just quietly places the pod wrongly. Host spread stays soft so pods still schedule when nodes are momentarily scarce; two pods sharing a node is a smaller problem than a pod not running. This became viable when Receive stopped using a PVC (see `persistence` below). A zonal volume cannot be attached from another zone, so a hard zone rule and an AZ-pinned pod pull in opposite directions: the rule says "go to the empty zone", the volume says "you may only run in zone A", and the pod goes Pending forever rather than for as long as it takes to add a node. On `emptyDir` there is nothing holding it back. The selector names both `app.kubernetes.io/name` and `app.kubernetes.io/component`, and the first of those is load-bearing: component names are not unique across backends. Loki also ships a `compactor`, a `query-frontend` and a `ruler`, so a component-only selector would make Thanos count Loki's pods when computing its own skew — silently, and only for the components whose names happen to collide. The release name is deliberately absent: the subchart renders this through `toYaml` rather than `tpl`, so a `{{ include ... }}` would land in the manifest literally. Spread is namespace-scoped and this chart assumes one instance of each backend per namespace, so name plus component is sufficient. See the Namespace layout section. Note this cannot move to `thanos.global.topologySpreadConstraints`: each constraint carries its own `labelSelector`, and a global one would make every Thanos component count Receive's pods when computing its own skew.
thanos.receive.persistence	object	{ "enabled": false }	Local TSDB storage for Receive: ephemeral, not a PVC. This is the same call the chart makes for Loki's ingesters, for the same reason, and it is worth spelling out because Receive looks stateful. Durability comes from `--receive.replication-factor=3`, not from disk. Blocks ship to object storage every 2h, so the window that exists only locally is at most 2h — and every replica uploads its own copy under a distinct `replica` external label, which the Compactor deduplicates. A pod that comes back with an empty volume has lost its copy of that window, and the query path still answers from the surviving replicas. A PVC makes an AZ failure worse, not merely less flexible: an EBS volume cannot be attached from another zone, so a pod whose zone is gone stays Pending until the zone returns rather than rescheduling into a healthy one. On the write path that turns a recoverable event into an outage that waits on the cloud provider — with RF 3 write quorum is 2, so two pods stuck Pending block writes outright. The trade accepted in exchange: Thanos Receive has no peer hand-off, so unlike Loki it will not backfill an emptied pod from its neighbours. That window stays at two copies instead of three until the next block ships.
thanos.storegateway	object	{ "enabled": true, "extraArgs": [ "--store.limits.request-series=5000000", "--store.limits.request-samples=200000000" ], "persistence": { "enabled": true, "size": "10Gi" }, "resources": { "requests": { "cpu": "500m", "memory": "3Gi" } }, "topologySpreadConstraints": [ { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "storegateway", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "topologyKey": "topology.kubernetes.io/zone", "whenUnsatisfiable": "ScheduleAnyway" }, { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "storegateway", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "topologyKey": "kubernetes.io/hostname", "whenUnsatisfiable": "ScheduleAnyway" } ] }	Thanos Store Gateway configuration. Store Gateway provides historical block querying. Autoscaling is available upstream but left off deliberately. Store Gateway is a PVC-backed StatefulSet that syncs the bucket's block index on startup, so scale-up is slow to become useful (it serves nothing until the index is warm) and CPU-triggered scaling reacts long after the load that triggered it. Scale-down also leaves orphaned PVCs behind, since StatefulSet volumes are not reclaimed. Size it deliberately instead.
thanos.storegateway.persistence	object	{ "enabled": true, "size": "10Gi" }	Index-header cache for Store Gateway. One of two Thanos components that keep a PVC — the Compactor is the other, and Receive alone is `emptyDir`-backed. The reason here is startup cost rather than durability or size: the on-disk binary index-headers are a read-through cache of the bucket and lose nothing when discarded, but rebuilding them means re-downloading from object storage, and the component serves no historical query until that finishes. On a large bucket that is minutes of degraded long-range dashboards on every restart. Kept on at every size deliberately, rather than flipping to ephemeral for small installs. A volume that appears and disappears with the profile is a surprise during an incident, and the AZ-pinning argument that drives Receive to `emptyDir` is much weaker here: Store Gateway is a read-only replica set with no quorum, so a pod stuck Pending in a dead zone costs read capacity rather than blocking writes. The `kind` profile is the one exception, where CI would rather not wait on volume provisioning for a cache.
thanos.storegateway.resources	object	{ "requests": { "cpu": "500m", "memory": "3Gi" } }	Resource requests for Store Gateway. Medium sizing. There is a floor here that is easy to trip over. Thanos defaults `--chunk-pool-size=2GB` and `--index-cache-size=250MB`, and the subchart passes neither, so a stock Store Gateway wants ~2.5Gi before it serves a single query. A request below that is an OOM, not a small install — the `thanos-small` profile shrinks the pools through `extraArgs` instead.
thanos.storegateway.topologySpreadConstraints	list	[ { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "storegateway", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "topologyKey": "topology.kubernetes.io/zone", "whenUnsatisfiable": "ScheduleAnyway" }, { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "storegateway", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "topologyKey": "kubernetes.io/hostname", "whenUnsatisfiable": "ScheduleAnyway" } ]	Topology spread for Store Gateway: soft on both axes. Soft (`ScheduleAnyway`) where Receive is hard, and the difference is the point rather than an inconsistency. Two things make it the right trade here: 1. Nothing depends on quorum. Store Gateway is a read-only replica set over object storage, so an unbalanced placement costs read capacity during a zone loss rather than correctness. Receive's hard rule protects write quorum; there is no equivalent to protect here. 2. It is the one Thanos component that still keeps a PVC, and a hard zone rule against a zonal volume can deadlock rather than merely wait. With `volumeBindingMode: WaitForFirstConsumer` — the default for the zonal CSI classes and what you want — the volume follows the pod and the two agree. With `Immediate`, the PVC's zone is chosen before scheduling and a hard rule pointing elsewhere leaves the pod Pending forever. Soft degrades instead of deadlocking on a StorageClass we do not control.
thanos.storegateway.extraArgs	list	[ "--store.limits.request-series=5000000", "--store.limits.request-samples=200000000" ]	Extra CLI arguments for Store Gateway. Read-path protection. Thanos has no per-tenant write limit to set; the realistic failure is one query fanning out across the bucket and taking this component down, which takes historical reads with it. These two caps make that query fail instead. The subchart ships this empty, so unlike `receive.extraArgs` there is nothing to lose by overriding it — but it is still a list, so restate both entries when adding a third.
thanos.compactor	object	enabled with default retention policies	Thanos Compactor configuration. Compactor provides block compaction and downsampling.
thanos.compactor.retention	object	{ "resolution1h": "365d", "resolution5m": "90d", "resolutionRaw": "30d" }	Retention policies for Thanos Compactor downsampled data The medium row of the retention table in Production Best Practices. Raw retention has a floor set by the downsampling thresholds: Thanos produces 5m downsamples only from blocks spanning 40h or more, and 1h downsamples only from blocks spanning 10d or more. Below those the tier is never created at all and long-range queries silently fall back to reading raw blocks — slower and more expensive, which is the opposite of the intent. 30d clears both comfortably.
thanos.compactor.vpa	object	{ "enabled": false }	VerticalPodAutoscaler for the Compactor. Off for the same reasons as `receive.vpa`, minus the block-loss argument — the Compactor's local volume is an idempotent working copy, so an eviction costs a restarted compaction rather than data. It still overwrites the requests below, and still behaves differently depending on whether the VPA CRD happens to be installed.
thanos.compactor.resources	object	{ "requests": { "cpu": 1, "memory": "2Gi" } }	Resource requests for the Compactor. Medium sizing. The Compactor is a singleton by necessity — concurrent compactors against one block set corrupt data — so it is the one component here with no horizontal escape hatch. Vertical is the only lever, which makes compaction falling behind something to alert on rather than discover. Downsampling is the memory-heavy phase, and `--compact.concurrency` multiplies it; leave that at 1 unless compaction is provably behind. No `ephemeral-storage` here: the Compactor keeps a PersistentVolume, so its scratch does not draw on the node's budget. See `persistence` for why.
thanos.compactor.persistence	object	{ "enabled": true, "size": "50Gi" }	Working directory for the Compactor: a PersistentVolume, unlike Receive. Scratch space for the block group under compaction. Nothing here is authoritative — the bucket is — and a Compactor killed mid-compaction simply redoes the work, so there is no durability argument for a volume. The argument is size. Compaction works on whole block groups, not on the recent window, so the requirement scales with days of data rather than hours. At the medium envelope a 2d group is roughly six 8h blocks at ~2.4Gi each, and the Compactor needs the sources and the output at once — on the order of 30Gi. A GKE node with a 47Gi boot disk offers about 18.8Gi of allocatable ephemeral storage, so that never fits: the pod stays Pending on `Insufficient ephemeral-storage`, and the cluster-autoscaler cannot help because no node of that shape would fit either. This is the reverse of the call made for Receive, and deliberately so. Receive holds hours of small, replicated data and is the better trade on `emptyDir`; the Compactor holds days of it and is not. The AZ-pinning cost is real — a zone outage stops compaction, and while compaction is stopped retention is not enforced and the bucket grows — but "compaction pauses during a zone outage" is a far better failure than "compaction never runs because the pod cannot be scheduled".
thanos.queryFrontend	object	{ "autoscaling": { "enabled": true, "maxReplicas": 5, "minReplicas": 2, "targetCPUUtilizationPercentage": 80 }, "enabled": false, "resources": { "requests": { "cpu": "250m", "memory": "512Mi" } }, "topologySpreadConstraints": [ { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "query-frontend", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "pod-template-hash" ], "maxSkew": 1, "topologyKey": "topology.kubernetes.io/zone", "whenUnsatisfiable": "ScheduleAnyway" }, { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "query-frontend", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "pod-template-hash" ], "maxSkew": 1, "topologyKey": "kubernetes.io/hostname", "whenUnsatisfiable": "ScheduleAnyway" } ] }	Thanos Query Frontend configuration. Query Frontend provides query parallelization and result caching. Only required for production. Note that enabling this is not sufficient on its own: point `connections.datasources.thanos.url` at the query-frontend Service too, or reads keep going straight to Query and the cache is never consulted. A validator warns when the two disagree.
thanos.queryFrontend.autoscaling	object	{ "enabled": true, "maxReplicas": 5, "minReplicas": 2, "targetCPUUtilizationPercentage": 80 }	Horizontal autoscaling for Query Frontend. Stateless like Query, so the same reasoning applies. Inert until `queryFrontend.enabled` is true.
thanos.queryFrontend.topologySpreadConstraints	list	[ { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "query-frontend", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "pod-template-hash" ], "maxSkew": 1, "topologyKey": "topology.kubernetes.io/zone", "whenUnsatisfiable": "ScheduleAnyway" }, { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "query-frontend", "app.kubernetes.io/name": "thanos" } }, "matchLabelKeys": [ "pod-template-hash" ], "maxSkew": 1, "topologyKey": "kubernetes.io/hostname", "whenUnsatisfiable": "ScheduleAnyway" } ]	Topology spread for Query Frontend: soft on both axes, same reasoning as Query. Inert until `enabled` is true.
thanos.queryFrontend.resources	object	{ "requests": { "cpu": "250m", "memory": "512Mi" } }	Resource requests for Query Frontend. Inert until it is enabled, which `thanos-large` does. Sized for splitting and result caching rather than query execution — the work happens in Query behind it.
thanos.ruler	object	{ "enabled": false }	Thanos Ruler configuration. Ruler provides alerting and recording rules evaluation.

#### Grafana Operator Bundled Grafana-operator for managing Grafana instances. Upstream references: * https://grafana.github.io/grafana-operator/docs/installation/helm/#values * https://github.com/grafana/grafana-operator/blob/master/deploy/helm/grafana-operator/values.yaml

Key	Type	Default	Description
grafana-operator.priorityClassName	string	`"monitoring-scalable"`	Scheduling priority. See the Priority classes section. Scalable tier: while the operator is down, dashboard and datasource reconciliation stops, but Grafana keeps serving what it already has.
grafana-operator.fullnameOverride	string	`"grafana-operator"`	Standard Helm full-name override. We use a static name for deterministic relations.
grafana-operator.namespaceOverride	string	`nil`	Namespace override.
grafana-operator.crds	object	{ "immutable": true }	CRD behavior. The Grafana Operator CRDs are owned by the `materialize-monitoring-crds` chart, which vendors a deflated copy of them. The operator chart offers no way to skip its own CRDs outright — `immutable` only chooses where they come from — so keep this `true`: that keeps them out of this chart's release manifest and leaves them install-only, which `helm install --skip-crds` drops entirely. Setting it `false` makes this chart template and upgrade the CRDs itself, fighting the CRDs chart for ownership.

#### Grafana Instance Bundled Grafana for dashboard rendering. Upstream reference: * https://github.com/grafana-community/helm-charts/blob/main/charts/grafana/values.yaml The defaults here are the *safe* shape, not the *production* shape, and the two differ in three places you have to close yourself: 1. **State.** Grafana stores users, service accounts and tokens, annotations, dashboard versions and permissions, preferences, and alert-rule state in a database of its own. The default is SQLite on an `emptyDir`, so all of it is lost on every restart, upgrade, and reschedule. The dashboards this chart installs come back on their own (grafana-operator re-pushes them every `resyncPeriod`); anything a human created does not. Apply the `grafana-postgres` profile, or `grafana-pvc` if you have no database. 2. **Reachability.** The Service is `ClusterIP`, so the only access path is `kubectl port-forward`. Set `ingress` or `service.type` — and read `connections.grafana.allowPublicAccess` before you do. 3. **Authentication.** The only account is the generated admin. Configure an identity provider under `grafana.ini` before exposing it to anyone but yourself. Everything under `grafana.ini` is passed through to Grafana's own config file verbatim, so any section Grafana understands can be set here — see the `[auth.generic_oauth]` example on that key. See [Production Best Practices > Grafana](https://materializeinc.github.io/materialize-monitoring/operating/production-best-practices/#grafana) for the full checklist.

Key	Type	Default	Description
grafana.fullnameOverride	string	`"grafana"`	Standard Helm full-name override. We use a static name for deterministic relations. `connections.grafana.mode: bundled` derives the URL it hands grafana-operator from this, so a release-name-derived name would leave the operator dialing a host that does not resolve.
grafana.namespaceOverride	string	`nil`	Namespace override.
grafana.priorityClassName	string	`"monitoring-scalable"`	Scheduling priority. See the Priority classes section.
grafana.image	object	{ "pullPolicy": "IfNotPresent", "registry": "docker.io", "repository": "grafana/grafana", "tag": "13.2.2" }	Grafana server image. Pinned here rather than left to track the subchart's `appVersion`, so Renovate can bump the server on its own cadence instead of only when a new chart release happens to carry one. A hardened rebuild is a drop-in swap: point `registry`/`repository` at it and keep the tag. Two published options track upstream Grafana versions and keep its entrypoint and layout, so a values change is the whole change: * Chainguard Images — https://images.chainguard.dev/directory/image/grafana/overview * Docker Hardened Images — https://docs.docker.com/dhi/ (subscription; images land in your own org namespace, so `registry` is yours) `profiles/registry/` does this for the whole stack rather than for Grafana alone, and carries the pull-secret wiring that goes with it. Both ship no shell and no package manager, which is the point — and which also means `plugins` cannot work, since that installs at container start. Bake plugins into the image instead. See the note on `plugins` below. Bitnami also publishes a hardened Grafana, but its images are built for Bitnami's own charts — Bitnami entrypoints, `/opt/bitnami` paths, UID 1001 — against an upstream subchart here, so it is a port rather than a swap. No profile ships for it.
grafana.replicas	int	`1`	Number of Grafana replicas. More than one requires a shared database (`grafana.ini.database`) — each replica otherwise carries its own divergent SQLite file, which is a correctness bug rather than availability. A render-time check enforces this.
grafana.autoscaling	object	{ "enabled": false, "maxReplicas": 5, "minReplicas": 1, "targetCPU": "60", "targetMemory": "" }	Horizontal autoscaling for the Grafana Deployment. Off by default because it is meaningless on the default SQLite backend: every replica would carry its own database. With `grafana.ini.database` pointed at PostgreSQL, Grafana is stateless and scales horizontally, so enable it then — `minReplicas` becomes the effective replica count and `replicas` stops being rendered at all. `targetCPU` is a percentage of the CPU request, so `resources.requests.cpu` has to be set or the HPA has no denominator and never scales. A render-time check warns when it is missing.
grafana.podDisruptionBudget	object	{ "maxUnavailable": 1 }	PodDisruptionBudget for Grafana. `maxUnavailable` rather than `minAvailable`, matching the convention the Loki and Thanos blocks use: it scales with the replica count, and on a single replica `minAvailable: 1` would permit no voluntary eviction at all and hang node drains.
grafana.resources	object	{ "limits": { "memory": "1Gi" }, "requests": { "cpu": "100m", "memory": "256Mi" } }	Resource requests and limits for the Grafana container. Requests are what the scheduler and the HPA both read — without a CPU request, `autoscaling.targetCPU` has no denominator and the HPA never scales. No CPU limit: query rendering is bursty and throttling it makes the UI feel broken. Raise the memory limit if you run many large dashboards or heavy plugins.
grafana.persistence	object	{ "accessModes": [ "ReadWriteOnce" ], "enabled": false, "size": "10Gi" }	Persistent volume for Grafana's SQLite database and plugin directory. Off by default. Turning it on survives restarts but caps you at one replica (SQLite tolerates one writer) and, on a `ReadWriteOnce` volume, requires `deploymentStrategy.type: Recreate` — a rolling update otherwise deadlocks with the new pod waiting for a volume the old pod has not released. Both are enforced at render time. The `grafana-pvc` profile is the assembled version. Prefer PostgreSQL (`grafana-postgres`) wherever a database is available: it is the only option that lifts both constraints.
grafana.deploymentStrategy	object	{ "type": "RollingUpdate" }	Deployment update strategy. Must be `Recreate` alongside a `ReadWriteOnce` PersistentVolume; see `persistence`.
grafana.service	object	{ "annotations": {}, "enabled": true, "loadBalancerSourceRanges": [], "port": 80, "targetPort": 3000, "type": "ClusterIP" }	Service exposing Grafana. `ClusterIP` is reachable only from inside the cluster, which is the safe default and the reason a fresh install needs `kubectl port-forward`. Prefer `ingress` over a `LoadBalancer` Service: Grafana is ordinary HTTP that wants host-based routing and a certificate, which is what Ingress is for. A `LoadBalancer` Service with no `loadBalancerSourceRanges` is an error at render time unless `connections.grafana.allowPublicAccess` is set.
grafana.ingress	object	{ "annotations": {}, "enabled": false, "hosts": null, "ingressClassName": "", "labels": {}, "path": "/", "pathType": "Prefix", "tls": null }	Ingress for Grafana. The preferred way to make Grafana reachable. Terminate TLS here: Grafana authenticates with a session cookie and, without TLS, that cookie and the admin password cross the network in the clear. A render-time check warns when an Ingress carries no `tls` block. Set `grafana.ini.server.root_url` to the same URL — Grafana builds share links, alert notification links, and OAuth redirect URIs from it, and all three break silently when it disagrees with the Ingress host.
grafana."grafana.ini"	object	{ "analytics": { "check_for_updates": false, "reporting_enabled": false }, "date_formats": { "default_timezone": "UTC" } }	Grafana's own configuration file, as YAML. Rendered verbatim into `grafana.ini`, so any section Grafana understands can be set here: authentication, SMTP, feature toggles, unified alerting, user provisioning. The chart sets only the handful below and merges anything you add over them. Secrets must NOT be inlined — this block renders into a ConfigMap, so a value here lands in plaintext in the release manifest, in `helm get values`, and in whatever Git repo holds your values file. Use Grafana's own expansion instead: `$__file{/path}` for a mounted Secret (see `extraSecretMounts`) or `$__env{VAR}` for one injected with `envValueFrom`. The subchart's `assertNoLeakedSecrets` check fails the render if you forget. OIDC single sign-on, as an example — this is the shape, not a default: ```yaml grafana: grafana.ini: server: root_url: https://grafana.example.com auth: oauth_auto_login: true auth.generic_oauth: enabled: true name: SSO allow_sign_up: true client_id: client_secret: $__file{/etc/secrets/grafana-oidc/client-secret} scopes: openid profile email groups auth_url: https://idp.example.com/oauth2/v1/authorize token_url: https://idp.example.com/oauth2/v1/token api_url: https://idp.example.com/oauth2/v1/userinfo allowed_domains: example.com # Map an IdP group claim onto a Grafana role, so membership is the # provisioning mechanism and nobody is added by hand. role_attribute_path: contains(groups[*], 'sre') && 'Admin' \|\| 'Viewer' extraSecretMounts: - name: grafana-oidc secretName: grafana-oidc mountPath: /etc/secrets/grafana-oidc readOnly: true ``` Every provider, the role-mapping rules, and the break-glass path are in [Dashboards > Grafana > Authentication](https://materializeinc.github.io/materialize-monitoring/dashboards/grafana/auth/).
grafana.networkPolicy	object	{ "allowExternal": true, "egress": { "enabled": false }, "enabled": true, "explicitIpBlocks": [], "explicitNamespacesSelector": {}, "ingress": true }	NetworkPolicy for Grafana. The subchart's policy is rigid: it emits exactly one ingress rule, on `service.targetPort`, and either allows every source or a fixed set of them. There is no way to add a second port through values. That constrains what this can say, so it is worth being precise about what it does say. Ingress: `3000` from anywhere, everything else closed. `allowExternal` stays true because Grafana is the one component in this stack a human is meant to reach, and the ways they reach it are all things a `podSelector` cannot see — an ingress-controller pod in another namespace, a cloud load balancer's source IPs, `kubectl port-forward`, which arrives from the API server. Turning it off is worthwhile once you know your path in; `explicitNamespacesSelector` takes the ingress controller's namespace and `explicitIpBlocks` takes a load balancer's health-check and client ranges. `profiles/grafana-ingress.values.yaml` is where that belongs. What it closes is the rest of the pod: `6060` (pprof) and `9094` (the unified-alerting gossip port) stop accepting connections. pprof should not be reachable in production, so that one is the point. `9094` is not — it matters as soon as you run multiple replicas and set `grafana.ini.unified_alerting.ha_peers`, which needs the pods to reach each other on it, and this template cannot express a second port. You do not have to turn this off to get HA alerting. The chart renders the missing rule itself, as `networkPolicies.grafanaGossip`, and it appears exactly when `ha_peers` is set. Policies are additive, so the two sit side by side. Turning that off while HA alerting is configured is what the render warns about; turning this one off takes the supplement with it, since a lone gossip policy would isolate the pods and leave only `9094` open. Egress is left unrestricted (`egress.enabled: false` renders a policy with no `Egress` policy type at all, rather than an empty allow). Grafana dials the things it is configured to dial: Loki and Thanos in-namespace, a PostgreSQL state database wherever it lives, an OIDC provider on the internet, `grafana.com` if `plugins` is non-empty. The first two are knowable here and the rest are not, and a partial allowlist would fail as a dashboard that renders empty with no error — the worst failure mode this stack has.
grafana.networkPolicy.ingress	bool	`true`	Emit the ingress rule. False renders a policy with no rules at all, which — with egress off too — is an empty spec, not a deny.
grafana.networkPolicy.allowExternal	bool	`true`	Accept from any source. See the note above before turning this off.
grafana.networkPolicy.explicitNamespacesSelector	object	{}	Namespaces allowed in when `allowExternal` is false. The ingress controller's namespace goes here.
grafana.networkPolicy.explicitIpBlocks	list	[]	CIDRs allowed in when `allowExternal` is false. Load-balancer client and health-check ranges go here.
grafana.networkPolicy.egress.enabled	bool	`false`	Leave off. See the note above: a partial egress allowlist for Grafana fails as empty panels rather than as an error.
grafana.plugins	list	[]	Grafana plugins to install at container start. Each entry is downloaded from grafana.com when the pod starts, so this needs egress to `grafana.com` on 443 and adds that download to every start and every restart — a startup dependency on a third-party service. For anything load-bearing, bake plugins into the image instead and point `image` at it. That is also the only option on a hardened base image, which ships no shell for `grafana cli` to run in. Pin an exact plugin version either way (`name@version`); an unpinned plugin silently changes underneath a pinned Grafana.
grafana.imageRenderer	object	{ "enabled": false }	Grafana Image Renderer, for server-side PNG rendering of panels. Deliberately off, and it should stay off in production: the renderer is a headless Chromium that fetches arbitrary URLs on Grafana's behalf, which makes it both a large attack surface and a server-side request forgery pivot into the cluster network. A render-time check warns when it is enabled.
grafana.testFramework	object	{ "enabled": false }	Helm test hooks shipped by the subchart. Off: it pulls a `bats` image this chart does not otherwise use or pin, and the e2e suite covers what it asserts.
grafana.assertNoLeakedSecrets	bool	`true`	Fail the render when a secret is inlined into `grafana.ini`. Leave this on. It is the guard that keeps database and OAuth credentials out of the ConfigMap; see the note on `grafana.ini`.
grafana.serviceMonitor	object	{ "enabled": true }	Scrape Grafana's own metrics.

#### Alertmanager Bundled Alertmanager for routing alerts emitted by the rule packages.

Key	Type	Default	Description
alertmanager.priorityClassName	string	`"monitoring-scalable"`	Scheduling priority. See the Priority classes section.
alertmanager.persistence.size	string	`"4Gi"`	Volume for silences and the notification log. Sized by cloud disk minimums, not by Alertmanager, which needs kilobytes. The subchart default of 50Mi is below the 4 GiB floor on GCP Hyperdisk and Azure managed disks, so provisioning fails there.

#### Kube State Metrics kube-state-metrics for Kubernetes resource-state metrics consumed by Materialize-adjacent dashboards. Upstream reference: * https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-state-metrics/values.yaml

Key	Type	Default	Description
kube-state-metrics.priorityClassName	string	`"monitoring-scalable"`	Scheduling priority. See the Priority classes section.
kube-state-metrics.networkPolicy	object	{ "egress": [ { "ports": [ { "port": 53, "protocol": "UDP" }, { "port": 53, "protocol": "TCP" } ], "to": [ { "namespaceSelector": {} } ] }, { "ports": [ { "port": 443, "protocol": "TCP" }, { "port": 6443, "protocol": "TCP" } ] } ], "enabled": true, "ingress": [ { "from": [ { "podSelector": { "matchLabels": { "app.kubernetes.io/name": "alloy-gateway" } } } ], "ports": [ { "port": 8080, "protocol": "TCP" }, { "port": 8081, "protocol": "TCP" } ] } ] }	NetworkPolicy for kube-state-metrics. Ingress is narrowed to the Alloy gateway, which is what scrapes it (`prometheus.operator.servicemonitors` in the gateway pipeline) — the same shape as `node-exporter.networkPolicy`, and unlike that one it is actually enforced, because these pods are not host-networked. `8081` is listed alongside `8080` even though nothing serves it at the default `selfMonitor.enabled: false`. A closed port that nothing dials costs nothing; the alternative is that turning `selfMonitor` on produces a scrape that fails silently against a policy nobody thought to update. Egress is restricted, and this is the one component where that is safe to do, because its destination set is closed: DNS, and the API server. It reads nothing else — the whole workload is a watch on the API and an exposition endpoint. Deny egress outright (as `node-exporter` does) and it never lists a single object; the pod stays Ready and every `kube_*` series simply disappears. `443`/`6443` to `0.0.0.0/0` rather than something tighter because the API server's address is the one thing a chart cannot derive. In-cluster it is the `kubernetes.default.svc` ClusterIP, which most CNIs evaluate against the endpoint behind it — and on a managed control plane that endpoint is outside the cluster network entirely. Narrow the CIDR to your control plane's where you know it.
kube-state-metrics.prometheus.monitor.http	object	{ "honorLabels": true }	Keep kube-state-metrics' own labels when they collide with the scrape's. Load-bearing, not a preference. Every `kube_` series carries its own `namespace`, `pod` and `container` labels describing the object it reports on* — that is the entire point of the exporter. Those names collide with the target labels of the scrape, which describe the kube-state-metrics pod itself. Without this, Prometheus resolves the collision by renaming the exporter's labels to `exported_namespace` / `exported_pod` and writing the target's identity into `namespace` / `pod`. The result is not missing data, which is why it went unnoticed: every series arrives, `up` is 1 and `scrape_samples_scraped` is healthy, but every series reads `namespace=""`. `kube_pod_info` collapses from one series per pod to a single identity, and any query written as `kube_*{namespace=…}` — which is all of ours — silently matches nothing. Both endpoints, so turning `selfMonitor` on does not reintroduce it on the half nobody was looking at. The subchart defaults both to `false`. Asserted by the e2e suite: `kube_state::labels_are_honored`.

#### Node Exporter node-exporter for hardware and OS metrics from every node. Upstream reference: * https://github.com/prometheus-community/helm-charts/blob/main/charts/prometheus-node-exporter/values.yaml * https://github.com/prometheus/node_exporter#collectors Run as its own DaemonSet rather than folded into the Alloy agent's `prometheus.exporter.unix`, so its resource envelope stays known and separate: a metrics regression cannot then starve log collection out of a shared limit. See the Terraform modules design doc for the full argument.

Key	Type	Default	Description
node-exporter.fullnameOverride	string	`"node-exporter"`	Standard Helm full-name override. We use a static name for deterministic relations.
node-exporter.namespaceOverride	string	`nil`	Namespace override.
node-exporter.priorityClassName	string	`"monitoring-critical"`	Scheduling priority. See the Priority classes section. Critical: a node with no node-exporter is a node nothing else reports on, and the gap does not backfill.
node-exporter.image	object	{ "distroless": true, "registry": "quay.io", "repository": "prometheus/node-exporter", "tag": "v1.12.1" }	Container image. Split into registry / repository / tag so Renovate's `helm-values` manager can bump the exporter independently of the subchart version — the two move on different cadences and a chart release is not a prerequisite for a node_exporter CVE fix. `distroless: true` appends `-distroless` to the tag, giving an image with no shell and no package manager. node_exporter reads the host's `/proc`, `/sys` and `/` — a shell in that container is a materially better foothold than a shell in most others, and nothing in our configuration needs one.
node-exporter.resources	object	{ "limits": { "memory": "64Mi" }, "requests": { "cpu": "10m", "memory": "64Mi" } }	Resource envelope, matching what Materialize runs in production today. Deliberately no CPU limit. This is a DaemonSet that does nothing between scrapes and then bursts for the length of one collection; a CPU limit turns that burst into CFS throttling, which shows up as scrape timeouts on exactly the loaded nodes where you most need the sample. The 10m request is what it averages, not what a scrape costs. Memory is request == limit, which is the opposite call for the opposite reason: node_exporter's working set is flat and bounded (~20-30 MiB for this collector set), so a limit equal to the request costs nothing and makes the per-node footprint a number the cluster autoscaler can actually plan with. This lands the pod in Burstable QoS. Guaranteed would require the CPU limit we just argued against; being evicted a little earlier under node pressure is the better trade, and `monitoring-critical` covers the ordering.
node-exporter.updateStrategy	object	{ "rollingUpdate": { "maxUnavailable": "10%" } }	Rolling update budget for the DaemonSet. Upstream's `maxUnavailable: 1` walks a large fleet one node at a time, which makes an image bump take hours. Metrics are gap-tolerant and the pod restarts in seconds, so a percentage keeps rollout time flat as the fleet grows.
node-exporter.kubeRBACProxy	object	{ "enabled": false }	kube-rbac-proxy sidecar, which would put the exporter behind TokenReview/SubjectAccessReview over HTTPS. Off deliberately. It is a second container on every node, and DaemonSet overhead is the constraint we are managing here — the sidecar's own request is comparable to node_exporter's, so it roughly doubles the per-node cost of node metrics to protect an endpoint that exposes no secrets. What it would buy, for when that trade changes: authenticated scrapes, and (via `kubeRBACProxy.tls` + `tlsSecret`) TLS with client-certificate auth, so only a scraper holding a cert signed by the configured CA can read `/metrics`. That is the shape an in-cluster-mTLS story would use, and it is parked, not rejected — see the cert-manager mTLS item on the roadmap. Note that with `hostNetwork: true` the exporter listens on the node's own interfaces, so today the real perimeter is the node firewall / security group, not anything Kubernetes enforces. Do not treat port 9100 as private just because a NetworkPolicy exists.
node-exporter.networkPolicy	object	{ "enabled": true, "ingress": [ { "from": [ { "podSelector": { "matchLabels": { "app.kubernetes.io/name": "alloy-gateway" } } } ], "ports": [ { "port": 9100, "protocol": "TCP" } ] } ] }	NetworkPolicy for the exporter. Ingress is narrowed to the Alloy gateway, which is what actually scrapes it (`prometheus.operator.servicemonitors` in the gateway pipeline). Egress is denied outright — node_exporter never initiates a connection. Read the hostNetwork caveat before relying on this. The pods run with `hostNetwork: true` (required: `netdev`, `netclass`, `netstat`, `sockstat` and `conntrack` all read namespaced files under `/proc/net`, so in a pod netns they would report the pod's traffic, not the node's). Most CNIs — Cilium and Calico among them — do not apply pod NetworkPolicy to host-networked pods, because that traffic belongs to the node identity. So this policy is a correct declaration of intent and is enforced where the CNI supports it, but it is not the control that keeps port 9100 private. The node's firewall is. The default `from` selects by pod label within the release namespace. Running the gateway in a different namespace (see `profiles/split-namespace.values.yaml`) requires replacing this list with one that adds a `namespaceSelector`.
node-exporter.prometheus.monitor.enabled	bool	`true`	Emit a ServiceMonitor. This is how collection actually happens: the Alloy gateway discovers ServiceMonitors cluster-wide and scrapes the endpoints behind them. Without this the DaemonSet runs and is never read.
node-exporter.extraArgs	list	[ "--collector.disable-defaults", "--collector.cpu", "--collector.cpufreq", "--collector.loadavg", "--collector.schedstat", "--collector.stat", "--collector.pressure", "--collector.meminfo", "--collector.vmstat", "--collector.swap", "--collector.vmstat.fields=^(oom_kill\|pgpg\|pswp\|pg.fault\|pgscan\|pgsteal\|workingset).$", "--collector.diskstats", "--collector.filesystem", "--collector.filesystem.mount-points-exclude=^/(dev\|proc\|run/credentials/.+\|sys\|var/lib/docker/.+\|var/lib/containers/storage/.+\|var/lib/kubelet/(pods\|plugins)/.+\|run/containerd/.+)($\|/)", "--collector.nvme", "--collector.xfs", "--collector.netdev", "--collector.netdev.device-exclude=^(lo\|lxc.\|veth.\|cali.\|azv.\|docker.\|br-.\|nodelocal.\|kube-ipvs.\|dummy.)$", "--collector.netclass", "--collector.netclass.ignored-devices=^(lo\|lxc.\|veth.\|cali.\|azv.\|docker.\|br-.\|nodelocal.\|kube-ipvs.\|dummy.)$", "--collector.netstat", "--collector.sockstat", "--collector.softnet", "--collector.udp_queues", "--collector.conntrack", "--collector.arp", "--collector.os", "--collector.uname", "--collector.time", "--collector.timex", "--collector.filefd", "--collector.entropy", "--collector.hwmon", "--collector.selinux", "--collector.kernel_hung" ]	Extra arguments to node_exporter, which is where the collector allowlist lives. `--collector.disable-defaults` turns everything off; each `--collector.` then turns one back on. An allowlist rather than a denylist because the default set drifts with each node_exporter release — a new default-on collector should not silently appear in our cardinality budget. Every name here must exist on Linux. An unknown `--collector.` flag is a parse error and node_exporter exits, so a typo or a platform-specific collector crash-loops the DaemonSet on every node at once. In particular `boottime` is a Darwin/BSD collector; on Linux `node_boot_time_seconds` comes from `stat`, which is why `stat` is in the list and `boottime` is not. Chosen for a network- and memory-hungry Rust database that swaps on purpose: Swap, explicitly. `swap` (per-device swap, off by default upstream) and `meminfo` (`SwapTotal`/`SwapFree`) give the level; `vmstat` gives the rate (`pswpin`/`pswpout`), which is the half that distinguishes healthy deliberate swapping from thrash. `pressure` (PSI) is the leading indicator for both. * Memory reclaim. The `vmstat` field filter is widened below to admit `pgscan_`/`pgsteal_`, which separate background reclaim (`kswapd`, fine) from direct reclaim (`direct`, an allocating thread is stalled) — the difference between swap working and swap hurting. * Network. `netdev`/`netclass` for the NICs, `netstat` for TCP retransmits, `sockstat` for socket and TCP-memory pressure, `softnet` for packets the kernel dropped before userspace saw them, `udp_queues` for DNS, and `conntrack` for table exhaustion behind NAT gateways. * CPU. `cpu` (including `mode="steal"`), plus `schedstat` for run-queue wait — "threads were runnable and did not run", which is the contention a tightly-optimized dataflow engine feels first — and `stat` for context switches and blocked processes. * Correctness-adjacent. `timex` for clock sync; a database with timestamp semantics on cloud VMs wants to know when NTP has given up. Left off on purpose, with the reason, since each is one line to re-add: * `slabinfo` — kernel slab accounting. Costs ~1.5k series per node (every slab cache × several metrics) and needs a root init container to chmod `/proc/slabinfo` (`permissionInitContainer.fixes.slabinfo`). `meminfo` already answers "how much kernel memory" via `Slab`/`SReclaimable`/ `SUnreclaim`; `slabinfo` only adds which cache, which is forensics you turn on during an investigation. * `ethtool` — on EC2 this is where the ENA allowance counters live (`bw__allowance_exceeded`, `pps_allowance_exceeded`, `conntrack_allowance_exceeded`): instance-level network throttling that is invisible in every other metric. Genuinely valuable on AWS and worth enabling there, paired with `--collector.ethtool.device-include=^(eth\|ens\|enp)` — held back only because the stat set is driver-specific (gVNIC and Azure expose different counters) and unverified on our images. `interrupts`, `softirqs` — per-CPU × per-IRQ, so thousands of series on a large instance. `softnet` covers the actionable part. * `tcpstat` — parses every socket in `/proc/net/tcp`, which is most expensive on exactly the busiest nodes. `sockstat` gives the aggregate cheaply. * `processes` — walks all of `/proc` each scrape. `stat` already reports running and blocked counts. * `edac`, `rapl`, `thermal_zone`, `dmi` — hypervisors do not expose ECC, powercap or thermals to guests, and instance type comes from node labels. Worth revisiting on bare metal. * `meminfo_numa`, `zoneinfo`, `buddyinfo` — NUMA and fragmentation detail for very large instances; opt in when investigating. * `ipvs`, `nfs`, `nfsd`, `bonding`, `mdadm`, `zfs`, `btrfs`, `bcache` — none apply to a cloud node running ext4/xfs on network block storage with Cilium replacing kube-proxy. `ipvs` matters if you run kube-proxy in IPVS mode; `nfs` if you mount EFS/Filestore. * `sysctl` — no default set, but `--collector.sysctl.include=vm.swappiness, vm.overcommit_memory,vm.overcommit_ratio` is a cheap guard against AMI drift changing swap behaviour underneath a swap-dependent workload. * `textfile` — the node-local extension point, but it needs a hostPath mount this chart does not create.

#### Metrics Server metrics-server for pod and node resource usage; only needed when the cluster does not already ship one. Upstream reference: * https://github.com/kubernetes-sigs/metrics-server/blob/master/charts/metrics-server/values.yaml

Key	Type	Default	Description
metrics-server.replicas	int	`1`	Number of replicas for metrics-server.