materialize-monitoring Helm Reference#

• Alloy reads global.image.pullSecrets and never this key. This chart honors both spellings for the workloads it renders itself, but the Alloy subchart only sees its own.
• Loki, grafana-operator, Alertmanager and metrics-server read no global at all and need their own <chart>.imagePullSecrets.

profiles/registry/pull-secret.values.yaml sets every one of those paths from a single Secret name; prefer it over setting this key alone, which leaves Loki — the largest pod count in the release — pulling anonymously.

It lives under global because two subcharts already read global.clusterDomain and build real addresses from it — Loki’s memberlist join address and canary URL, and Thanos’s in-cluster endpoints. Helm propagates global into every subchart, so setting it here is what makes one value mean one thing rather than three keys that can disagree.

One subchart is not covered by that propagation. metrics-server reads its own metrics-server.tls.clusterDomain, which is set to the same default below; a render-time check warns when the two disagree, because the failure is a certificate whose SANs name a domain the cluster does not use.

Load-bearing for certificates: the SAN ladder ends in $svc.$ns.svc.$clusterDomain, so a wrong value here produces certificates that fail verification against the endpoints this chart itself dials.

One value, read everywhere the stack labels a signal: the Alloy agent and gateway (through pipeline.env.CLUSTER_NAME, which defaults to it), the gateway’s remote-write external_labels, and both rulers’ alerts (through the ruler-env ConfigMap). Terraform’s cluster_name sets it.

Give each cluster its own. The label is what tells clusters apart once they share a log or metric store, a notification channel or an incident tool, and the default distinguishes nothing. Letters, digits, ., _ and -, starting with a letter or digit: it is substituted unquoted into Loki’s configuration and into an Alloy template.

Ingress covers 9093 (the API — Loki’s ruler, the Alloy gateway’s scrape, and Grafana’s alerting all arrive here) and 9094, the gossip port a multi-replica Alertmanager uses to deduplicate notifications between peers.

9094 is listed twice, TCP and UDP, and both are required. Alertmanager’s mesh is memberlist: it joins and pushes state over TCP and gossips over UDP, and the upstream chart’s Service declares clusterpeer-tcp and clusterpeer-udp on the same port for exactly that reason. A TCP-only rule produces the worst version of this failure — the peers find each other, the cluster forms, and it never converges, so every notification goes out once per replica.

Both are listed even at the default replicaCount: 1, where nothing dials either: the cost is an open port that no pod connects to, and the alternative is a cluster that silently sends every notification twice the day someone scales it up.

Egress is deliberately wide. Alertmanager’s job is to reach PagerDuty, Slack, an SMTP relay, or a webhook on someone’s internal network, and this chart cannot know which — 443 and the SMTP ports are the defaults because they are what receivers use, not because they are a boundary.

Ingress is only the metrics port. 8888 (pprof) is deliberately left out — it is a debugging surface, and a policy that opens it by default defeats the point of having one.

Egress carries 443/6443 to 0.0.0.0/0 because the operator is an API client first: it watches Grafana, GrafanaDashboard and GrafanaDatasource resources continuously, and a policy that cannot reach the API server leaves it running, healthy, and reconciling nothing. The API server’s address is not something a chart can derive — in-cluster it is a Service ClusterIP that most CNIs evaluate against the real endpoint behind it, which on a managed control plane is outside the cluster network entirely.

connections.grafana.mode: external puts the Grafana it writes to outside the cluster as well; 443 already covers the usual case, but a self-hosted Grafana on another port needs an egress.extra entry. The render warns when it can tell that applies.

A supplement, not a replacement: grafana.networkPolicy stays on and keeps doing what it does. That template emits exactly one ingress rule, on service.targetPort, with no way to add a second port through values — so a multi-replica Grafana running HA alerting has its 9094 gossip closed by the very policy that protects its UI. The failure is quiet in the worst way: the replicas start healthy, never find each other, and every Grafana-managed alert notifies once per replica.

This renders the missing rule from the umbrella chart, which can. TCP and UDP, because memberlist uses both — TCP for the join and for state pushes, UDP for the gossip itself — and a TCP-only rule produces a cluster that forms and then does not converge.

It renders only when grafana.ini.unified_alerting.ha_peers is set, which is what makes the replicas dial 9094 in the first place; profiles/grafana-postgres.values.yaml is the shipped shape that sets it. Nothing is created for a single-replica Grafana that never gossips.

Both directions are broad, and both for the same reason: the peers are the control plane and the nodes, neither of which has an address this chart can select. allowExternal is true by default here, unlike the other two — the aggregation layer dials 10250 from the API server, which on a managed control plane is not a pod, not in the cluster’s pod CIDR, and not reachable by any podSelector. A policy that closed that port would break every kubectl top and every HPA in the cluster, not just this release.

What the policy still buys is the rest of the surface: no other port on the pod accepts a connection, and egress is limited to the kubelet port plus the API server rather than anywhere at all.

Egress 10250 to 0.0.0.0/0 is the kubelet summary API on each node. Node IPs are not selectable either, and narrowing this to the node CIDR is a good idea wherever you know it — egress.extra with an ipBlock, and drop 10250 from external.ports.

Supply an issuer with issuerRef, or let the chart bootstrap a self-signed root with selfSigned.enabled. The two are mutually exclusive and the render refuses both at once.

Prefer an issuer scoped to this stack over the cluster’s general-purpose one. None of the receiving components here implement per-client authorization — Loki, Thanos receive and Alloy’s receivers can be told to require a certificate signed by a given CA, and none of them can be told that this identity may write and that one may not. The whole authorization decision is “is this signed by the CA we trust”, so the size of the trust domain is the security property. Reusing a ClusterIssuer that signs for every workload in the cluster reduces mTLS to “has any certificate”, which is a real improvement over “can reach the Service” and much narrower than it sounds.

Renders the three resources cert-manager needs for a private CA: a selfSigned issuer, a CA Certificate signed by it, and an issuer backed by that CA which every component certificate then references.

Convenient rather than principled — a real deployment usually has a PKI already, and issuerRef above is the path for it. This exists so that trying the feature does not start with a cert-manager tutorial, and so the chart-only test tier has a root to issue from.

kind: ClusterIssuer is the default deliberately. A namespaced Issuer can only sign for Certificate resources in its own namespace, and profiles/split-namespace.values.yaml puts components in several — so a namespaced issuer silently signs some components and leaves the rest stuck Pending. Issuer is correct and cheaper for the single-namespace default; the render warns when it sees the combination that does not work.

Only read for kind: ClusterIssuer. A namespaced Issuer has the opposite rule and takes its own namespace; the chart handles both.

Only Grafana takes an external certificate today, because it is the only component in the stack meant for a human and the only one this chart helps expose. Set dnsNames to the hostname the load balancer answers on; it must match what users actually type, and what grafana.ini.server.root_url says.

Leave issuerRef.name empty and nothing external is rendered, which is the right answer for an L7 load balancer holding a cloud-managed certificate.

services is the list the SAN ladder is built from, and it is a list rather than a single name because Loki and Thanos are several Services behind one trust boundary. The defaults name exactly the Services this chart’s own URLs dial — a render check asserts that every in-cluster destination URL matches a SAN on the corresponding certificate, so a Service added to a destination and forgotten here fails the render rather than the handshake.

Each entry takes:

A mounted file does not have that problem: the kubelet refreshes Secret contents atomically, and Alloy’s client paths pick up the new material on the next connection. Set these to paths under /etc/mzmon/tls, which is where the chart mounts the certificate it issues for this component.

Inline PEM stays supported as the bring-your-own-PKI escape hatch.

The gateway has three listeners, not two, and they are split across the two pipeline trees the same way the listeners are:

receive_http is the one to check last: it lives in the metrics tree and is easy to leave behind, and forgetting it leaves the metrics write path wide open behind a logs path that looks secured. A validator refuses that combination rather than letting it ship.

clientAuth is exposed rather than implied because the middle state is what a safe rollout needs, and an operator who cannot name that state cannot perform one. Values are Go’s tls.ClientAuthType:

Unlike Loki’s server, Alloy’s listeners are not probed by the kubelet on the same port — the readiness probe is on 12345 — so phase 3 is reachable here.

A mounted file does not have that problem: the kubelet refreshes Secret contents atomically, and Alloy’s client paths pick up the new material on the next connection. Set these to paths under /etc/mzmon/tls, which is where the chart mounts the certificate it issues for this component.

Inline PEM stays supported as the bring-your-own-PKI escape hatch.

The gateway scrapes /metrics/cadvisor on every kubelet rather than the agent running its own cAdvisor. The kubelet already computes these stats, so running our own meant computing them twice per node — measured at ~750Mi per agent against a 200Mi logs-only envelope. Scraping the kubelet removes the housekeeping cost entirely and puts the remaining scrape cost on the gateway, where it is shared across replicas rather than reserved on every node — the distinction that matters when bin packing.

Coverage is not the tradeoff it was assumed to be: a GKE kubelet serves 69 distinct container_* metrics against the 70 an in-process cAdvisor produced, including the descriptor / socket / thread family that otherwise needs the host PID namespace. Every container_* metric this chart’s queries reference is present.

Needs nodes and nodes/metrics RBAC, which the Alloy subchart’s default clusterRules already grant.

Left false: on GKE the kubelet certificate verifies against the in-cluster CA, which the chart passes as ca_file. A distribution that signs kubelet certs with a CA the pod does not trust needs this true, and the failure is quiet — scrapes fail and container metrics stop rather than anything erroring at install. Check up{job="cadvisor"} when bringing up a new distribution.

kube-proxy turns a Service into rules on the node, so its sync latency is the answer when connections to a Service fail while every pod behind it is healthy. The infra-net dashboard’s kube-proxy row reads it.

Discovery is a server-side pod selector, so a cluster without kube-proxy returns no targets and this costs nothing. GKE Dataplane V2 and any Cilium install in kube-proxy-replacement mode are exactly that case. No enable toggle, for the same reason kubelet above has none: the pipeline is pre-rendered, so a key here would have to be read by something to mean anything, and the scrape already disables itself where kube-proxy is absent. A knob written to the env ConfigMap and read by nothing is a bug this repo has shipped once already.

The gateway can pull what a cloud provider’s monitoring API publishes about the managed database and the buckets a deployment depends on, and write it beside every other metric. The result shares retention, PromQL and alerting with the rest of the stack, so a provider series is joinable with mz_persist_* in one expression. The external-dependency design records why this is a pull rather than a Grafana datasource.

Every provider is off by default. Provider collection is an addition to what the clients already report about the same dependencies, not a replacement for it.

Resources are named, not discovered. Each service lists the instances or buckets to watch, and nothing else is pulled. Tag discovery would find every resource in the account and bill for each of them. Where a resource cannot be named in advance, a named scope stands in for it: EKS nodes are found by their cluster, AKS node VMs and load balancers by theirs, and GCP load balancers by their VPC network. A provider that is enabled with nothing listed fails at render.

Credentials never travel through values. The pull uses the gateway pod’s own cloud identity, bound through alloy-gateway.serviceAccount.annotations:

The data is minutes old when it arrives. CloudWatch publishes RDS metrics a few minutes late and S3 storage metrics once a day. Cloud Monitoring stamps each sample with its own time: Cloud SQL samples arrive about three minutes old, and GCS storage samples over ten. Azure Monitor values are a minute or two behind, and blob capacity is refreshed daily. Query these families with last_over_time(...[15m]) or wider, and do not page on them.

Each pull is billed by the provider. The cost is a function of the interval and of how many resources and metrics are listed, not of how many dashboards are open. The interval is the main lever.

Every replica runs the exporter, and one scrapes it. The scrape is clustered, so a provider’s API is called once per interval however many gateway replicas there are.

The metric sets are fixed, and live in the pipeline, not here. Each pull is a custom component in the chart’s gateway-provider pipeline, instantiated per resource (CloudWatch) or per service (GCP and Azure) for what is listed below. Changing what is pulled is a change to that pipeline.

The gateway has three listeners, not two, and they are split across the two pipeline trees the same way the listeners are:

receive_http is the one to check last: it lives in the metrics tree and is easy to leave behind, and forgetting it leaves the metrics write path wide open behind a logs path that looks secured. A validator refuses that combination rather than letting it ship.

clientAuth is exposed rather than implied because the middle state is what a safe rollout needs, and an operator who cannot name that state cannot perform one. Values are Go’s tls.ClientAuthType:

Unlike Loki’s server, Alloy’s listeners are not probed by the kubelet on the same port — the readiness probe is on 12345 — so phase 3 is reachable here.

Metrics go over OTLP to Google’s Telemetry API (telemetry.googleapis.com) and land as prometheus.googleapis.com/ metric types, queryable with PromQL in Cloud Monitoring. They are billed as Prometheus samples ingested, not per byte as custom workload.googleapis.com/ metrics are.

Names. The metric type carries the point kind: prometheus.googleapis.com/<name>/gauge, /counter, /histogram, /summary, and /unknown plus /unknown:counter for an untyped series. The gateway’s scrapes honor metadata, so the kind is the one the target declares. PromQL takes the bare name, as it does in Thanos, and a histogram answers to <name>_bucket, _count and _sum. There is no prefix to set: the Telemetry API accepts only prometheus.googleapis.com or no domain at all.

Labels. Cloud Monitoring files each series under a prometheus_target resource whose labels are project_id, location, cluster, namespace, job and instance. job and instance are the scrape target’s. namespace is the series’ own, as in Thanos. cluster is CLUSTER_NAME (the cluster label every other destination carries) unless the series has one of its own. location and the project come from the GKE metadata server unless location and project below say otherwise. A series that already carries a location, project_id, job or instance label keeps it as exported_<label>. Every series also carries collected_by="materialize-monitoring", which sets it apart from other Prometheus data in the project: GKE’s managed kube-state-metrics, for one, writes the same metric types under the same job.

Auth is the gateway’s ambient Google identity (Workload Identity on GKE) through otelcol.auth.google, which needs roles/monitoring.metricWriter and the telemetry.googleapis.com API enabled on the project. That component is public preview, which the gateway’s experimental stability level already allows.

For backends that authenticate with an API-key header rather than a bearer token — Honeycomb’s x-honeycomb-team, for instance. Before this existed the only way to send one was authType: custom and a hand-written otelcol.auth.headers block, which no validator could check.

Each entry names a header with key and sets exactly one of:

• value — a literal. It renders into the gateway’s pipeline ConfigMap in plaintext, so use it only for non-secret routing headers such as a dataset or tenant name.
• valueEnv — the name of an environment variable, read with sys.env() when the gateway starts. Use it for anything secret and supply the variable through alloy-gateway.alloy.envFrom (a secretRef). The name is yours to choose; nothing else in the chart depends on it.

Unlike the Loki and Prometheus destinations, there is no value/valueEnv pair per header where the value is copied into the env ConfigMap. A header is either a literal or a lookup, and routing a literal through the env ConfigMap would only move it from one plaintext ConfigMap into another.

Setting this true is the escape hatch for a deployment whose allowlist lives somewhere the chart cannot read — a security group, an egress firewall, a service mesh, an authenticating proxy in front. It suppresses the error, not the exposure: the render still warns, because a Grafana on the open internet is protected by nothing but its admin password until you configure an identity provider under grafana.ini.

Nothing in it is validated or defaulted, and the production guidance on the grafana block above does not reach it. Prefer mode: bundled, where the grafana subchart owns the Deployment and all of that applies.

connections:
  grafana:
    mode: operator
    operator:
      spec:
        version: "13.0.2"
        deployment:
          spec:
            template:
              spec:
                containers:
                  - name: grafana
                    resources:
                      requests:
                        cpu: 100m
                        memory: 256Mi
        config:
          database:
            type: postgres
            host: grafana-db.example.internal:5432

Reference: https://grafana.github.io/grafana-operator/docs/api/#grafanaspec

caSecret names a Secret in the Grafana instance’s namespace holding the roots to trust. Under split-namespace that is grafana, not the release namespace, and the certificate machinery already issues <release>-grafana-tls there — its ca.crt key is the internal CA, so the default below works as-is once certificates.enabled is on.

Prefer this over jsonData.tlsSkipVerify. Skipping verification leaves the connection encrypted and unauthenticated, which on the read path means Grafana will happily talk to anything that answers on that address — and the failure mode of getting it wrong is a dashboard that renders, from the wrong source.

Takes precedence over caSecret, and needs no operator involvement at all, which makes it the right choice when the CA is already in hand at render time. It does not track a rotation: the PEM here is a copy, and re-issuing the CA means re-rendering. Prefer caSecret when the material lives in the cluster.

The Secret must be in the Grafana instance’s namespace, which is not necessarily this release’s under split-namespace.

key is load-bearing beyond naming the field: grafana-operator substitutes ${<key>} — the secretKeyRef key verbatim, dots and all — so the chart derives the placeholder from it. A mismatch there fails in the least visible way available: the CR applies, the operator logs nothing, secureJsonFields reports tlsCACert: true, and only at query time does Grafana say failed to parse TLS CA PEM certificate, because what it stored is the literal placeholder text.

timeInterval is the scrape interval Grafana assumes when it sizes $__rate_interval. The chart sets it to the slower of pipeline.metrics.kubelet.scrapeInterval and pipeline.metrics.kubeProxy.scrapeInterval (60s by default). rate() needs two samples in its window, and with Grafana’s own 15s assumption the window is one minute, which holds one sample of a 60s scrape. Set it here only when this datasource reads metrics scraped more slowly than that; the render warns about a shorter one.

caSecret names a Secret in the Grafana instance’s namespace holding the roots to trust. Under split-namespace that is grafana, not the release namespace, and the certificate machinery already issues <release>-grafana-tls there — its ca.crt key is the internal CA, so the default below works as-is once certificates.enabled is on.

Prefer this over jsonData.tlsSkipVerify. Skipping verification leaves the connection encrypted and unauthenticated, which on the read path means Grafana will happily talk to anything that answers on that address — and the failure mode of getting it wrong is a dashboard that renders, from the wrong source.

Takes precedence over caSecret, and needs no operator involvement at all, which makes it the right choice when the CA is already in hand at render time. It does not track a rotation: the PEM here is a copy, and re-issuing the CA means re-rendering. Prefer caSecret when the material lives in the cluster.

The Secret must be in the Grafana instance’s namespace, which is not necessarily this release’s under split-namespace.

key is load-bearing beyond naming the field: grafana-operator substitutes ${<key>} — the secretKeyRef key verbatim, dots and all — so the chart derives the placeholder from it. A mismatch there fails in the least visible way available: the CR applies, the operator logs nothing, secureJsonFields reports tlsCACert: true, and only at query time does Grafana say failed to parse TLS CA PEM certificate, because what it stored is the literal placeholder text.

The derived ones are materialize, materialize-sql, materialize-operator, kube-state-metrics, cadvisor, node-exporter, loki and alloy, each present when this chart runs the component and collects its metrics, and cloudwatch, cloud-monitoring and azure-monitor, present when the gateway runs that provider pull (pipeline.metrics.provider.*). List one here as well when something outside the chart provides it — your own kube-state-metrics, say. An unknown name fails the render.

Selecting a rule does not bypass its capabilities: a selected rule whose capabilities are missing is still not installed, and the render warns.

An override never changes a rule’s expression. The usual reason for one is a workload whose normal behaviour a default does not fit: a large cluster can take hours to hydrate when nothing is wrong.

rules:
  overrides:
    cluster-hydration-stuck:
      for: 6h
    cluster-cpu-high:
      labels:
        severity: notice
        team: analytics

severity must stay one of critical, warning and notice, and audience one of platform, workload and development, so the routes still match. An unknown alert name fails the render, and an override for a rule that is not installed renders with a warning.

A Loki rule reads one tenant’s logs, and the ruler does not evaluate across tenants, so the gateway writes every LogQL PrometheusRule into each tenant listed here. Under the default static tenancy every log line is in staticTenant, and the default is right.

Under byEnvironment each Materialize environment’s logs are in a tenant named for its environment id, which the chart cannot know at render time. List those tenants here, and restate the list when an environment is added. Under byNamespace the tenant set grows with every namespace, and no list stays complete.

Rules about an environment’s pods (pending, restarting, OOMKilled) scope to these. Empty uses materialize.namespaces, and if that is empty too, materialize-system.namespace.

The infrastructure rules (memory, OOMKills, restarts, file descriptors, Deployment availability, daemonset budgets) grade a workload by the tier it is in. Each entry is an RE2 fragment matched against the whole name: container names, and for the k8s-deployment-unavailable-* rules, Deployment names. That is why a tier lists both where they differ, such as ebs-plugin and ebs-csi-controller. It may not contain a quote, a backslash or whitespace.

The defaults are the Kubernetes add-ons common on EKS and GKE, plus this chart’s own collectors. Replace a tier to describe your cluster; an empty tier matches nothing. Container names shared across unrelated workloads (Karpenter’s is controller) are left out, since the match is on the name alone.

externalDependencies:
  consensus:
    - flavor: rds
      resourceId: mzmon-prod-db

The resource has to be watched by its provider pull or the exporter as well, or there is nothing to record, and the render warns. Each recorded series carries the resource as its resource label. Matching ignores case.

The three shipped presets say how much this deployment depends on Materialize. important is the default because it is the assumption that is wrong in the least damaging direction: a critical-infrastructure deployment left on it gets a notification where it wanted a page, while an evaluation deployment that pages on critical gets an operator who turns alerting off.

A class names a kind of delivery — page, high, normal, low — rather than a receiver. Each receiver declares the classes it serves, so one preset works for a deployment with a pager and one with a single chat channel. suppressed is reserved: the alert still fires and still shows in Alertmanager and Grafana, and no notification is sent.

This is a map, so one cell can be changed without restating the rest, a preset can carry a severity of its own, and a preset of your own is one more key. Class names are free-form. The render fails when the selected preset names a class no receiver serves, since an unroutable severity is otherwise discovered during the incident it should have reported.

The chart does not model receiver types, so every integration Alertmanager documents works as written in its configuration reference. Credentials are referenced through a *_file field against a mounted Secret; see alertmanager.extraSecretMounts.

Several receivers can serve one class, and every one of them is notified.

group_by: [alertname, cluster, namespace] sends one notification per condition per Materialize environment. cluster is constant within one Alertmanager, and is there for the incident tools downstream: PagerDuty’s dedup_key and Opsgenie’s alias are hashes of the group key, so without it two clusters sending the same condition open one incident, and either cluster’s resolution closes it. receiver, routes and matchers are the chart’s and are rejected here.

An alert is tested against these first, so a specific match wins and the preset remains the fallback. A route that should also reach the preset sets continue: true. Each receiver has to name one under receivers.

routes:
  extra:
    - matchers: ['component="storage"', 'severity="critical"']
      receiver: data-team

The chart ships its templates as mzmon.gotmpl whatever this says, as mzmon.* definitions a receiver can reference. On, they also replace Alertmanager’s built-in defaults, so a receiver that sets no title, text or link of its own gets them:

A template under templates that defines one of these names again still wins, because the chart’s file is loaded first. Off restores Alertmanager’s own defaults, whose links name the pod’s address.

Notifications link to the alert group and to a pre-filled silence in Grafana’s alerting pages, which read the bundled Alertmanager through the connections.datasources.alertmanager datasource. Empty takes the URL Grafana builds its own links from:

Set this when that address is not the one people reach Grafana at: an external Grafana addressed in-cluster, or a root_url written with Grafana’s %(domain)s placeholders, which the chart cannot resolve. With no URL, notifications carry no Grafana links, and the render warns once a receiver exists.

Off by default. profiles/mtls.values.yaml turns it on and assembles the settings that have to move with it, and mtls-phase2 verifies client certificates. Set these directly only alongside the same changes; the render refuses each half-applied combination.

The certificate, the client-auth policy and the CA are re-read on every connection, so a renewal or a move between NoClientCert and VerifyClientCertIfGiven applies without a restart.

RequireAndVerifyClientCert is refused while Alertmanager’s own probes or config reloader would be locked out. Both reach port 9093 from inside the pod and neither can present a client certificate: a Kubernetes httpGet probe has no field for one, and the config reloader’s client has no option for one. So VerifyClientCertIfGiven is the terminal state here, as it is for Loki’s HTTP port: certificates that are presented are verified, and a client that presents none is still served.

Leave this on. Values files are committed, diffed, pasted into support threads and rendered into Terraform plans, so a token written into one is a token published. Nearly every Alertmanager credential field has a _file variant — api_url_file, routing_key_file, credentials_file, password_file — that reads it from a mounted Secret instead.

instance (the default) selects everything pointed at the Grafana this release created, by the labels in connections.grafana.labels — the same ones the resources carry and their instanceSelector matches.

That deliberately reaches beyond this release. The dashboards ship in the materialize-monitoring-dashboards* charts, releases of their own whose GrafanaManifests carry grafana-operator’s finalizer, and nothing in those releases’ teardown runs when this one is being removed. Take the operator away without clearing them and they wedge in Terminating with no remover. It is still bounded: it cannot reach resources aimed at a Grafana this release did not create.

release is the narrow form — only what this release created. Correct when grafana-operator is not ours to remove, since it survives the uninstall and clears the rest itself, and the escape hatch if the broader sweep ever reaches something it should not.

Only instance consults mzmon.grafanaOperator.enabled; with the operator unmanaged here, both scopes fall back to the release.

GOMEMLIMIT at ~80% of the memory limit is the part worth keeping regardless of the number. Go’s GC has no knowledge of a cgroup limit, so it grows the heap toward a ceiling the kernel enforces by killing the process. Telling the runtime about the ceiling turns an OOM-kill into GC pressure — the failure mode becomes “slower”, not “dead”. Keep it in step with the memory limit above.

Where the journal lives is host-specific, so both roots have to be covered. Observed:

Bottlerocket also has an empty /run/log/journal, and kind has no /var/log/journal at all — neither is a problem, but neither alone is enough either.

Deliberately not mounting /var/log/journal directly: journald’s default Storage=auto switches to persistent storage as soon as that directory exists, so a hostPath mount that creates it would change how the host journals. varlog already exposes it wherever it legitimately exists.

/etc/machine-id is required, and its absence is silent. systemd stores the journal under <path>/<machine-id>/, and the container has no machine-id of its own — the Alloy image ships none. Without the host’s, loki.source.journal starts, reports healthy, and reads zero lines.

Measured on a Bottlerocket node, controlling for the restart that a mount change forces: with the mount, loki_source_journal_target_lines_total went 0 -> 1244; removed again, back to 0 on a freshly started pod.

A kind node happens to work without it, which makes this a bad thing to test on kind alone — the two differ in which directory holds the journal (/run/log/journal there, /var/log/journal on Bottlerocket), and only the latter needs the ID to resolve. Do not remove this because kind stays green.

That holds only while nothing reads the material, which is why the mTLS profiles flip it to optional: false. Once a config names a path inside this mount, an empty mount is not a pod serving plaintext: Alloy fails its initial load and enters CrashLoopBackOff, backing off to five minutes, on an error naming a file rather than the Secret behind it. Required, the same race is a container the kubelet holds in ContainerCreating with a FailedMount event, started on the next volume re-sync. Do not set this to false here — with certificates off there is no Secret to wait for and the pod would never start.

The kubelet refreshes the contents in place on renewal, which is why the tls.*File carriers are preferred over the inline PEMs.

Sized for the logs-only agent. This needed ~750Mi while an in-process cAdvisor lived here; that moved to the gateway, so the envelope is back to what the log path actually costs, with headroom over the ~200Mi it ran at before.

The CPU limit stays above the request: the agent is bursty, and a limit equal to the request turns that into CFS throttling on the busiest nodes.

Ingress is one port from one peer. 12345 is Alloy’s HTTP server — it serves /metrics and the live debugging UI, which shows the running pipeline and samples of the data moving through it — and the only pod that needs to reach it is the gateway performing the serviceMonitor scrape.

Egress is not restricted, and the policyTypes list says so rather than pairing an Egress type with an allow-everything rule. Two of the agent’s three destinations are unaddressable from here: the API server backs discovery.kubernetes, and on a managed control plane it is not in the pod network at all. The third, the gateway, is in-namespace and would be easy — but a policy that allows only the parts we happen to know is a policy that breaks the first time someone adds a scrape to the agent pipeline, and does it silently. Narrow it deliberately, with egress and policyTypes, once you know what your agent actually collects.

The default from selects by pod label within the release namespace. Running the gateway in a different namespace (see profiles/split-namespace.values.yaml) requires replacing this list with one that adds a namespaceSelector.

experimental, because every gateway scrape sets honor_metadata, which Alloy refuses to build below that level. It is what types the metrics the OpenTelemetry destinations receive. The Google Cloud exporter’s otelcol.auth.google is public preview and needs no more than this.

A lower level renders an error rather than a gateway that crashloops: alloy validate, and so the pre-validate job, does not check a component’s arguments against it.

That holds only while nothing reads the material, which is why the mTLS profiles flip it to optional: false. Once a config names a path inside this mount, an empty mount is not a pod serving plaintext: Alloy fails its initial load and enters CrashLoopBackOff, backing off to five minutes, on an error naming a file rather than the Secret behind it. Required, the same race is a container the kubelet holds in ContainerCreating with a FailedMount event, started on the next volume re-sync. Do not set this to false here — with certificates off there is no Secret to wait for and the pod would never start.

The kubelet refreshes the contents in place on renewal, which is why the tls.*File carriers are preferred over the inline PEMs.

Memory is the gateway’s binding constraint. Its live heap is about 200MiB of fixed cost plus about 2.6KiB for every series the pod holds. Each series is held three times: by the scrape cache, the OTLP-to-Prometheus bridge and the remote-write queue. The gateways split the scrape targets, so at minReplicas each pod holds its share of every series in the cluster. It also holds, until its write-ahead log next checkpoints, the series of targets a recent resharding moved away from it.

Keep that live heap under about half of GOMEMLIMIT. Past that the GC runs constantly: measured with a 600MiB GOMEMLIMIT, a pod with 300MiB live collected about 3 times a minute on 0.07 cores, and one with 450MiB live collected 30 or more times a minute on 0.3 cores. That CPU is what the autoscaler scales on, so a gateway short of memory also scales out and back for no other reason.

The default fits a medium install. It was measured on 9 nodes and ten Materialize replicas, a quarter of a million series in all. At 2 replicas a pod there holds 120k to 160k series, about 500 to 600MiB live, against a 1.6GiB GOMEMLIMIT. Raise this as series grow. GOMEMLIMIT and the memory limiter’s thresholds follow it (see extraEnv).

Every scale event reshards the scrape targets. A scale-down hands each remaining pod a larger share, and every pod keeps the series of the targets it gave up until its write-ahead log next checkpoints, hours later. A gateway that scales down and back up within the hour pays for both shares at once. The scale-up path stays fast, because it relieves load rather than concentrating it.

Two ingress rules, because the gateway’s ports fall into two groups with very different audiences.

The ingest ports — 3100 (Loki push), 4317/4318 (OTLP), 9090 (Prometheus remote write) — are open to every pod in the cluster. That is deliberate and it is not laziness: these are the endpoints the stack exists to offer. The agent pushes logs to 3100, and anything in the cluster with telemetry to send — a Materialize workload emitting OTLP, an application remote-writing metrics — is a legitimate client this chart cannot enumerate. namespaceSelector: {} still means in-cluster: a sender outside the pod network is denied, and the gateway is not meant to be internet-facing.

12345 is not an ingest port and is closed to everything but the gateway itself. It carries the clustering gossip between gateway replicas (alloy.clustering.enabled) and serves /metrics and the debugging UI, which exposes the pipeline and samples of the telemetry passing through it. The gateway scrapes its own serviceMonitor, so its own pods are the only peer.

Egress is not restricted, and cannot usefully be. The gateway scrapes the kubelet cAdvisor endpoint on every node — port 10250 on a node IP, which no podSelector matches — and it discovers ServiceMonitor and PodMonitor targets cluster-wide, so its destination set is every pod and every port that anyone in the cluster has ever pointed a monitor at. On top of that it is the stack’s egress point for external backends: Amazon Managed Prometheus, Honeycomb, a Datadog or OTLP destination under connections. Enumerating that is not a policy, it is a copy of the cluster’s inventory that goes stale.

So Renovate bumps Loki on its own cadence, with Loki’s own release notes, instead of only when a chart release happens to carry a new appVersion. Every Loki component runs this image unless its own image overrides it. The canary is pinned separately in lokiCanary.image, and Renovate groups the two so that they move together.

In the Loki chart, global.imageRegistry outranks the registry of every image, this one included. The profiles under profiles/registry/ repoint registry and repository and keep this tag.

The ruler deployment has been on since this chart had one. What it has never had is somewhere to send an alert: without alertmanager_url it evaluates its rules correctly and drops every alert on the floor, which is indistinguishable from nothing being wrong. That is what this block fixes.

Rule storage is configured and is not here. With loki.storage.use_thanos_objstore on, the subchart renders a top-level ruler_storage block pointing at loki.storage.bucketNames.ruler. The chart’s log-derived rules are written into it through the ruler API, by the alloy-gateway’s loki.rules.kubernetes, from PrometheusRule resources labelled mzmon.materialize.cloud/flavor: logql. A rule store that refuses writes, such as local, leaves them out.

Everything here is tpl-evaluated by the subchart, so .Release.* resolves. It does not see the umbrella’s values, which is why these addresses are spelled out rather than built from the helpers the Thanos side uses — and why split-namespace overrides both of them.

Without it every replica loads every group: each alert is sent once per replica, and each recording-rule sample is written once per replica.

The replicas divide the groups through a hash ring. Loki puts every ring on memberlist when memberlist.join_members is set, which the subchart does, so the ruler’s ring rides the same gossip cluster as the rest of Loki and needs no configuration of its own.

This is what the ruler’s PVC exists for: the WAL buffers derived samples when the metric store is unreachable, which is exactly the run-up to an incident. Same destination as the Thanos ruler — the alloy-gateway’s prometheus.receive_http listener — so both rulers’ results reach the destination fan-out rather than only the bundled metric store.

Grafana’s alerting UI lists a datasource’s rules from the datasource’s own URL. In distributed mode the query frontend answers queries and the ruler answers for rules, so a datasource pointed at either one sees half of Loki. The gateway routes by path and gives Grafana one address for both.

The routing is this chart’s own (nginxConfig.file), and it is narrower than the subchart’s:

Every request resolves its backend’s name through resolver, so a Service recreated under the same name is followed without a restart. Turning the gateway off needs connections.datasources.loki.url pointed back at the query frontend, and Grafana then shows no Loki rules.

Read by this chart’s nginx.conf; the subchart does not know the key. profiles/mtls.values.yaml sets these, alongside ssl and schema.

Requiring a client certificate is not offered. The kubelet’s readiness probe dials the same listener and cannot present one, the same limit that holds Loki’s own HTTP port at phase 2.

Plaintext exporters are excluded from it. The subchart renders a single ServiceMonitor covering everything it labels, with one scheme shared by every target. Four of those targets never speak TLS whatever Loki is configured to do — the canary’s own /metrics server, the two memcached exporters, and the gateway’s access-log exporter — so under profiles/mtls, which sets scheme: https here, all four fail the scrape and their series vanish. For the canary that means the end-to-end write→read check goes quiet rather than red, which is the worst way for a canary to fail.

Each of the four therefore carries prometheus.io/service-monitor: "false", which the subchart’s selector excludes, plus a monitoring.materialize.cloud/scrape-scheme: plaintext opt-in that this chart’s own monitor selects on (templates/scrapers/monitor-loki-plaintext.yaml). The split is unconditional so the two modes share one code path.

So Renovate bumps Thanos on its own cadence, with Thanos’s own release notes, instead of only when a chart release happens to carry a new appVersion. The subchart has no per-component image, so this one tag moves Query, Store Gateway, Compactor, Query Frontend and Ruler together. The profiles under profiles/registry/ repoint registry and repository and keep this tag.

Upstream ships these off. One switch turns them on for all components, and the per-component pdb.enabled cannot opt back out — the subchart templates test or <component>.pdb.enabled global.pdb.enabled.

maxUnavailable rather than minAvailable, deliberately:

• It scales with replica count instead of pinning an absolute floor.
• On the single-replica Compactor, minAvailable: 1 would block every voluntary eviction and hang node drains indefinitely. maxUnavailable: 1 permits the eviction, so the singleton is a harmless no-op rather than a drain deadlock.
• On Receive it must stay within what write quorum tolerates (replicationFactor - ((replicationFactor / 2) + 1)), which is 1 at the replication factor of 3 set below. A validator enforces this.

This matches the Loki convention (ingester maxUnavailable: 1).

One switch, no per-component knobs and no selectors — the subchart renders a fixed policy per component and exposes nothing to tune. What each one says is: ingress on that component’s own service ports, from anywhere; egress unrestricted.

So this is the weakest policy in the stack, and worth being clear about what it does and does not buy. It does not restrict who may reach Query, Receive or Store Gateway — any pod in the cluster still can. It does close every port that is not a declared service port, which is the surface an unrelated workload would otherwise find on these pods.

It is on because it costs nothing to be right about. The rules cannot break a working install: Receive’s grpc, http and remote-write ports are all in its own ingress list, so hashring replication between Receive pods and the gateway’s writes both keep working, and unrestricted egress leaves the object store reachable. Narrowing the ingress sources needs an upstream change, or your own policy layered on top.

The subchart ships this empty. Its container context is already read-only, non-root and drops every capability; this adds RuntimeDefault seccomp.

It reaches only the components whose own podSecurityContext is empty. The subchart picks the first non-empty of component, parent and global without merging them, and Compactor, Store Gateway, Receive and Ruler each ship an fsGroup of their own. Those four carry the same two keys in their own blocks below, which merge over the subchart’s fsGroup.

pod-template-hash, not controller-revision-hash: Query is a Deployment, and that is the label its ReplicaSets carry. Using the StatefulSet label here would match nothing and quietly disable the same-revision guard.

This is a recent-query cache, not a durability window: blocks ship to object storage every 2h and the Store Gateway serves everything older, so all that shortens here is how far back a query is answered from local TSDB rather than from the bucket. 6h covers the common dashboard ranges and leaves 4h of margin after a block closes, uploads, and the Store Gateway syncs it.

Raising it back is a real cost rather than just a number. At the medium envelope (1.5M series) 24h is ~7.3Gi per pod, and a GKE node with a 47Gi boot disk offers only ~18.8Gi of allocatable ephemeral storage — one pod would claim 40% of the node’s budget, and Loki’s ingesters draw on the same pool.

1. It rewrites the requests below, so a sizing profile would not survive contact with it.
2. updateMode: Auto evicts pods to apply new requests, and Receive holds up to tsdb.retention (6h) of blocks on an emptyDir whose only redundancy is the replication factor — an eviction discards that pod’s copy of the window.
3. The subchart’s VPA template is CRD-gated, so leaving it on makes the stack behave differently on clusters with the VPA CRD installed than on those without — silently, in both directions.

If you want its recommendations without its actions, set updateMode: "Off" rather than re-enabling Auto.

No memory limit, deliberately, and for a sharper reason than elsewhere in this chart: an OOM-kill drops the un-uploaded block window, and unlike Loki’s ingesters that window has nothing protecting it but the replication factor. Alert on usage instead.

ephemeral-storage is declared because the TSDB lives on an emptyDir (see persistence). The request is what the scheduler places against, so a node without room refuses the pod instead of filling up silently. The limit carries deliberate headroom over the request: the kubelet’s response to exceeding it is eviction, and evicting Receive destroys exactly the un-uploaded window this sizing exists to protect.

Size these against a node’s allocatable ephemeral storage, not its disk size. The two are nowhere near each other: GKE reserves most of the boot disk for the image filesystem, so a 47Gi disk offers roughly 18.8Gi allocatable. A request above that cannot schedule anywhere, and the cluster-autoscaler will not rescue it — no node of that shape would fit either, so the pod sits Pending indefinitely on Insufficient ephemeral-storage. Check before raising these:

kubectl get nodes -o custom-columns=‘NODE:.metadata.name,EPH:.status.allocatable.ephemeral-storage’

4Gi holds 6h of the medium envelope (~1.8Gi) with room for the WAL and compaction churn, and leaves the rest of the node for everything sharing that pool — including Loki’s ingesters, which are also emptyDir-backed.

Hard (DoNotSchedule) on zones, deliberately. A pod that cannot satisfy it goes Pending, which is the signal Karpenter or the cluster-autoscaler uses to add a node in the deficient zone — a soft rule cannot summon capacity that way, it just quietly places the pod wrongly. Host spread stays soft so pods still schedule when nodes are momentarily scarce; two pods sharing a node is a smaller problem than a pod not running.

This became viable when Receive stopped using a PVC (see persistence below). A zonal volume cannot be attached from another zone, so a hard zone rule and an AZ-pinned pod pull in opposite directions: the rule says “go to the empty zone”, the volume says “you may only run in zone A”, and the pod goes Pending forever rather than for as long as it takes to add a node. On emptyDir there is nothing holding it back.

The selector names both app.kubernetes.io/name and app.kubernetes.io/component, and the first of those is load-bearing: component names are not unique across backends. Loki also ships a compactor, a query-frontend and a ruler, so a component-only selector would make Thanos count Loki’s pods when computing its own skew — silently, and only for the components whose names happen to collide.

The release name is deliberately absent: the subchart renders this through toYaml rather than tpl, so a {{ include ... }} would land in the manifest literally. Spread is namespace-scoped and this chart assumes one instance of each backend per namespace, so name plus component is sufficient. See the Namespace layout section.

Note this cannot move to thanos.global.topologySpreadConstraints: each constraint carries its own labelSelector, and a global one would make every Thanos component count Receive’s pods when computing its own skew.

Durability comes from --receive.replication-factor=3, not from disk. Blocks ship to object storage every 2h, so the window that exists only locally is at most 2h — and every replica uploads its own copy under a distinct receive_replica external label, which the Compactor deduplicates (see compactor.extraArgs). A pod that comes back with an empty volume has lost its copy of that window, and the query path still answers from the surviving replicas.

A PVC makes an AZ failure worse, not merely less flexible: an EBS volume cannot be attached from another zone, so a pod whose zone is gone stays Pending until the zone returns rather than rescheduling into a healthy one. On the write path that turns a recoverable event into an outage that waits on the cloud provider — with RF 3 write quorum is 2, so two pods stuck Pending block writes outright.

The trade accepted in exchange: Thanos Receive has no peer hand-off, so unlike Loki it will not backfill an emptied pod from its neighbours. That window stays at two copies instead of three until the next block ships.

Autoscaling is available upstream but left off deliberately. Store Gateway is a PVC-backed StatefulSet that syncs the bucket’s block index on startup, so scale-up is slow to become useful (it serves nothing until the index is warm) and CPU-triggered scaling reacts long after the load that triggered it. Scale-down also leaves orphaned PVCs behind, since StatefulSet volumes are not reclaimed. Size it deliberately instead.

Kept on at every size deliberately, rather than flipping to ephemeral for small installs. A volume that appears and disappears with the profile is a surprise during an incident, and the AZ-pinning argument that drives Receive to emptyDir is much weaker here: Store Gateway is a read-only replica set with no quorum, so a pod stuck Pending in a dead zone costs read capacity rather than blocking writes.

The kind profile is the one exception, where CI would rather not wait on volume provisioning for a cache.

1. Nothing depends on quorum. Store Gateway is a read-only replica set over object storage, so an unbalanced placement costs read capacity during a zone loss rather than correctness. Receive’s hard rule protects write quorum; there is no equivalent to protect here.
2. It is the one Thanos component that still keeps a PVC, and a hard zone rule against a zonal volume can deadlock rather than merely wait. With volumeBindingMode: WaitForFirstConsumer — the default for the zonal CSI classes and what you want — the volume follows the pod and the two agree. With Immediate, the PVC’s zone is chosen before scheduling and a hard rule pointing elsewhere leaves the pod Pending forever. Soft degrades instead of deadlocking on a StorageClass we do not control.

The subchart ships this empty, so unlike receive.extraArgs there is nothing to lose by overriding it — but it is still a list, so restate both entries when adding a third.

Raw retention has a floor set by the downsampling thresholds: Thanos produces 5m downsamples only from blocks spanning 40h or more, and 1h downsamples only from blocks spanning 10d or more. Below those the tier is never created at all and long-range queries silently fall back to reading raw blocks — slower and more expensive, which is the opposite of the intent. 30d clears both comfortably.

--compact.enable-vertical-compaction lets the Compactor merge overlapping blocks. Without it, any overlap halts compaction, retention and downsampling until someone repairs the bucket by hand. Receive produces overlaps on every restart: the block it flushes on shutdown ends at its last sample, and after restart it accepts replicated and retried samples a few seconds older than that. Overlaps of 5–40s were observed after ordinary rollouts.

--deduplication.replica-label=receive_replica drops the per-pod label from block grouping, so the replication factor’s copies compact into one block. The bucket then holds one copy of each series rather than RF. The default merge function deduplicates only identical samples, which is what Receive replication writes. Thanos Query still deduplicates the recent window on receive_replica, so reads are unchanged. A deduplicated group holds every series, so above replicaCount == replicationFactor a compaction group is larger than one replica’s was.

Both flags are irreversible for the blocks they touch. The replica label alone also turns on vertical compaction (Thanos cmd/thanos/compact.go); the explicit flag keeps overlaps from halting compaction if an override drops the replica label.

Downsampling is the memory-heavy phase, and --compact.concurrency multiplies it; leave that at 1 unless compaction is provably behind.

No ephemeral-storage here: the Compactor keeps a PersistentVolume, so its scratch does not draw on the node’s budget. See persistence for why.

Compaction works on whole block groups, not on the recent window, so the requirement scales with days of data rather than hours. At the medium envelope a 2d group is roughly six 8h blocks at ~2.4Gi each, and the Compactor needs the sources and the output at once — on the order of 30Gi. A GKE node with a 47Gi boot disk offers about 18.8Gi of allocatable ephemeral storage, so that never fits: the pod stays Pending on Insufficient ephemeral-storage, and the cluster-autoscaler cannot help because no node of that shape would fit either.

This is the reverse of the call made for Receive, and deliberately so. Receive holds hours of small, replicated data and is the better trade on emptyDir; the Compactor holds days of it and is not. The AZ-pinning cost is real — a zone outage stops compaction, and while compaction is stopped retention is not enforced and the bucket grows — but “compaction pauses during a zone outage” is a far better failure than “compaction never runs because the pod cannot be scheduled”.

Note that enabling this is not sufficient on its own: point connections.datasources.thanos.url at the query-frontend Service too, or reads keep going straight to Query and the cache is never consulted. A validator warns when the two disagree.

This is the switch that makes PromQL alerting exist. The chart installs the Prometheus Operator CRDs but runs no Prometheus and no operator, and Alloy — the one component that consumes ServiceMonitor and PodMonitor — is a collector with no rule evaluator. So the single consumer of a PrometheusRule in this stack is the Ruler’s import sidecar, and until the Ruler is on, a PrometheusRule renders, applies, passes every check, and does nothing.

Evaluation is a PromQL query over the network to Thanos Query. There is no local TSDB to fall back on, which means an outage in the query path stops alert evaluation, and stopped evaluation looks exactly like nothing being wrong. Size Query with that in mind.

Query, not Query Frontend, deliberately. The frontend splits and caches, which is right for a dashboard and wrong for an evaluator: a cached range served to a rule is an alert firing — or failing to fire — on data up to a cache TTL stale, with nothing in either component saying so. Grafana’s datasource follows the frontend when it is enabled; this does not.

The name is bare rather than fully qualified because Ruler and Query are components of one subchart and always land in one namespace, so it stays correct under split-namespace.

tpl-evaluated by the subchart, so .Release.* resolves. alertmanager-headless follows alertmanager.fullnameOverride, and the render warns when the two disagree. split-namespace overrides this because it moves Alertmanager out of the release namespace.

dns+ against the headless Service, so the Ruler resolves one target per ready replica and sends every alert to each of them. Alertmanager gossips silences and the notification log, not alerts, and a replica that never received an alert cannot notify for it when its peer is down. The Ruler re-resolves every --alertmanagers.sd-dns-interval (30s).

This empties the subchart’s example-alerts.yaml, which ships an ExampleAlwaysFiring rule built on vector(1). Left in place it notifies on every evaluation, forever, through whatever receiver an operator has configured.

It is emptied rather than removed because the subchart’s JSON Schema lists example-alerts.yaml as a required property, so neither rules: {} (which would not clear a subchart default anyway) nor example-alerts.yaml: null renders — the second fails schema validation before any template runs. A rule file declaring no groups is the only spelling that both satisfies the schema and evaluates nothing, until the schema is fixed upstream.

The render fails if the example rule comes back.

The chart’s own rules do not arrive here. They arrive as PrometheusRule resources, through the import sidecar below.

A kubectl sidecar lists PrometheusRule resources every 60s, writes each one’s .spec into the Ruler’s rule directory, and POSTs /-/reload when the set changes. No Prometheus Operator controller is involved.

labelSelector imports every PrometheusRule in the cluster except the LogQL ones, including any belonging to a co-resident kube-prometheus-stack. That is kept deliberately: it is what makes a customer’s own PrometheusRule work with no chart configuration. Set a narrower selector here if this cluster runs another rule owner whose alerts should not reach this Alertmanager.

The exclusion is load-bearing. The chart’s log-derived rules are PrometheusRule resources too, labelled mzmon.materialize.cloud/flavor: logql, and the Thanos ruler reads every file it imports as PromQL; one LogQL file fails the reload, and the ruler keeps evaluating whatever it loaded before. The subchart joins this map into key=value pairs, so a key ending in ! renders as key!=value, which also keeps a PrometheusRule carrying no flavor label. A selector replacing this one keeps the exclusion, or selects mzmon.materialize.cloud/flavor: promql; the render fails otherwise.

The image is pinned rather than left on the upstream latest, so a default install does not track a floating tag. It is the only Docker Hub image the Thanos subchart pulls, which is why the registry profiles each carry a line for it.

It cannot be the distroless registry.k8s.io/kubectl this chart uses for the cleanup hook, tempting as sharing one image is. The sidecar’s entrypoint is /bin/sh /scripts/import.sh, and the script shells out to curl to POST /-/reload. alpine/kubectl is Alpine plus curl plus kubectl; the distroless build has neither a shell nor curl, so it crash-loops immediately. The same applies to any hardened kubectl image — see the note in profiles/registry/.

The minor tracks the cleanup hook’s kubectl, and the same version-skew advice applies: keep it within one minor of your API server.

Three reasons stateless is the right mode here, in ascending order of how much they matter. It removes a stateful workload. It puts rule results on the same path as every other series in this stack, rather than making the Ruler a second writer into object storage with its own compaction interaction. And it is what makes alert state forwardable: a Ruler shipping blocks puts ALERTS in Thanos and out of reach of every gateway destination except Thanos.

With enabled, the subchart passes --remote-write.config-file and drops --objstore.config-file, so the Ruler runs no block shipper. The configuration targets the alloy-gateway’s metrics listener, and the chart renders it as a Secret (templates/thanos-ruler-remote-write.yaml) rather than through config, because the gateway’s address follows umbrella values that the subchart’s tpl cannot read.

secretName is one of two Secrets the chart renders, -tls for when pipeline.metrics.gateway.server.tls is on, which profiles/mtls.values.yaml selects and the render enforces. The Ruler reads the file once at startup, so the names carry the mode and a revision: switching Secrets changes the pod template, which is what rolls it.

Writing elsewhere means naming another Secret here, or setting secretName: "" with createSecret: true and the configuration in config. The render’s gateway checks then no longer apply.

--label=cluster is load-bearing. It is an external label, which the Ruler adds to every alert it sends that does not already carry cluster, and to every sample it writes. $(CLUSTER_NAME) is expanded by Kubernetes from extraEnv below, which reads the ruler-env ConfigMap the chart renders from clusterName. It has to survive an override of this list; the render warns when it goes missing.

mzmon-thanos-tls is the certificate the chart issues for Thanos when certificates is on. The Ruler presents it to Alertmanager when alerting.server.tls is on and to the gateway when its metrics listener serves TLS, and trusts its ca.crt. Mounted unconditionally and optionally, so the same values work before issuance, during it and after. This is a list: restate the entry when adding one.

The Loki ruler keeps a volume for the same WAL, on the argument that buffering derived samples through a metric-store outage is worth a disk. That argument applies here too and is deliberately not taken: the recording rules this Ruler evaluates are the ext:* layer, a few low-cardinality series, and the WAL holds them through a store outage on the pod’s own volume. What a PVC would add is keeping them across a pod rescheduled during that outage, which loses minutes of history rather than any alert, and a 10Gi PVC per replica is not a default worth shipping for that. Revisit if a recording rule’s history becomes something an alert or a report depends on.

Soft rather than DoNotSchedule, unlike Receive: both replicas evaluate every rule and Alertmanager deduplicates, so co-locating them costs a simultaneous loss rather than a quorum. A Ruler that cannot schedule is a Ruler that is not evaluating, which is the worse outcome.

controller-revision-hash because this is a StatefulSet; Deployments in this file use pod-template-hash.

Unlike the other pins, this one is not meant to move on its own. The operator’s CRDs are vendored from the subchart (make grafana-operator-crds), and upstream releases the chart and the image under one version. Renovate groups this tag with the subchart in Chart.yaml so the two land in one PR. The pin is what lets Renovate see the image, and lets the registry profiles repoint registry and repository while keeping the tag.

A hardened rebuild is a drop-in swap: point registry/repository at it and keep the tag. Two published options track upstream Grafana versions and keep its entrypoint and layout, so a values change is the whole change:

• Chainguard Images — https://images.chainguard.dev/directory/image/grafana/overview
• Docker Hardened Images — https://docs.docker.com/dhi/ (subscription; images land in your own org namespace, so registry is yours)

profiles/registry/ does this for the whole stack rather than for Grafana alone, and carries the pull-secret wiring that goes with it.

Both ship no shell and no package manager, which is the point — and which also means plugins cannot work, since that installs at container start. Bake plugins into the image instead. See the note on plugins below.

Bitnami also publishes a hardened Grafana, but its images are built for Bitnami’s own charts — Bitnami entrypoints, /opt/bitnami paths, UID 1001 — against an upstream subchart here, so it is a port rather than a swap. No profile ships for it.

targetCPU is a percentage of the CPU request, so resources.requests.cpu has to be set or the HPA has no denominator and never scales. A render-time check warns when it is missing.

A read-only root filesystem needs two things, and Grafana breaks quietly without either:

Everything else Grafana writes lands on the storage and search volumes the subchart mounts at /var/lib/grafana and /var/lib/grafana-search. The one exception is GF_AWS_PROFILES: the image’s entrypoint writes those profiles to /usr/share/grafana/.aws, which then needs a mount of its own through grafana.extraEmptyDirMounts.

A second mount at /tmp, or a second volume named tmp, makes the API server reject the Deployment. Neither grafana.extraEmptyDirMounts nor grafana.extraVolumeMounts may add one.

Validators fail the render on the plugin setting, a missing .aws mount and a duplicate /tmp, and warn when this is turned off.

Prefer PostgreSQL (grafana-postgres) wherever a database is available: it is the only option that lifts both constraints.

A LoadBalancer Service with no loadBalancerSourceRanges is an error at render time unless connections.grafana.allowPublicAccess is set.

Set grafana.ini.server.root_url to the same URL — Grafana builds share links, alert notification links, and OAuth redirect URIs from it, and all three break silently when it disagrees with the Ingress host.

Secrets must NOT be inlined — this block renders into a ConfigMap, so a value here lands in plaintext in the release manifest, in helm get values, and in whatever Git repo holds your values file. Use Grafana’s own expansion instead: $__file{/path} for a mounted Secret (see extraSecretMounts) or $__env{VAR} for one injected with envValueFrom. The subchart’s assertNoLeakedSecrets check fails the render if you forget.

OIDC single sign-on, as an example — this is the shape, not a default:

grafana:
  grafana.ini:
    server:
      root_url: https://grafana.example.com
    auth:
      oauth_auto_login: true
    auth.generic_oauth:
      enabled: true
      name: SSO
      allow_sign_up: true
      client_id: <client-id>
      client_secret: $__file{/etc/secrets/grafana-oidc/client-secret}
      scopes: openid profile email groups
      auth_url: https://idp.example.com/oauth2/v1/authorize
      token_url: https://idp.example.com/oauth2/v1/token
      api_url: https://idp.example.com/oauth2/v1/userinfo
      allowed_domains: example.com
      # Map an IdP group claim onto a Grafana role, so membership is the
      # provisioning mechanism and nobody is added by hand.
      role_attribute_path: contains(groups[*], 'sre') && 'Admin' || 'Viewer'
  extraSecretMounts:
    - name: grafana-oidc
      secretName: grafana-oidc
      mountPath: /etc/secrets/grafana-oidc
      readOnly: true

Every provider, the role-mapping rules, and the break-glass path are in Dashboards > Grafana > Authentication.

The subchart’s policy is rigid: it emits exactly one ingress rule, on service.targetPort, and either allows every source or a fixed set of them. There is no way to add a second port through values. That constrains what this can say, so it is worth being precise about what it does say.

Ingress: 3000 from anywhere, everything else closed. allowExternal stays true because Grafana is the one component in this stack a human is meant to reach, and the ways they reach it are all things a podSelector cannot see — an ingress-controller pod in another namespace, a cloud load balancer’s source IPs, kubectl port-forward, which arrives from the API server. Turning it off is worthwhile once you know your path in; explicitNamespacesSelector takes the ingress controller’s namespace and explicitIpBlocks takes a load balancer’s health-check and client ranges. profiles/grafana-ingress.values.yaml is where that belongs.

What it closes is the rest of the pod: 6060 (pprof) and 9094 (the unified-alerting gossip port) stop accepting connections. pprof should not be reachable in production, so that one is the point. 9094 is not — it matters as soon as you run multiple replicas and set grafana.ini.unified_alerting.ha_peers, which needs the pods to reach each other on it, and this template cannot express a second port.

You do not have to turn this off to get HA alerting. The chart renders the missing rule itself, as networkPolicies.grafanaGossip, and it appears exactly when ha_peers is set. Policies are additive, so the two sit side by side. Turning that off while HA alerting is configured is what the render warns about; turning this one off takes the supplement with it, since a lone gossip policy would isolate the pods and leave only 9094 open.

Egress is left unrestricted (egress.enabled: false renders a policy with no Egress policy type at all, rather than an empty allow). Grafana dials the things it is configured to dial: Loki and Thanos in-namespace, a PostgreSQL state database wherever it lives, an OIDC provider on the internet, and grafana.com on every start, for the plugins it preinstalls (see grafana.ini.plugins) and any listed in plugins. The first two are knowable here and the rest are not, and a partial allowlist would fail as a dashboard that renders empty with no error — the worst failure mode this stack has.

For anything load-bearing, bake plugins into the image instead and point image at it. That is also the only option on a hardened base image, which ships no shell for grafana cli to run in. Pin an exact plugin version either way (name@version); an unpinned plugin silently changes underneath a pinned Grafana.

Matches Loki, Thanos, Grafana and Alloy, whose names are pinned the same way. Everything that addresses Alertmanager — both rulers, the Grafana datasource, the certificate SANs — can then name it literally, and the subchart’s rule of collapsing <release>-alertmanager to <release> when the release name contains alertmanager no longer matters. Changing it renames the StatefulSet, whose volumes do not follow; see Upgrading.

So Renovate bumps Alertmanager on its own cadence, with Alertmanager’s own release notes, instead of only when a chart release happens to carry a new appVersion. Grafana is pinned the same way. This chart’s repository carries the registry host; there is no separate registry key. The profiles under profiles/registry/ repoint repository and keep this tag.

A single notifier is lost to an ordinary node drain, holds the only copy of every silence, and cannot report its own absence. Two replicas gossip silences and the notification log to each other and deduplicate, so each notification goes out once and either replica can send it.

Two rather than three. Gossip tolerates a partition by notifying from both sides rather than by electing a leader, so a third replica adds little availability, and it adds a third copy of every notification whenever gossip is broken.

Above one, the subchart passes --cluster.peer for every ordinal and opens the mesh ports on the headless Service. Both rulers address that headless Service, not the load-balanced one, so that every replica receives every alert: gossip replicates silences and the notification log, not alerts. See thanos.ruler.alertmanagers and loki.loki.rulerConfig.

The subchart renders config into a ConfigMap with toYaml and no tpl, so nothing in it can be computed. The severity presets, the receiver classes and the render-time checks all have to be, so the chart renders the configuration itself from alerting, as the alertmanager-config Secret, and mounts it below. Turning this back on fails the render: the subchart would mount its ConfigMap over /etc/alertmanager, the directory that holds the chart’s.

The second volume puts an amtool configuration at /etc/amtool/config.yml, where amtool looks by default, rendered from the same Secret. It carries the URL and, when alerting.server.tls is on, the CA that issued certFile (the ca.crt beside it), so kubectl exec ... -- amtool silence query works unchanged in every TLS phase. It presents a client certificate only when clientAuth requires one. Mounted as a directory rather than with subPath, so it follows changes to the Secret.

Receiver credentials are referenced by path, never inlined; see alerting.assertNoInlineCredentials. One Secret is mounted by default: create alertmanager-receivers in the Alertmanager namespace, and reference each of its keys as /etc/alertmanager/secrets/alertmanager-receivers/<key>.

It is optional, so the pods start before it exists. Alertmanager reads a *_file credential each time it sends, so creating or rotating the Secret needs no restart. The kubelet refreshes the mounted copy within about a minute.

To mount further Secrets, add entries under /etc/alertmanager/secrets/. This is a list, and Helm replaces lists, so restate the default entries when adding one. The render fails when a receiver names a *_file path that no mount covers.

The second entry is Alertmanager’s own certificate, which the chart issues as mzmon-alertmanager-tls when certificates is on. Mounted unconditionally and optionally, so the same values work before issuance, during it and after, and turning alerting.server.tls on changes no volume.

The subchart rolls its pods on a checksum of the ConfigMap it renders, and it no longer renders one, so without this sidecar a routing change would reach the running pods only at their next restart. The sidecar watches the mounted directory and POSTs /-/reload. A configuration Alertmanager rejects leaves the previous one running, and alertmanager_config_last_reload_successful drops to 0.

The image is pinned here, like Alertmanager’s, so Renovate tracks it directly. Every profile under profiles/registry/ repoints it.

A read-only root filesystem is safe: Alertmanager writes only under --storage.path, which is the volume, and reads its configuration and credentials from mounts. The subchart ships readOnlyRootFilesystem commented out, and a validator warns when it is turned off.

Amazon SNS is the one Alertmanager integration that authenticates with the pod’s own cloud identity; everything else, including email through SES or Azure Communication Services, takes a credential from a Secret. On EKS, eks.amazonaws.com/role-arn names a role with sns:Publish on the topic, trusting system:serviceaccount:<namespace>:alertmanager. EKS Pod Identity needs no annotation. See Alert Channels.

A starting replica joins the mesh and pulls its peer’s silences and notification log before it serves, so a Ready replica already holds the replicated state. A StatefulSet rollout replaces one replica at a time and waits on this, which is what keeps a rolling update from losing silences.

Alertmanager holds active alerts, silences and the notification log in memory, and a quiet install idles near 15Mi. Two replicas carry the alert volume a Materialize deployment produces, so there is no sizing profile. No CPU limit, following the rest of the chart. The memory limit is the one setting that can take both replicas down together during an alert storm, so it is set well above the request; raise it for very large alert or silence populations rather than lowering it.

Two replicas in one zone are one notifier as far as a zone outage is concerned, and a zone outage is the failure a second replica exists to survive. So the zone rule is DoNotSchedule, like Thanos Receive’s. A replica that cannot satisfy it goes Pending, which is the signal the cluster autoscaler reads to add a node in the deficient zone.

Each replica’s volume is zonal, and that is compatible with a hard rule only under volumeBindingMode: WaitForFirstConsumer, the default for every managed-cloud CSI class. The volume is then created where the scheduler placed the pod, so each replica lands in its own zone and stays there. Under Immediate binding both volumes can be provisioned in one zone before scheduling, and the second replica stays Pending while the first keeps notifying.

Two deliberate differences from Receive:

• No minDomains. A single-zone cluster schedules both replicas, so only a cluster whose nodes carry no zone label needs no-zone-spread (or min_zones = 0 on Terraform).
• No matchLabelKeys: [controller-revision-hash]. It would let a new replica ignore an old-revision peer when it is first scheduled — which is what an upgrade from one replica to two does — and land in that peer’s zone. When the peer is then replaced, its volume pins it to the same zone, the skew is 2, and it stays Pending permanently. A StatefulSet replaces a pod before scheduling its successor, so the rollout deadlock matchLabelKeys prevents on a Deployment does not arise here.

The selector names app.kubernetes.io/name alone because the subchart renders this through toYaml, not tpl, so the release name is not available. Spread is namespace-scoped, and this chart assumes one Alertmanager per namespace.

Gossip replicates both between the replicas, so the volume is what survives losing both at once — an uninstall and reinstall, a namespace-wide restart, a cluster-wide outage. Silences are human-created and exist nowhere else. The notification log is what stops the first evaluation after such an event from re-sending every notification that already went out.

This and size are immutable after install. They render into the StatefulSet’s volumeClaimTemplates, which Kubernetes refuses to change, so an upgrade that changes either fails. Changing them means deleting the StatefulSet with --cascade=orphan first.

Delivery failures (alertmanager_notifications_failed_total), mesh membership (alertmanager_cluster_members) and reload health (alertmanager_config_last_reload_successful) are observable only with this on.

Both Services the subchart renders carry identical labels, so the monitor selects the headless one too and would scrape every replica twice. The relabeling drops the headless Service’s targets.

Alertmanager’s built-in notification templates link to its alert list (<baseURL>/#/alerts) from it. The chart’s templates (alerting.defaultTemplates) link to Grafana instead whenever Grafana’s URL is known, and fall back to the alert list and a pre-filled silence (<baseURL>/#/silences/new) only when it is not and this is set.

Do not point it at Grafana. Grafana serves neither path, so every link lands on Grafana’s home page. Set alerting.grafanaURL, or Grafana’s root_url, for links into Grafana.

Unset, Alertmanager’s own links name the pod’s address, which is unreachable from outside the cluster, and the chart’s templates leave them out. Set this when Alertmanager is exposed, behind authentication, to the address operators reach it at. A path is fine: extraArgs.web.route-prefix keeps the endpoints at /, so an ingress serving a sub-path strips the prefix before forwarding.

kube-state-metrics publishes each label as label_<key>, with every character outside [a-zA-Z0-9_] mapped to _. It publishes neither family until an entry names a label for that resource.

The pods labels arrive under the canonical names cluster_id, replica_id, replica_size and environment_id rather than as label_*. prometheus.monitor.http.metricRelabelings renames them. Only clusterd pods carry the first three. No released Materialize operator sets materialize.cloud/environment-id yet, so environment_id is absent until one does.

Cardinality. Each family has one series per object, whatever the list names, because the labels are carried on that one series. A broader list costs index size and series churn rather than series count: a label whose value changes on a running object ends one series and starts another. pods=[*] copies every label on every pod in the cluster, including the per-rollout hashes and whatever a workload owner chose to set, so name labels individually instead.

Extending the list. Helm replaces a list rather than merging it, so an override restates both entries. Leaving out nodes empties every per-pool panel with nothing else failing. The Terraform module’s kube_state_metrics_pod_labels appends to the pods entry without restating it. An ownership label for cost-center allocation is the expected addition, and arrives as label_<key>.

kube-state-metrics publishes one series of each per materializes.materialize.cloud resource:

The active generation is the one the operator last promoted. During a rollout it is the older generation, until the new one has caught up and been promoted. A rollout that never promotes leaves the older one serving, so the highest generation in the environment’s pod names doesn’t stand in for it. generation lets a query join the series to the generation in a pod’s name.

The environment’s namespace is environment_namespace. namespace and pod on these series name the kube-state-metrics pod, as on every node-level kube_* series. kube-state-metrics also adds customresource_group, customresource_kind and customresource_version. Read kube_mz_environment_info through group by (cluster, environment_id, environment_name, environment_namespace), which drops those labels.

Pinned to v1alpha1. kube-state-metrics takes one entry per resource, and a wildcard version lists every resource once for each version the CRD serves. Every Materialize operator serves v1alpha1, and where the v1 CRD is installed too, listing v1alpha1 returns the resources created as v1 as well.

kube-state-metrics watches the cluster’s CRDs, so it publishes nothing until the Materialize CRD exists, and picks it up when it does. rbac.extraRules grants it list and watch on materializes.

Ingress is narrowed to the Alloy gateway, which is what scrapes it (prometheus.operator.servicemonitors in the gateway pipeline) — the same shape as node-exporter.networkPolicy, and unlike that one it is actually enforced, because these pods are not host-networked.

8081 is listed alongside 8080 even though nothing serves it at the default selfMonitor.enabled: false. A closed port that nothing dials costs nothing; the alternative is that turning selfMonitor on produces a scrape that fails silently against a policy nobody thought to update.

Egress is restricted, and this is the one component where that is safe to do, because its destination set is closed: DNS, and the API server. It reads nothing else — the whole workload is a watch on the API and an exposition endpoint. Deny egress outright (as node-exporter does) and it never lists a single object; the pod stays Ready and every kube_* series simply disappears.

443/6443 to 0.0.0.0/0 rather than something tighter because the API server’s address is the one thing a chart cannot derive. In-cluster it is the kubernetes.default.svc ClusterIP, which most CNIs evaluate against the endpoint behind it — and on a managed control plane that endpoint is outside the cluster network entirely. Narrow the CIDR to your control plane’s where you know it.

Load-bearing, not a preference.

Every kube_* series carries its own namespace, pod and container labels describing the object it reports on — that is the entire point of the exporter. Those names collide with the target labels of the scrape, which describe the kube-state-metrics pod itself. Without this, Prometheus resolves the collision by renaming the exporter’s labels to exported_namespace / exported_pod and writing the target’s identity into namespace / pod.

The result is not missing data, which is why it went unnoticed: every series arrives, up is 1 and scrape_samples_scraped is healthy, but every series reads namespace="<the monitoring namespace>". kube_pod_info collapses from one series per pod to a single identity, and any query written as kube_*{namespace=…} — which is all of ours — silently matches nothing.

Both endpoints, so turning selfMonitor on does not reintroduce it on the half nobody was looking at. The subchart defaults both to false.

Asserted by the e2e suite: kube_state::labels_are_honored.

cluster_id and replica_id are the names mz_cluster_info and mz_replica_info already carry, so a join through kube_pod_labels needs no label_replace. Without the rename the keys would be label_cluster_environmentd_materialize_cloud_cluster_id and its siblings. The label_* form is dropped rather than kept beside the canonical one, because nothing read it before these labels were allowlisted.

Each copy matches (.+), so it fires only where the pod label is present. With the default (.*) it would also fire where the label is absent and write an empty value, which removes the canonical label from a series that carries its own: kube_mz_environment_info would lose its environment_id. Restate these rules when overriding the list, since Helm replaces it.

Deliberately no CPU limit. This is a DaemonSet that does nothing between scrapes and then bursts for the length of one collection; a CPU limit turns that burst into CFS throttling, which shows up as scrape timeouts on exactly the loaded nodes where you most need the sample. The 10m request is what it averages, not what a scrape costs.

Memory is request == limit, which is the opposite call for the opposite reason: node_exporter’s working set is flat and bounded (~20-30 MiB for this collector set), so a limit equal to the request costs nothing and makes the per-node footprint a number the cluster autoscaler can actually plan with. This lands the pod in Burstable QoS. Guaranteed would require the CPU limit we just argued against; being evicted a little earlier under node pressure is the better trade, and monitoring-critical covers the ordering.