materialize-monitoring Helm Reference#

• Alloy reads global.image.pullSecrets and never this key. This chart honors both spellings for the workloads it renders itself, but the Alloy subchart only sees its own.
• Loki, grafana-operator, Alertmanager and metrics-server read no global at all and need their own <chart>.imagePullSecrets.

profiles/registry/pull-secret.values.yaml sets every one of those paths from a single Secret name; prefer it over setting this key alone, which leaves Loki — the largest pod count in the release — pulling anonymously.

It lives under global because two subcharts already read global.clusterDomain and build real addresses from it — Loki’s memberlist join address and canary URL, and Thanos’s in-cluster endpoints. Helm propagates global into every subchart, so setting it here is what makes one value mean one thing rather than three keys that can disagree.

One subchart is not covered by that propagation. metrics-server reads its own metrics-server.tls.clusterDomain, which is set to the same default below; a render-time check warns when the two disagree, because the failure is a certificate whose SANs name a domain the cluster does not use.

Load-bearing for certificates: the SAN ladder ends in $svc.$ns.svc.$clusterDomain, so a wrong value here produces certificates that fail verification against the endpoints this chart itself dials.

Ingress covers 9093 (the API — Loki’s ruler, the Alloy gateway’s scrape, and Grafana’s alerting all arrive here) and 9094, the gossip port a multi-replica Alertmanager uses to deduplicate notifications between peers.

9094 is listed twice, TCP and UDP, and both are required. Alertmanager’s mesh is memberlist: it joins and pushes state over TCP and gossips over UDP, and the upstream chart’s Service declares clusterpeer-tcp and clusterpeer-udp on the same port for exactly that reason. A TCP-only rule produces the worst version of this failure — the peers find each other, the cluster forms, and it never converges, so every notification goes out once per replica.

Both are listed even at the default replicaCount: 1, where nothing dials either: the cost is an open port that no pod connects to, and the alternative is a cluster that silently sends every notification twice the day someone scales it up.

Egress is deliberately wide. Alertmanager’s job is to reach PagerDuty, Slack, an SMTP relay, or a webhook on someone’s internal network, and this chart cannot know which — 443 and the SMTP ports are the defaults because they are what receivers use, not because they are a boundary.

Ingress is only the metrics port. 8888 (pprof) is deliberately left out — it is a debugging surface, and a policy that opens it by default defeats the point of having one.

Egress carries 443/6443 to 0.0.0.0/0 because the operator is an API client first: it watches Grafana, GrafanaDashboard and GrafanaDatasource resources continuously, and a policy that cannot reach the API server leaves it running, healthy, and reconciling nothing. The API server’s address is not something a chart can derive — in-cluster it is a Service ClusterIP that most CNIs evaluate against the real endpoint behind it, which on a managed control plane is outside the cluster network entirely.

connections.grafana.mode: external puts the Grafana it writes to outside the cluster as well; 443 already covers the usual case, but a self-hosted Grafana on another port needs an egress.extra entry. The render warns when it can tell that applies.

A supplement, not a replacement: grafana.networkPolicy stays on and keeps doing what it does. That template emits exactly one ingress rule, on service.targetPort, with no way to add a second port through values — so a multi-replica Grafana running HA alerting has its 9094 gossip closed by the very policy that protects its UI. The failure is quiet in the worst way: the replicas start healthy, never find each other, and every Grafana-managed alert notifies once per replica.

This renders the missing rule from the umbrella chart, which can. TCP and UDP, because memberlist uses both — TCP for the join and for state pushes, UDP for the gossip itself — and a TCP-only rule produces a cluster that forms and then does not converge.

It renders only when grafana.ini.unified_alerting.ha_peers is set, which is what makes the replicas dial 9094 in the first place; profiles/grafana-postgres.values.yaml is the shipped shape that sets it. Nothing is created for a single-replica Grafana that never gossips.

Both directions are broad, and both for the same reason: the peers are the control plane and the nodes, neither of which has an address this chart can select. allowExternal is true by default here, unlike the other two — the aggregation layer dials 10250 from the API server, which on a managed control plane is not a pod, not in the cluster’s pod CIDR, and not reachable by any podSelector. A policy that closed that port would break every kubectl top and every HPA in the cluster, not just this release.

What the policy still buys is the rest of the surface: no other port on the pod accepts a connection, and egress is limited to the kubelet port plus the API server rather than anywhere at all.

Egress 10250 to 0.0.0.0/0 is the kubelet summary API on each node. Node IPs are not selectable either, and narrowing this to the node CIDR is a good idea wherever you know it — egress.extra with an ipBlock, and drop 10250 from external.ports.

Supply an issuer with issuerRef, or let the chart bootstrap a self-signed root with selfSigned.enabled. The two are mutually exclusive and the render refuses both at once.

Prefer an issuer scoped to this stack over the cluster’s general-purpose one. None of the receiving components here implement per-client authorization — Loki, Thanos receive and Alloy’s receivers can be told to require a certificate signed by a given CA, and none of them can be told that this identity may write and that one may not. The whole authorization decision is “is this signed by the CA we trust”, so the size of the trust domain is the security property. Reusing a ClusterIssuer that signs for every workload in the cluster reduces mTLS to “has any certificate”, which is a real improvement over “can reach the Service” and much narrower than it sounds.

Renders the three resources cert-manager needs for a private CA: a selfSigned issuer, a CA Certificate signed by it, and an issuer backed by that CA which every component certificate then references.

Convenient rather than principled — a real deployment usually has a PKI already, and issuerRef above is the path for it. This exists so that trying the feature does not start with a cert-manager tutorial, and so the chart-only test tier has a root to issue from.

kind: ClusterIssuer is the default deliberately. A namespaced Issuer can only sign for Certificate resources in its own namespace, and profiles/split-namespace.values.yaml puts components in several — so a namespaced issuer silently signs some components and leaves the rest stuck Pending. Issuer is correct and cheaper for the single-namespace default; the render warns when it sees the combination that does not work.

Only read for kind: ClusterIssuer. A namespaced Issuer has the opposite rule and takes its own namespace; the chart handles both.

Only Grafana takes an external certificate today, because it is the only component in the stack meant for a human and the only one this chart helps expose. Set dnsNames to the hostname the load balancer answers on; it must match what users actually type, and what grafana.ini.server.root_url says.

Leave issuerRef.name empty and nothing external is rendered, which is the right answer for an L7 load balancer holding a cloud-managed certificate.

services is the list the SAN ladder is built from, and it is a list rather than a single name because Loki and Thanos are several Services behind one trust boundary. The defaults name exactly the Services this chart’s own URLs dial — a render check asserts that every in-cluster destination URL matches a SAN on the corresponding certificate, so a Service added to a destination and forgotten here fails the render rather than the handshake.

Each entry takes:

A mounted file does not have that problem: the kubelet refreshes Secret contents atomically, and Alloy’s client paths pick up the new material on the next connection. Set these to paths under /etc/mzmon/tls, which is where the chart mounts the certificate it issues for this component.

Inline PEM stays supported as the bring-your-own-PKI escape hatch.

The gateway has three listeners, not two, and they are split across the two pipeline trees the same way the listeners are:

receive_http is the one to check last: it lives in the metrics tree and is easy to leave behind, and forgetting it leaves the metrics write path wide open behind a logs path that looks secured. A validator refuses that combination rather than letting it ship.

clientAuth is exposed rather than implied because the middle state is what a safe rollout needs, and an operator who cannot name that state cannot perform one. Values are Go’s tls.ClientAuthType:

Unlike Loki’s server, Alloy’s listeners are not probed by the kubelet on the same port — the readiness probe is on 12345 — so phase 3 is reachable here.

A mounted file does not have that problem: the kubelet refreshes Secret contents atomically, and Alloy’s client paths pick up the new material on the next connection. Set these to paths under /etc/mzmon/tls, which is where the chart mounts the certificate it issues for this component.

Inline PEM stays supported as the bring-your-own-PKI escape hatch.

The gateway scrapes /metrics/cadvisor on every kubelet rather than the agent running its own cAdvisor. The kubelet already computes these stats, so running our own meant computing them twice per node — measured at ~750Mi per agent against a 200Mi logs-only envelope. Scraping the kubelet removes the housekeeping cost entirely and puts the remaining scrape cost on the gateway, where it is shared across replicas rather than reserved on every node — the distinction that matters when bin packing.

Coverage is not the tradeoff it was assumed to be: a GKE kubelet serves 69 distinct container_* metrics against the 70 an in-process cAdvisor produced, including the descriptor / socket / thread family that otherwise needs the host PID namespace. Every container_* metric this chart’s queries reference is present.

Needs nodes and nodes/metrics RBAC, which the Alloy subchart’s default clusterRules already grant.

Left false: on GKE the kubelet certificate verifies against the in-cluster CA, which the chart passes as ca_file. A distribution that signs kubelet certs with a CA the pod does not trust needs this true, and the failure is quiet — scrapes fail and container metrics stop rather than anything erroring at install. Check up{job="cadvisor"} when bringing up a new distribution.

The gateway has three listeners, not two, and they are split across the two pipeline trees the same way the listeners are:

receive_http is the one to check last: it lives in the metrics tree and is easy to leave behind, and forgetting it leaves the metrics write path wide open behind a logs path that looks secured. A validator refuses that combination rather than letting it ship.

clientAuth is exposed rather than implied because the middle state is what a safe rollout needs, and an operator who cannot name that state cannot perform one. Values are Go’s tls.ClientAuthType:

Unlike Loki’s server, Alloy’s listeners are not probed by the kubelet on the same port — the readiness probe is on 12345 — so phase 3 is reachable here.

For backends that authenticate with an API-key header rather than a bearer token — Honeycomb’s x-honeycomb-team, for instance. Before this existed the only way to send one was authType: custom and a hand-written otelcol.auth.headers block, which no validator could check.

Each entry names a header with key and sets exactly one of:

• value — a literal. It renders into the gateway’s pipeline ConfigMap in plaintext, so use it only for non-secret routing headers such as a dataset or tenant name.
• valueEnv — the name of an environment variable, read with sys.env() when the gateway starts. Use it for anything secret and supply the variable through alloy-gateway.alloy.envFrom (a secretRef). The name is yours to choose; nothing else in the chart depends on it.

Unlike the Loki and Prometheus destinations, there is no value/valueEnv pair per header where the value is copied into the env ConfigMap. A header is either a literal or a lookup, and routing a literal through the env ConfigMap would only move it from one plaintext ConfigMap into another.

Setting this true is the escape hatch for a deployment whose allowlist lives somewhere the chart cannot read — a security group, an egress firewall, a service mesh, an authenticating proxy in front. It suppresses the error, not the exposure: the render still warns, because a Grafana on the open internet is protected by nothing but its admin password until you configure an identity provider under grafana.ini.

Nothing in it is validated or defaulted, and the production guidance on the grafana block above does not reach it. Prefer mode: bundled, where the grafana subchart owns the Deployment and all of that applies.

connections:
  grafana:
    mode: operator
    operator:
      spec:
        version: "13.0.2"
        deployment:
          spec:
            template:
              spec:
                containers:
                  - name: grafana
                    resources:
                      requests:
                        cpu: 100m
                        memory: 256Mi
        config:
          database:
            type: postgres
            host: grafana-db.example.internal:5432

Reference: https://grafana.github.io/grafana-operator/docs/api/#grafanaspec

caSecret names a Secret in the Grafana instance’s namespace holding the roots to trust. Under split-namespace that is grafana, not the release namespace, and the certificate machinery already issues <release>-grafana-tls there — its ca.crt key is the internal CA, so the default below works as-is once certificates.enabled is on.

Prefer this over jsonData.tlsSkipVerify. Skipping verification leaves the connection encrypted and unauthenticated, which on the read path means Grafana will happily talk to anything that answers on that address — and the failure mode of getting it wrong is a dashboard that renders, from the wrong source.

Takes precedence over caSecret, and needs no operator involvement at all, which makes it the right choice when the CA is already in hand at render time. It does not track a rotation: the PEM here is a copy, and re-issuing the CA means re-rendering. Prefer caSecret when the material lives in the cluster.

The Secret must be in the Grafana instance’s namespace, which is not necessarily this release’s under split-namespace.

key is load-bearing beyond naming the field: grafana-operator substitutes ${<key>} — the secretKeyRef key verbatim, dots and all — so the chart derives the placeholder from it. A mismatch there fails in the least visible way available: the CR applies, the operator logs nothing, secureJsonFields reports tlsCACert: true, and only at query time does Grafana say failed to parse TLS CA PEM certificate, because what it stored is the literal placeholder text.

caSecret names a Secret in the Grafana instance’s namespace holding the roots to trust. Under split-namespace that is grafana, not the release namespace, and the certificate machinery already issues <release>-grafana-tls there — its ca.crt key is the internal CA, so the default below works as-is once certificates.enabled is on.

Prefer this over jsonData.tlsSkipVerify. Skipping verification leaves the connection encrypted and unauthenticated, which on the read path means Grafana will happily talk to anything that answers on that address — and the failure mode of getting it wrong is a dashboard that renders, from the wrong source.

Takes precedence over caSecret, and needs no operator involvement at all, which makes it the right choice when the CA is already in hand at render time. It does not track a rotation: the PEM here is a copy, and re-issuing the CA means re-rendering. Prefer caSecret when the material lives in the cluster.

The Secret must be in the Grafana instance’s namespace, which is not necessarily this release’s under split-namespace.

key is load-bearing beyond naming the field: grafana-operator substitutes ${<key>} — the secretKeyRef key verbatim, dots and all — so the chart derives the placeholder from it. A mismatch there fails in the least visible way available: the CR applies, the operator logs nothing, secureJsonFields reports tlsCACert: true, and only at query time does Grafana say failed to parse TLS CA PEM certificate, because what it stored is the literal placeholder text.

GOMEMLIMIT at ~80% of the memory limit is the part worth keeping regardless of the number. Go’s GC has no knowledge of a cgroup limit, so it grows the heap toward a ceiling the kernel enforces by killing the process. Telling the runtime about the ceiling turns an OOM-kill into GC pressure — the failure mode becomes “slower”, not “dead”. Keep it in step with the memory limit above.

Where the journal lives is host-specific, so both roots have to be covered. Observed:

Bottlerocket also has an empty /run/log/journal, and kind has no /var/log/journal at all — neither is a problem, but neither alone is enough either.

Deliberately not mounting /var/log/journal directly: journald’s default Storage=auto switches to persistent storage as soon as that directory exists, so a hostPath mount that creates it would change how the host journals. varlog already exposes it wherever it legitimately exists.

/etc/machine-id is required, and its absence is silent. systemd stores the journal under <path>/<machine-id>/, and the container has no machine-id of its own — the Alloy image ships none. Without the host’s, loki.source.journal starts, reports healthy, and reads zero lines.

Measured on a Bottlerocket node, controlling for the restart that a mount change forces: with the mount, loki_source_journal_target_lines_total went 0 -> 1244; removed again, back to 0 on a freshly started pod.

A kind node happens to work without it, which makes this a bad thing to test on kind alone — the two differ in which directory holds the journal (/run/log/journal there, /var/log/journal on Bottlerocket), and only the latter needs the ID to resolve. Do not remove this because kind stays green.

That holds only while nothing reads the material, which is why the mTLS profiles flip it to optional: false. Once a config names a path inside this mount, an empty mount is not a pod serving plaintext: Alloy fails its initial load and enters CrashLoopBackOff, backing off to five minutes, on an error naming a file rather than the Secret behind it. Required, the same race is a container the kubelet holds in ContainerCreating with a FailedMount event, started on the next volume re-sync. Do not set this to false here — with certificates off there is no Secret to wait for and the pod would never start.

The kubelet refreshes the contents in place on renewal, which is why the tls.*File carriers are preferred over the inline PEMs.

Sized for the logs-only agent. This needed ~750Mi while an in-process cAdvisor lived here; that moved to the gateway, so the envelope is back to what the log path actually costs, with headroom over the ~200Mi it ran at before.

The CPU limit stays above the request: the agent is bursty, and a limit equal to the request turns that into CFS throttling on the busiest nodes.

Ingress is one port from one peer. 12345 is Alloy’s HTTP server — it serves /metrics and the live debugging UI, which shows the running pipeline and samples of the data moving through it — and the only pod that needs to reach it is the gateway performing the serviceMonitor scrape.

Egress is not restricted, and the policyTypes list says so rather than pairing an Egress type with an allow-everything rule. Two of the agent’s three destinations are unaddressable from here: the API server backs discovery.kubernetes, and on a managed control plane it is not in the pod network at all. The third, the gateway, is in-namespace and would be easy — but a policy that allows only the parts we happen to know is a policy that breaks the first time someone adds a scrape to the agent pipeline, and does it silently. Narrow it deliberately, with egress and policyTypes, once you know what your agent actually collects.

The default from selects by pod label within the release namespace. Running the gateway in a different namespace (see profiles/split-namespace.values.yaml) requires replacing this list with one that adds a namespaceSelector.

That holds only while nothing reads the material, which is why the mTLS profiles flip it to optional: false. Once a config names a path inside this mount, an empty mount is not a pod serving plaintext: Alloy fails its initial load and enters CrashLoopBackOff, backing off to five minutes, on an error naming a file rather than the Secret behind it. Required, the same race is a container the kubelet holds in ContainerCreating with a FailedMount event, started on the next volume re-sync. Do not set this to false here — with certificates off there is no Secret to wait for and the pod would never start.

The kubelet refreshes the contents in place on renewal, which is why the tls.*File carriers are preferred over the inline PEMs.

Two ingress rules, because the gateway’s ports fall into two groups with very different audiences.

The ingest ports — 3100 (Loki push), 4317/4318 (OTLP), 9090 (Prometheus remote write) — are open to every pod in the cluster. That is deliberate and it is not laziness: these are the endpoints the stack exists to offer. The agent pushes logs to 3100, and anything in the cluster with telemetry to send — a Materialize workload emitting OTLP, an application remote-writing metrics — is a legitimate client this chart cannot enumerate. namespaceSelector: {} still means in-cluster: a sender outside the pod network is denied, and the gateway is not meant to be internet-facing.

12345 is not an ingest port and is closed to everything but the gateway itself. It carries the clustering gossip between gateway replicas (alloy.clustering.enabled) and serves /metrics and the debugging UI, which exposes the pipeline and samples of the telemetry passing through it. The gateway scrapes its own serviceMonitor, so its own pods are the only peer.

Egress is not restricted, and cannot usefully be. The gateway scrapes the kubelet cAdvisor endpoint on every node — port 10250 on a node IP, which no podSelector matches — and it discovers ServiceMonitor and PodMonitor targets cluster-wide, so its destination set is every pod and every port that anyone in the cluster has ever pointed a monitor at. On top of that it is the stack’s egress point for external backends: Amazon Managed Prometheus, Honeycomb, a Datadog or OTLP destination under connections. Enumerating that is not a policy, it is a copy of the cluster’s inventory that goes stale.

Upstream ships these off. One switch turns them on for all components, and the per-component pdb.enabled cannot opt back out — the subchart templates test or <component>.pdb.enabled global.pdb.enabled.

maxUnavailable rather than minAvailable, deliberately:

• It scales with replica count instead of pinning an absolute floor.
• On the single-replica Compactor, minAvailable: 1 would block every voluntary eviction and hang node drains indefinitely. maxUnavailable: 1 permits the eviction, so the singleton is a harmless no-op rather than a drain deadlock.
• On Receive it must stay within what write quorum tolerates (replicationFactor - ((replicationFactor / 2) + 1)), which is 1 at the replication factor of 3 set below. A validator enforces this.

This matches the Loki convention (ingester maxUnavailable: 1).

One switch, no per-component knobs and no selectors — the subchart renders a fixed policy per component and exposes nothing to tune. What each one says is: ingress on that component’s own service ports, from anywhere; egress unrestricted.

So this is the weakest policy in the stack, and worth being clear about what it does and does not buy. It does not restrict who may reach Query, Receive or Store Gateway — any pod in the cluster still can. It does close every port that is not a declared service port, which is the surface an unrelated workload would otherwise find on these pods.

It is on because it costs nothing to be right about. The rules cannot break a working install: Receive’s grpc, http and remote-write ports are all in its own ingress list, so hashring replication between Receive pods and the gateway’s writes both keep working, and unrestricted egress leaves the object store reachable. Narrowing the ingress sources needs an upstream change, or your own policy layered on top.

pod-template-hash, not controller-revision-hash: Query is a Deployment, and that is the label its ReplicaSets carry. Using the StatefulSet label here would match nothing and quietly disable the same-revision guard.

This is a recent-query cache, not a durability window: blocks ship to object storage every 2h and the Store Gateway serves everything older, so all that shortens here is how far back a query is answered from local TSDB rather than from the bucket. 6h covers the common dashboard ranges and leaves 4h of margin after a block closes, uploads, and the Store Gateway syncs it.

Raising it back is a real cost rather than just a number. At the medium envelope (1.5M series) 24h is ~7.3Gi per pod, and a GKE node with a 47Gi boot disk offers only ~18.8Gi of allocatable ephemeral storage — one pod would claim 40% of the node’s budget, and Loki’s ingesters draw on the same pool.

1. It rewrites the requests below, so a sizing profile would not survive contact with it.
2. updateMode: Auto evicts pods to apply new requests, and Receive holds up to tsdb.retention (6h) of blocks on an emptyDir whose only redundancy is the replication factor — an eviction discards that pod’s copy of the window.
3. The subchart’s VPA template is CRD-gated, so leaving it on makes the stack behave differently on clusters with the VPA CRD installed than on those without — silently, in both directions.

If you want its recommendations without its actions, set updateMode: "Off" rather than re-enabling Auto.

No memory limit, deliberately, and for a sharper reason than elsewhere in this chart: an OOM-kill drops the un-uploaded block window, and unlike Loki’s ingesters that window has nothing protecting it but the replication factor. Alert on usage instead.

ephemeral-storage is declared because the TSDB lives on an emptyDir (see persistence). The request is what the scheduler places against, so a node without room refuses the pod instead of filling up silently. The limit carries deliberate headroom over the request: the kubelet’s response to exceeding it is eviction, and evicting Receive destroys exactly the un-uploaded window this sizing exists to protect.

Size these against a node’s allocatable ephemeral storage, not its disk size. The two are nowhere near each other: GKE reserves most of the boot disk for the image filesystem, so a 47Gi disk offers roughly 18.8Gi allocatable. A request above that cannot schedule anywhere, and the cluster-autoscaler will not rescue it — no node of that shape would fit either, so the pod sits Pending indefinitely on Insufficient ephemeral-storage. Check before raising these:

kubectl get nodes -o custom-columns=‘NODE:.metadata.name,EPH:.status.allocatable.ephemeral-storage’

4Gi holds 6h of the medium envelope (~1.8Gi) with room for the WAL and compaction churn, and leaves the rest of the node for everything sharing that pool — including Loki’s ingesters, which are also emptyDir-backed.

Hard (DoNotSchedule) on zones, deliberately. A pod that cannot satisfy it goes Pending, which is the signal Karpenter or the cluster-autoscaler uses to add a node in the deficient zone — a soft rule cannot summon capacity that way, it just quietly places the pod wrongly. Host spread stays soft so pods still schedule when nodes are momentarily scarce; two pods sharing a node is a smaller problem than a pod not running.

This became viable when Receive stopped using a PVC (see persistence below). A zonal volume cannot be attached from another zone, so a hard zone rule and an AZ-pinned pod pull in opposite directions: the rule says “go to the empty zone”, the volume says “you may only run in zone A”, and the pod goes Pending forever rather than for as long as it takes to add a node. On emptyDir there is nothing holding it back.

The selector names both app.kubernetes.io/name and app.kubernetes.io/component, and the first of those is load-bearing: component names are not unique across backends. Loki also ships a compactor, a query-frontend and a ruler, so a component-only selector would make Thanos count Loki’s pods when computing its own skew — silently, and only for the components whose names happen to collide.

The release name is deliberately absent: the subchart renders this through toYaml rather than tpl, so a {{ include ... }} would land in the manifest literally. Spread is namespace-scoped and this chart assumes one instance of each backend per namespace, so name plus component is sufficient. See the Namespace layout section.

Note this cannot move to thanos.global.topologySpreadConstraints: each constraint carries its own labelSelector, and a global one would make every Thanos component count Receive’s pods when computing its own skew.

Durability comes from --receive.replication-factor=3, not from disk. Blocks ship to object storage every 2h, so the window that exists only locally is at most 2h — and every replica uploads its own copy under a distinct replica external label, which the Compactor deduplicates. A pod that comes back with an empty volume has lost its copy of that window, and the query path still answers from the surviving replicas.

A PVC makes an AZ failure worse, not merely less flexible: an EBS volume cannot be attached from another zone, so a pod whose zone is gone stays Pending until the zone returns rather than rescheduling into a healthy one. On the write path that turns a recoverable event into an outage that waits on the cloud provider — with RF 3 write quorum is 2, so two pods stuck Pending block writes outright.

The trade accepted in exchange: Thanos Receive has no peer hand-off, so unlike Loki it will not backfill an emptied pod from its neighbours. That window stays at two copies instead of three until the next block ships.

Autoscaling is available upstream but left off deliberately. Store Gateway is a PVC-backed StatefulSet that syncs the bucket’s block index on startup, so scale-up is slow to become useful (it serves nothing until the index is warm) and CPU-triggered scaling reacts long after the load that triggered it. Scale-down also leaves orphaned PVCs behind, since StatefulSet volumes are not reclaimed. Size it deliberately instead.

Kept on at every size deliberately, rather than flipping to ephemeral for small installs. A volume that appears and disappears with the profile is a surprise during an incident, and the AZ-pinning argument that drives Receive to emptyDir is much weaker here: Store Gateway is a read-only replica set with no quorum, so a pod stuck Pending in a dead zone costs read capacity rather than blocking writes.

The kind profile is the one exception, where CI would rather not wait on volume provisioning for a cache.

1. Nothing depends on quorum. Store Gateway is a read-only replica set over object storage, so an unbalanced placement costs read capacity during a zone loss rather than correctness. Receive’s hard rule protects write quorum; there is no equivalent to protect here.
2. It is the one Thanos component that still keeps a PVC, and a hard zone rule against a zonal volume can deadlock rather than merely wait. With volumeBindingMode: WaitForFirstConsumer — the default for the zonal CSI classes and what you want — the volume follows the pod and the two agree. With Immediate, the PVC’s zone is chosen before scheduling and a hard rule pointing elsewhere leaves the pod Pending forever. Soft degrades instead of deadlocking on a StorageClass we do not control.

The subchart ships this empty, so unlike receive.extraArgs there is nothing to lose by overriding it — but it is still a list, so restate both entries when adding a third.

Raw retention has a floor set by the downsampling thresholds: Thanos produces 5m downsamples only from blocks spanning 40h or more, and 1h downsamples only from blocks spanning 10d or more. Below those the tier is never created at all and long-range queries silently fall back to reading raw blocks — slower and more expensive, which is the opposite of the intent. 30d clears both comfortably.

Downsampling is the memory-heavy phase, and --compact.concurrency multiplies it; leave that at 1 unless compaction is provably behind.

No ephemeral-storage here: the Compactor keeps a PersistentVolume, so its scratch does not draw on the node’s budget. See persistence for why.

Compaction works on whole block groups, not on the recent window, so the requirement scales with days of data rather than hours. At the medium envelope a 2d group is roughly six 8h blocks at ~2.4Gi each, and the Compactor needs the sources and the output at once — on the order of 30Gi. A GKE node with a 47Gi boot disk offers about 18.8Gi of allocatable ephemeral storage, so that never fits: the pod stays Pending on Insufficient ephemeral-storage, and the cluster-autoscaler cannot help because no node of that shape would fit either.

This is the reverse of the call made for Receive, and deliberately so. Receive holds hours of small, replicated data and is the better trade on emptyDir; the Compactor holds days of it and is not. The AZ-pinning cost is real — a zone outage stops compaction, and while compaction is stopped retention is not enforced and the bucket grows — but “compaction pauses during a zone outage” is a far better failure than “compaction never runs because the pod cannot be scheduled”.

Note that enabling this is not sufficient on its own: point connections.datasources.thanos.url at the query-frontend Service too, or reads keep going straight to Query and the cache is never consulted. A validator warns when the two disagree.

A hardened rebuild is a drop-in swap: point registry/repository at it and keep the tag. Two published options track upstream Grafana versions and keep its entrypoint and layout, so a values change is the whole change:

• Chainguard Images — https://images.chainguard.dev/directory/image/grafana/overview
• Docker Hardened Images — https://docs.docker.com/dhi/ (subscription; images land in your own org namespace, so registry is yours)

profiles/registry/ does this for the whole stack rather than for Grafana alone, and carries the pull-secret wiring that goes with it.

Both ship no shell and no package manager, which is the point — and which also means plugins cannot work, since that installs at container start. Bake plugins into the image instead. See the note on plugins below.

Bitnami also publishes a hardened Grafana, but its images are built for Bitnami’s own charts — Bitnami entrypoints, /opt/bitnami paths, UID 1001 — against an upstream subchart here, so it is a port rather than a swap. No profile ships for it.

targetCPU is a percentage of the CPU request, so resources.requests.cpu has to be set or the HPA has no denominator and never scales. A render-time check warns when it is missing.

Prefer PostgreSQL (grafana-postgres) wherever a database is available: it is the only option that lifts both constraints.

A LoadBalancer Service with no loadBalancerSourceRanges is an error at render time unless connections.grafana.allowPublicAccess is set.

Set grafana.ini.server.root_url to the same URL — Grafana builds share links, alert notification links, and OAuth redirect URIs from it, and all three break silently when it disagrees with the Ingress host.

Secrets must NOT be inlined — this block renders into a ConfigMap, so a value here lands in plaintext in the release manifest, in helm get values, and in whatever Git repo holds your values file. Use Grafana’s own expansion instead: $__file{/path} for a mounted Secret (see extraSecretMounts) or $__env{VAR} for one injected with envValueFrom. The subchart’s assertNoLeakedSecrets check fails the render if you forget.

OIDC single sign-on, as an example — this is the shape, not a default:

grafana:
  grafana.ini:
    server:
      root_url: https://grafana.example.com
    auth:
      oauth_auto_login: true
    auth.generic_oauth:
      enabled: true
      name: SSO
      allow_sign_up: true
      client_id: <client-id>
      client_secret: $__file{/etc/secrets/grafana-oidc/client-secret}
      scopes: openid profile email groups
      auth_url: https://idp.example.com/oauth2/v1/authorize
      token_url: https://idp.example.com/oauth2/v1/token
      api_url: https://idp.example.com/oauth2/v1/userinfo
      allowed_domains: example.com
      # Map an IdP group claim onto a Grafana role, so membership is the
      # provisioning mechanism and nobody is added by hand.
      role_attribute_path: contains(groups[*], 'sre') && 'Admin' || 'Viewer'
  extraSecretMounts:
    - name: grafana-oidc
      secretName: grafana-oidc
      mountPath: /etc/secrets/grafana-oidc
      readOnly: true

Every provider, the role-mapping rules, and the break-glass path are in Dashboards > Grafana > Authentication.

The subchart’s policy is rigid: it emits exactly one ingress rule, on service.targetPort, and either allows every source or a fixed set of them. There is no way to add a second port through values. That constrains what this can say, so it is worth being precise about what it does say.

Ingress: 3000 from anywhere, everything else closed. allowExternal stays true because Grafana is the one component in this stack a human is meant to reach, and the ways they reach it are all things a podSelector cannot see — an ingress-controller pod in another namespace, a cloud load balancer’s source IPs, kubectl port-forward, which arrives from the API server. Turning it off is worthwhile once you know your path in; explicitNamespacesSelector takes the ingress controller’s namespace and explicitIpBlocks takes a load balancer’s health-check and client ranges. profiles/grafana-ingress.values.yaml is where that belongs.

What it closes is the rest of the pod: 6060 (pprof) and 9094 (the unified-alerting gossip port) stop accepting connections. pprof should not be reachable in production, so that one is the point. 9094 is not — it matters as soon as you run multiple replicas and set grafana.ini.unified_alerting.ha_peers, which needs the pods to reach each other on it, and this template cannot express a second port.

You do not have to turn this off to get HA alerting. The chart renders the missing rule itself, as networkPolicies.grafanaGossip, and it appears exactly when ha_peers is set. Policies are additive, so the two sit side by side. Turning that off while HA alerting is configured is what the render warns about; turning this one off takes the supplement with it, since a lone gossip policy would isolate the pods and leave only 9094 open.

Egress is left unrestricted (egress.enabled: false renders a policy with no Egress policy type at all, rather than an empty allow). Grafana dials the things it is configured to dial: Loki and Thanos in-namespace, a PostgreSQL state database wherever it lives, an OIDC provider on the internet, grafana.com if plugins is non-empty. The first two are knowable here and the rest are not, and a partial allowlist would fail as a dashboard that renders empty with no error — the worst failure mode this stack has.

For anything load-bearing, bake plugins into the image instead and point image at it. That is also the only option on a hardened base image, which ships no shell for grafana cli to run in. Pin an exact plugin version either way (name@version); an unpinned plugin silently changes underneath a pinned Grafana.

Ingress is narrowed to the Alloy gateway, which is what scrapes it (prometheus.operator.servicemonitors in the gateway pipeline) — the same shape as node-exporter.networkPolicy, and unlike that one it is actually enforced, because these pods are not host-networked.

8081 is listed alongside 8080 even though nothing serves it at the default selfMonitor.enabled: false. A closed port that nothing dials costs nothing; the alternative is that turning selfMonitor on produces a scrape that fails silently against a policy nobody thought to update.

Egress is restricted, and this is the one component where that is safe to do, because its destination set is closed: DNS, and the API server. It reads nothing else — the whole workload is a watch on the API and an exposition endpoint. Deny egress outright (as node-exporter does) and it never lists a single object; the pod stays Ready and every kube_* series simply disappears.

443/6443 to 0.0.0.0/0 rather than something tighter because the API server’s address is the one thing a chart cannot derive. In-cluster it is the kubernetes.default.svc ClusterIP, which most CNIs evaluate against the endpoint behind it — and on a managed control plane that endpoint is outside the cluster network entirely. Narrow the CIDR to your control plane’s where you know it.

Load-bearing, not a preference.

Every kube_* series carries its own namespace, pod and container labels describing the object it reports on — that is the entire point of the exporter. Those names collide with the target labels of the scrape, which describe the kube-state-metrics pod itself. Without this, Prometheus resolves the collision by renaming the exporter’s labels to exported_namespace / exported_pod and writing the target’s identity into namespace / pod.

The result is not missing data, which is why it went unnoticed: every series arrives, up is 1 and scrape_samples_scraped is healthy, but every series reads namespace="<the monitoring namespace>". kube_pod_info collapses from one series per pod to a single identity, and any query written as kube_*{namespace=…} — which is all of ours — silently matches nothing.

Both endpoints, so turning selfMonitor on does not reintroduce it on the half nobody was looking at. The subchart defaults both to false.

Asserted by the e2e suite: kube_state::labels_are_honored.

Deliberately no CPU limit. This is a DaemonSet that does nothing between scrapes and then bursts for the length of one collection; a CPU limit turns that burst into CFS throttling, which shows up as scrape timeouts on exactly the loaded nodes where you most need the sample. The 10m request is what it averages, not what a scrape costs.

Memory is request == limit, which is the opposite call for the opposite reason: node_exporter’s working set is flat and bounded (~20-30 MiB for this collector set), so a limit equal to the request costs nothing and makes the per-node footprint a number the cluster autoscaler can actually plan with. This lands the pod in Burstable QoS. Guaranteed would require the CPU limit we just argued against; being evicted a little earlier under node pressure is the better trade, and monitoring-critical covers the ordering.

Off deliberately. It is a second container on every node, and DaemonSet overhead is the constraint we are managing here — the sidecar’s own request is comparable to node_exporter’s, so it roughly doubles the per-node cost of node metrics to protect an endpoint that exposes no secrets.

What it would buy, for when that trade changes: authenticated scrapes, and (via kubeRBACProxy.tls + tlsSecret) TLS with client-certificate auth, so only a scraper holding a cert signed by the configured CA can read /metrics. That is the shape an in-cluster-mTLS story would use, and it is parked, not rejected — see the cert-manager mTLS item on the roadmap.

Note that with hostNetwork: true the exporter listens on the node’s own interfaces, so today the real perimeter is the node firewall / security group, not anything Kubernetes enforces. Do not treat port 9100 as private just because a NetworkPolicy exists.

Ingress is narrowed to the Alloy gateway, which is what actually scrapes it (prometheus.operator.servicemonitors in the gateway pipeline). Egress is denied outright — node_exporter never initiates a connection.

Read the hostNetwork caveat before relying on this. The pods run with hostNetwork: true (required: netdev, netclass, netstat, sockstat and conntrack all read namespaced files under /proc/net, so in a pod netns they would report the pod’s traffic, not the node’s). Most CNIs — Cilium and Calico among them — do not apply pod NetworkPolicy to host-networked pods, because that traffic belongs to the node identity. So this policy is a correct declaration of intent and is enforced where the CNI supports it, but it is not the control that keeps port 9100 private. The node’s firewall is.

The default from selects by pod label within the release namespace. Running the gateway in a different namespace (see profiles/split-namespace.values.yaml) requires replacing this list with one that adds a namespaceSelector.

--collector.disable-defaults turns everything off; each --collector.<name> then turns one back on. An allowlist rather than a denylist because the default set drifts with each node_exporter release — a new default-on collector should not silently appear in our cardinality budget.

Every name here must exist on Linux. An unknown --collector.* flag is a parse error and node_exporter exits, so a typo or a platform-specific collector crash-loops the DaemonSet on every node at once. In particular boottime is a Darwin/BSD collector; on Linux node_boot_time_seconds comes from stat, which is why stat is in the list and boottime is not.

Chosen for a network- and memory-hungry Rust database that swaps on purpose:

• Swap, explicitly. swap (per-device swap, off by default upstream) and meminfo (SwapTotal/SwapFree) give the level; vmstat gives the rate (pswpin/pswpout), which is the half that distinguishes healthy deliberate swapping from thrash. pressure (PSI) is the leading indicator for both.
• Memory reclaim. The vmstat field filter is widened below to admit pgscan_*/pgsteal_*, which separate background reclaim (kswapd, fine) from direct reclaim (direct, an allocating thread is stalled) — the difference between swap working and swap hurting.
• Network. netdev/netclass for the NICs, netstat for TCP retransmits, sockstat for socket and TCP-memory pressure, softnet for packets the kernel dropped before userspace saw them, udp_queues for DNS, and conntrack for table exhaustion behind NAT gateways.
• CPU. cpu (including mode="steal"), plus schedstat for run-queue wait — “threads were runnable and did not run”, which is the contention a tightly-optimized dataflow engine feels first — and stat for context switches and blocked processes.
• Correctness-adjacent. timex for clock sync; a database with timestamp semantics on cloud VMs wants to know when NTP has given up.

Left off on purpose, with the reason, since each is one line to re-add:

• slabinfo — kernel slab accounting. Costs ~1.5k series per node (every slab cache × several metrics) and needs a root init container to chmod /proc/slabinfo (permissionInitContainer.fixes.slabinfo). meminfo already answers “how much kernel memory” via Slab/SReclaimable/ SUnreclaim; slabinfo only adds which cache, which is forensics you turn on during an investigation.
• ethtool — on EC2 this is where the ENA allowance counters live (bw_*_allowance_exceeded, pps_allowance_exceeded, conntrack_allowance_exceeded): instance-level network throttling that is invisible in every other metric. Genuinely valuable on AWS and worth enabling there, paired with --collector.ethtool.device-include=^(eth|ens|enp) — held back only because the stat set is driver-specific (gVNIC and Azure expose different counters) and unverified on our images.
• interrupts, softirqs — per-CPU × per-IRQ, so thousands of series on a large instance. softnet covers the actionable part.
• tcpstat — parses every socket in /proc/net/tcp, which is most expensive on exactly the busiest nodes. sockstat gives the aggregate cheaply.
• processes — walks all of /proc each scrape. stat already reports running and blocked counts.
• edac, rapl, thermal_zone, dmi — hypervisors do not expose ECC, powercap or thermals to guests, and instance type comes from node labels. Worth revisiting on bare metal.
• meminfo_numa, zoneinfo, buddyinfo — NUMA and fragmentation detail for very large instances; opt in when investigating.
• ipvs, nfs, nfsd, bonding, mdadm, zfs, btrfs, bcache — none apply to a cloud node running ext4/xfs on network block storage with Cilium replacing kube-proxy. ipvs matters if you run kube-proxy in IPVS mode; nfs if you mount EFS/Filestore.
• sysctl — no default set, but --collector.sysctl.include=vm.swappiness, vm.overcommit_memory,vm.overcommit_ratio is a cheap guard against AMI drift changing swap behaviour underneath a swap-dependent workload.
• textfile — the node-local extension point, but it needs a hostPath mount this chart does not create.