Stack-wide settings#

This page explains the settings that cut across every component: the namespace layout, scheduling priority, network policies, image registries, and the zone count. What to do about each one is in the production checklist, ordered by what goes wrong if you skip it.

Namespace layout#

Each subchart uses a deterministic name, a static fullnameOverride such as loki or thanos, so no resource carries a release-name prefix. That assumes one instance of each backend per namespace, which is a reasonable constraint for an umbrella infrastructure chart. Two layouts are supported; both are fine, so pick per your isolation needs.

Shared namespace (default). Everything deploys into the release namespace (recommended: monitoring). This is the default and the path most installs should take.

Split namespace. One namespace per subchart, via the split-namespace profile (-f charts/materialize-monitoring/profiles/split-namespace.values.yaml), which sets a namespaceOverride per component (loki, thanos, grafana, …). Support is best-effort.

The layout changes the namespace half of every workload-identity binding: an IRSA / GKE / Azure trust-policy subject targets system:serviceaccount:<namespace>:<sa>, where <namespace> is the release namespace under the shared layout and the per-component namespace under split. The ServiceAccount name is unaffected (it’s the deterministic fullnameOverride). See Logs & Events > Storing and Metrics > Storing.

[consumer] selects the layout at install time; [operator] owns the namespace/RBAC policy it implies.

Scheduling priority#

The chart creates two PriorityClasses and assigns every long-running workload to one of them. The split is about what losing a pod costs you, not about how important the component feels.

The per-node collectors are singletons per node. When one is evicted, nothing else reports that node and the gap never backfills — there is no equivalent of a replica picking up the slack. The backends are replicated, buffered, or both: Alloy retries its writes, so an evicted Loki ingester costs latency rather than data. The gateway is graded critical despite being a Deployment, because it is the single egress choke point for every signal in the stack.

Both classes set preemptionPolicy: Never, which is the load-bearing choice. Priority still decides scheduling-queue order and, more importantly, which pod the kubelet evicts first under node pressure — but monitoring will never evict a Materialize pod to make room for itself. Monitoring that takes down the thing it monitors is worse than monitoring that is late. Both sit well below system-cluster-critical (2000000000) and system-node-critical (2000001000), so cluster plumbing still outranks the stack.

Network policies#

Every workload in the stack carries a NetworkPolicy, and every one of them is on by default. Most come from the subchart that owns the workload; the three subcharts that ship no policy of their own get one rendered by this chart, from templates/networkpolicies.yaml and configured under networkPolicies.

Ingress is tight and egress mostly is not, and that asymmetry is deliberate rather than unfinished. Ingress is the direction that protects these endpoints — unauthenticated metrics, log-push, and query APIs — its allowed peers are knowable from the chart, and a wrong rule fails loudly at the first scrape. Egress destinations are cluster- and account-specific: the API server’s address, kubelet ports on node IPs, an object-storage endpoint, a notification provider, an external Grafana. A wrong egress rule fails silently, minutes or hours later, somewhere other than the pod that changed. So egress is narrowed only where the chart genuinely knows the destination set — node-exporter (nothing), kube-state-metrics (DNS and the API server), loki (in-namespace, DNS, object storage) — and left open elsewhere, with egress.extra and the subcharts’ own egress lists as the place to narrow it against facts only your cluster has.

The gateway is the clearest case for why: it scrapes the kubelet cAdvisor endpoint on every node — port 10250 on a node IP, which no podSelector matches — and it discovers ServiceMonitor and PodMonitor targets cluster-wide, so its destination set is every pod and port anyone has ever pointed a monitor at, plus whatever external backend connections names.

Images and registries#

Every image in the stack is pinned by registry, repository, and tag: in this chart’s values.yaml where it matters enough to own the cadence, and in a subchart’s defaults otherwise. Repointing at a mirror or a hardened rebuild is therefore a values change rather than a fork. Four overlays under profiles/registry/ do exactly that. mirror moves every image to a private mirror without changing any of them, which is the common case and the only one that can carry the Alloy build. chainguard and docker-hardened-images swap in a hardened rebuild — one per vendor that publishes the whole set and keeps the upstream entrypoint and layout (Chainguard Images, Docker Hardened Images). pull-secret carries the credential half on its own and composes with any of them.

Bitnami Secure Images is deliberately absent — it rebuilds around its own charts, so it is a port rather than a retag; see Getting Started > Helm for what that costs. If your mirroring happens in containerd or via an ImageDigestMirrorSet, none of this applies: manifests keep naming the upstream registry and the redirect is invisible to Helm. See Getting Started > Helm for the composition order.

No single global reaches every subchart#

This is the part that looks solved and is not. Helm propagates global into every subchart, but each subchart decides for itself whether to read it, and these do not agree:

Setting only global.imagePullSecrets therefore leaves Loki — the largest pod count in the release — pulling anonymously. It does not fail the render or the install: it surfaces as ImagePullBackOff on the ingesters some minutes after helm install reports success. The pull-secret profile sets all three groups, which is the whole reason it exists as a file rather than as two --set flags.

The matching trap on the registry side is global.imageRegistry. Loki’s image helper coalesces it ahead of every per-component image.registry, so setting it to reach one mirror silently flattens all of Loki’s images — memcached and the canary included — onto that host and overrides whatever else was configured. Set per-component registries instead.

Where a retag is not enough#

Hardened images keep the upstream binary and drop everything around it, which is the point and also the failure mode:

• The UID rarely matches the chart’s. Grafana’s chart writes runAsUser: 472 and Loki’s 10001, while a hardened rebuild runs as whatever the vendor chose. A mismatch renders and schedules cleanly, then crash-loops on a volume the container cannot write. Check the image’s UID and align the relevant securityContext before pointing one at a cluster that holds data.
• No package manager means no start-time plugin installation. See Grafana plugins.
• A vendor outside the two profiled here may not be a retag at all. Both shipped profiles keep the upstream entrypoint, arguments, and layout, which is what makes them a values change. A vendor that rebuilds around its own charts is a port, and the render can fail before you get as far as the UID — a tag that is not plain semver breaks prometheus-node-exporter outright unless node-exporter.version is set alongside it.

The three hook Jobs#

The pre-delete cleanup hook and the two Alloy pre-install validator Jobs are workloads this chart renders itself rather than subchart pods, so they take their pull secrets from both globals directly and need nothing extra set. Each also accepts a local override appended to the globals — cleanup.grafanaOperator.image.pullSecrets, and for the validators the Alloy pods’ own alloy-{agent,gateway}.image.pullSecrets, since they run those images. Names are deduplicated across all of them, so pointing every path at one Secret gives one entry.

The cleanup hook is worth confirming by hand. It runs at pre-delete, so an image it cannot pull does not fail an install you can retry — it hangs helm uninstall on the hook, with the operator still up and the release half-removed. helm uninstall --no-hooks is the way out, and needing it is a worse day than checking first.

Fewer than two zones#

Loki’s ingesters, Thanos Receive, and Alertmanager spread across zones with a hard constraint, so that replication survives losing a zone. For how that constraint is built, see Spreading across zones. It assumes at least two availability zones, which is every managed cloud default and not much else. Below that it fails closed: the stack doesn’t come up, and the cause reads like a chart bug rather than a property of the cluster.

One zone is exactly as broken as none, and that surprises people. When the number of eligible domains is below minDomains, Kubernetes treats the global minimum as 0 rather than skipping the check — so a single zone holding all three replicas computes a skew of 3 against a maxSkew of 1, and every pod stays Pending. Nothing about the symptom points at a zone count. Alertmanager’s zone rule sets no minDomains, so it schedules on a single zone, but with no zone labels both of its replicas stay Pending too.

Zero-zone clusters are not an edge case worth dismissing: kind labels no node this way, and neither do many on-premises distributions. Hand-labelling nodes to satisfy a monitoring chart is the wrong answer, which is why min_zones exists.

Both fixes drop the hard zone rule and keep the soft host rule, which matters most on exactly these clusters — with no zones to lose, the node is the only failure domain there is. On the Terraform path that falls out for free, because min_zones filters the one constraint rather than replacing the list; on the Helm path the profile has to restate the host rule, since Helm overwrites lists rather than merging them.

Nothing reminds you when you gain a zone. The pods keep scheduling happily, spread across nothing, and RF 3 goes on reading like AZ resilience on every dashboard. min_zones is the better habit than the profile for this reason alone: it is a number you update when the cluster changes, not a file you have to remember to stop passing.