Production checklist#

This checklist is for you if you install or operate the materialize-monitoring stack in production. It covers the collection tier (Alloy), node metrics (node-exporter), and the bundled Loki, Thanos, Grafana, and Alertmanager.

The checklist is ordered by what goes wrong if you skip an item, most severe first:

Exposure

Someone outside the trust boundary can read your data or change your configuration.

Silent failure

Part of the stack stops working without an error, so nothing tells you.

Data loss

An ordinary restart, node loss, or upgrade loses telemetry or state for good.

Outage

Part of the stack fails to install, upgrade, or stay up. These failures are loud.

Capacity and cost

The stack runs short of resources, or costs more than it needs to.

Hardening and extras

Nothing breaks. Each item adds defense in depth, or extra signal where your environment calls for it.

Each item is tagged with its owner under the shared responsibility model, and names the component it touches. The component links to the page that explains the setting. Open items are yours to act on. What the chart already ships is checked, and collapsed at the end of each tier.

For the trust boundaries, the cluster permissions the stack holds, and where credentials live, see Securing.

Shared responsibility model#

Four parties share responsibility for a production deployment. Checklist items are tagged with the primary owner:

The following table shows how the parties split each area:

If you install with Terraform#

The Terraform modules are a [consumer] implementation, so several items are already done when you use them. Those are marked (Terraform: automatic); read them to understand what’s happening, not as work to do.

The modules satisfy the following:

The following stay yours on any install path:

• A usable StorageClass, for the four PVC-backed workloads: Alertmanager, the Loki ruler, and the Thanos Store Gateway and Compactor. The modules don’t create one. For what “usable” means on GCP, see the StorageClass item under Outage.
• Sizing and retention budgets, node-pool capacity, and the profile choice.
• In-cluster TLS between components. Certificates and the mTLS profiles ship off by default on both paths; for which hops each phase covers, see Securing > What is not there yet.
• Everything tagged [operator].

On Azure, the module turns on no blob versioning, so the disaster-recovery item under Data loss stays yours there in full.

Exposure#

Skip one of these and someone outside the trust boundary can read your data or change your configuration.

• [operator] Network policies: Confirm your CNI enforces NetworkPolicy. Cilium and Calico do. kindnet, the default in kind and therefore in the E2E tiers, ignores NetworkPolicy entirely, so a green E2E run isn’t evidence that any of this is enforced.
• [operator] node-exporter: Confirm port 9100 isn’t reachable from outside the cluster. The pods are host-networked, which puts them outside pod NetworkPolicy on most CNIs, so this is a security-group or firewall question, not a Kubernetes one.
• [consumer] Object storage: (Terraform: automatic) Grant Loki and Thanos bucket access by workload identity (IRSA, GKE Workload Identity, or Azure Workload Identity) rather than static keys, and supply any storage secret by reference. Static keys are a documented escape hatch only. For Thanos, see Metrics > Storing.
• [operator] Loki: Know that isolation within the Loki tenant is label-based. The hard isolation boundary is the install. That’s fine for trusted internal consumers; per-team or customer-facing access needs Grafana label-based access control or tenant-per-environment writes.
• [operator] Grafana: Configure an identity provider under grafana.ini before exposing Grafana to anyone but yourself. Any auth.* section Grafana supports works. A render-time check warns when an exposed Grafana has none.
• [consumer] Grafana: Provide the OAuth client secret as a Secret, referenced with $__file{} or $__env{}. Never put it in grafana.ini as a literal, because that renders into a ConfigMap.
• [operator] Grafana: Map an IdP group claim onto Grafana roles with role_attribute_path, so group membership does the provisioning and nobody is added by hand. After it’s right, set role_attribute_strict: true, so a broken claim fails the login instead of quietly falling back to Viewer.
• [operator] Grafana: Know that Grafana’s permissions aren’t a data boundary. Every data source is queryable by anyone who can reach it, so a Viewer in Grafana still reads every metric in Thanos and every log in the tenant.
• [consumer] Grafana: Choose the protocol layer deliberately. A LoadBalancer Service gives you L4 on every cloud and terminates no TLS. An Ingress gives you L7, terminates TLS, and needs a controller you might not have. L7 is what a public Grafana wants, because a WAF and edge authentication are the two things L4 can’t do, but L4 has no request-timeout ceiling, which matters for the long Thanos and Loki queries a panel makes. The grafana-ingress profile is the L7 shape; the Terraform wrappers use the Service.
• [consumer] Grafana: Terminate TLS in front of Grafana. It authenticates with a session cookie, so without TLS that cookie and the admin password cross the network in the clear. Use either an Ingress tls block or termination at the load balancer against a cloud-held certificate (ACM on AWS, a managed certificate on GKE). A render-time check warns on an Ingress with no tls; it can’t see the second case, so confirm rather than dismiss it. A bare LoadBalancer Service with no controller in front is the one shape with no TLS anywhere.
• [operator] Grafana: Supply the certificate or the issuer. The consumer wires it; where the trust comes from, such as a cert-manager ClusterIssuer, an ACM ARN, or an uploaded certificate, is a policy decision.
• [consumer] Grafana: Set security.cookie_secure: true after TLS is in place. Before that, the session cookie is never sent and nobody can log in.
• [operator] Grafana: Rotate the generated admin password, or supply your own with grafana.admin.existingSecret. Grafana reads it once at startup, so a rotation needs a restart.
• [operator] Grafana: Never inline the database password into grafana.ini, because it renders into a ConfigMap. Use $__file{} against a mounted Secret, or $__env{} against envValueFrom. The subchart’s assertNoLeakedSecrets check fails the render if you forget; leave it on.
• [operator] Alertmanager: Don’t expose Alertmanager without authentication in front of it. It has none of its own, and its API creates silences. Grafana’s Alertmanager data source gives you the alert list and the silence editor behind Grafana’s authentication.
• [consumer] Alertmanager: Publish to Amazon SNS with workload identity, not keys: an IRSA role or EKS Pod Identity association for the alertmanager ServiceAccount, allowing sns:Publish on the topic (plus kms:GenerateDataKey* and kms:Decrypt for an encrypted topic). SNS is the only integration that can use one; SES and Azure Communication Services email authenticate over SMTP with a credential in the Secret.

Silent failure#

Skip one of these and part of the stack stops working without an error, so nothing tells you.

• [consumer] Alloy: (Terraform: automatic) Restart both Alloy roles after any pipeline or filter change. A config change doesn’t roll the pods, and a reload doesn’t pick up the filter half of the config. This is the one place the chart can’t own its own rollout.
• [operator] Alloy: Change pipeline behavior (stages, label families, endpoints) through the YAML sources, never by hand-editing rendered .alloy in a running deployment. Edits there are untracked and lost on the next render.
• [operator] Alloy: Alert on the pipeline’s own shedding (drop_counter_reason), so a rate limit that’s dropping lines is visible rather than silent data loss. The shipped alloy-log-drops alert covers only entries loki.write fails to deliver.
• [operator] Alloy and node-exporter: Verify coverage after any node-pool change. The agent’s pod count and count(up{job="node-exporter"}) should both equal count(kube_node_info). A persistent gap is a pool the DaemonSets aren’t reaching.
• [operator] Alloy and node-exporter: If you use NoExecute taints for routine pool separation, add a toleration or lose those nodes. Neither DaemonSet tolerates them by default. Add the agent’s through the Terraform module’s tolerations, which appends rather than replaces.
• [operator] Alloy and node-exporter: Record every pool you exclude. Excluding either DaemonSet from a pool, or narrowing its tolerations, is a deliberate blind spot, not a saving. Do it only where the per-node budget can’t absorb the pod, about 64Mi for node-exporter.
• [consumer] Namespaces: Under the split-namespace layout, add namespaceSelector entries to the network policies. A podSelector without a namespaceSelector doesn’t cross a namespace boundary, so the pod-label rules match nothing and the traffic they allow is dropped, with no error. Add them through networkPolicies.<component>.ingress.extra and egress.extra, and the subcharts’ own ingress lists, such as node-exporter.networkPolicy.ingress; or turn the affected policies off. The render warns when it sees the combination.
• [consumer] Cluster name: Give every cluster its own name: cluster_name on Terraform, clusterName on Helm. Every log line carries it as cluster, and the default alert group_by includes it. The default, default, distinguishes nothing once two clusters share a log store, a channel, or an incident tool. Two clusters sending one condition to one PagerDuty service then share an incident and resolve each other’s.
• [operator] node-exporter: If your cluster already runs a node-exporter, turn this one off with node-exporter.enabled: false, or install_node_exporter = false on Terraform. Running both double-counts silently in any sum() or rate().
• [operator] node-exporter: Alert on collectors failing, not only on the exporter being up. node_scrape_collector_success == 0 means that collector returned nothing while the scrape as a whole succeeded, so up stays green and the metrics never appear. Some failures are expected and worth allowlisting rather than chasing: hwmon is usually empty on cloud VMs, nvme needs NVMe devices, and kernel_hung needs Linux 6.13+ for /proc/sys/kernel/hung_task_detect_count.
• [operator] node-exporter: schedstat has two kernel gates and fails quietly on the second. /proc/schedstat needs CONFIG_SCHEDSTATS, and since Linux 4.6 the counters are also off at runtime unless kernel.sched_schedstats=1. A missing file shows up in node_scrape_collector_success; the sysctl being off doesn’t, because the file reads as zeros.
• [operator] Loki: Wire tier-0 alerts to paging: ingester unhealthy or flush failures, compactor not running, discarded samples (loki_discarded_samples_total), object-store errors, and disk usage. Loki down is its own incident.
• [operator] Thanos: Alert on Receive write failures and quorum errors. With a replication factor of 1 or 2, these are the first sign of a lost pod, and the dashboards don’t show it.
• [operator] Thanos: Keep the Store Gateway enabled. It’s how queries reach historical blocks; without it, reads are limited to what Receive holds locally.
• [operator] Grafana: Set grafana.ini.server.root_url to the URL users actually reach. Share links, alert notification links, and OAuth redirect URIs are all built from it, and all three break silently when it disagrees with the host. A render-time check covers it.
• [operator] Grafana: A hardened Grafana can’t install plugins at start, so grafana.plugins silently gets you a Grafana without them. Bake them into a derived image instead.
• [operator] Grafana: Alert on Grafana being down. It isn’t a data-loss incident, but it’s the interface every other alert is investigated through.
• [operator] Alertmanager: Configure at least one receiver. Until one exists, every alert reaches mzmon-null, and the render warns.
• [operator] Alertmanager: Choose alerting.preset deliberately. important is the default because it errs in the least damaging direction. A deployment for which Materialize is critical infrastructure wants critical-infrastructure, which routes critical to page. A preset of your own is one more key under alerting.presets.
• [operator] Alertmanager: Test a receiver end to end after configuring it, with amtool alert add from inside a pod. The render proves the configuration is well-formed, not that a vendor accepts the notification.
• [consumer] Alertmanager: Provision the alertmanager-receivers Secret in the namespace the Alertmanager pods run in, with External Secrets Operator, Vault Agent, SOPS, or a CSI driver. The chart consumes it by name and never creates it. It’s mounted as optional, so a missing Secret shows up as failed notifications, not as a pod that won’t start. Rotating a credential in it needs no restart: Alertmanager reads a *_file credential on each send, and the kubelet refreshes the mount within about a minute.
• [operator] Alertmanager: Leave alertmanager.baseURL empty unless Alertmanager itself is exposed, and then set it to that address. Never point it at Grafana, which doesn’t serve Alertmanager’s UI paths, so every link in a notification breaks. The render warns when the two share a host.
• [operator] Alertmanager: Under split-namespace, the profile opens Alertmanager to the ruler and Grafana namespaces. A receiver listening on a port other than 443, 587, or 465 needs that port added to networkPolicies.alertmanager.egress.external.ports.
• [operator] Alertmanager: Add meta-alerts on the notifier. The shipped rule set has none on alertmanager_cluster_members, alertmanager_config_last_reload_successful, or alertmanager_notifications_failed_total, so a notifier that can’t deliver is visible on a dashboard and pages nobody.
• [operator] Alertmanager: Route a deadman’s switch to something outside the cluster. Stopped evaluation looks like nothing being wrong, and only an external heartbeat monitor can tell the two apart.

Data loss#

Skip one of these and an ordinary restart, node loss, or upgrade loses telemetry or state for good.

• [operator] Thanos: Keep the Receive replication factor odd, and restate receive.extraArgs in full whenever you set it. The chart sets the factor to 3 through that list, and Helm overwrites lists, so an override that adds any flag drops it to Thanos’s default of 1. At 1, the un-uploaded window exists in one copy on one node. A render-time check warns at factor 1, warns harder at 2, and errors when the factor exceeds replicaCount.
• [operator] Thanos: Restate compactor.extraArgs in full too. It carries the vertical compaction and deduplication flags, and dropping the first halts the Compactor at the first overlapping block.
• [operator] Thanos: Know that deduplication is irreversible on an existing bucket. Downsampled blocks written before it was turned on can’t be vertically compacted, and keep their per-replica copies until retention expires them.
• [operator] Alloy: Confirm the loki.write durability settings (WAL, retry, and backoff) survive gateway restarts to your RPO.
• [operator] Alloy: Persist the agent’s file positions and journal cursor on a hostPath, so a restart resumes where it left off instead of re-tailing (duplicate lines) or skipping (gaps).
• [consumer] Loki: (Terraform: automatic) Align the object-store lifecycle policy with Loki’s retention, or make it longer, so the compactor owns deletion. The modules leave bucket expiry off by default, which is the safe end of that alignment.
• [operator] Loki: Treat schema periods as append-only. A future format change goes in a new period with a from date ahead of now, never in an edit to a past period.
• [consumer] [operator] Loki: Plan disaster recovery as object versioning, cross-region replication, and, for audit, Object Lock (WORM). Restoring means repointing at the bucket; there’s no native snapshot. The Terraform modules enable versioning by default on AWS and GCP, but not on Azure. Replication and Object Lock remain yours on every cloud.
• [operator] Grafana: Move Grafana’s state off emptyDir before anyone depends on it. Use the grafana-postgres profile for production, or grafana-pvc for a single small instance with no database available.
• [operator] Grafana: Back the database up. It holds every service-account token, alert rule, and annotation. Nothing in this chart replicates it, and a PVC snapshot is the only copy on the grafana-pvc path.
• [operator] Grafana: Switching an existing install from SQLite to PostgreSQL doesn’t carry state over, because Grafana has no migration between them. Export what matters through the HTTP API first.
• [operator] Grafana: Treat the running Grafana as a cache, not a source. Dashboards edited in the UI are overwritten at the next resync; change them in packages/dashboards/ instead.

Outage#

Skip one of these and part of the stack fails to install, upgrade, or stay up. These failures are loud, but they can read like chart bugs.

• [consumer] StorageClass: (Terraform: storage_class) A usable StorageClass must exist for the PVC-backed workloads: Alertmanager, the Loki ruler (for its remote-write WAL), and the Thanos Store Gateway and Compactor. “Usable” isn’t the same as “present”. On GCP’s C4 and N4 machine families, which accept only Hyperdisk, every StorageClass GKE creates by default is Persistent Disk, and none of them attaches. Use volumeBindingMode: WaitForFirstConsumer, the default for managed-cloud CSI classes; under Immediate, a PVC’s zone is chosen before scheduling, and the hard zone rules leave pods Pending. On GKE, the legacy standard class is Immediate.
• [operator] Zones: Tell the chart how many zones you have. The default assumes two or more, and below that it fails closed: Loki’s ingesters and Thanos Receive stay Pending, and on a cluster with no zone labels so does Alertmanager. On Terraform, set min_zones; on Helm, apply the no-zone-spread profile.
• [operator] Zones: Aim for three or more zones, and raise minDomains to the count your node pools can launch in. The chart’s floor is 2, so with three zones a two-zone placement still satisfies it. Setting it above your real count leaves pods Pending forever. With two zones, an AZ loss can break Loki and Thanos write quorum (2 of 3) until the ring recovers.
• [operator] Thanos: Check Receive’s ephemeral-storage request against your nodes’ allocatable ephemeral storage, not their disk size, before you pick a profile or raise Receive’s local retention. A request no node shape fits never schedules, and the cluster autoscaler doesn’t rescue it.
• [consumer] Images: Create the pull Secret in every namespace the stack renders into. Helm doesn’t create it, and a Secret is namespaced, so a split-namespace install needs one copy per namespace, not one for the release.
• [operator] Images: Verify the vendor’s tag strings before install. Hardened-image vendors track upstream versions without always spelling them identically: a leading v is commonly dropped, and a DHI mirror carries only the tags you chose to mirror. The profiles inherit the chart’s tags rather than restating them, so add an explicit tag for any image whose string differs.
• [operator] Images: Alloy is a rebuild, not a retag. The chart runs a Materialize build of Alloy carrying the custom components the pipelines are written against, and a stock vendor Alloy fails config validation at startup. A mirror carries it fine; the mirror profile moves it along with everything else, digest intact.
• [operator] Images: Confirm the pre-delete cleanup hook can pull its image. An image it can’t pull hangs helm uninstall with the release half-removed, and helm uninstall --no-hooks is the way out.
• [operator] Scheduling: PriorityClasses are cluster-scoped, so two releases of this chart in one cluster fight over them. Set priorityClasses.create: false on all but one, or rename them per release with priorityClasses.critical.name and priorityClasses.scalable.name.
• [operator] Scheduling: If you rename or disable the PriorityClasses, update the subcharts’ priorityClassName values to match. A priorityClassName naming a class that doesn’t exist doesn’t degrade: the API server rejects the pod, and the only evidence is an admission error on a ReplicaSet nobody is watching. A render-time check warns when it can tell.
• [operator] Scheduling: If your platform already defines a priority scheme, point the subchart values at your own classes instead. metrics-server stays on the upstream system-cluster-critical deliberately: it backs the metrics API that HPAs and the Materialize Console read, so it’s cluster plumbing rather than monitoring.
• [operator] Network policies: Nothing here allows the kubelet’s liveness and readiness probes, which arrive from the node’s own IP. Cilium and Calico both exempt host-to-pod traffic, so probes keep working. A CNI that doesn’t would fail every pod in the namespace at once, which is the symptom to recognize rather than a per-component bug.
• [consumer] Version pinning: (Terraform: automatic) Pin the chart version, and with it every subchart and image, so upgrades are deliberate.
• [consumer] Loki: (Terraform: automatic) Provision the object-storage bucket (S3-compatible, GCS, or Azure Blob): a single bucket, with prefixes /loki/chunks and /loki/ruler.
• [consumer] Loki: (Terraform: automatic) On any backend other than S3, name it in all three load-bearing places: storage.object_store.type, the newest schemaConfig period, and compactor.delete_request_store. A stale one crash-loops the component that reads it.
• [operator] Loki: Cover the credential endpoint as well as the bucket in Loki’s egress, and declare it explicitly rather than relying on another workload’s broad 443 rule. Missing STS egress crash-loops the compactor with misleading memberlist timeouts.
• [operator] Loki: Budget the ingester roll and raise the deploy timeout. The serial roll takes about 1 minute per ingester and overruns Helm’s default 5-minute --wait. Set helm upgrade --timeout, Flux spec.timeout, or Pulumi customTimeouts; ArgoCD is async and tolerant. For more information, see Upgrading.
• [consumer] Grafana: Provision the database and an owning user. Grafana runs its own schema migrations at startup, so a read/write-only grant fails the migration.
• [consumer] Grafana: Provide the database password as a Secret in the namespace the Grafana pod runs in. Under split-namespace, that’s grafana, not the release namespace. The chart consumes it by name and doesn’t mint it.
• [operator] Grafana: Create the DNS record. Neither the Ingress nor the Service publishes the hostname, and the chart can’t, because it has no view of your zone. Unless something like external-dns already reconciles records in the cluster, this is a manual step that’s easy to forget until the certificate fails to issue.
• [operator] Grafana: Keep the local admin as break-glass. disable_login_form: true still leaves /login?disableAutoLogin reachable; know that before an IdP outage takes your incident tooling with it.
• [operator] Grafana: Don’t update a bundled data source plugin from Grafana’s plugin catalog. On the read-only root filesystem it leaves the data source unloaded until the pod restarts.
• [operator] Grafana: Where Grafana can’t reach grafana.com, set grafana.ini.plugins.preinstall_disabled: true, which removes the Drilldown apps’ download at every start.
• [operator] Alertmanager: Treat persistence.enabled and persistence.size as immutable. An upgrade that changes either fails until the StatefulSet is deleted with --cascade=orphan.
• [operator] Alertmanager: Know the recovery for a replica whose zone is lost: delete its PVC, then its pod, so it reschedules into a healthy zone and resyncs from its peer.

Capacity and cost#

Skip one of these and the stack runs short of resources, or costs more than it needs to.

• [operator] Loki: Pick the Loki size from the throughput table, and record the measured sustained rate and 5-minute burst it’s based on.
• [operator] Thanos: Pick the Thanos profile from inventory, and record the inventory it’s based on. thanos-small and thanos-large are deltas from the chart defaults, which target medium.
• [operator] Thanos: Re-measure head series against the envelope after install, and after any change in collection count.
• [operator] Thanos: To cut metrics cost, tighten minMetricImportance, not the profile. Downshift only after head series have fallen.
• [operator] Loki: Set the per-tenant ingestion_rate_mb, ingestion_burst_size_mb, and max_global_streams_per_user from the table. These apply per environment.
• [operator] Loki and Thanos: Set retention to your storage budget: Loki’s global retention_period, and Thanos’s per-resolution retention.
• [operator] Loki: Configure tiered (per-stream) retention: keep ERROR and audit streams long, and expire high-volume INFO fast.
• [operator] Thanos: Keep raw retention above the downsampling thresholds (40h for the 5m tier, 10d for the 1h tier). Below them the tier is never produced, and long-range queries silently fall back to raw blocks.
• [operator] Loki: Add ingesters (N > RF) when per-ingester memory or stream count climbs, or to spread the regression burst. One tenant concentrates cardinality, so watch per-ingester memory against the total fleet stream count.
• [operator] Loki: Size memcached per the table.
• [operator] Loki: Scale queriers and frontends, not ingesters, when dashboards feel slow.
• [operator] Loki: If you bring nodes up tainted until DaemonSets are healthy, model the taint as a Karpenter startupTaint so it doesn’t over-provision. The ingester spread already sets nodeTaintsPolicy: Honor, so tainted nodes stay out of the skew math.
• [operator] Alloy: Size gateway memory vertically, from the series each pod holds, rather than adding memory to the HPA.
• [operator] Alloy: Understand the backpressure semantics before tuning the rate limits: drop = false queues, drop = true sheds.
• [operator] node-exporter: Don’t add a CPU limit to node-exporter. It turns each scrape’s burst into CFS throttling, which surfaces as scrape timeouts on the loaded nodes whose samples matter most.
• [operator] Thanos: Keep replicaCount at or above the replication factor. At equality every pod holds every series, which is good availability but no horizontal capacity.
• [operator] Thanos: Enable Query Frontend for production read paths (splitting and result caching), and repoint connections.datasources.thanos.url at it, or the cache is deployed and bypassed. A render-time check warns on that mismatch. The thanos-large profile does both; at S and M, Query is the data source target.
• [operator] Thanos: Keep Query’s replicaCount equal to autoscaling.minReplicas. The subchart templates a static replicas alongside the HPA, so every upgrade or GitOps reconcile writes it back; matching the floor makes that reset a no-op. A validator warns when the two disagree.
• [operator] Thanos: Keep resource requests on anything you autoscale. Without a request the HPA has no CPU target to measure against, so it never scales.

Hardening and extras#

Skip one of these and nothing breaks. Each adds defense in depth, or extra signal where your environment calls for it.

• [operator] Network policies: Narrow the 0.0.0.0/0 egress rules where your infrastructure lets you. kube-state-metrics and grafana-operator reach the API server on 443 and 6443 to everywhere, because a chart can’t derive the control plane’s address, and Loki reaches object storage the same way. A VPC endpoint’s CIDR, or Cilium’s toFQDNs, is the tighter form; for Loki, see Object-storage egress.
• [operator] TLS: Turn on in-cluster TLS where your environment requires it. Certificates and the mTLS profiles ship off by default.
• [operator] Alertmanager: TLS on the API port comes with the mTLS profiles: alerting.server.tls moves 9093 to TLS, and the profiles move the probes, the reloader, the scrape, both rulers, and the Grafana data source with it. VerifyClientCertIfGiven at phase 2 is as far as it goes, because the kubelet’s probes and the reloader can’t present a certificate.
• [operator] node-exporter: TLS client authentication on /metrics is available and parked (kubeRBACProxy.tls.tlsClientAuth plus tlsSecret). Turn it on if your environment requires authenticated scrapes, accepting a second container on every node.
• [operator] Images: Mirror at the node if you can. A containerd hosts.toml or an ImageDigestMirrorSet redirects pulls without touching any chart value, which keeps one mapping in one place. Reach for mirror.values.yaml when you can’t configure nodes, or when policy wants the mirror named in the manifest.
• [operator] Grafana: Pin plugin versions (name@version) or bake them in, so a plugin can’t change underneath a pinned Grafana. A validator warns on an unpinned entry.
• [operator] Thanos: If you re-enable VPA deliberately, use updateMode: "Off" for recommendations only, and know that its numbers disagree with the profile by design.
• [operator] node-exporter: On AWS, consider enabling ethtool with --collector.ethtool.device-include=^(eth|ens|enp). It’s where the ENA allowance counters live (bw_in_allowance_exceeded, bw_out_allowance_exceeded, pps_allowance_exceeded, conntrack_allowance_exceeded): instance-level network throttling that’s invisible in every other metric. It’s off by default because the stat set is driver-specific, so validate it on your AMI before relying on it.
• [operator] node-exporter: On bare metal, edac (ECC errors) and rapl are worth adding; hypervisors don’t expose either to guests, which is why they’re off.
• [operator] node-exporter: Turn slabinfo on during an investigation, not as a default. It costs about 1.5k series per node and needs a root init container to chmod /proc/slabinfo, and meminfo already reports Slab, SReclaimable, and SUnreclaim.