Production Best Practices#

Production guidance for the materialize-monitoring stack, organized by backend. Every checklist item is tagged with its primary owner under the shared responsibility model, and is checked ([x]) when the chart already ships it as a default — unchecked items are the deployment-time actions (or still-to-build chart work) that remain.

Today this covers the collection tier (Alloy), the bundled logging backend (Loki), and the bundled metrics backend (Thanos); Grafana and Alertmanager sections will follow the same shape.

Shared responsibility model#

Four parties share responsibility for a production deployment. Checklist items are tagged with the primary owner.

If you install with Terraform#

The Terraform modules are a [consumer] implementation, so several items below are already done when you use them. Those are marked (Terraform: automatic) — read them to understand what is happening, not as work to do.

Satisfied by the modules today:

Still yours, on any install path:

• A usable StorageClass must exist — five workloads are PVC-backed and the modules do not create one. “Usable” is not the same as “present”: on GCP’s C4 and N4 machine families, which accept only Hyperdisk, every StorageClass GKE creates by default is Persistent Disk and none of them will attach. The Terraform modules take storage_class; see Getting Started > Terraform.
• Sizing and retention budgets, node-pool capacity, and the profile choice.
• Basic-auth or mTLS secrets between components, which are not yet wired on any path.
• Everything tagged [operator].

Azure has no wrapper module yet, so an Azure install is a plain [consumer] — every item below applies.

Namespace layout#

Each subchart uses a deterministic name (a static fullnameOverride, e.g. loki, thanos, so no release-name prefix), which assumes one instance of each backend per namespace — a reasonable constraint for an umbrella infrastructure chart. Two layouts are supported; both are fine, so pick per your isolation needs.

Shared namespace (default). Everything deploys into the release namespace (recommended: monitoring). This is the default and the path most installs should take.

Split namespace. One namespace per subchart, via the split-namespace profile (-f charts/materialize-monitoring/profiles/split-namespace.values.yaml), which sets a namespaceOverride per component (loki, thanos, grafana, …). Support is best-effort.

The layout changes the namespace half of every workload-identity binding: an IRSA / GKE / Azure trust-policy subject targets system:serviceaccount:<namespace>:<sa>, where <namespace> is the release namespace under the shared layout and the per-component namespace under split. The ServiceAccount name is unaffected (it’s the deterministic fullnameOverride). See Logs & Events > Storing and Metrics > Storing.

[consumer] selects the layout at install time; [operator] owns the namespace/RBAC policy it implies.

Collection (Alloy)#

The Alloy tier collects and processes telemetry before it reaches a backend. It runs in two roles — the alloy-agent DaemonSet (one per node) and the alloy-gateway Deployment — configured as code (see the logging pipeline reference (internal)). The gateway is where the dominant cost/stability lever lives, so most of the care goes there.

Configuration & change management#

• [chart] Pipelines are authored as code (packages/alloy-pipelines/*.yaml) and rendered to .alloy; the rendered output is committed and CI asserts it matches a fresh render, so config drift is caught at review time.
• [chart] alloy validate runs on every rendered pipeline in the build.
• [chart] Pre-install/pre-upgrade alloy validate hook so a bad config fails the release rather than a running pod.
• [operator] Change pipeline behavior (stages, label families, endpoints) through the YAML sources, never by hand-editing rendered .alloy in a running deployment — edits there are lost on the next render and untracked.
• [consumer] (Terraform: automatic) Restart Alloy after a config change. This is the one place the chart cannot own its own rollout, and it is the reverse of what a chart normally guarantees — see below.

Alloy config changes do not roll the pods. Everywhere else in this chart, changing a value changes a pod template and Kubernetes rolls it. Alloy is the exception, and a helm upgrade that reports success can leave both roles serving the previous configuration indefinitely.

The bundled alloy subchart stamps a checksum/config annotation only when it creates the config ConfigMap itself. This chart renders the pipeline ConfigMaps in the umbrella (mzmon-alloy-{agent,gateway} and their -env pair) and points the subchart at them, so that guard never fires. The parent chart can compute the correct hash — it can read all of .Values and .Files — but a subchart value is static YAML, so there is nowhere to put it that reaches the pod template. Only a consumer can close this, because it holds the values before Helm renders them.

It must be a restart, not a reload. The -env ConfigMaps are consumed with envFrom, and environment variables are fixed at container start — so a metric-filter change (minMetricImportance) is invisible to both Alloy’s /-/reload endpoint and the config-reloader sidecar. Enabling either would silently no-op on that half of the config surface, which is worse than doing nothing.

Installing with the Terraform module needs no action: it stamps mzmon.materialize.cloud/values-hash onto both pod templates, so a config change rolls them and an unchanged apply does not. Installing with Helm directly, restart both roles yourself after any pipeline or filter change:

kubectl -n monitoring rollout restart deployment/alloy-gateway daemonset/alloy-agent

Cardinality & rate control (the lever)#

• [chart] The gateway promotes only a small, stable label set (level, app, container, namespace + their k8s_-prefixed forms, environment_id) and routes everything else identifying to structured metadata. Adding a Loki label multiplies stream cardinality — default to structured metadata.
• [chart] Per-level rate limits keep INFO/unknown chatter bounded while letting ERROR/CRITICAL through; oversized and stale lines are dropped (longer_than, older_than).
• [chart] The agent applies a per-node pod-log rate cap (AGENT_POD_LOG_RATE_LIMIT / AGENT_POD_LOG_BURST) so one noisy node can’t starve the pipeline.
• [operator] Understand the backpressure semantics before tuning: stage.limit with drop = false queues (backpressures the sender), drop = true sheds load. The final safety limit sheds; per-node and per-level limits are the tuning surface.

Gateway availability & delivery#

• [operator] Run the gateway with ≥2 replicas behind its Service — it holds in-memory buffers, so a single replica is a delivery gap during restarts and node churn.
• [operator] Confirm loki.write durability settings (WAL + ret/backoff) survive gateway restarts to your RPO; the write endpoint is set with GATEWAY_LOKI_DEST and the ingress port with ALLOY_LOKI_PORT.
• [consumer] If sending OTLP, target :4317 (gRPC) or :4318 (HTTP); if chaining gateways, point the upstream writer at the downstream :3100. See Collecting.
• [operator] loki.write auth to a secured/remote destination (basic_auth/headers) is not yet wired — provide it before shipping to a destination that requires it.

Agent placement & durability#

• [consumer] Tolerate node taints so the DaemonSet actually lands on every node you want logs from (tainted/spot/system pools included) — a missing toleration is a silent per-node blind spot.
• [operator] Persist the agent’s file positions and journal cursor (hostPath) so a restart resumes where it left off instead of re-tailing (duplicate lines) or skipping (gaps).
• [consumer] Set CLUSTER_NAME on the agent so every line carries a stable cluster label when several clusters share a log store.

Security & meta-monitoring#

• [chart] Distroless Alloy image: FIPS boringcrypto, multi-arch, non-root, GHCR-published.
• [chart] ServiceMonitor/PodMonitor (or GCP PodMonitoring) for both Alloy roles — scrape Alloy’s own component metrics (received/sent bytes, dropped lines, write failures).
• [operator] Alert on gateway write failures and drop counters (drop_counter_reason) so shedding or a broken destination is visible rather than silent data loss.

Logging (Loki)#

For the architecture these items configure, see Logs & Events.

Sizing the logging backend#

Size by throughput and burst, not by stored-bucket size. Bucket size is a derived output (sustained throughput × retention), not an input. Different parts of Loki are sized off different points of the load envelope:

• Ingest path (distributors, ingesters, WAL, ingestion limits) → the 5-minute burst, with headroom to the regression ceiling.
• Storage / retention / bucket → sustained throughput.
• Read path (frontend, queriers, caches) → query load, independent of ingest.

The 5-minute burst typically runs several times the peak-hour rate, so averages must not drive sizing. The three tiers are defined by the ingest envelope:

The regression ceiling is the burst you must degrade gracefully against — typically the volume before Alloy’s cardinality/series reduction is applied. Size the ingest path so a reduction regression throttles via limits rather than crashing the fleet.

Measure to validate#

Size and re-measure off the distributor — the same signal sizing is derived from. [operator]

# sustained (size storage + retention to this)
max_over_time( sum(rate(loki_distributor_bytes_received_total[1h]))[7d:1h] )

# 5-min burst (size ingest path + WAL + limits to this)
max_over_time( sum(rate(loki_distributor_bytes_received_total[5m]))[7d:5m] )

If the 5-minute figure climbs toward the regression ceiling, that is the signal to add ingesters (see §4) before it bites.

Per-size resources#

Starting points — replicas × (cpu request / memory request); tune from real usage. [chart] defaults, [operator]/[consumer] override per profile.

• Ingesters = 3 minimum at every size (the replication_factor 3 floor), and the compactor is always a singleton.
• Scale ingesters past 3 on memory/cardinality, not bytes — with N = RF every ingester holds every stream; run N > RF to shard streams and spread the burst.
• Ingesters are ephemeral (node-local emptyDir, no PVC) — durability is replication_factor 3, not disk. This makes ingesters freely reschedulable and sidesteps EBS zonal pinning / slow volume reattach on node replacement.
• Do not set a tight memory limit on ingesters — an OOM-kill drops in-memory/WAL-buffered logs. Use generous limits (or none) and alert on usage.

Protective limits#

Per tenant (per environment). The aggregate burst is a fleet-capacity concern handled by ingester count, not by these. [operator] sets per profile.

Checklist#

1. Topology & sizing#

• [operator] Select the profile/size from the table above; record the measured sustained + 5-min burst it is based on.
• [chart] Microservice/distributed mode; ingesters ≥ 3; compactor = 1 singleton.
• [operator] Query-scheduler is enabled by default at M/L; the small profile omits it (the query-frontend’s own queue suffices there).
• [chart] Ingester memory limit left unset; requests set for scheduling.

2. Schema & storage#

• [chart] schema_config: TSDB, schema v13, 24h index period.
• [consumer] (Terraform: automatic) Provision the object-storage bucket (S3-compatible / GCS / Azure Blob); single bucket, prefixes /loki/chunks, /loki/ruler.
• [consumer] (Terraform: automatic) Object-store lifecycle policy aligned with (or longer than) Loki retention, so the compactor owns deletion. The modules leave bucket expiry off by default, which is the safe end of that alignment.
• [operator] Treat schema periods as append-only — future format changes go in a new period with a from date ahead of now, never by editing a past period.
• [consumer] (Terraform: automatic) On any backend other than S3, name it in all three load-bearing places — storage.object_store.type, the newest schemaConfig period, and compactor.delete_request_store. The chart’s defaults are S3-shaped, and a stale one crash-loops the component that reads it with no s3 endpoint in config file rather than degrading. A render-time check refuses a mismatched set, and warns on the inert fourth (storage.type); see Selecting the backend.

3. Replication, ring & placement#

• [chart] replication_factor: 3; ring backend = memberlist (no Consul/etcd).
• [chart] Ingester topologySpreadConstraints: hard across zones (DoNotSchedule) so an un-spread pod goes Pending and Karpenter provisions the missing zone; soft across hosts (ScheduleAnyway), with the chart’s default hard per-host anti-affinity dropped by nulling its rule list.
• [operator] Aim for ≥3 zones for true AZ resilience under RF 3; set minDomains to the zone count your node pool can launch in. With 2 zones, know an AZ loss can break write quorum until the ring recovers.
• [operator] If you bring nodes up tainted until DaemonSets are healthy, the spread already sets nodeTaintsPolicy: Honor (tainted nodes stay out of the skew math) — model the taint as a Karpenter startupTaint so it doesn’t over-provision. Taints only gate placement.
• [chart] PodDisruptionBudget on ingesters (maxUnavailable: 1), with a render-time warning if it or the rollout maxUnavailable is set > 1.
• [operator] priorityClassName so ingesters/compactor are not evicted under node pressure.

4. Ingester durability & rollouts#

• [chart] Ingesters are ephemeral — node-local emptyDir, no PVC. Durability is replication_factor 3, so a killed/rescheduled ingester’s un-flushed data is recovered from its peers.
• [chart] Index-gateway and compactor are also ephemeral — their local disk is a read-through cache / idempotent working copy of the object-store index, so they reschedule freely across zones (the compactor singleton in particular is never PVC-pinned to one AZ).
• [chart] flush_on_shutdown: true (best-effort) with a modest terminationGracePeriodSeconds (~60s). Do not rely on a long grace period — enterprise force-kill windows (120/300s) are harmless because replication covers a truncated flush.
• [chart] StatefulSet rolls ingesters one at a time (maxUnavailable: 1); a burst rollout needs zoneAwareReplication (zone-at-a-time) or the alpha MaxUnavailableStatefulSet gate — neither is in play, and PDBs govern drains, not rollout speed.
• [operator] Budget the roll and raise the deploy timeout. The serial, readiness-gated ingester roll takes ~1 min per ingester (so a 6-ingester roll ≈ 5 min) and is not bounded by node provisioning — it overruns Helm’s default 5-min --wait. Set helm upgrade --timeout (or Flux spec.timeout / Pulumi customTimeouts; ArgoCD is async and tolerant). A wait-timeout here means “still rolling,” not “failed.” See Upgrading.
• [operator] Add ingesters (N > RF) when per-ingester memory/stream-count climbs or to spread the regression burst — streams shard across the ring only when N > RF.
• [consumer] (Terraform: storage_class) A dynamic-provisioning StorageClass still needs to exist for the one PVC-backed Loki component, the ruler (it keeps a PVC for its remote-write WAL) — CSI driver installed, not safe to assume on bare clusters, and on GCP C4/N4 no default class is attachable at all.

5. Limits & cardinality#

• [operator] Set per-tenant ingestion_rate_mb / ingestion_burst_size_mb / max_global_streams_per_user from the table; remember these are per environment.
• [chart] reject_old_samples: true + reject_old_samples_max_age set.
• [chart] The Alloy gateway keeps the label set small and routes high-cardinality fields to structured metadata — the dominant cost/stability lever. See Collection (Alloy).
• [operator] Alert on loki_discarded_samples_total so a limit hit is visible.

6. Retention & compaction#

• [chart] Compactor retention_enabled: true, delete_request_store, retention_delete_delay configured.
• [operator] Set the global retention_period to the storage budget.
• [operator] Configure tiered (per-stream) retention — keep ERROR/audit streams long, expire high-volume INFO fast.

7. Caching#

• [chart] Results cache (query-frontend), chunks cache, and index/stats cache on the bundled memcached.
• [chart] Query-result caching enabled (cache_results, max_cache_freshness_per_query).
• [operator] Size memcached per the table.

8. Read path#

• [chart] Query-frontend ≥ 2 for queue fairness; query splitting/parallelism configured.
• [chart] Grafana Loki datasource provisioned, pointing at the query-frontend Service (bundled nginx loki-gateway is off; datasource wiring still to land).
• [operator] Scale queriers/frontends — not ingesters — when dashboards feel slow.

9. Tenancy & auth#

• [chart] One logical tenant per install: auth_enabled: true with a single named X-Scope-OrgID (not the implicit fake tenant), so a future split is config — not a data migration, since the tenant ID is baked into the object-storage path.
• [operator] Isolation is label-based within the tenant (environment_id, …); the hard isolation boundary is the install (per region/stack). Fine for trusted internal consumers — revisit if per-team or customer-facing access is required (then Grafana LBAC, or tenant-per-environment writes + multi-tenant reads).
• [operator] Per-environment controls are label-based, not tenant-based: per-stream retention (environment_id) and per-label rate limits — both static config, which is why no runtime_config live reload is needed.
• [operator] Watch per-ingester memory against total fleet stream count — one tenant concentrates cardinality, bringing the N > RF lever (§4) forward.
• [consumer] Provide the basic-auth (or mTLS) Secret by name; the chart consumes, it does not mint.

10. Meta-monitoring#

• [chart] ServiceMonitor/PodMonitor (or GCP PodMonitoring) for every Loki component.
• [chart] loki-canary enabled for end-to-end write→read verification.
• [chart] Loki mixin dashboards + alerts installed.
• [operator] Tier-0 alerts wired to paging: ingester unhealthy/flush failures, compactor not running, discarded samples, object-store errors, disk usage. Loki down is its own incident.

11. Security & credentials#

• [consumer] (Terraform: automatic on AWS and GCP) Object-store access via workload identity (IRSA / GKE WI / Azure WI) — see Storing > Granting object-storage access for the per-provider setup; static keys only as a documented escape hatch.
• [consumer] (Terraform: automatic on AWS and GCP) No long-lived credentials in the chart; storage secret by reference.
• [chart] runAsNonRoot, read-only root filesystem, dropped capabilities on all components.
• [operator] Optional inter-component TLS where the cluster requires it.

NetworkPolicy egress (if networkPolicy.enabled)#

Enabling the NetworkPolicy denies egress by default except what it explicitly allows. Loki needs external egress that the base policy does not grant, so you must permit it or Loki hangs.

• [chart] Egress to object storage AND the credential endpoint on 443 via networkPolicy.externalStorage (default ports: [443], cidrs: ["0.0.0.0/0"]). Tighten cidrs per environment (see below).
• [operator] The credential path must be covered, not just the bucket: IRSA fetches credentials from AWS STS on 443 (GKE/Azure WI have their own token endpoints). Missing STS egress is the classic failure — the compactor blocks at startup fetching credentials for the delete-requests store, its HTTP server never serves, the liveness probe kills it every ~5 min, and it crashloops with misleading memberlist … WriteTo … i/o timeout noise. The block is silent (a hanging TCP connect), so it looks like anything but a network policy.
• [operator] Do not rely on an ambient broad-443 egress rule from another workload (e.g. an application-level “reach the external kube API server” rule). It won’t select the Loki namespace, and even where it does it’s a load-bearing coincidence: the day someone scopes that rule down, Loki breaks. Declare Loki’s egress explicitly.
• [operator] Tighten for production: prefer VPC endpoints (PrivateLink) for S3 + STS and scope cidrs to the VPC / endpoint CIDRs — 0.0.0.0/0:443 is a reasonable default only when egress is already governed at the infra layer (SGs, NAT, egress firewall). On Cilium, FQDN egress (toFQDNs) to the S3/STS hostnames is a good tight-but-not-brittle alternative.

12. Day-2: upgrades, migration, DR#

• [operator] Upgrade ingesters one-at-a-time with flush (see §4).
• [operator] Schema changes = new period, never in place (see §2).
• [consumer]/[operator] DR = object versioning + cross-region replication + (for audit) Object Lock/WORM; restore = repoint at the bucket. No native snapshot — see Storing. The Terraform modules enable versioning by default; replication and Object Lock remain yours.
• [consumer] (Terraform: automatic) Pin the Loki chart/image version; upgrades are deliberate.

13. Validation#

• [chart] loki -verify-config runs in CI and as an initContainer before a component serves.
• [chart] helm template | kubeconform + helm lint in CI.
• [chart] Smallest integration profile = single-binary + filesystem (no object store) for hermetic e2e tests.

See also#

• Logs & Events — the logging architecture these items configure.
• Storing — storage, retention, and disaster recovery in depth.
• Upgrading — cross-cutting upgrade guidance.
• Loki production deployment (official) — Grafana’s reference production config (built for far larger volumes; read it for the patterns, not the magnitudes).

Metrics (Thanos)#

For the architecture these items configure, see Metrics.

Thanos is less finished than Loki in this chart, and the checklist says so honestly: there are no sizing profiles, no resource requests, and no topology spread on any Thanos component today. Treat the unchecked [chart] items below as the current work list rather than as guidance you are expected to satisfy by hand.

Receive: replication is the availability lever, not mode#

The single most consequential setting, and the one most often misread.

receive.mode is a topology choice:

• standalone (default) is RouterIngestor mode — one workload that both routes and ingests. It still builds a ketama hashring across the StatefulSet pods (receive.hashrings.autogen), so replicaCount: 3 shards writes across three pods.
• split separates Router (Deployment) from Ingester (StatefulSet) so they scale independently.

Neither choice, by itself, gives you redundancy. The replication factor does, and the chart passes --receive.replication-factor only in split mode — so standalone runs at Thanos’s default of 1 unless receive.extraArgs sets it.

Thanos write quorum is (rf / 2) + 1:

Use an odd factor. A factor of 2 requires both copies and tolerates nothing — the same as 1, and on a small ring worse: on three pods, losing one fails ~1/3 of series at RF 1 but ~2/3 at RF 2, because more series depend on any given pod while quorum still demands all of their copies. Even factors above 2 tolerate no more losses than the odd factor below them; the extra copy buys durability, not availability.

thanos:
  receive:
    replicaCount: 3
    extraArgs:
      - --receive.replication-factor=3

Note this differs from Loki, where ingesters are deliberately ephemeral and durability comes from RF 3 alone. Thanos Receive is PVC-backed (10Gi RWO per pod by default) and therefore AZ-pinned, and it holds up to receive.tsdb.retention (24h) of not-yet-uploaded blocks. With RF 1 that window exists in exactly one copy, on one volume, in one zone.

Split mode is not recommended yet. It landed upstream only recently, and receive.ingester does not inherit from the top-level receive.* defaults — upstream considers the non-merging behavior intentional, so this is unlikely to change. In practice that means restating ~31 keys, of which the values schema hard-requires eight sub-objects (hashrings, service, vpa, persistence, podSecurityContext, probes, serviceMonitor, pdb) before the chart will render at all. Prefer standalone with an odd replication factor until that changes.

Checklist#

1. Ingestion topology & replication#

• [chart] receive.mode: standalone with replicaCount: 3 and an auto-generated ketama hashring.
• [operator] Set an odd --receive.replication-factor (3) via receive.extraArgs; the chart cannot set it in standalone mode. A render-time check warns at factor 1, warns harder at 2, and errors when the factor exceeds replicaCount.
• [operator] Keep replicaCount >= replicationFactor. At replicaCount == replicationFactor every pod holds every series — good availability, no horizontal capacity.
• [chart] PodDisruptionBudgets on every component (thanos.global.pdb, maxUnavailable: 1) — matching the Loki ingester convention. maxUnavailable rather than minAvailable deliberately: it scales with the replica count, and on the single-replica Compactor minAvailable: 1 would permit no eviction at all and hang node drains. A validator errors when the Receive budget exceeds what write quorum tolerates, and warns on minAvailable for the singleton.
• [chart] topologySpreadConstraints across zones for Receive, so RF 3 actually survives an AZ loss rather than landing three copies in one zone. Receive is PVC-backed and therefore AZ-pinned, so this matters more here than for Loki’s ephemeral ingesters.
• [operator] priorityClassName so Receive and Compactor are not evicted under node pressure.

2. Object storage & credentials#

• [chart] Objstore config rendered into a Secret (global.objstore.createSecret), consumed by every component.
• [consumer] (Terraform: automatic on AWS and GCP) Supply the bucket and grant access by workload identity (IRSA / GKE Workload Identity / Azure Workload Identity) rather than static keys. A render-time check errors when the identity annotation names a different cloud than the objstore backend, and warns when a cloud backend has neither an annotation nor inline credentials.
• [consumer] (Terraform: storage_class) A dynamic-provisioning StorageClass must exist and be attachable by the nodes these land on — Receive, Store Gateway, and Compactor are all PVC-backed. On GCP C4/N4 that rules out every default class; see Getting Started > Terraform.

3. Components & read path#

• [chart] Query, Receive, Store Gateway, and Compactor enabled by default.
• [operator] Enable Query Frontend for production read paths (splitting and result caching) — and repoint connections.datasources.thanos.url at it, or the cache is deployed and bypassed. A render-time check warns on exactly that mismatch.
• [operator] Store Gateway is how queries reach historical blocks; disabling it limits reads to what Receive still holds locally.
• [chart] Horizontal autoscaling on Query (2–5 replicas, 80% CPU), and on Query Frontend once it is enabled — both are stateless, with no ring membership or local state. Store Gateway autoscaling is deliberately off: it is a PVC-backed StatefulSet that syncs the bucket index on startup, so scale-up serves nothing until it is warm, and scale-down orphans PVCs.
• [operator] Keep replicaCount equal to autoscaling.minReplicas. The subchart templates a static replicas even alongside an HPA, so every upgrade or GitOps reconcile writes it back — matching the floor makes that reset a no-op instead of a scale blip. A validator warns when the two disagree.

4. Retention & compaction#

• [chart] Compactor enabled with downsampling retention: raw 30d, 5m 90d, 1h 365d.
• [operator] Set those to your storage budget. Retention is enforced by the Compactor — with it disabled nothing expires and bucket cost grows without bound.
• [chart] Receive TSDB WAL retention 24h with compression, so blocks ship to object storage promptly.

5. Sizing#

• [chart] No resource requests or limits are set on any Thanos component. Sizing profiles (thanos-small / thanos-large, mirroring the Loki convention where the chart defaults are medium) are outstanding work.
• [operator] Until they land, set requests explicitly. Thanos sizes off different axes than Loki — active series and samples/sec for Receive, block volume and retention for Store Gateway and Compactor, query concurrency for Query and Query Frontend.
• [operator] Autoscaling on Query does not remove the need for requests: without them the HPA has no CPU target to measure against, so it never scales.

6. Meta-monitoring#

• [chart] ServiceMonitors for every Thanos component (thanos.global.serviceMonitor).
• [operator] Alert on Receive write failures and quorum errors — with RF 1 or 2 these are the first sign of a lost pod, and they are silent from the dashboards’ point of view.

7. Validation#

• [chart] Render-time validators cover objstore placeholders, backend/identity mismatch, component topology, replication-factor quorum, and writers or datasources aimed at a Thanos that is not deployed.
• [chart] helm template | kubeconform in CI (shared with the Loki checklist).

See also#

• Metrics — the metrics architecture these items configure.
• Storing — object storage and retention in depth.
• Thanos Receive documentation (official) — hashring, replication, and quorum semantics.