Upgrading materialize-monitoring#

StatefulSet fields that cannot be patched#

Most of a StatefulSet is mutable, but volumeClaimTemplates and selector are not. A release that changes either — a component gaining, losing, or resizing its volume — cannot be applied by helm upgrade, which fails partway with:

cannot patch "thanos-compactor" with kind StatefulSet: StatefulSet.apps
"thanos-compactor" is invalid: spec: Forbidden: updates to statefulset spec for
fields other than 'replicas', 'ordinals', 'template', 'updateStrategy',
'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Nothing in the chart can smooth this over — it is a Kubernetes API constraint, not a Helm one — so the StatefulSet has to be deleted and recreated by the upgrade.

The two Thanos storage changes in chart v0.16.0 require this, and only for installs upgrading across it:

# Compactor: local state is scratch, so recreating the pod costs nothing but a
# restarted compaction.
kubectl -n monitoring delete statefulset thanos-compactor
helm upgrade mzmon ... # recreates it with the volumeClaimTemplate

# Receive: the same, plus cleaning up volumes nothing will reclaim. Do this one
# pod at a time if you want to preserve write quorum through the change.
kubectl -n monitoring delete statefulset thanos-receive
helm upgrade mzmon ...
kubectl -n monitoring delete pvc -l app.kubernetes.io/component=receive

Deleting a StatefulSet does not delete its PVCs. The default persistentVolumeClaimRetentionPolicy is Retain, so volumes created from volumeClaimTemplates outlive it. That cuts both ways, and both directions matter here:

• Recreating a StatefulSet rebinds the existing PVCs by name (data-<sts>-0), which is what makes the Compactor migration safe to repeat.
• Moving away from a volume leaves PVCs with nothing to reclaim them, still provisioned and still billed. Receive’s must be deleted explicitly — which is safe, because that data was never the durable copy.

--cascade=orphan is the wrong tool for a volume change. It keeps the pods running while the StatefulSet is replaced, which is useful when a restart is what you are avoiding — but an orphaned pod keeps the volumes it started with, so the new claim only takes effect once the pod is replaced anyway.

Losing Receive’s local data during this is not a data-loss event: at most 2h of it had not yet reached object storage, every write exists on three replicas, and the Store Gateway serves everything older. The Compactor’s scratch is not authoritative at all — the bucket is, and compaction is idempotent.

If a Compactor later gets wedged in an unavailable zone by the volume it just gained, that is recoverable by hand; see The Thanos Compactor is stuck in a zone for the ordering, which matters because two Compactors running at once is the one thing that corrupts data.

Alertmanager: two replicas, renamed#

The release that made Alertmanager highly available (DEP-226) also pinned its resource names to alertmanager, which were previously derived from the release name as <release>-alertmanager. helm upgrade applies it without error.

Thanos: selectors name the chart#

The release that moves the Thanos subchart to 0.46.0 (DEP-229) adds app.kubernetes.io/name: thanos to every Thanos selector. Earlier releases selected on component and release name alone. Loki ships a compactor, a query-frontend and a ruler into the same namespace under the same release, so those three Thanos components also selected Loki’s pods.

Services, budgets and policies take the new selectors in place. Deployment and StatefulSet selectors are immutable, so helm upgrade fails, naming every Thanos workload it could not patch, as described under StatefulSet fields that cannot be patched. The workloads are deleted with --cascade=orphan before the upgrade.

kubectl -n monitoring delete statefulset,deployment \
  -l app.kubernetes.io/part-of=thanos,app.kubernetes.io/instance=mzmon \
  --cascade=orphan
helm upgrade mzmon ... # or terraform apply

mzmon is the release name the Terraform module installs under. A Helm install substitutes its own.

Orphaning is the right tool here, unlike for a volume change. The pods already carry every label in the new selectors, and the chart leaves their templates unchanged, so the recreated workloads adopt the running pods without restarting them. No PVC is touched.

A workload whose pod template was changed outside Helm rolls once instead. kubectl rollout restart is the usual cause: it stamps a kubectl.kubernetes.io/restartedAt annotation that Helm preserves across upgrades, and a recreated workload starts from the chart’s template without it. A StatefulSet rolls one pod at a time and waits for each to be ready.

An upgrade that already failed on this recovers the same way: delete the workloads with --cascade=orphan, then run it again. The failed attempt has already applied everything except the workloads, including the new Service, budget and policy selectors. The Terraform module installs with atomic = false, so nothing was rolled back.

Resource names, pod labels and the thanos-thanos ServiceAccount that workload identity bindings name are all unchanged.

Ingester rollouts: duration and deploy timeouts#

Any change to the ingester pod spec — image, resources, or scaling the replica count — rolls the ingester StatefulSet. That roll is ordered and readiness-gated: pods cycle one at a time, and the next is not touched until the previous re-joins the ring and reports Ready. This is deliberate — with replication_factor: 3, one-at-a-time keeps at most one of three replicas down, preserving write quorum throughout the roll.

Budget roughly one minute per ingester, plus headroom for any new nodes. Each pod pays: graceful ring-leave + best-effort flush (up to terminationGracePeriodSeconds, ~60s) → start → memberlist join → ring ACTIVE → readiness stabilization. A 6-ingester fleet therefore takes ~5 minutes to roll, and it is not bounded by how fast new nodes appear.

That overruns tools that wait on the rollout with a short default:

A --wait timeout on a rollout you deliberately triggered means “still rolling,” not “failed” — the rollout completes correctly and the client just stopped watching.

That is the narrow exception, not the rule. On any other timeout, assume something is broken and inspect events before touching the timeout: see Troubleshooting.

Do not speed the roll by allowing more than one ingester down at a time (updateStrategy.rollingUpdate.maxUnavailable > 1, via the alpha MaxUnavailableStatefulSet gate). With RF 3 that can drop you to a single healthy replica and break write quorum mid-roll. zoneAwareReplication (roll a whole zone at once) is the only quorum-safe burst, at the cost of cross-AZ replication traffic — see the logging architecture.