Upgrading materialize-monitoring#

StatefulSet fields that cannot be patched#

Most of a StatefulSet is mutable, but volumeClaimTemplates and selector are not. A release that changes either — a component gaining, losing, or resizing its volume — cannot be applied by helm upgrade, which fails partway with:

cannot patch "thanos-compactor" with kind StatefulSet: StatefulSet.apps
"thanos-compactor" is invalid: spec: Forbidden: updates to statefulset spec for
fields other than 'replicas', 'ordinals', 'template', 'updateStrategy',
'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden

Nothing in the chart can smooth this over — it is a Kubernetes API constraint, not a Helm one — so the StatefulSet has to be deleted and recreated by the upgrade.

The two Thanos storage changes in chart v0.16.0 require this, and only for installs upgrading across it:

# Compactor: local state is scratch, so recreating the pod costs nothing but a
# restarted compaction.
kubectl -n monitoring delete statefulset thanos-compactor
helm upgrade mzmon ... # recreates it with the volumeClaimTemplate

# Receive: the same, plus cleaning up volumes nothing will reclaim. Do this one
# pod at a time if you want to preserve write quorum through the change.
kubectl -n monitoring delete statefulset thanos-receive
helm upgrade mzmon ...
kubectl -n monitoring delete pvc -l app.kubernetes.io/component=receive

Deleting a StatefulSet does not delete its PVCs. The default persistentVolumeClaimRetentionPolicy is Retain, so volumes created from volumeClaimTemplates outlive it. That cuts both ways, and both directions matter here:

• Recreating a StatefulSet rebinds the existing PVCs by name (data-<sts>-0), which is what makes the Compactor migration safe to repeat.
• Moving away from a volume leaves PVCs with nothing to reclaim them, still provisioned and still billed. Receive’s must be deleted explicitly — which is safe, because that data was never the durable copy.

--cascade=orphan is the wrong tool for a volume change. It keeps the pods running while the StatefulSet is replaced, which is useful when a restart is what you are avoiding — but an orphaned pod keeps the volumes it started with, so the new claim only takes effect once the pod is replaced anyway.

Losing Receive’s local data during this is not a data-loss event: at most 2h of it had not yet reached object storage, every write exists on three replicas, and the Store Gateway serves everything older. The Compactor’s scratch is not authoritative at all — the bucket is, and compaction is idempotent.

If a Compactor later gets wedged in an unavailable zone by the volume it just gained, that is recoverable by hand; see The Thanos Compactor is stuck in a zone for the ordering, which matters because two Compactors running at once is the one thing that corrupts data.

Ingester rollouts: duration and deploy timeouts#

Any change to the ingester pod spec — image, resources, or scaling the replica count — rolls the ingester StatefulSet. That roll is ordered and readiness-gated: pods cycle one at a time, and the next is not touched until the previous re-joins the ring and reports Ready. This is deliberate — with replication_factor: 3, one-at-a-time keeps at most one of three replicas down, preserving write quorum throughout the roll.

Budget roughly one minute per ingester, plus headroom for any new nodes. Each pod pays: graceful ring-leave + best-effort flush (up to terminationGracePeriodSeconds, ~60s) → start → memberlist join → ring ACTIVE → readiness stabilization. A 6-ingester fleet therefore takes ~5 minutes to roll, and it is not bounded by how fast new nodes appear.

That overruns tools that wait on the rollout with a short default:

A --wait timeout on a rollout you deliberately triggered means “still rolling,” not “failed” — the rollout completes correctly and the client just stopped watching.

That is the narrow exception, not the rule. On any other timeout, assume something is broken and inspect events before touching the timeout: see Troubleshooting.

Do not speed the roll by allowing more than one ingester down at a time (updateStrategy.rollingUpdate.maxUnavailable > 1, via the alpha MaxUnavailableStatefulSet gate). With RF 3 that can drop you to a single healthy replica and break write quorum mid-roll. zoneAwareReplication (roll a whole zone at once) is the only quorum-safe burst, at the cost of cross-AZ replication traffic — see the logging architecture.