Kubernetes runbook#

This page is for you if you operate the Kubernetes cluster that materialize-monitoring runs on. It holds the procedures that apply to more than one component. The component runbooks link here for those steps.

Storage#

These procedures change the PersistentVolumeClaims (PVCs) that the stack’s StatefulSets create from their volumeClaimTemplates.

Temporarily increase storage of working PVCs#

A working PVC holds only a cache or a scratch copy of data that lives elsewhere, usually the bucket. Losing one costs the component time to rebuild it, and loses no data. A one-off job, such as the Thanos Compactor’s first run after a long halt, can need more space than the component needs in steady state. You can grow a working PVC for that job, and later return it to the chart’s size.

The following table lists the stack’s PVCs and whether each is a working PVC:

Kubernetes can grow a PVC in place, but never shrinks one. Nothing reconciles an existing PVC against its StatefulSet’s volumeClaimTemplates, so a helm upgrade leaves a grown PVC at its new size.

Grow the PVC#

To grow a working PVC, follow these steps. In each snippet, set PVC_NAME to the PVC from the table earlier on this page, and NEW_SIZE to the size you want.

1. Check that the PVC’s StorageClass allows expansion:

   MONITORING_NAMESPACE=monitoring
   KUBECONTEXT=my-cluster
   PVC_NAME=data-thanos-compactor-0
   
   STORAGE_CLASS=$(kubectl --context "$KUBECONTEXT" -n "$MONITORING_NAMESPACE" get pvc "$PVC_NAME" \
     -o jsonpath='{.spec.storageClassName}')
   kubectl --context "$KUBECONTEXT" get storageclass "$STORAGE_CLASS" \
     -o jsonpath='{.allowVolumeExpansion}{"\n"}'

   If the command prints nothing or false, you can’t grow the PVC in place. Raise the component’s persistence.size in the chart values and upgrade the release instead. Then recreate the PVC, which creates it at the new size.

2. Request the new size:

   MONITORING_NAMESPACE=monitoring
   KUBECONTEXT=my-cluster
   PVC_NAME=data-thanos-compactor-0
   NEW_SIZE=50Gi
   
   kubectl --context "$KUBECONTEXT" -n "$MONITORING_NAMESPACE" patch pvc "$PVC_NAME" \
     -p "$(printf '{"spec":{"resources":{"requests":{"storage":"%s"}}}}' "$NEW_SIZE")"

3. Wait until the PVC reports the new size:

   MONITORING_NAMESPACE=monitoring
   KUBECONTEXT=my-cluster
   PVC_NAME=data-thanos-compactor-0
   
   kubectl --context "$KUBECONTEXT" -n "$MONITORING_NAMESPACE" get pvc "$PVC_NAME" \
     -o jsonpath='{.status.capacity.storage} {.status.conditions[*].type}{"\n"}'

   While the volume grows, the conditions show Resizing and then FileSystemResizePending. Most CSI drivers grow the filesystem while the pod runs. If the condition stays at FileSystemResizePending, delete the pod, and the filesystem grows when the new pod mounts the volume.

Some cloud providers limit how often you can resize a volume. For example, Amazon EBS allows one modification of a volume every six hours. Choose a size with room to spare, so that you don’t need a second resize.

Return the PVC to the chart’s size#

After the job that needed the space finishes, you can return a working PVC to the size its StatefulSet requests by recreating it. Recreate only a working PVC.

Deleting a PVC discards everything on it, and you can’t undo it. With the default reclaimPolicy of Delete, the StorageClass also deletes the disk behind the PVC.

• Alertmanager: the replica loses its silences and notification log. Replicas share both by gossip, so a new replica recovers them from its peers. If every replica loses its PVC, silenced alerts notify again, and recent notifications can repeat.
• Loki ruler: the replica loses the recording-rule samples it hasn’t sent yet. Those samples are missing from the metrics store for good.

To recreate a working PVC, follow these steps. In each snippet, set PVC_NAME to the PVC, and POD_NAME to the pod that mounts it. The pod’s name is the PVC’s name without its prefix, such as thanos-compactor-0 for data-thanos-compactor-0.

1. Delete the PVC without waiting. The PVC stays Terminating until no pod mounts it:

   MONITORING_NAMESPACE=monitoring
   KUBECONTEXT=my-cluster
   PVC_NAME=data-thanos-compactor-0
   
   kubectl --context "$KUBECONTEXT" -n "$MONITORING_NAMESPACE" delete pvc "$PVC_NAME" --wait=false

2. Delete the pod that mounts it:

   MONITORING_NAMESPACE=monitoring
   KUBECONTEXT=my-cluster
   POD_NAME=thanos-compactor-0
   
   kubectl --context "$KUBECONTEXT" -n "$MONITORING_NAMESPACE" delete pod "$POD_NAME"

3. Check that the StatefulSet recreated the pod and a new PVC at the requested size:

   MONITORING_NAMESPACE=monitoring
   KUBECONTEXT=my-cluster
   PVC_NAME=data-thanos-compactor-0
   POD_NAME=thanos-compactor-0
   
   kubectl --context "$KUBECONTEXT" -n "$MONITORING_NAMESPACE" get pod "$POD_NAME"
   kubectl --context "$KUBECONTEXT" -n "$MONITORING_NAMESPACE" get pvc "$PVC_NAME" \
     -o jsonpath='{.status.capacity.storage}{"\n"}'

   If the new pod stays Pending with an event that says its PVC is being deleted, it started before the old PVC was gone. Delete the pod again.

For a StatefulSet with more than one replica, such as the Store Gateway, recreate one PVC at a time. Wait for each pod to be Ready before you start the next.

A recreated working PVC starts empty, so the component pays to rebuild it. A Store Gateway replica rebuilds its index headers from the bucket, and serves no historical query until it finishes. On a large bucket, that takes minutes. The Compactor downloads the blocks of a compaction in progress again.