Install an ad-hoc stack for troubleshooting#

This page is for you if you need a Grafana with metrics in it, on a cluster you don’t run day to day, as soon as possible. Typical cases are a day-1 install, an on-premises cluster you haven’t seen before, or a node pool that Materialize already fills.

The install uses profiles/adhoc.values.yaml, one file that covers the whole stack:

Every component runs one replica, with a replication factor of 1.
Nothing claims a PersistentVolume, so the cluster doesn’t need a StorageClass.
Loki runs as one process on its own filesystem, so it needs no bucket.
Topology spread constraints are removed, so pods schedule on nodes with no zone labels.
NetworkPolicies are off.

Losing a pod loses that pod’s data, so don’t run this profile in production. Grafana keeps users, API keys, and anything else changed in its UI only until it restarts. The dashboards, data sources, and folders the charts install come back on their own.

For a lasting install, see Helm.

Before you begin#

You need the following:

helm and kubectl, with access to the cluster.
An object store bucket for Thanos. Thanos has no local-disk mode that works across its components, so even an ad-hoc install needs one. S3, an S3-compatible store such as Ceph RGW or MinIO, GCS, and Azure Blob Storage all work.
An address you can reach Grafana on, such as a node IP.

Install the stack#

Pull the chart, which includes every profile:

helm pull oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring --version VERSION --untar

Replace VERSION with the chart version you install.

Create the Secret that Thanos reads its bucket from. For S3 or an S3-compatible store:
```
kubectl create namespace monitoring
kubectl create secret generic thanos-objstore-config --namespace monitoring \
  --from-file=objstore.yml=/dev/stdin <<'EOF'
type: S3
config:
  bucket: BUCKET
  endpoint: HOST:PORT
  access_key: ACCESS_KEY
  secret_key: SECRET_KEY
  insecure: true
EOF
```
Replace the following:
- BUCKET: the bucket for Thanos blocks.
- HOST:PORT: the store’s endpoint, with no scheme.
- ACCESS_KEY and SECRET_KEY: the store’s credentials.
Set insecure: false when the store serves HTTPS. GCS and Azure take the same Secret with their own type and config.

Install the CRDs:

helm install mzmon-crds oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-crds \
  --namespace monitoring --version VERSION --wait

Install the chart with the profile:

helm install mzmon oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring \
  --namespace monitoring --skip-crds --version VERSION \
  -f materialize-monitoring/profiles/adhoc.values.yaml \
  --timeout 15m

The install output warns about every production setting the profile turns off. Read past those for anything else.

Install the dashboards, which ship in charts of their own, one per Grafana folder:

helm install mzmon-dashboards oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-dashboards \
  --namespace monitoring --version VERSION
helm install mzmon-dashboards-infra oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-dashboards-infra \
  --namespace monitoring --version VERSION
helm install mzmon-dashboards-meta oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-dashboards-meta \
  --namespace monitoring --version VERSION

The dashboards charts share one version, but the other charts each have their own, so replace VERSION in each command. Skip any dashboards chart you don’t want: -infra carries the platform dashboards, and -meta carries the ones that watch the monitoring stack itself.

Check that the pods are running and that the dashboards reached Grafana:

kubectl get pods --namespace monitoring
kubectl get grafana,grafanamanifest --namespace monitoring

Reach Grafana#

A port-forward works, but it drops connections when a dashboard opens many requests at once. On a cluster whose nodes you can reach, a NodePort Service is steadier. kubectl expose copies the selector and ports from the existing Service:

kubectl expose service grafana --namespace monitoring --name grafana-nodeport --type NodePort
kubectl get service grafana-nodeport --namespace monitoring -o jsonpath='{.spec.ports[0].nodePort}'

Open http://NODE_IP:NODE_PORT. Replace NODE_IP with the address of any node, and NODE_PORT with the port the second command prints.

The admin password is in the grafana Secret:

kubectl get secret grafana --namespace monitoring -o jsonpath='{.data.admin-password}' | base64 -d

Delete the grafana-nodeport Service when you’re done; Helm doesn’t manage it.

Adjust for the cluster#

The profile marks each setting you’re likely to change with SWITCH. Edit your copy of adhoc.values.yaml, or pass the key with --set:

NetworkPolicies: Each key that renders one is listed, so you can turn a single component’s policy back on. If you turn them on, see grafana-operator times out reaching Grafana for clusters that run NodeLocal DNSCache.
DNS Service name: Loki’s gateway resolves through a Service named kube-dns. Set loki.global.dnsService when the cluster names it something else. For more information, see host not found in resolver.
metrics-server: Off, as at the chart default, because most clusters already serve metrics.k8s.io. Turn it on when kubectl get apiservice v1beta1.metrics.k8s.io finds nothing.
Loki: Turn it off when the cluster can’t fit one more pod. Also turn off the Alloy gateway’s Loki destination, which the profile lists beside it.

To keep logs in object storage, add -f materialize-monitoring/profiles/s3-compatible.values.yaml after adhoc. Its header has the commands for the two Secrets it reads.

Clean up#

Uninstall the releases in the reverse order you installed them:

helm uninstall mzmon-dashboards-meta --namespace monitoring
helm uninstall mzmon-dashboards-infra --namespace monitoring
helm uninstall mzmon-dashboards --namespace monitoring
helm uninstall mzmon --namespace monitoring
helm uninstall mzmon-crds --namespace monitoring

The following outlive the releases:

The CRDs, which the CRDs release leaves in place. Other installs in the cluster might use them.
The Secrets you created, and any NodePort Service.
The Thanos blocks in the bucket.

For when and how to remove the CRDs, see Uninstalling.