Installing via Helm#

If you are not using the terraform module to install materialize-monitoring, you can use the provided Helm charts to install the monitoring stack in your Kubernetes cluster.

The Terraform modules are the recommended path: they create the buckets and workload-identity bindings, derive every backend key from one object_storage object, and roll Alloy when its config changes.

The charts are the full-fidelity surface — everything Terraform does is a layer over them — so reach for Helm when you need a setting the modules do not model, or when Terraform is not how you deploy.

What you take on by installing directly is the [consumer] half of the shared responsibility model: cloud resources, secrets, version pinning, and the Alloy rollout below.

Dependency: Installing CRDs#

materialize-monitoring relies on several Custom Resource Definitions (CRDs) to function properly.

A second materialize-monitoring-crds Helm chart is provided to install these CRDs separately from the main materialize-monitoring chart, which is recommended to manage the lifecycle of these CRDs separately from the main chart.

Install it first, then install materialize-monitoring with --skip-crds:

helm install mzmon-crds oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-crds --namespace monitoring

The bundled Grafana Operator ships its own copy of the Grafana CRDs and offers no way to opt out of them, so without --skip-crds a fresh install of the main chart would create them behind the CRDs chart’s back. See Dependencies for the full list of CRDs.

Dependency: Setting Up Storage#

You will likely need to set up storage for your metrics and logs before you can start using materialize-monitoring. The specific steps for setting up storage will depend on your environment and the storage solution you choose.

If you are using both external metric storage and external log storage, you will not need an object storage bucket.

Two buckets (or containers) are the norm, one per backend — Loki and Thanos want different lifecycle rules, and per-bucket IAM keeps each backend out of the other’s data.

Cloud Managed Kubernetes Service (AWS EKS, Google Cloud GKE, Azure AKS, etc.)#

Grant access through workload identity rather than static keys. The binding is per-cloud and documented step by step, including the trust-policy subjects that must match the chart’s rendered ServiceAccount names:

• Logs & Events > Storing — IRSA, GKE Workload Identity, and Entra Workload ID, in tabs
• Metrics > Storing — the same three for Thanos

Then name the backend in your values. This is the step that most often goes wrong: the chart’s defaults are S3-shaped, so any other backend has to be named in three load-bearing places and none of them fails softly — the client is chosen by name and validated against a config that was never populated, so the component crash-loops. See Selecting the backend.

The example profiles do all of it correctly and are the shortest path:

Read the Azure profile’s header before copying it. Entra Workload ID needs a pod label as well as the ServiceAccount annotation, and the two subcharts take that label under different keys — loki.loki.podLabels for Loki, thanos.global.commonLabels for Thanos, which has no podLabels of its own.

On-Premises Kubernetes Cluster with Access to Cloud Object Storage (S3, GCS, Azure Blob Storage, etc.)#

Without a cloud identity provider there is no workload identity to bind, so credentials become static keys supplied as a Secret — the documented escape hatch rather than the recommended path. Both backends read them by reference:

• Loki takes them under loki.loki.storage.object_store.<backend>; prefer supplying the values through loki.<component>.extraEnvFrom (a secretRef) over inlining them, since inline values render into a ConfigMap in plaintext.
• Thanos takes them inside thanos.global.objstore.config, which becomes a Secret when createSecret: true.

Any S3-compatible endpoint works — set object_storage.endpoint in Loki’s config and endpoint in Thanos’s. The chart’s own tier-2 E2E runs against rustfs this way, so a self-hosted MinIO, Ceph, or rustfs is a supported shape rather than an untested one.

The chart’s validators grade credential handling and will tell you when a cloud backend has neither an identity annotation nor inline credentials — at which point the component falls back to ambient node credentials, which usually means it silently works in one cluster and fails in another. Read the render output rather than assuming silence.

Customizing your Helm Installation#

The materialize-monitoring Helm chart is designed to be highly customizable, so you can easily integrate with your existing observability infrastructure.

Typically, you would want to create a values.yaml file that has your specific configurations. You may start fresh or you can copy a preset from the charts/materialize-monitoring/profiles/ directory in this repository.

Be aware that when merging examples together that you do not have multiple of the same key on the same level since they do not automatically merge. YAML is whitespace sensitive.

You must specify -f YOUR_VALUES.yaml in your helm install/helm upgrade command to apply these customizations. These are automatically overlaid on top of the default values of the chart, so you only need to specify the values that are different from the default.

This documentation may refer to values in dotted notation (e.g., component.subcomponent.key=value) which corresponds roughly to this YAML structure:

component:
  subcomponent:
    key: value

Choosing which components run, via tags#

Which subcharts install is driven by tags, and the semantics are OR: a component runs if any tag covering it is true. There is no need to disable a group before enabling one member.

Per-chart overrides — tags.loki, tags.thanos, tags.alloy-agent, tags.alloy-gateway, tags.grafana-standalone, tags.grafana-operator, tags.alertmanager, tags.kube-state-metrics, tags.metrics-server — are OR’d on top.

So a Loki-only install is tags.default: false plus tags.loki: true, and adding Thanos to a default install needs nothing at all.

Disabling a Component#

Each subchart also has an enabled circuit breaker that takes precedence over every tag. Setting loki.enabled: false turns Loki off even with tags.default: true, which is the reliable way to subtract one component from an otherwise default install.

Sizing#

The chart’s defaults target a medium deployment, so the sizing profiles are deltas in either direction rather than a full configuration:

Thanos sizing profiles are not written yet, so small and large currently size Loki only. See Production Best Practices for the throughput envelope each tier assumes.

Other shape overlays#

Disabling a Component#

If you want further control of the managed components, you can selectively disable components in the materialize-monitoring Helm chart by setting the enabled field for that component to false in your YOUR_VALUES.yaml file or via --set in your helm install/helm upgrade command.

Initial Installation#

CRDs first, then the chart with --skip-crds. Both releases go in the same namespace; the CRDs are cluster-scoped, so that namespace only holds the release metadata.

helm install mzmon-crds oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring-crds \
  --namespace monitoring --create-namespace \
  --version X.Y.Z --wait

helm install mzmon oci://ghcr.io/materializeinc/helm-charts/materialize-monitoring \
  --namespace monitoring --skip-crds \
  --version X.Y.Z \
  -f my-values.yaml \
  --timeout 15m

Pin --version on both. The two charts have deliberately separate lifecycles, so they do not share a version — see Compatibility for which pairs are tested together.

A few flags earn their place:

• --timeout 15m. Helm’s default is 5 minutes, and a first install brings up Loki, Thanos, Grafana, Alertmanager, and both Alloy roles together. If it still times out, do not raise it again — a timeout almost always means a pod is broken rather than slow. See Troubleshooting.
• --wait on the CRDs release only. The main chart runs pre-install validation Jobs; if you add --wait there, add --wait-for-jobs too, or their verdict is never observed and a bad config rolls anyway.
• Not --atomic on a first install. A rollback destroys the evidence of which component failed, and this stack has enough moving parts that the diagnostic is usually worth more than the cleanup.

Verify it came up#

The chart’s validators print to NOTES.txt, so read the install output rather than assuming silence means success — warnings there are the earliest signal that a backend key or a quorum setting is wrong.

kubectl --namespace monitoring get pods
kubectl --namespace monitoring get grafana,grafanadatasource,grafanamanifest

Grafana is ClusterIP, so reach it with a port-forward:

kubectl --namespace monitoring port-forward svc/grafana 3000:80

The admin credentials are in the grafana Secret (keys admin-user and admin-password).

The chart generates that password and reuses it across helm upgrade, because the Grafana chart looks the existing Secret up before generating.

That lookup returns nothing during helm template and --dry-run, so any render-only pipeline — most GitOps setups — regenerates the password on every sync. Supply your own Secret and set grafana.grafana.admin.existingSecret there. The Terraform module does exactly this, for exactly this reason.

After changing values#

Restart Alloy after any pipeline or filter change. This is the one place the chart cannot own its own rollout, and it is the reverse of what a chart normally guarantees: a helm upgrade that reports success can leave both Alloy roles serving the previous configuration indefinitely.

kubectl --namespace monitoring rollout restart deployment/alloy-gateway daemonset/alloy-agent

The mechanism, and why a reload cannot substitute for a restart, is in Production Best Practices. The short version: the pipeline’s -env ConfigMaps are consumed with envFrom, and environment variables are fixed at container start, so neither Alloy’s /-/reload nor a config-reloader sidecar can pick up a filter change.

Going to production#

Production Best Practices is the deployment checklist, organized by backend and tagged with who owns each item — the chart, you as the chart consumer, or you as the operator.

On the Helm path every [consumer] item is yours, including the buckets and workload-identity bindings that the Terraform modules would otherwise create.

Upgrading and uninstalling#

• Upgrading — ingester rollouts are ordered and readiness-gated, so budget roughly a minute per ingester and raise your client’s timeout accordingly.
• Uninstalling — read this before tearing down. Deleting the release without first removing the Grafana custom resources deadlocks on finalizers that only grafana-operator can remove, and the namespace never finishes terminating.

Reference#

• Chart values — every value, generated from values.yaml
• CRDs — what the CRDs chart installs
• Compatibility — chart, CRDs chart, and Terraform module versions