`materialize-monitoring` Helm Reference#

Unified observability stack for Materialize: Alloy-based pipeline, optional bundled backends (Loki / Thanos / Grafana / Alertmanager), and the dashboards / alerts / rules that consume the Materialize metrics surface.

Source Code#

https://github.com/MaterializeInc/materialize-monitoring/tree/main/charts/materialize-monitoring

Requirements#

Kubernetes: >=1.27.0-0

Storage Requirements#

Using Thanos or Loki (both enabled by default) requires object storage (such as an AWS S3 bucket) for long-term storage.

In development, Thanos may use a PVC for less-reliable storage. Loki however will not run without object storage. You may consider Garage or RustFS or MinIO for manually provisioned object storage.

Included Subcharts#

Repository	Name	Version
https://grafana.github.io/helm-charts	alloy(alloy-agent)	^1.8.0
https://grafana.github.io/helm-charts	alloy(alloy-gateway)	^1.8.0
https://kubernetes-sigs.github.io/metrics-server	metrics-server	^3.13.0
oci://ghcr.io/grafana-community/helm-charts	grafana	^12.3.3
oci://ghcr.io/grafana-community/helm-charts	loki	^15.0.0
oci://ghcr.io/grafana/helm-charts	grafana-operator	^5.22.2
oci://ghcr.io/prometheus-community/charts	alertmanager	^1.25.0
oci://ghcr.io/prometheus-community/charts	kube-state-metrics	^7.3.0
oci://ghcr.io/thanos-community/helm-charts	thanos	^0.11.0

Values#

Globals#

Values that are passed to all subcharts

Key	Type	Default	Description
global.imageRegistry	string	`""`	Override the registry for all images in this chart and its subcharts. Leave empty to use upstream defaults.
global.imagePullSecrets	list	[]	Image pull secrets applied to every workload in this chart and its subcharts.

Main chart configuration#

Configuration for the main chart

Key	Type	Default	Description
nameOverride	string	`""`	Standard Helm name override. Note that in umbrella charts, fullname may be shorted. See fullnameOverride.
fullnameOverride	string	`"mzmon"`	Standard Helm fullname override. Note that in umbrella charts, this may be shortened to avoid long prefixes. (and have more stable names in resources)
namespaceOverride	string	`""`	Namespace override for default workloads.

Subchart enablement (Helm tags)#

Group toggles for subchart enablement. Subchart enablement has two layers, in precedence order:

Per-chart circuit breaker (<chart>.enabled). Each subchart dependency carries a condition: <chart>.enabled in Chart.yaml. The corresponding enabled: key is left commented-out under each subchart block in values.yaml; uncommenting it force-includes (true) or force-excludes (false) that chart regardless of any tag state. Helm evaluates condition: first and only falls through to tags when the path is absent from values, so this is a hard override.
Helm tags (the tags: block). When no <chart>.enabled is set, enablement is decided by tags. Each dependency in Chart.yaml carries the master default tag, a group tag (e.g. bundled-backends), and a per-chart tag (e.g. loki); a dependency is included if any of its tags evaluates true (OR semantics). To opt one chart out of an otherwise-enabled group, set the group tag false and flip the per-chart tags individually — or use the circuit breaker.

Group ↔ chart mapping:

Group tag	Charts
`default`	`pipeline` + `bundled-backends` + `managed-grafana`
	groups + `kube-state-metrics` (recommended stack)
`pipeline`	`alloy-agent`, `alloy-gateway`
`bundled-backends`	`loki`, `thanos`, `alertmanager`
`managed-grafana`	`grafana`, `grafana-operator`
`cluster-metrics`	`kube-state-metrics`, `metrics-server`
`crds`	`prometheus-operator-crds`, `grafana-operator-crds`
	(in the sibling `materialize-monitoring-crds` chart)

default is the only group on by default and enables the full recommended stack. --set tags.default=false turns everything off, so you can enable a single group (bundled-backends, managed-grafana, pipeline, cluster-metrics) or individual charts on top. Profile preset values files under profiles/ flip these appropriately. (kube-state-metrics is in default, but metrics-server is not — most clusters already run metrics-server; enable it via tags.cluster-metrics or tags.metrics-server if yours doesn’t.)

Key	Type	Default	Description
tags.default	bool	`true`	Enable all recommended defaults. You can `--set tags.default=false` to disable all services and explicitly enable others.
tags.pipeline	bool	`false`	Enable both Alloy releases (agent + gateway).
tags.bundled-backends	bool	`false`	Enable Loki, Thanos, and Alertmanager as a group. (Grafana is in the `managed-grafana` group.)
tags.cluster-metrics	bool	`false`	Enable kube-state-metrics and metrics-server as a group.
tags.alloy-agent	bool	`false`	Per-chart override: enable just the Alloy agent. OR'd with `tags.default` / `tags.pipeline`.
tags.alloy-gateway	bool	`false`	Per-chart override: enable just the Alloy gateway. OR'd with `tags.default` / `tags.pipeline`.
tags.loki	bool	`false`	Per-chart override: enable just Loki. OR'd with `tags.default` / `tags.bundled-backends`.
tags.thanos	bool	`false`	Per-chart override: enable just Thanos. OR'd with `tags.default` / `tags.bundled-backends`.
tags.grafana-standalone	bool	`false`	Per-chart override: enable just Grafana standalone. OR'd with `tags.default` / `tags.managed-grafana`.
tags.grafana-operator	bool	`false`	Per-chart override: enable just Grafana operator. OR'd with `tags.default` / `tags.managed-grafana`.
tags.alertmanager	bool	`false`	Per-chart override: enable just Alertmanager. OR'd with `tags.default` / `tags.bundled-backends`.
tags.kube-state-metrics	bool	`false`	Per-chart override: enable just kube-state-metrics. OR'd with `tags.default` / `tags.cluster-metrics`.
tags.metrics-server	bool	`false`	Per-chart override: enable just metrics-server. OR'd with `tags.cluster-metrics`.

Materialize Integration#

Materialize-specific configuration values.

Key	Type	Default	Description
materialize.namespaces	list	[]	Namespaces to scrape Materialize workloads from. Empty list means all namespaces the chart can read.
materialize.environmentd.serviceMonitor	object	{ "enabled": true, "metricEndpoints": [], "selector": {}, "targetLabels": [] }	ServiceMonitor/PodMonitor configuration for environmentd
materialize.environmentd.serviceMonitor.enabled	bool	`true`	Deploy a PodMonitor to collect Materialize metrics.
materialize.environmentd.serviceMonitor.selector	object	{}	Override for default selector
materialize.environmentd.serviceMonitor.targetLabels	list	[]	Override for default labels to copy from the pod into metrics
materialize.environmentd.serviceMonitor.metricEndpoints	list	[]	Override for default metric endpoints
materialize.environmentdSQL.serviceMonitor	object	{ "enabled": true, "metricEndpoints": [], "selector": {}, "targetLabels": [] }	ServiceMonitor/PodMonitor configuration for environmentd-sql
materialize.environmentdSQL.serviceMonitor.enabled	bool	`true`	Deploy a PodMonitor to collect Materialize metrics.
materialize.environmentdSQL.serviceMonitor.selector	object	{}	Override for default selector
materialize.environmentdSQL.serviceMonitor.targetLabels	list	[]	Override for default labels to copy from the pod into metrics
materialize.environmentdSQL.serviceMonitor.metricEndpoints	list	[]	Override for default metric endpoints
materialize.clusterd.serviceMonitor	object	{ "enabled": true, "metricEndpoints": [], "selector": {}, "targetLabels": [] }	ServiceMonitor/PodMonitor configuration for clusterd
materialize.clusterd.serviceMonitor.enabled	bool	`true`	Deploy a PodMonitor to collect Materialize metrics.
materialize.clusterd.serviceMonitor.selector	object	{}	Override for default selector
materialize.clusterd.serviceMonitor.targetLabels	list	[]	Override for default labels to copy from the pod into metrics
materialize.clusterd.serviceMonitor.metricEndpoints	list	[]	Override for default metric endpoints
materialize.deploymentMode	string	`"self-managed"`	Deployment mode normalization hint. One of: `self-managed`, `cloud`. Drives relabeling rules in the pipeline.
materialize-system.namespace	string	`"materialize-environment"`	The namespace materialize-system was deployed into
materialize-system.serviceMonitor	object	{ "enabled": true }	ServiceMonitor/PodMonitor configuration TODO: nothing uses this yet
materialize-operator.namespace	string	`"materialize"`	The namespace materialize-operator was deployed into
materialize-operator.serviceMonitor	object	{ "enabled": true, "metricEndpoints": [], "selector": {}, "targetLabels": [] }	ServiceMonitor/PodMonitor configuration
materialize-operator.serviceMonitor.selector	object	{}	Override for default selector
materialize-operator.serviceMonitor.targetLabels	list	[]	Override for default labels to copy from the pod into metrics
materialize-operator.serviceMonitor.metricEndpoints	list	[]	Override for default metric endpoints

Pipeline configuration#

Pipeline configuration values that drive Alloy behavior and defaults.

The Alloy pipeline is the flagship deliverable of this chart. Pipeline configuration is expressed in values, rendered into ConfigMaps under templates/pipelines/, and consumed by the Alloy agent and gateway release instances.

Key	Type	Default	Description
pipeline.preValidateJob.enabled	bool	`true`	Run a hook/dependency to validate rendered configs before rolling them out
pipeline.preValidateJob.annotations	object	{ "helm.sh/hook": "pre-install,pre-upgrade", "helm.sh/hook-delete-policy": "before-hook-creation" }	Job specific annotations This should be used to control when the job is executed. The default is as a helm hook, but setting any other annotation overrides this.
pipeline.preValidateJob.backoffLimit	int	`1`	Number of times to retry the job before failing the release.
pipeline.preValidateJob.podSecurityContext	object	{ "fsGroup": 473, "runAsGroup": 473, "runAsUser": 473 }	Security context for the pre-validate job pod. This is the hardened recommendation with the alloy user.
pipeline.preValidateJob.containerSecurityContext	object	{ "allowPrivilegeEscalation": false, "capabilities": { "drop": [ "ALL" ] }, "readOnlyRootFilesystem": true, "runAsGroup": 473, "runAsNonRoot": true, "runAsUser": 473 }	Security context for the pre-validate job container. This is the hardened recommendation with the alloy user.
pipeline.env	object	{}	Environment variables to set in agent and gateway pods. These support tpl rendering. Since these are injected into an envFrom, any env entries take precedence.
pipeline.env.CLUSTER_NAME	string	`"default"`	Name of the cluster to discriminate workloads from different sources.
pipeline.env.GATEWAY_LOG_LEVEL	string	`"info"`	Level for the gateway logs.

Log configuration#

Configuration for log behavior

Key	Type	Default	Description
pipeline.logging.agent.rateLimit	int	`5000`	Rate limit for alloy agent incoming pod logs. This is per agent.
pipeline.logging.agent.burst	int	`20000`	Burst limit for alloy agent incoming pod logs.
pipeline.logging.agent.destination.loki.url	string	`"http://alloy-gateway.{{ include \"mzmon.alloyGateway.namespace\" $ }}.svc:3100/loki/api/v1/push"`	alloy-gateway push endpoint URL.
pipeline.logging.agent.destination.loki.retries	object	{ "maxBackoffPeriod": "5m", "maxBackoffRetries": 10, "minBackoffPeriod": "1s", "retryOnHttp429": true }	Retry configuration.
pipeline.logging.agent.destination.loki.authType	string	`"none"`	Type of authentication to use with the alloy-gateway endpoint. Use none if no authentication is required.
pipeline.logging.agent.destination.loki.tls.enabled	bool	`false`	Whether to enable TLS for alloy-gateway dest.
pipeline.logging.agent.destination.loki.tls.verify	bool	`true`	Whether to verify the TLS certificate for the alloy-gateway dest.
pipeline.logging.agent.destination.loki.tls.ca	string	`""`	Certificate Authority (CA) PEM contents for TLS.
pipeline.logging.agent.destination.loki.tls.cert	string	`""`	Client certificate PEM contents for TLS.
pipeline.logging.agent.destination.loki.tls.key	string	`""`	Client private key PEM contents for TLS.
pipeline.logging.agent.destination.loki.tls.serverName	string	`""`	Alternative SNI (Server Name Indication) to specify.
pipeline.logging.agent.destination.loki.tls.minVersion	string	`"TLS13"`	Minimum TLS version to allow. Use TLS12 if you need better compat. TLS11 and TLS10 are not recommended for production.
pipeline.logging.gateway.destination.loki.enabled	bool	`true`	Enable writing to a loki destination. By default, we use the in-cluster loki
pipeline.logging.gateway.destination.loki.url	string	`"http://loki-distributor.{{ include \"mzmon.loki.namespace\" $ }}.svc:3100/loki/api/v1/push"`	Loki push endpoint URL.
pipeline.logging.gateway.destination.loki.retries	object	{ "maxBackoffPeriod": "5m", "maxBackoffRetries": 10, "minBackoffPeriod": "1s", "retryOnHttp429": true }	Retry configuration.
pipeline.logging.gateway.destination.loki.authType	string	`"none"`	Type of authentication to use with the loki endpoint. Use none if no authentication is required. Use basicAuth for username/password. Use bearer for bearer token. Use oauth2 for OAuth2 client credentials.
pipeline.logging.gateway.destination.loki.basicAuth	object	{ "password": "", "passwordEnv": "GATEWAY_LOKI_DEST_PASSWORD", "username": "", "usernameEnv": "GATEWAY_LOKI_DEST_USERNAME" }	Configuration for auth when using authType=basicAuth You will need to provide alloy-gateway.alloy.agent.extraEnv TODO: add a check for this
pipeline.logging.gateway.destination.loki.bearer	object	{ "token": "", "tokenEnv": "GATEWAY_LOKI_DEST_BEARER_TOKEN" }	Configuration for bearer token when using authType=bearer This is used for bearer type tokens.
pipeline.logging.gateway.destination.loki.oauth2	object	{ "clientId": "", "clientIdEnv": "GATEWAY_LOKI_DEST_OAUTH2_CLIENT_ID", "clientSecret": "", "clientSecretEnv": "GATEWAY_LOKI_DEST_OAUTH2_CLIENT_SECRET", "scopes": [], "tokenUrl": "", "tokenUrlEnv": "GATEWAY_LOKI_DEST_OAUTH2_TOKEN_URL" }	Configuration for OAuth2 when using authType=oauth2
pipeline.logging.gateway.destination.loki.tls.enabled	bool	`false`	Whether to enable TLS for the loki destination.
pipeline.logging.gateway.destination.loki.tls.verify	bool	`true`	Whether to verify the TLS certificate for the loki destination.
pipeline.logging.gateway.destination.loki.tls.ca	string	`""`	Certificate Authority (CA) PEM contents for TLS.
pipeline.logging.gateway.destination.loki.tls.cert	string	`""`	Client certificate PEM contents for TLS.
pipeline.logging.gateway.destination.loki.tls.key	string	`""`	Client private key PEM contents for TLS.
pipeline.logging.gateway.destination.loki.tls.serverName	string	`""`	Alternative SNI (Server Name Indication) to specify.
pipeline.logging.gateway.destination.loki.tls.minVersion	string	`"TLS13"`	Minimum TLS version to allow. Use TLS12 if you need better compat. TLS11 and TLS10 are not recommended for production.
pipeline.logging.gateway.destination.otel.enabled	bool	`false`	Enable writing to an OpenTelemetry/OTLP destination. By default, we do not use the OTLP destination. NOTE: This is configured in the pipeline.metrics.gateway.destination.otlp block, not the logging block.
pipeline.logging.tenancy.staticTenant	string	`"loki"`	Default tenant to write logs to. This is used when tenantMap values is set to `static`.
pipeline.logging.tenancy.tenantMap	object	{ "audit": "static", "default": "static", "environment": "static", "infra": "static" }	Type of tenancy used to write logs. Use static to write to the staticTenant value (recommended). Use byEnvironment to separate by the environment name. Use byNamespace to use the namespace of the source pod as the tenant. Use byLabel to use a label (tenantLabel) to specifically identify the tenant. Use none to disable tenancy (only use if loki does not have tenancy).

Metrics configuration#

Configuration for metrics behavior

Key	Type	Default	Description
pipeline.metrics.gateway.denyMetrics	list	[]	Denylist of metrics that are excluded from being exported These are \|'d in a regex pattern
pipeline.metrics.gateway.destination.prometheusRemoteWrite.enabled	bool	`true`	Enable writing to a Prometheus remote write destination. We default to in-cluster Thanos.
pipeline.metrics.gateway.destination.prometheusRemoteWrite.url	string	`"http://thanos-receive.{{ include \"mzmon.thanos.namespace\" $ }}.svc:10908/api/v1/receive"`	Prometheus remote write endpoint URL.
pipeline.metrics.gateway.destination.prometheusRemoteWrite.minMetricImportance	string	`"all"`	Only export metrics with the specified importance level. Values are "essential", "recommended", "extended", "diagnostic", "all"
pipeline.metrics.gateway.destination.prometheusRemoteWrite.authType	string	`"none"`	Type of authentication to use with the loki endpoint. Use none if no authentication is required. Use basicAuth for username/password. Use bearer for bearer token. Use oauth2 for OAuth2 client credentials. Use sigv4 for AWS SigV4 signing (For AMP).
pipeline.metrics.gateway.destination.prometheusRemoteWrite.basicAuth	object	{ "password": "", "passwordEnv": "GATEWAY_PROMETHEUS_DEST_PASSWORD", "username": "", "usernameEnv": "GATEWAY_PROMETHEUS_DEST_USERNAME" }	Configuration for auth when using authType=basicAuth
pipeline.metrics.gateway.destination.prometheusRemoteWrite.bearer	object	{ "token": "", "tokenEnv": "GATEWAY_PROMETHEUS_DEST_BEARER_TOKEN" }	Configuration for bearer token when using authType=bearer This is used for bearer type tokens.
pipeline.metrics.gateway.destination.prometheusRemoteWrite.oauth2	object	{ "clientId": "", "clientIdEnv": "GATEWAY_PROMETHEUS_DEST_OAUTH2_CLIENT_ID", "clientSecret": "", "clientSecretEnv": "GATEWAY_PROMETHEUS_DEST_OAUTH2_CLIENT_SECRET", "scopes": [], "tokenUrl": "", "tokenUrlEnv": "GATEWAY_PROMETHEUS_DEST_OAUTH2_TOKEN_URL" }	Configuration for OAuth2 when using authType=oauth2
pipeline.metrics.gateway.destination.prometheusRemoteWrite.sigv4	object	{ "region": "", "roleArn": "" }	Configuration for AWS signatures when using authType=sigv4 This generally does not need to be set (it is derived from IRSA).
pipeline.metrics.gateway.destination.prometheusRemoteWrite.tls.enabled	bool	`false`	Whether to enable TLS for the prometheus destination.
pipeline.metrics.gateway.destination.prometheusRemoteWrite.tls.verify	bool	`true`	Whether to verify the TLS certificate for the prometheus destination.
pipeline.metrics.gateway.destination.prometheusRemoteWrite.tls.ca	string	`""`	Certificate Authority (CA) PEM contents for TLS.
pipeline.metrics.gateway.destination.prometheusRemoteWrite.tls.cert	string	`""`	Client certificate PEM contents for TLS.
pipeline.metrics.gateway.destination.prometheusRemoteWrite.tls.key	string	`""`	Client private key PEM contents for TLS.
pipeline.metrics.gateway.destination.prometheusRemoteWrite.tls.serverName	string	`""`	Alternative SNI (Server Name Indication) to specify.
pipeline.metrics.gateway.destination.prometheusRemoteWrite.tls.minVersion	string	`"TLS13"`	Minimum TLS version to allow. Use TLS12 if you need better compat. TLS11 and TLS10 are not recommended for production.

OpenTelemetry/OTLP destinations.#

OpenTelemetry/OTLP destination configuration for metrics (and logging).

This supports several components and allows further customization of endpoints. Multiple exporters can be enabled at once.

Key	Type	Default	Description
pipeline.metrics.gateway.destination.otel.enabled	bool	`false`	Enable writing metrics to an OpenTelemetry/OTLP destination. By default, we do not use the OTel destination. WARNING: if logging otel destination is enabled, this block will still be used for configuration! (just not for metrics)
pipeline.metrics.gateway.destination.otel.googleCloudExporter.enabled	bool	`false`	Enable writing to a Google Cloud Monitoring / Cloud Logging destination.
pipeline.metrics.gateway.destination.otel.googleCloudExporter.compression	string	`"gzip"`	Compression for logs/metrics Only gzip is supported for Google Cloud Monitoring / Logging.
pipeline.metrics.gateway.destination.otel.googleCloudExporter.minMetricImportance	string	`"recommended"`	Only export metrics with the specified importance level. Values are "essential", "recommended", "extended", "diagnostic", "all"
pipeline.metrics.gateway.destination.otel.googleCloudExporter.handlers	list	[ "otelcol.exporter.googlecloud.destination.input" ]	Handlers to use for the Google Cloud exporter.
pipeline.metrics.gateway.destination.otel.googleCloudExporter.config	string	An `otelcol.exporter.googlecloud.destination` definition.	Raw configuration for an otelcol.exporter.googlecloud block. The default config uses Workload Identity Federation (WIF) to authenticate to GCP.
pipeline.metrics.gateway.destination.otel.datadogExporter.enabled	bool	`false`	Enable writing to a Datadog destination.
pipeline.metrics.gateway.destination.otel.datadogExporter.compression	string	`"gzip"`	Compression for logs/metrics Only gzip is supported for Datadog.
pipeline.metrics.gateway.destination.otel.datadogExporter.minMetricImportance	string	`"recommended"`	Only export metrics with the specified importance level. Values are "essential", "recommended", "extended", "diagnostic", "all"
pipeline.metrics.gateway.destination.otel.datadogExporter.hostMetadata.enabled	bool	`false`	Whether to include host metadata in the Datadog exporter. FIXME: how do we support this in an Agent->Gateway architecture?
pipeline.metrics.gateway.destination.otel.datadogExporter.handlers	list	[ "otelcol.exporter.datadog.destination.input" ]	Handlers to use for the Datadog exporter.
pipeline.metrics.gateway.destination.otel.datadogExporter.config	string	An `otelcol.exporter.datadog.destination` definition.	Raw configuration for an otelcol.exporter.datadog block.
pipeline.metrics.gateway.destination.otel.otlpExporter.enabled	bool	`false`	Enable writing to an OpenTelemetry/OTLP destination. This is the generic fallback for other destinations. You can also use this if you need to set a custom destination block.
pipeline.metrics.gateway.destination.otel.otlpExporter.url	string	`""`	OTLP push endpoint URL. This does not need a protocol prefix (http:// or https://)
pipeline.metrics.gateway.destination.otel.otlpExporter.protocol	string	`"grpc"`	Protocol to use for OTLP. Use grpc for gRPC protocol. Use http for HTTP protocol.
pipeline.metrics.gateway.destination.otel.otlpExporter.compression	string	`"gzip"`	Compression to use Use gzip for better compatibility. Use snappy for better performance.
pipeline.metrics.gateway.destination.otel.otlpExporter.minMetricImportance	string	`"all"`	Only export metrics with the specified importance level. Values are "essential", "recommended", "extended", "diagnostic", "all"
pipeline.metrics.gateway.destination.otel.otlpExporter.handlers	list	[ "otelcol.exporter.otlp[http].destination.input" ]	Handlers to use for the OTLP exporter. Update this if your config was customized.
pipeline.metrics.gateway.destination.otel.otlpExporter.config	string	{{- $exporterType := ternary "otlp" "otlphttp" ( eq .Values.pipeline.metrics.gateway.destination.otel.otlpExporter.protocol "grpc" ) }} otelcol.exporter.{{ $exporterType }} "destination" { client { endpoint = {{ .Values.pipeline.metrics.gateway.destination.otel.otlpExporter.url \| required "destination.otlp.url must be set" \| quote }} compression = {{ .Values.pipeline.metrics.gateway.destination.otel.otlpExporter.compression \| quote }} {{- if ( include "mzmon.alloyGateway.otelDest.authEnabled" $ ) }} auth = {{ include "mzmon.alloyGateway.otelDest.authHandler" $ }} {{- end }} } }	Raw configuration for an otelcol.exporter.otlp block. Use this if you need to configure multiple destinations or use an alternative endpoint entirely.

OpenTelemetry Authentication Configuration#

Configuration for OpenTelemetry/OTLP destinations. This is only needed for destinations that require an auth handler (like otlpExporter).

Key	Type	Default	Description
pipeline.metrics.gateway.destination.otel.auth.authType	string	`"none"`	Type of authentication to use with the OpenTelemetry destination. Valid values are: `none`, `basic`, `bearer`, `awsSigv4`, and `custom`.
pipeline.metrics.gateway.destination.otel.auth.basic.config	string	An `otelcol.auth.basic.oteldest` definition.	Raw configuration for an otelcol.auth.basic block. This uses the GATEWAY_OTEL_DEST_USERNAME/GATEWAY_OTEL_DEST_PASSWORD env vars.
pipeline.metrics.gateway.destination.otel.auth.bearer.config	string	An `otelcol.auth.bearer.oteldest` definition.	Raw configuration for an otelcol.auth.bearer block. This uses the GATEWAY_OTEL_DEST_BEARER_TOKEN env var.
pipeline.metrics.gateway.destination.otel.auth.awsSigv4.region	string	`""`	Override the region to sign requests for.
pipeline.metrics.gateway.destination.otel.auth.awsSigv4.roleArn	string	`""`	Override the role ARN to assume for signing requests.
pipeline.metrics.gateway.destination.otel.auth.awsSigv4.config	string	An `otelcol.auth.sigv4.oteldest` definition.	Raw configuration for an otelcol.auth.sigv4 block.
pipeline.metrics.gateway.destination.otel.auth.custom.handler	string	`"{{ fail \"Be sure to set this\" }}"`	Handler for a custom auth handler (Advanced escape hatch). This should point to the definition you used. Most auth types use `.handler` for their capsule export. If you need multiple handlers, you can skip this field and instead modify your otlpExporter.config to not use "mzmon.alloyGateway.otelDest.authHandler".
pipeline.metrics.gateway.destination.otel.auth.custom.config	string	// THIS IS AN EXAMPLE otelcol.auth.basic "oteldest" { client_auth { username = sys.env("GATEWAY_OTEL_DEST_USERNAME") password = sys.env("GATEWAY_OTEL_DEST_PASSWORD") } }	Raw configuration for a custom auth handler.

Monitoring configurations#

Configuration for dashboards, rules, and alerts

Underlying content is generated into pre-rendered/ from the sources under packages/ and embedded via .Files.Get.

Key	Type	Default	Description
dashboards.config.grafana.enabled	bool	`true`	Install the bundled Grafana dashboards. Requires the Grafana operator or a writable Grafana instance.
dashboards.config.grafana.mode	string	`"operator"`	Grafana deployment mode, either "standalone" (the bundled Grafana chart) or "operator" (a separate Grafana Operator instance).
dashboards.config.grafana.manifest.resyncPeriod	string	`"5m"`	Time to sync the dashboard from the manifest
dashboards.config.grafana.manifest.instanceSelector	object	{}	Non-default label selector for a Grafana-operator Grafana instance. Defaults to the labels on the `Grafana` instance this chart creates (see `connections.grafana.labels`), so the two cannot drift.
dashboards.config.grafana.manifest.allowCrossNamespaceImport	string	`inferred`	Allow dashboards to match a Grafana instance outside their own namespace. Left unset, this is inferred — it turns on only when the `Grafana` resource lands in a different namespace than the dashboards, as it does under the `split-namespace` profile. Set it explicitly when pointing `instanceSelector` at an instance this chart does not create. Note that the CRDs forbid turning this back off in place; the resource has to be recreated.
dashboards.config.grafana.manifest.apiTarget	string	`"dashboard.grafana.app/v2"`	Dashboard API Version (v2 or v2beta1)
dashboards.config.datadog.enabled	bool	`false`	Install the bundled Datadog dashboards. Requires Datadog API credentials configured out-of-band.
dashboards.selected	list	[ "env-*" ]	List of dashboard patterns to render

Rule configuration#

Configuration for rules

Key	Type	Default	Description
config.rules.prometheus.enabled	bool	`true`	Install the bundled Prometheus recording and alerting rules as PrometheusRule resources.
config.rules.loki.enabled	bool	`false`	Install the bundled Loki rules.
config.rules.thanos.enabled	bool	`false`	Install the bundled Thanos rules.

Alert configuration#

Configuration for alerts

Key	Type	Default	Description
config.alerts.enabled	bool	`true`	Install the bundled Alertmanager routing and templates.

Scraper configuration#

Configuration for scrapers

Key	Type	Default	Description
config.scrapers.enabled	bool	`true`	Install ServiceMonitors / Alloy scrape configs for Materialize and adjacent components.

Grafana connection configuration#

How to talk to a grafana instance

Key	Type	Default	Description
connections.grafana.mode	string	`"bundled"`	How this establishes its connection to Grafana. `bundled` (default) targets the Grafana deployed by the bundled `grafana` subchart; the URL and admin-credential Secret are derived from it. `external` targets a Grafana you already run — Grafana Cloud, a shared platform Grafana, another cluster — and requires `external.url` plus either `external.apiKey` or `external.adminUser` + `external.adminPassword`. `operator` hands the instance lifecycle to grafana-operator itself, which builds it from the operator's own defaults and is not yet production-ready.
connections.grafana.labels	object	{}	Additional labels applied to the Grafana instance, and to the `instanceSelector` of every Grafana resource this chart targets at it. Merged over a static `monitoring.materialize.cloud/grafana-instance: mzmon` label, which is what keeps the selector non-empty — grafana-operator reads an empty `matchLabels` as every instance, not none. Add to this to narrow the selector further, e.g. to scope per release when two `materialize-monitoring` releases share a cluster.
connections.grafana.external.url	string	`""`	External grafana uri
connections.grafana.external.adminPassword	object	{}	Secret for Grafana admin password
connections.grafana.external.adminUser	object	{}	Secret for Grafana admin user
connections.grafana.external.apiKey	object	{}	Secret for Grafana API key

Datasource configuration#

Datasources provisioned into the Grafana instance

Provisioned as GrafanaDatasource resources, pushed into the same instance the dashboards target. url values are rendered with tpl, so they may reference chart helpers.

Key	Type	Default	Description
connections.datasources.enabled	bool	`true`	Install the bundled datasources. Requires the Grafana operator.
connections.datasources.resyncPeriod	string	`"5m"`	How often the operator re-pushes each datasource.
connections.datasources.editable	bool	`false`	Whether the datasources can be edited in the Grafana UI. Edits are reverted on the next resync either way; this only hides the controls so the reversion is not a surprise.
connections.datasources.thanos.enabled	string	follows `thanos.enabled	Provision the Thanos datasource. Unset follows whether the bundled Thanos is enabled. Set it explicitly to point Grafana at metrics storage this chart does not deploy.
connections.datasources.thanos.name	string	`"Thanos"`	Datasource name, as shown in Grafana.
connections.datasources.thanos.uid	string	`"mzmon-thanos"`	Stable datasource UID.
connections.datasources.thanos.url	string	`"http://thanos-query.{{ include \"mzmon.thanos.namespace\" $ }}.svc:9090"`	Thanos Query endpoint. Rendered with `tpl`.
connections.datasources.thanos.isDefault	bool	`true`	Make this Grafana's default datasource. The bundled dashboards deliberately do not pin a datasource: their `${metricsDatasource}` variable resolves to whichever Prometheus-type datasource is default. With no default, every panel renders empty and Grafana reports no error. Only turn this off if something else in the instance is already the default Prometheus datasource.
connections.datasources.thanos.jsonData	object	{}	Extra `jsonData`, merged over the chart's defaults (`prometheusType: Thanos`, `httpMethod: POST`).
connections.datasources.thanos.secureJsonData	object	{}	Inline `secureJsonData`. Prefer `valuesFrom` for real secrets — this renders into the release manifest.
connections.datasources.thanos.valuesFrom	list	[]	Secret- or ConfigMap-sourced field injection, passed through to the `GrafanaDatasource`. This is the supported way to supply credentials. `valuesFrom: - targetPath: secureJsonData.basicAuthPassword valueFrom: secretKeyRef: name: thanos-basic-auth key: password`
connections.datasources.loki.enabled	string	follows `loki.enabled	Provision the Loki datasource. Unset follows whether the bundled Loki is enabled.
connections.datasources.loki.name	string	`"Loki"`	Datasource name, as shown in Grafana.
connections.datasources.loki.uid	string	`"mzmon-loki"`	Stable datasource UID.
connections.datasources.loki.url	string	`"http://loki-query-frontend.{{ include \"mzmon.loki.namespace\" $ }}.svc:3100"`	Loki read endpoint. Rendered with `tpl`. The Loki gateway is disabled by default, so reads go to the query frontend directly (see `loki.gateway.enabled`).
connections.datasources.loki.tenant	string	follows `pipeline.logging.tenancy.staticTenant	Tenant to read as, sent in the `X-Scope-OrgID` header. The bundled Loki runs `auth_enabled: true`, so reads without this header fail with `no org id`. Unset follows the tenant the pipeline writes to. Set to `""` to send no header, which is only correct against a Loki with `auth_enabled: false`.
connections.datasources.loki.jsonData	object	{}	Extra `jsonData`, merged over the chart's defaults (the tenant header name, and `timeout`).
connections.datasources.loki.secureJsonData	object	{}	Inline `secureJsonData`. Prefer `valuesFrom` for real secrets — this renders into the release manifest.
connections.datasources.loki.valuesFrom	list	[]	Secret- or ConfigMap-sourced field injection, passed through to the `GrafanaDatasource`. This is the supported way to supply credentials.

Bundled subchart configurations#

Configuration for bundled subcharts

Alloy Agent#

Alloy collector instance running close to scrape targets. Pre-egress shaping happens here.

Upstream reference:

Key	Type	Default	Description
alloy-agent.fullnameOverride	string	`"alloy-agent"`	Standard Helm full-name override. We use a static name for deterministic relations.
alloy-agent.namespaceOverride	string	`nil`	Namespace override.
alloy-agent.crds	object	{ "create": false }	Control for the PodLogs crd.
alloy-agent.global.podSecurityContext	object	{ "fsGroup": 473, "runAsGroup": 473, "runAsUser": 473 }	Security context for the alloy agent pods.
alloy-agent.alloy.stabilityLevel	string	`"generally-available"`	Stability level of alloy components.
alloy-agent.alloy.extraEnv	list	[]	Extra environment variables to pass to the alloy agent pod.
alloy-agent.alloy.envFrom	list	[ { "configMapRef": { "name": "mzmon-alloy-agent-env" } }, { "secretRef": { "name": "mzmon-alloy-agent-env", "optional": true } } ]	Environment variable configmaps/secrets to pass to the alloy agent pod.
alloy-agent.alloy.mounts	object	{ "extra": [ { "mountPath": "/tmp", "name": "tmp" } ], "varlog": true }	Volume mounts to expose to alloy agent.
alloy-agent.alloy.securityContext	object	{ "allowPrivilegeEscalation": false, "capabilities": { "drop": [ "ALL" ] }, "readOnlyRootFilesystem": true, "runAsGroup": 473, "runAsNonRoot": false, "runAsUser": 0 }	Security context for the alloy agent containers. The agent MUST run as root in order to be able to read container logs.
alloy-agent.alloy.resources	object	{ "limits": { "cpu": "100m", "memory": "200Mi" }, "requests": { "cpu": "100m", "memory": "200Mi" } }	Resources for the alloy agent containers.
alloy-agent.controller.extraAnnotations	object	{}	Extra annotations to apply to the alloy agent pod. If you are using pulumi, be sure to add `config.kubernetes.io/depends-on: job/mzmon-validate-agent`

Alloy Gateway#

Alloy gateway instance. Cardinality reduction and backend-specific egress happen here.

Upstream reference:

Key	Type	Default	Description
alloy-gateway.fullnameOverride	string	`"alloy-gateway"`	Standard Helm full-name override. We use a static name for deterministic relations.
alloy-gateway.namespaceOverride	string	`nil`	Namespace override.
alloy-gateway.crds	object	{ "create": false }	Control for the PodLogs crd.
alloy-gateway.global.podSecurityContext	object	{ "fsGroup": 473, "runAsGroup": 473, "runAsUser": 473 }	Security context for the alloy gateway pods.
alloy-gateway.alloy.stabilityLevel	string	`"generally-available"`	Stability level of alloy components.
alloy-gateway.alloy.extraEnv	list	[]	Extra environment variables to pass to the alloy gateway pod.
alloy-gateway.alloy.envFrom	list	[ { "configMapRef": { "name": "mzmon-alloy-gateway-env" } }, { "secretRef": { "name": "mzmon-alloy-gateway-env", "optional": true } } ]	Environment variable configmaps/secrets to pass to the alloy gateway pod.
alloy-gateway.alloy.extraPorts	list	[ { "name": "loki", "port": 3100, "protocol": "TCP", "targetPort": 3100 }, { "name": "otlp-grpc", "port": 4317, "protocol": "TCP", "targetPort": 4317 }, { "name": "otlp-http", "port": 4318, "protocol": "TCP", "targetPort": 4318 }, { "name": "prom", "port": 9090, "protocol": "TCP", "targetPort": 9090 } ]	Ports to expose from alloy-gateway.
alloy-gateway.alloy.mounts	object	{ "extra": [ { "mountPath": "/tmp", "name": "tmp" } ], "varlog": false }	Volume mounts to expose to alloy gateway.
alloy-gateway.alloy.securityContext	object	{ "allowPrivilegeEscalation": false, "capabilities": { "drop": [ "ALL" ] }, "readOnlyRootFilesystem": true, "runAsGroup": 473, "runAsNonRoot": true, "runAsUser": 473 }	Security context for the alloy gateway containers.
alloy-gateway.alloy.resources	object	{ "limits": { "cpu": "500m", "memory": "512Mi" }, "requests": { "cpu": "500m", "memory": "512Mi" } }	Resources for the alloy gateway containers.
alloy-gateway.controller.extraAnnotations	object	{}	Extra annotations to apply to the alloy gateway pod. If you are using pulumi, be sure to add `config.kubernetes.io/depends-on: job/mzmon-validate-gateway`
alloy-gateway.serviceAccount.create	bool	`true`	Create a service account for alloy-gateway.
alloy-gateway.serviceAccount.annotations	object	{}	Extra annotations to set on the alloy-gateway service account. Use `eks.amazonaws.com/role-arn` to set up IRSA. Use `iam.gke.io/gcp-service-account` to set up Workload Identity Federation.

Loki#

Bundled Loki backend for logs.

Upstream reference:

Key	Type	Default	Description
loki.fullnameOverride	string	`"loki"`	Standard Helm full-name override. We use a static name for deterministic relations.
loki.namespaceOverride	string	`nil`	Namespace override.
loki.deploymentMode	string	`"Distributed"`	How loki is deployed. We prefer to run in Distributed/Microservices mode.
loki.networkPolicy.enabled	bool	`true`	Whether to enable a network policy for loki In production, this is recommended to be enabled.
loki.networkPolicy.metrics	object	{ "namespaceSelector": {} }	Selector for incoming traffic to metric endpoints. This must be configured manually (usually set to `kubernetes.io/metadata.name: monitoring`).
loki.networkPolicy.ingress	object	{ "namespaceSelector": {} }	Selector for incoming traffic to the read/write endpoints. This must be configured manually (usually set to `kubernetes.io/metadata.name: monitoring`).
loki.networkPolicy.externalStorage	object	{ "cidrs": [ "0.0.0.0/0" ], "ports": [ 443, 80 ] }	Outgoing traffic from loki to the object store. This is usually unrestricted, even in many production settings. Adjust if you have a VPCEndpoint in front of S3/STS or are using a non-standard port.
loki.loki.storage.bucketNames	object	{ "chunks": "\u003cREPLACE-ME\u003e", "ruler": "\u003cREPLACE-ME\u003e" }	Bucket names for object storage. These are required to be populated.
loki.loki.storage.use_thanos_objstore	bool	`true`	Use the thanos object store client
loki.loki.storage.object_store	object	{ "type": "s3" }	Object storage configuration. Modify as needed.
loki.loki.schemaConfig	object	{ "configs": [ { "from": "2024-01-01", "index": { "period": "24h", "prefix": "loki_index_" }, "object_store": "s3", "schema": "v13", "store": "tsdb" } ] }	Schema configuration for the loki TSDB. This is append-only and MUST be copied into projects and mutated on upgrades (if v13 stops being in use).
loki.loki.commonConfig	object	{ "replication_factor": 3 }	Cluster-wide replication factor. Three is our HA floor and the reason at least three ingesters are required. (This matches the upstream default; surfaced here because it is load-bearing.)
loki.loki.limits_config	object	{ "ingestion_burst_size_mb": 20, "ingestion_rate_mb": 10, "max_global_streams_per_user": 10000, "reject_old_samples": true, "reject_old_samples_max_age": "168h", "retention_period": "30d", "volume_enabled": true }	Per-tenant limits, sized for a medium install. These are protective caps (per environment), not the expected volume — see Operating > Production Best Practices for the throughput envelope.
loki.loki.limits_config.retention_period	string	`"30d"`	Default retention before the compactor deletes logs. Upstream defaults to infinite retention; we set a real bound.
loki.loki.limits_config.ingestion_rate_mb	int	`10`	Per-tenant ingestion rate / burst, in MB. Per environment, not the fleet aggregate.
loki.loki.limits_config.max_global_streams_per_user	int	`10000`	Active-stream ceiling per tenant; a cardinality guard.
loki.loki.limits_config.reject_old_samples	bool	`true`	Reject writes too far in the past.
loki.loki.limits_config.volume_enabled	bool	`true`	Enable the log-volume endpoints Grafana's logs drilldown relies on.
loki.loki.compactor	object	{ "delete_request_store": "s3", "retention_delete_delay": "8h", "retention_enabled": true }	Compactor configuration (distinct from the compactor deployment below). Retention is enforced by the compactor and is OFF in upstream defaults.
loki.loki.compactor.delete_request_store	string	`"s3"`	Where delete requests are stored; matches the object-store backend.
loki.loki.compactor.retention_delete_delay	string	`"8h"`	Grace period before retention/deletes actually remove data.
loki.gateway.enabled	bool	`false`	Disable gateway by default. We recommend using alloy-gateway for loki writes. Use the query-frontend for loki reads.

Loki Microservice Configuration#

Configuration for each loki microservice.

https://grafana.com/docs/loki/latest/get-started/components/

Key	Type	Default	Description
loki.distributor.enabled	bool	`true`	Enable the distributor microservice. Distributor is required. The Distributor is the stateless front door for writes. It validates incoming streams against per-tenant limits, enforces rate limits, and normalizes labels, then splits the batch into individual streams and forwards each to the owning ingesters.
loki.distributor.replicas	string	`nil`	Number of replicas for the distributor microservice. If autoscaling is enabled, this should be set to null.
loki.distributor.kind	string	`"Deployment"`	Type of workload for the distributor.
loki.distributor.autoscaling	object	{ "enabled": true, "maxReplicas": 4, "minReplicas": 2, "targetCPUUtilizationPercentage": 60, "targetMemoryUtilizationPercentage": 75 }	Configuration for autoscaling of distributor For production, this is recommended to be enabled. We provide opinionated defaults.
loki.distributor.service.type	string	`"ClusterIP"`	Service type for the distributor microservice. Without a gateway, this is the exposed write component.
loki.distributor.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	Configuration for pod disruption budget for the distributor microservice.
loki.distributor.resources	object	{ "requests": { "cpu": "150m", "memory": "256Mi" } }	Resource requests (medium install). Tune per profile.
loki.ingester.enabled	bool	`true`	Enable the ingester microservice. Ingester is required. The Ingester is the stateful heart of the write path, and it also serves the most recent reads. It buffers incoming entries into per-stream in-memory chunks, compresses them, and periodically flushes those chunks and their index to object storage.
loki.ingester.replicas	int	`3`	Number of ingester replicas. Three is the floor for replication_factor 3. Scale UP on memory / stream-cardinality (so streams shard across the ring), not on bytes — with replicas == replication_factor every ingester holds every stream.
loki.ingester.kind	string	`"StatefulSet"`	Ingesters are stateful.
loki.ingester.persistence.enabled	bool	`false`	Run ingesters ephemerally: no PVC, node-local emptyDir for the WAL and not-yet-flushed chunks. Durability comes from replication_factor 3, not from disk — a rescheduled ingester starts fresh and the ring backfills from its peers. This avoids EBS zonal pinning and the slow volume detach/attach that PVCs incur during node replacement.
loki.ingester.terminationGracePeriodSeconds	int	`60`	Termination grace period for ingesters. flush-on-shutdown is best-effort; if the node force-kills sooner, the other replicas still flush their copies, so do not depend on a long grace period for durability. Kept modest to survive enterprise force-kill windows.
loki.ingester.affinity	object	{ "podAntiAffinity": { "requiredDuringSchedulingIgnoredDuringExecution": null } }	Pod affinity for ingesters. Drop the chart's default hard per-host anti-affinity so host spread can be soft (see topologySpreadConstraints); zone spread stays hard. We null the nested list rather than the whole `affinity` map: `affinity: {}` is a no-op against the subchart default, and `affinity: null` clears it but makes helm-unittest log a noisy "cannot overwrite table" warning — nulling the list avoids both.
loki.ingester.topologySpreadConstraints	list	[ { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "ingester", "app.kubernetes.io/instance": "{{ .Release.Name }}", "app.kubernetes.io/name": "{{ include \"loki.name\" . }}" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "minDomains": 2, "nodeTaintsPolicy": "Honor", "topologyKey": "topology.kubernetes.io/zone", "whenUnsatisfiable": "DoNotSchedule" }, { "labelSelector": { "matchLabels": { "app.kubernetes.io/component": "ingester", "app.kubernetes.io/instance": "{{ .Release.Name }}", "app.kubernetes.io/name": "{{ include \"loki.name\" . }}" } }, "matchLabelKeys": [ "controller-revision-hash" ], "maxSkew": 1, "topologyKey": "kubernetes.io/hostname", "whenUnsatisfiable": "ScheduleAnyway" } ]	Topology spread for ingesters: hard across zones, soft across hosts. A pod that cannot satisfy the hard zone rule goes Pending, which is the signal Karpenter (or the cluster-autoscaler) uses to add a node in the deficient zone — soft rules cannot summon capacity that way. Host spread is soft so pods still schedule when nodes are momentarily scarce.
loki.ingester.zoneAwareReplication.enabled	bool	`false`	Enable zone-aware replication for the ingester microservice. Not needed at our size: topologySpreadConstraints give the AZ spread without the complexity.
loki.ingester.podDisruptionBudget	object	{ "enabled": true, "maxUnavailable": 1 }	PDB for the ingester. Protect ingest quorum across rollouts and node drains. Do not set maxUnavailable >= 2 to avoid potential quorum loss.
loki.ingester.resources	object	{ "requests": { "cpu": "500m", "memory": "1Gi" } }	Resource requests (medium install). The memory limit is intentionally left unset: an OOM-kill drops in-memory / WAL-buffered logs, so we alert on usage rather than cap hard.
loki.querier.enabled	bool	`true`	Enable the querier microservice. Querier is required. It executes LogQL, reading recent data from ingesters and historical data from object storage.
loki.querier.replicas	string	`nil`	Replicas are managed by autoscaling; leave null.
loki.querier.autoscaling	object	{ "enabled": true, "maxReplicas": 4, "minReplicas": 2, "targetCPUUtilizationPercentage": 60, "targetMemoryUtilizationPercentage": 75 }	Autoscale the stateless read workers with query load.
loki.querier.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for querier.
loki.querier.resources	object	{ "requests": { "cpu": "250m", "memory": "512Mi" } }	Resource requests. This assumes a medium install by default.
loki.queryFrontend.enabled	bool	`true`	Enable the query-frontend microservice. Query-frontend is required. It queues and splits queries and caches results.
loki.queryFrontend.replicas	string	`nil`	Static replicas for query-frontend when autoscaling is not enabled.
loki.queryFrontend.kind	string	`"Deployment"`	Kind of workload for query-frontend. Deployments are recommended for stateless components.
loki.queryFrontend.autoscaling	object	{ "enabled": true, "maxReplicas": 4, "minReplicas": 2, "targetCPUUtilizationPercentage": 60, "targetMemoryUtilizationPercentage": 75 }	Configure autoscaling for query-frontend.
loki.queryFrontend.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for query-frontend.
loki.queryFrontend.resources	object	{ "requests": { "cpu": "100m", "memory": "256Mi" } }	Resource requests for query-frontend.
loki.queryScheduler.enabled	bool	`true`	Enable the query-scheduler microservice. Recommended for scaled deployments: it decouples the read queue from the query-frontend so the two scale independently. Two replicas for availability.
loki.queryScheduler.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for query-scheduler.
loki.queryScheduler.resources	object	{ "requests": { "cpu": "100m", "memory": "256Mi" } }	Resources for the query-scheduler microservice.
loki.compactor.enabled	bool	`true`	Enable the compactor. Compactor is required and MUST be a singleton: it compacts the index and enforces retention against shared object storage.
loki.compactor.replicas	int	`1`	Compactor replicas. This MUST be one, since it runs as a singleton.
loki.compactor.kind	string	`"StatefulSet"`	Kind of workload for the compactor. This must be a statefulset.
loki.compactor.persistence.enabled	bool	`false`	Run the compactor ephemerally (no PVC, node-local emptyDir). Its local dir is just a working copy of the object-store index, and compaction is idempotent and off the critical path — losing it only restarts the current cycle. Ephemeral also lets the singleton float freely between zones instead of being pinned by a PVC. Our production index is ~30MB, so re-downloading each cycle is negligible.
loki.compactor.resources	object	{ "requests": { "cpu": "250m", "memory": "512Mi" } }	Resources for the compactor.
loki.indexGateway.enabled	bool	`true`	Enable the index-gateway. Index-gateway is required: it serves index lookups so queriers do not each download the whole index from object storage.
loki.indexGateway.persistence	object	{ "enabled": false }	Run the index-gateway ephemerally (no PVC, node-local emptyDir). The local index is a read-through cache of object storage, not authoritative — a fresh pod just re-downloads what it queries. This keeps the read path off EBS zonal pinning; the only cost is a little cold-start warmup and a burst of index reads after a reschedule.
loki.indexGateway.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for index-gateway.
loki.indexGateway.resources	object	{ "requests": { "cpu": "200m", "memory": "512Mi" } }	Resources for the index-gateway.
loki.ruler.enabled	bool	`true`	Enable the ruler. The ruler evaluates LogQL alerting and recording rules. Recording-rule samples are remote-written back through alloy-gateway to the metric store.
loki.ruler.persistence.enabled	bool	`true`	Keep a PVC for the ruler (unlike the other components, which are ephemeral). Rule definitions come from object storage, but the ruler's remote-write WAL buffers recording-rule samples when the metric store is unreachable — genuinely useful durability in the run-up to an incident, exactly when you don't want to drop derived signals.
loki.ruler.podDisruptionBudget	object	{ "enabled": true, "minAvailable": 1 }	PDB for the ruler.
loki.ruler.resources	object	{ "requests": { "cpu": "100m", "memory": "256Mi" } }	Resources for the ruler.
loki.chunksCache	h5	`{"allocatedMemory":2048}`	Chunk cache (memcached). Default allocation is sized for very large installs; we shrink it to match our volumes. The results cache keeps its upstream default.
loki.monitoring.serviceMonitor.enabled	bool	`true`	Enable a ServiceMonitor for the loki microservices.
loki.lokiCanary	h5	`{"enabled":true, "kind":"Deployment", "lokiurl":"loki-query-frontend:3100", "push":false}`	End-to-end write→read canary for meta-monitoring. On by default upstream; surfaced here because self-monitoring the log store is a first-class requirement for us.

Thanos#

Bundled Thanos backend for long-term metrics.

Upstream reference:

Key	Type	Default	Description
thanos.fullnameOverride	string	`"thanos"`	Standard Helm full-name override. We use a static name for deterministic relations.
thanos.namespaceOverride	string	`nil`	Namespace override.
thanos.global.pdb	object	{ "enabled": true, "maxUnavailable": 1 }	PodDisruptionBudgets for every Thanos component. Upstream ships these off. One switch turns them on for all components, and the per-component `pdb.enabled` cannot opt back out — the subchart templates test `or <component>.pdb.enabled global.pdb.enabled`. `maxUnavailable` rather than `minAvailable`, deliberately: It scales with replica count instead of pinning an absolute floor. On the single-replica Compactor, `minAvailable: 1` would block every voluntary eviction and hang node drains indefinitely. `maxUnavailable: 1` permits the eviction, so the singleton is a harmless no-op rather than a drain deadlock. On Receive it must stay within what write quorum tolerates (`replicationFactor - ((replicationFactor / 2) + 1)`), which is 1 at the replication factor of 3 set below. A validator enforces this. This matches the Loki convention (ingester `maxUnavailable: 1`).
thanos.query	object	{ "autoscaling": { "enabled": true, "maxReplicas": 5, "minReplicas": 2, "targetCPUUtilizationPercentage": 80 }, "enabled": true }	Thanos Query configuration. Query provides a PromQL query endpoint.
thanos.query.autoscaling	object	{ "enabled": true, "maxReplicas": 5, "minReplicas": 2, "targetCPUUtilizationPercentage": 80 }	Horizontal autoscaling for Query. Query is a stateless PromQL fan-out, so it is the natural place to autoscale: no local state, no ring membership, no PVC.
thanos.receive.enabled	bool	`true`	Enable Thanos receiver. Receive provides a Prometheus remote_write-compatible endpoint.
thanos.receive.mode	string	`"standalone"`	Whether to split receive distributors from ingesters. mode=split is not very stable in the helm chart
thanos.receive.replicaCount	int	`3`	Number of receive replicas.
thanos.storegateway	object	{ "enabled": true }	Thanos Store Gateway configuration. Store Gateway provides historical block querying. Autoscaling is available upstream but left off deliberately. Store Gateway is a PVC-backed StatefulSet that syncs the bucket’s block index on startup, so scale-up is slow to become useful (it serves nothing until the index is warm) and CPU-triggered scaling reacts long after the load that triggered it. Scale-down also leaves orphaned PVCs behind, since StatefulSet volumes are not reclaimed. Size it deliberately instead.
thanos.compactor	object	enabled with default retention policies	Thanos Compactor configuration. Compactor provides block compaction and downsampling.
thanos.compactor.retention	object	{ "resolution1h": "365d", "resolution5m": "90d", "resolutionRaw": "30d" }	Retention policies for Thanos Compactor downsampled data
thanos.queryFrontend	object	{ "autoscaling": { "enabled": true, "maxReplicas": 5, "minReplicas": 2, "targetCPUUtilizationPercentage": 80 }, "enabled": false }	Thanos Query Frontend configuration. Query Frontend provides query parallelization and result caching. Only required for production. Note that enabling this is not sufficient on its own: point `connections.datasources.thanos.url` at the query-frontend Service too, or reads keep going straight to Query and the cache is never consulted. A validator warns when the two disagree.
thanos.queryFrontend.autoscaling	object	{ "enabled": true, "maxReplicas": 5, "minReplicas": 2, "targetCPUUtilizationPercentage": 80 }	Horizontal autoscaling for Query Frontend. Stateless like Query, so the same reasoning applies. Inert until `queryFrontend.enabled` is true.
thanos.ruler	object	{ "enabled": false }	Thanos Ruler configuration. Ruler provides alerting and recording rules evaluation.

Grafana Operator#

Bundled Grafana-operator for managing Grafana instances.

Upstream references:

Key	Type	Default	Description
grafana-operator.fullnameOverride	string	`"grafana-operator"`	Standard Helm full-name override. We use a static name for deterministic relations.
grafana-operator.namespaceOverride	string	`nil`	Namespace override.
grafana-operator.crds	object	{ "immutable": true }	CRD behavior. The Grafana Operator CRDs are owned by the `materialize-monitoring-crds` chart, which vendors a deflated copy of them. The operator chart offers no way to skip its own CRDs outright — `immutable` only chooses where they come from — so keep this `true`: that keeps them out of this chart's release manifest and leaves them install-only, which `helm install --skip-crds` drops entirely. Setting it `false` makes this chart template and upgrade the CRDs itself, fighting the CRDs chart for ownership.

Grafana Instance#

Bundled Grafana for dashboard rendering.

Upstream reference:

https://github.com/grafana-community/helm-charts/blob/main/charts/grafana/values.yaml

Key	Type	Default	Description
grafana.fullnameOverride	string	`"grafana"`	Standard Helm full-name override. We use a static name for deterministic relations. `connections.grafana.mode: bundled` derives the URL it hands grafana-operator from this, so a release-name-derived name would leave the operator dialing a host that does not resolve.
grafana.namespaceOverride	string	`nil`	Namespace override.

Alertmanager#

Bundled Alertmanager for routing alerts emitted by the rule packages.

Key	Type	Default	Description
alertmanager.persistence.size	string	`"4Gi"`	Volume for silences and the notification log. Sized by cloud disk minimums, not by Alertmanager, which needs kilobytes. The subchart default of 50Mi is below the 4 GiB floor on GCP Hyperdisk and Azure managed disks, so provisioning fails there.

Kube State Metrics#

kube-state-metrics for Kubernetes resource-state metrics consumed by Materialize-adjacent dashboards.

Upstream reference:

https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-state-metrics/values.yaml

Metrics Server#

metrics-server for pod and node resource usage; only needed when the cluster does not already ship one.

Upstream reference:

https://github.com/kubernetes-sigs/metrics-server/blob/master/charts/metrics-server/values.yaml

Key	Type	Default	Description
metrics-server.replicas	int	`1`	Number of replicas for metrics-server.

materialize-monitoring Helm Reference#

Source Code#

Requirements#

Storage Requirements#

Included Subcharts#

Values#

Globals#

Main chart configuration#

Subchart enablement (Helm tags)#

Materialize Integration#

Pipeline configuration#

Log configuration#

Metrics configuration#

OpenTelemetry/OTLP destinations.#

OpenTelemetry Authentication Configuration#

Monitoring configurations#

Rule configuration#

Alert configuration#

Scraper configuration#

Grafana connection configuration#

Datasource configuration#

Bundled subchart configurations#

Alloy Agent#

Alloy Gateway#

Loki#

Loki Microservice Configuration#

Thanos#

Grafana Operator#

Grafana Instance#

Alertmanager#

Kube State Metrics#

Metrics Server#

`materialize-monitoring` Helm Reference#