Installing via Terraform#

For teams standing up self-managed Materialize with materialize-terraform-self-managed. The observability stack comes up with the cluster, from the same modules, with no separate install step.

Shipped, and on by default. Since materialize-terraform-self-managed v11 the monitoring stack is opt-out rather than opt-in — enable_observability defaults to true in every example root. All three clouds are wired: AWS, GCP, and Azure. The legacy Prometheus + Grafana modules are retired.

What you get#

One module per cloud creates the storage and identity the stack needs, then installs the Helm charts at a pinned version:

Cloud-side, per backend: one bucket, and one IAM role (AWS) or Google service account with a Workload Identity binding (GCP). Plus, unless you turn it off, one small PostgreSQL instance for Grafana — see Reaching Grafana.

Grafana gets an internal L4 load balancer with the rest of the stack on all three clouds. DNS, TLS, and an identity provider are still yours — see Reaching Grafana.

Usage#

Observability is a module block in each example root, gated on one variable:

module "monitoring" {
  count  = var.enable_observability ? 1 : 0
  source = "../../modules/monitoring"

  prefix     = var.name_prefix
  project_id = var.project_id
  region     = var.region

  namespace        = "monitoring"
  create_namespace = false # the operator module creates it

  node_selector = local.generic_node_labels

  materialize_instance_namespace = local.materialize_instance_namespace
  materialize_operator_namespace = local.materialize_operator_namespace

  depends_on = [module.operator, module.gke, module.generic_nodepool, module.coredns]
}

If you start from an example root, that block is already there and already on. To opt out, set this in your terraform.tfvars:

enable_observability = false

See the tfvars reference below for the rest.

tfvars reference#

Variables you would realistically set in a terraform.tfvars at an example root. Everything else has a default that suits the supported topology.

Turning it off#

Sizing and placement#

These are set on the monitoring module block rather than as root variables, so change them there or add a matching root variable.

Storage and retention#

StorageClass on GCP C4 and N4 node pools#

The C4 and N4 machine families accept only Hyperdisk. They cannot attach Persistent Disk of any type, and GKE’s default standard-rwo class is pd-balanced, so every PVC-backed workload hangs:

AttachVolume.Attach failed for volume "pvc-...":
  pd-balanced disk type cannot be used by c4-standard-8 machine type, badRequest

The other classes GKE creates by default — premium-rwo (pd-ssd) and standard (pd-standard) — are Persistent Disk too, so none of them work either. GKE does not create a Hyperdisk class for you:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hyperdisk-balanced
provisioner: pd.csi.storage.gke.io
parameters:
  type: hyperdisk-balanced
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: true

Then storage_class = "hyperdisk-balanced".

Switching the class on an existing install does not migrate the volumes. A StatefulSet’s volumeClaimTemplates are immutable, so the old PVCs have to be deleted before the new class takes effect — and deleting them discards their contents. That is cheap for these five (Thanos re-downloads blocks from object storage, and the Alertmanager and ruler volumes hold local state, not the rules themselves), but it is not a no-op.

Extra metrics destinations#

Thanos is always the primary metric store. These fan out in addition to it, from the Alloy gateway.

min_importance is a cost control, not a filter for convenience. Cloud Monitoring bills per custom metric and per sample, so the tier you pick sets the bill. recommended covers the dashboards; all sends the full surface and is a diagnostic setting, not a steady state.

The tiers come from the same metric-tiers.yaml the chart uses, so a Terraform-selected tier and a Helm-selected one always mean the same set. See Metrics > Storing for what each tier contains.

Authentication is ADC only — there is no key-file path. Without the Workload Identity binding the module creates, the exporter falls back to the node’s service account, which works only if that account happens to hold roles/monitoring.metricWriter.

Datadog and generic OTLP (Honeycomb, Grafana Cloud, your own collector) are inputs on the materialize-monitoring module itself — datadog_metrics / datadog_api_key and otlp_metrics / otlp_auth_header_secrets / otlp_auth_bearer_token. Neither needs cloud resources, so they are not surfaced as flat wrapper variables the way Google Cloud Monitoring is; set them on the wrapper’s monitoring module block. Credentials reach the gateway through a Secret the module creates rather than through the Helm values, and rotating one rolls the gateway. See Metrics > Storing and the module README.

Integration#

Grafana persistence and exposure#

Set on the wrapper’s monitoring module block, with root variables for them in the examples. See Reaching Grafana for what each one implies.

Anything else#

additional_values = [
  yamlencode({
    thanos = {
      receive = { replicaCount = 5 }
    }
  }),
]

The full input and output list is in the Terraform variable reference. Every chart value is reachable through additional_values; see the Helm values reference.

Outputs#

$ terraform output grafana_url
"http://grafana.monitoring.svc.cluster.local"

$ terraform output -raw grafana_admin_password

Reaching Grafana#

Out of the box Grafana is ClusterIP, so access is a port-forward:

kubectl -n monitoring port-forward svc/grafana 3000:80

Log in as admin with the password from terraform output -raw grafana_admin_password.

That is the right default, not a gap: the only account is the generated admin, and every datasource behind Grafana reads every metric in Thanos and every log in the tenant. Exposing it is a deliberate step with three parts.

1. Give it somewhere to keep state#

Do this first. Grafana stores users, service accounts and tokens, annotations, dashboard versions and permissions, preferences, and alert-rule state in a database of its own, and the chart default is SQLite on an emptyDir — lost on every restart, upgrade, and reschedule. An exposed Grafana that silently discards everything its users build in it is worse than an unreachable one.

The per-cloud wrapper modules provision a dedicated small PostgreSQL instance for it, and the examples turn that on wherever enable_observability is on:

Dedicated rather than a database inside the Materialize instance, for reasons that differ per cloud and happen to agree — RDS has no API to add a database to an existing instance, and an Azure Flexible Server has one administrator login and no ARM resource for extra roles. The default sizes are the smallest each cloud offers, which is enough: Grafana’s state is small and its query rate is a handful per page load.

Switching an existing install from SQLite to PostgreSQL does not carry state over — Grafana has no migration between the two. Export what matters through the HTTP API first.

2. Put a load balancer in front#

All three clouds take the same variable, grafana_load_balancer, and produce an L4 load balancer. How it is built differs, because the deterministic option does:

On AWS that means the address is a Terraform attribute rather than something read back after apply, so grafana_url is known at plan time — and the allowlist is security-group rules on the NLB rather than loadBalancerSourceRanges.

The examples pass it unconditionally, alongside the Materialize console and balancerd load balancers, and host is optional — the load balancer answers on an address of its own either way, and the hostname only fixes root_url.

The load balancer does not terminate TLS. An L4 load balancer passes bytes through, so TLS terminates at the Grafana pod or not at all — and until Grafana has a certificate it serves plain HTTP, session cookie and admin password included.

The modules can issue that certificate; see Giving Grafana a certificate below. Until it is in place, treat exposure as internal-only, and do not set security.cookie_secure — it marks the cookie Secure, the browser then refuses to send it over the plain-HTTP connection that works, and login stops working entirely.

L7 — an ALB, a GCP Application Load Balancer, an Azure Application Gateway — is the intended end state for public exposure, because a WAF and authentication at the edge are the two things L4 cannot do. It is deferred rather than rejected: Azure has no ingress-controller module yet, and the chart’s Gateway API support is still BETA, so adopting Ingress now would mean migrating twice. See Ingress and Service are not interchangeable.

On AWS the Load Balancer Controller has to already be installed — it reconciles the TargetGroupBinding that attaches the NLB’s target group to the Grafana Service — and the examples install it as module.aws_lbc.

Both are internal by default, and both read the internal_load_balancer and ingress_cidr_blocks variables the Materialize load balancers already use, with the presence check enforced by a validation block copied from the repo’s nlb module rather than left as a convention.

# At an example root
enable_observability = true

# Optional on all three — the load balancer answers on an address of its own
# either way, and this only fixes `root_url` and the certificate name.
grafana_host = "grafana.example.com"

# Public instead of internal. Narrow the allowlist yourself: it defaults to
# ["0.0.0.0/0"], inherited from the variable the Materialize load balancers use,
# which is only a sensible default while the load balancer is internal.
internal_load_balancer = false
ingress_cidr_blocks    = ["203.0.113.0/24"]

DNS is the one thing the modules cannot do for you. Neither the Ingress nor the Service publishes the hostname, and Terraform has no view of your zone. An ACME challenge against a name that does not resolve is the usual way this gets noticed.

3. Give Grafana a certificate#

Behind an L4 load balancer the certificate has to exist in the cluster, because the pod is the only thing left that can terminate TLS. The module issues it from cert-manager:

certificates_enabled       = true
issuer_ref                 = { name = "letsencrypt-prod", kind = "ClusterIssuer" }
grafana_external_dns_names = ["grafana.example.com"]

issuer_ref covers both the browser-facing certificate and the in-cluster ones, unless internal_issuer_ref overrides the latter. Leaving it null with certificates_enabled = true bootstraps a self-signed root scoped to the release — right for the internal mesh, but meaningless to a browser, so a public name needs a real issuer.

Setting grafana_external_dns_names with no issuer_ref issues nothing, and the plan says so. That combination is a validation error, not a silent no-op, precisely because it looks configured.

If instead an L7 load balancer terminates with a cloud-managed certificate (ACM, Google Certificate Manager, Azure Key Vault), the key never enters the cluster — leave this empty and attach the certificate by annotation through grafana.service.annotations in additional_values.

The module issues the certificate; wiring Grafana to serve it is grafana.ini.server plus a secret mount, through additional_values. See Terminating TLS.

Separately, internal_tls (off / encrypt / present / authenticate) moves the in-cluster hops off plaintext. It also requires certificates_enabled. On a fresh install go straight to authenticate; on a running stack step through present first, or a server that rolls before its writers stops ingestion until they catch up.

4. Configure an identity provider#

Until you do, the generated admin password is the whole of the access control. Everything under grafana.ini is passed through verbatim, so any provider Grafana supports is reachable — through additional_values on the Terraform path. The chart warns at render time when an exposed Grafana has none.

See Authentication for the wiring, including why client secrets cannot live in grafana.ini and how to map an IdP group claim onto Grafana roles.

Scheduling#

node_selector reaches every centralized workload — Loki, Thanos, Grafana, Alertmanager, kube-state-metrics, and the Alloy gateway — but deliberately not the Alloy agent.

The agent is a DaemonSet whose job is to collect from every node. A node selector would confine it to one workload pool and silently stop collecting logs and node metrics from everywhere else. tolerations do reach it, because tolerations widen where a pod may run, which is exactly what a DaemonSet wants.

Capacity#

The stack is meaningfully larger than the Prometheus + Grafana pair it replaces: microservice Loki, Thanos, Grafana, Alertmanager, kube-state-metrics, and two Alloy roles.

If your node pool is sized for the old shape, the first apply lands unschedulable pods. Either grow the pool or start at sizing = "small".

Going to production#

The modules cover the cloud-resource half of a production deployment — buckets, workload identity, version pinning — but not the sizing, retention, and capacity decisions that only you can make.

Production Best Practices is the checklist, tagged by owner. Start with what the Terraform path already handles, then work the items still tagged [operator].

Three that catch people out on a first production install:

• A default StorageClass must exist. Several components are PVC-backed and the modules do not create one.
• Retention is enforced in-cluster, not by the bucket. metrics_retention_days defaults to off for a reason — see the Thanos checklist.
• Grafana has no identity provider until you configure one. The modules give it a durable database and can put a load balancer in front of it, but not an IdP — see Authentication and the Grafana checklist.

Pinning a different version#

Each wrapper module pins the common module to a released tag:

# In {aws,gcp,azure}/modules/monitoring/main.tf
source = "github.com/MaterializeInc/materialize-monitoring//terraform/modules/materialize-monitoring?ref=materialize-monitoring/v0.17.0"

To try an unreleased change, point source at a branch instead — ?ref=my-branch.

The ref is the only version you set. The module reads its chart version from the Chart.yaml shipped beside it, so the two cannot disagree — chart_version exists only to pin something different deliberately.

A local path works too, but it must be relative. Terraform copies a module referenced by an absolute path into .terraform/modules/ without the chart directory beside it, and the sizing profiles stop resolving. The module raises a precondition rather than failing quietly.

Migrating from the previous stack#

The prometheus and grafana modules are replaced. Applying the change:

• Destroys the prometheus and grafana releases and their PersistentVolumeClaims. Up to 15 days of local Prometheus data goes with them — there is no backfill, and collection restarts at install. Anything hand-created in the old Grafana does not carry over.
• Replaces the prometheus_url output with metrics_url and logs_url.
• Creates new cloud resources: buckets and identities per backend.

enable_observability keeps its name and defaults.

Tearing it down#

terraform destroy works unaided now: the chart’s pre-delete hook removes the Grafana custom resources while grafana-operator is still running to clear its finalizers, and blocks until they are gone. The module already orders its two releases so the main release is destroyed before the CRDs release, and the hook covers the ordering within a release that Terraform cannot control.

Two cases still deadlock:

• A release applied before the hook existed. Helm runs pre-delete hooks from the stored release manifest, so bump the module and terraform apply once before destroying.
• Removing the CRDs first, which takes the resource types with it and leaves the hook failing on an unknown type.

In either case, delete them by hand first:

kubectl -n monitoring delete grafanadatasources,grafanamanifests --all

See Operating > Uninstalling for the mechanism and for recovering a teardown that is already stuck.

If you are not using Terraform#

The Helm charts are the full-fidelity surface, and everything above is a thin layer over them. Terraform’s job is the part Helm cannot do: creating the buckets, granting workload identity, and pinning a version.