Configuring Alerting through Terraform#

The materialize-monitoring Terraform module takes the same alerting configuration as the chart, as four inputs. This page shows how each one is written in HCL. What the settings mean is covered on Configuring Alerting for the rules and Alert Channels for where alerts go, and applies unchanged here.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

The inputs#

Every attribute of alert_rules and alerting is optional. An attribute left unset is left out of the Helm values, so the chart’s default applies. A map the chart already fills, such as alerting.presets, merges with the keys a caller sets rather than being replaced by them.

A default install configures no receiver, so every alert reaches mzmon-null, which notifies nobody.

Receivers and routing#

The Alertmanager-native parts of alerting pass through as written: receivers, routes, inhibit_rules, time_intervals, templates and global. Their field names are Alertmanager’s own, so every example on Alert Channels translates by writing its YAML as HCL. The module types these parts as any, so an integration Alertmanager supports needs no module release.

The Alert Channels pager-and-channel example, with a route sending workload alerts to the team that owns the clusters:

module "monitoring" {
  # ...

  alerting = {
    preset = "critical-infrastructure"

    receivers = {
      oncall = {
        class = "page"
        route = { group_wait = "10s", repeat_interval = "1h" }
        config = {
          pagerduty_configs = [{
            routing_key_file = "/etc/alertmanager/secrets/alertmanager-receivers/pagerduty-key"
            send_resolved    = true
          }]
        }
      }
      platform = {
        class = ["high", "normal"]
        config = {
          slack_configs = [{
            channel       = "#platform-alerts"
            api_url_file  = "/etc/alertmanager/secrets/alertmanager-receivers/slack-url"
            send_resolved = true
          }]
        }
      }
      data-team = {
        config = {
          slack_configs = [{
            channel      = "#data-platform"
            api_url_file = "/etc/alertmanager/secrets/alertmanager-receivers/data-team-slack-url"
          }]
        }
      }
    }

    routes = {
      extra = [{
        matchers = ["audience=\"workload\""]
        receiver = "data-team"
        continue = true
      }]
    }
  }

  alerting_receiver_secrets = {
    "pagerduty-key"       = var.pagerduty_routing_key
    "slack-url"           = var.platform_slack_webhook
    "data-team-slack-url" = var.data_team_slack_webhook
  }
}

Every bundled alert carries an audience label, platform or workload, which is what the extra route matches. A matcher is an Alertmanager string, so the quotes inside it are escaped in HCL.

Credentials#

A receiver’s credentials MUST NOT be written into alerting. Anything in the Helm values is readable with helm get values by anyone who can read the release Secret, and the chart fails the render on an inline credential.

A receiver instead names a file under /etc/alertmanager/secrets/alertmanager-receivers/, through its field’s _file variant: api_url_file, routing_key_file, credentials_file. alerting_receiver_secrets supplies those files. Each of its keys becomes a key of the alertmanager-receivers Secret, which the chart mounts at that path, so a receiver reads the key slack-url as /etc/alertmanager/secrets/alertmanager-receivers/slack-url.

The credentials are still stored in Terraform state, as every Terraform-managed Secret is. State SHOULD be kept where only the people who could read the Secret itself can read it.

A credential kept in a Secret the module does not create is mounted through alertmanager.extraSecretMounts in additional_values; see Further Secrets.

Under split-namespace#

The chart’s split-namespace profile runs Alertmanager in a namespace of its own, alertmanager. The Secret has to be created there, so alertmanager_namespace = "alertmanager" MUST be set alongside the profile. The module cannot infer it, because the profile is applied through additional_values, and the namespace has to exist before apply.

Tuning the bundled rules#

alert_rules maps onto the chart’s rules.*. Configuring Alerting describes the default set, capabilities, the audience split and the workload tiers.

module "monitoring" {
  # ...

  alert_rules = {
    # This deployment runs a dedicated CockroachDB for its metadata.
    capabilities = ["crdb-dedicated"]

    # Beyond the default set: the rules for that database, user-cluster
    # freshness, which is opt-in because some clusters are behind by design,
    # and the infrastructure memory rules, which read the workload tiers.
    selected = ["crdb", "cluster-falling-behind", "cluster-stale", "infra_memory"]
    disabled = ["pods-stuck-in-waiting"]

    overrides = {
      # Large clusters here take hours to hydrate with nothing wrong.
      cluster-hydration-stuck = { for_duration = "6h" }
      cluster-cpu-high        = { labels = { severity = "notice" } }
    }

    excluded_namespaces = ["materialize-scratch"]

    # Replaces the chart's list for this tier only.
    infra_workloads = {
      core = ["coredns", "kube-proxy", "aws-node", "karpenter"]
    }
  }
}

An override’s duration is for_duration, the rule’s for. HCL reads { for = "6h" } as a for expression, so the attribute has a name of its own. An override’s severity MUST stay critical, warning or notice, and its audience platform or workload, so the routes still match.

Checking the result#

Most mistakes fail before anything is installed, and the rest fail the Helm render at apply.

terraform plan shows the composed Helm values in the helm_release resource, with alerting_receiver_secrets as (sensitive value). After apply, Checking a configuration shows how to confirm what the running Alertmanager loaded and which receiver an alert would reach.

Through the per-cloud wrappers#

The per-cloud wrappers in materialize-terraform-self-managed pass all four inputs through under the same names. A deployment built from one of their example roots sets them on the wrapper’s monitoring module block, written exactly as in the examples above. alertmanager_namespace defaults to the wrapper’s namespace, and is set to alertmanager under the split-namespace profile, as described above.