materialize-monitoring Changelog#

Container Images v0.10.0 (Unreleased)#

Changes Pending

mzmon-lib (shared library) v0.15.0 (Unreleased)#

Changes Pending

Dashboards (Helm chart) v0.22.0 (Unreleased)#

Changes Pending

Dashboards (Helm chart) v0.21.1#

  • Name the Datadog dashboard files datadog-.json, unblocking the dashboards release
    • materialize-monitoring#633
    • The Datadog dashboard downloads are named datadog-<stem>.json, such as datadog-env-top.json, so they no longer share a file name with the Grafana render of the same dashboard.

Dependencies#

  • Included mzmon-lib (shared library) @ v0.13.0..v0.14.0

Container Images v0.9.0#

materialize-monitoring (Helm chart + Terraform module) v1.7.0 (Unreleased)#

Changes Pending

Prometheus Scrapers v0.6.0 (Unreleased)#

Changes Pending

Dashboards (Helm chart) v0.21.0#

  • Datadog dashboards: port every env-top panel, with a parity ledger
    • materialize-monitoring#630
    • Datadog: env-top, the Materialize Environment Overview, holds the Grafana dashboard’s panels, downloadable as JSON from the Available Dashboards page
      • Environment, namespace, cluster, and replica template variables. Pick one environment and the namespace it runs in: environments reuse cluster and replica ids, and the Kubernetes panels read the namespace.
      • The peek and Iceberg commit latency panels need percentiles enabled for their metrics in Datadog.
      • The Datadog dashboards page lists the panels it draws differently from Grafana or leaves out, such as the alert panels.
  • Add a GCS section to the persist dashboard, and drop its always-zero storage series
    • materialize-monitoring#625
    • The Materialize Persist dashboard’s Object Store Internals tab has a GCS section when the GCS Cloud Monitoring pull is on
      • Live data, objects, and average object size per bucket, their daily change, and the live, noncurrent, and soft-deleted split
    • The Materialize Persist dashboard’s Storage tab no longer shows orphaned data or uploads in flight, which Materialize always reported as zero
      • mz_persist_shard_usage_leaked_bytes and mz_persist_shard_usage_not_leaked_not_referenced_bytes leave the extended metric tier
  • Datadog dashboards: split widget queries into bare queries and formulas
  • Datadog dashboards: the build machinery and an env-top preview
    • materialize-monitoring#587
    • Datadog: a preview of the Materialize Environment Overview (env-top), with its cluster and replica counts, downloadable as JSON from the Available Dashboards page. Import it through Datadog’s API or Terraform’s datadog_dashboard_json, and pick one environment, because the counts merge environments when several are selected.

Dependencies#

  • Included mzmon-lib (shared library) @ v0.13.0..v0.14.0
    • Build dev dependencies without debug info
    • Update Rust crate jsonschema to 0.60.0
    • Update Rust crate zip to v9 [SECURITY]
    • Lock file maintenance (cargo)
    • Render canonical label names from labels.yaml in the Materialize monitors (DEP-421)
      • materialize-monitoring#582
      • The Materialize PodMonitors, and the classic and Google Cloud Managed Prometheus scrape configs, add canonical label names beside the labels Materialize emits: cluster_id, replica_id, cluster_name, cluster_size, replica_name, replica_size, object_name, object_type, collection_id and shard_id. Each is copied only where it’s absent, so it adds no series. Canonical label names lists where each comes from.
      • Deprecated: the labels they’re copied from. Both names stay on every series for at least 30 days after this release; move queries and alert routes to the canonical names.
        • instance_id, compute_instance, compute_cluster_id and cluster_environmentd_materialize_cloud_cluster_id are cluster_id
        • compute_replica_id and cluster_environmentd_materialize_cloud_replica_id are replica_id
        • compute_cluster_name, and name on mz_cluster_info, are cluster_name; size on mz_cluster_info is cluster_size
        • name and size on mz_replica_info are replica_name and replica_size
        • name and type on mz_object_info are object_name and object_type
        • global_id is collection_id, and shard is shard_id
      • If you replace materialize.environmentd.serviceMonitor.metricEndpoints, materialize.environmentdSQL.serviceMonitor.metricEndpoints, materialize.clusterd.serviceMonitor.metricEndpoints or materialize-operator.serviceMonitor.metricEndpoints, restate the default metricRelabelings, or those series lack the canonical labels.

materialize-monitoring (Helm chart + Terraform module) v1.6.0#

  • Render canonical label names from labels.yaml in the Materialize monitors (DEP-421)
    • materialize-monitoring#582
    • The Materialize PodMonitors, and the classic and Google Cloud Managed Prometheus scrape configs, add canonical label names beside the labels Materialize emits: cluster_id, replica_id, cluster_name, cluster_size, replica_name, replica_size, object_name, object_type, collection_id and shard_id. Each is copied only where it’s absent, so it adds no series. Canonical label names lists where each comes from.
    • Deprecated: the labels they’re copied from. Both names stay on every series for at least 30 days after this release; move queries and alert routes to the canonical names.
      • instance_id, compute_instance, compute_cluster_id and cluster_environmentd_materialize_cloud_cluster_id are cluster_id
      • compute_replica_id and cluster_environmentd_materialize_cloud_replica_id are replica_id
      • compute_cluster_name, and name on mz_cluster_info, are cluster_name; size on mz_cluster_info is cluster_size
      • name and size on mz_replica_info are replica_name and replica_size
      • name and type on mz_object_info are object_name and object_type
      • global_id is collection_id, and shard is shard_id
    • If you replace materialize.environmentd.serviceMonitor.metricEndpoints, materialize.environmentdSQL.serviceMonitor.metricEndpoints, materialize.clusterd.serviceMonitor.metricEndpoints or materialize-operator.serviceMonitor.metricEndpoints, restate the default metricRelabelings, or those series lack the canonical labels.
  • Split the dashboards chart into one chart per Grafana folder
    • materialize-monitoring#585
    • The dashboards ship in three charts, one per Grafana folder: materialize-monitoring-dashboards for the Materialize environment (env-*), materialize-monitoring-dashboards-infra for the platform underneath it, and materialize-monitoring-dashboards-meta for Alloy, Loki, and Thanos.
      • The Terraform module installs all three, with no variable changes.
      • If you install with Helm directly, upgrading materialize-monitoring-dashboards removes the infrastructure and meta-monitoring dashboards from it. To keep them, install materialize-monitoring-dashboards-infra and materialize-monitoring-dashboards-meta after that upgrade.
      • Each chart’s selected defaults to ["*"], and its grafana.folderUids default names only its own folder. Pass the same values file to all three; each ignores folder entries it doesn’t use.
      • Dashboard UIDs, titles, and links don’t change.
  • Update alloy Helm Chart to v1.14.0
  • Update thanos Helm Chart to v0.47.3
  • Production best practices: order the checklist by consequence
  • Update loki Helm Chart to v19
  • Add adhoc, s3-compatible and no-storage-class profiles
    • materialize-monitoring#558
    • New profiles: adhoc (the whole stack, single replica, no volumes, for troubleshooting), s3-compatible (Ceph RGW, MinIO, or rustfs with static keys from Secrets), and no-storage-class (emptyDir in place of every PVC, Grafana’s included).
    • The install NOTES give the dashboards chart’s OCI address in place of a <repo> placeholder.
  • Update docker.io/nginxinc/nginx-unprivileged Docker tag to v1.31.7

Dependencies#

materialize-monitoring (Helm chart + Terraform module) v1.5.0#

  • Find GCP load balancers by their VPC network
    • materialize-monitoring#571
    • pipeline.metrics.provider.gcp.loadBalancing.networks lists VPC networks whose passthrough load balancers to pull, external or internal, beside the forwarding rules in forwardingRules. A rule either list names is pulled. Use it for a rule that cannot be named beforehand, such as one GKE makes for a Service the same release creates. On a network that holds only the cluster, as the Terraform wrappers create it, that is the cluster’s load balancers and nothing else.
  • Add a noop profile, and let the Grafana resource be turned off
    • materialize-monitoring#572
    • Turn off the chart’s Grafana resource, and install a release that creates nothing
      • connections.grafana.enabled (default true): set it to false and the chart creates no Grafana resource in any connections.grafana.mode. Use it when no grafana-operator acts on the release, or when dashboards.config.grafana.manifest.instanceSelector points the folders and datasources at a Grafana resource you manage.
      • With connections.grafana.enabled: false, the chart refuses to render folders or datasources that still select its own instance. Turn them off, or set dashboards.config.grafana.manifest.instanceSelector.
      • profiles/noop.values.yaml: a release that creates no resources. Install with it first, then turn components on over it one tag at a time. The materialize-monitoring-crds chart is unaffected.

Dependencies#

materialize-monitoring (Helm chart + Terraform module) v1.4.0#

  • Pull load balancer and NAT gateway metrics, and fill infra-net’s Cloud Networking tab
    • materialize-monitoring#552
    • Six new cloud provider pulls cover the load balancers in front of the cluster and the NAT gateways its traffic leaves through. Each is off until resources are listed.
      • pipeline.metrics.provider.cloudwatch.nlb.loadBalancers: Network Load Balancers, each as arn with its targetGroups, for open and new connections, bytes, resets by sender, and healthy and unhealthy targets. An ARN or its CloudWatch form, such as net/<name>/<id>, is accepted; Application Load Balancers are refused.
      • pipeline.metrics.provider.cloudwatch.natGateway.ids: NAT gateway IDs, for port allocation errors, dropped packets, idle timeouts, and connections.
      • pipeline.metrics.provider.gcp.loadBalancing.forwardingRules: passthrough load balancer forwarding rules, external or internal, for packets and bytes in each direction.
      • pipeline.metrics.provider.gcp.cloudNat.gateways: Cloud NAT gateway names, for allocation failures, per-VM port use, open connections, and dropped packets by reason.
      • pipeline.metrics.provider.azure.loadBalancer.clusters: AKS cluster names, whose kubernetes and kubernetes-internal load balancers are pulled for data path availability, backend health probes, bytes, and new connections.
      • pipeline.metrics.provider.azure.natGateway.names: NAT gateway names, for data path availability, SNAT connections attempted and failed, and dropped packets.
    • The AWS pulls need only cloudwatch:GetMetricStatistics, and the GCP pulls only roles/monitoring.viewer, which the existing pulls already use. On Azure, the identity needs Monitoring Reader on the AKS cluster’s node resource group and on each NAT gateway.
    • Series from the new pulls carry a provider_pull label, up included. Series from the existing pulls are unchanged.
    • The new families are tiered by each provider’s metricImportance, like the existing provider families.
    • Infrastructure Networking’s Cloud Networking tab draws the new pulls: target and backend health, connections, resets, traffic, and NAT port exhaustion. Each row renders only where its pull runs, with a note on how to turn on the rest. The two placeholder rows are gone.
  • Update grafana monorepo
  • Truncate oversized log lines and add a development alert tier
    • materialize-monitoring#549
    • Oversized log lines are cut rather than dropped, and the cut line ends in a suffix that names the rule: [TRUNCATED:SAFETYCLAMP], [TRUNCATED:LINE], or [TRUNCATED:FIELDS].
      • The agent and the gateway clamp every line to PIPELINE_LOG_LINE_SAFETY_CLAMP (default 1MiB). The gateway then cuts lines to GATEWAY_LOG_LINE_SIZE_LIMIT (default 240KiB), with a fixed 250KiB backstop. Set either variable in pipeline.env, with a unit such as 512KiB; a bare number stops Alloy from loading its configuration.
      • The gateway cuts label values to 2KiB and structured metadata to a per-entry budget: 512B for each key it sets, 16KiB for msg and for keys a Loki push client sends, and 16KiB held back for a push client’s own structured metadata. Entries then fit Loki’s default limits rather than being refused, as long as a push client’s own structured metadata stays under 16KiB.
      • A clamped JSON or logfmt line no longer parses, so it reaches Loki without its extracted fields.
    • loki_process_dropped_lines_total no longer reports the too large reason. Cuts count in loki_process_truncated_fields_total, labeled by field, which the default metric tier keeps.
    • Alloy Meta Monitoring: the Truncated Lines panel counts those cuts by field, beside the agents’ CRI reassembly cuts.
    • Alerts can carry audience: development, for conditions the stack has already contained that the people who write the software can remove. An extra route on audience="development" sends them to a development channel, and rules.overrides accepts the value.
      • oversized-log-lines (severity: notice, not in the default set) fires when Alloy cut a log line in the past hour. Select it with rules.selected.
  • postgres-exporter: log at warn, and note Azure’s pg_monitor
    • materialize-monitoring#546
    • The postgres-exporter logs at warn by default (postgres-exporter.config.logLevel). Set it to info to see each probe’s connection again.
  • Update grafana Helm Chart to v14

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
    • Lead the Loki, Alloy and environment overviews with firing alerts
      • materialize-monitoring#551
      • Loki Meta Monitoring, Alloy Meta Monitoring, and the Summary tab of Materialize Environment Overview now count the alerts firing about their subject, and list every alert firing or pending in an Alerts row. Each alert’s name links to its entry under Common Alerts.
      • An alert count shows the current value. It reads zero only while the Thanos ruler, and every Loki rule group where a Loki ruler runs, are being evaluated and written back. Otherwise it’s empty.
      • The recommended metric tier adds loki_prometheus_rule_group_last_evaluation_timestamp_seconds, loki_prometheus_rule_group_interval_seconds, loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds, and loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds.
    • alerts: count memory-limiter exits as replica oom kills
      • materialize-monitoring#555
      • cluster-replica-oomkilled now also fires when clusterd’s memory limiter kills a user cluster replica (exit code 167), not only on a kernel OOM kill. Expect it to fire more often on clusters with swap enabled.

  • Included Prometheus Scrapers @ v0.4.0..v0.5.0

  • Included mzmon-lib (shared library) @ v0.13.0..v0.14.0

Container Images v0.8.0#

Dashboards (Helm chart) v0.20.0#

  • Split the dashboards chart into one chart per Grafana folder
    • materialize-monitoring#585
    • The dashboards ship in three charts, one per Grafana folder: materialize-monitoring-dashboards for the Materialize environment (env-*), materialize-monitoring-dashboards-infra for the platform underneath it, and materialize-monitoring-dashboards-meta for Alloy, Loki, and Thanos.
      • The Terraform module installs all three, with no variable changes.
      • If you install with Helm directly, upgrading materialize-monitoring-dashboards removes the infrastructure and meta-monitoring dashboards from it. To keep them, install materialize-monitoring-dashboards-infra and materialize-monitoring-dashboards-meta after that upgrade.
      • Each chart’s selected defaults to ["*"], and its grafana.folderUids default names only its own folder. Pass the same values file to all three; each ignores folder entries it doesn’t use.
      • Dashboard UIDs, titles, and links don’t change.
  • Validate the Datadog translations and translate the rest of the registry
    • materialize-monitoring#586
    • The Datadog form of every registry query on Common Queries is validated against a live Datadog organization
      • Counter rates read .as_rate().rollup(sum, 60). .as_rate() alone overstates a rate on short time ranges by the scrape interval over 10 seconds.
      • Prometheus job and instance labels are service and service.instance.id in Datadog, and tag values are lowercase.
      • Percentile queries (p50:, p99:) need percentiles enabled for their metric in Datadog.
      • The ALERTS panels have no Datadog form.
  • Design doc: Datadog dashboards from the query registry
  • Find GCP load balancers by their VPC network
    • materialize-monitoring#571
    • pipeline.metrics.provider.gcp.loadBalancing.networks lists VPC networks whose passthrough load balancers to pull, external or internal, beside the forwarding rules in forwardingRules. A rule either list names is pulled. Use it for a rule that cannot be named beforehand, such as one GKE makes for a Service the same release creates. On a network that holds only the cluster, as the Terraform wrappers create it, that is the cluster’s load balancers and nothing else.
  • Pull load balancer and NAT gateway metrics, and fill infra-net’s Cloud Networking tab
    • materialize-monitoring#552
    • Six new cloud provider pulls cover the load balancers in front of the cluster and the NAT gateways its traffic leaves through. Each is off until resources are listed.
      • pipeline.metrics.provider.cloudwatch.nlb.loadBalancers: Network Load Balancers, each as arn with its targetGroups, for open and new connections, bytes, resets by sender, and healthy and unhealthy targets. An ARN or its CloudWatch form, such as net/<name>/<id>, is accepted; Application Load Balancers are refused.
      • pipeline.metrics.provider.cloudwatch.natGateway.ids: NAT gateway IDs, for port allocation errors, dropped packets, idle timeouts, and connections.
      • pipeline.metrics.provider.gcp.loadBalancing.forwardingRules: passthrough load balancer forwarding rules, external or internal, for packets and bytes in each direction.
      • pipeline.metrics.provider.gcp.cloudNat.gateways: Cloud NAT gateway names, for allocation failures, per-VM port use, open connections, and dropped packets by reason.
      • pipeline.metrics.provider.azure.loadBalancer.clusters: AKS cluster names, whose kubernetes and kubernetes-internal load balancers are pulled for data path availability, backend health probes, bytes, and new connections.
      • pipeline.metrics.provider.azure.natGateway.names: NAT gateway names, for data path availability, SNAT connections attempted and failed, and dropped packets.
    • The AWS pulls need only cloudwatch:GetMetricStatistics, and the GCP pulls only roles/monitoring.viewer, which the existing pulls already use. On Azure, the identity needs Monitoring Reader on the AKS cluster’s node resource group and on each NAT gateway.
    • Series from the new pulls carry a provider_pull label, up included. Series from the existing pulls are unchanged.
    • The new families are tiered by each provider’s metricImportance, like the existing provider families.
    • Infrastructure Networking’s Cloud Networking tab draws the new pulls: target and backend health, connections, resets, traffic, and NAT port exhaustion. Each row renders only where its pull runs, with a note on how to turn on the rest. The two placeholder rows are gone.
  • Lead the Loki, Alloy and environment overviews with firing alerts
    • materialize-monitoring#551
    • Loki Meta Monitoring, Alloy Meta Monitoring, and the Summary tab of Materialize Environment Overview now count the alerts firing about their subject, and list every alert firing or pending in an Alerts row. Each alert’s name links to its entry under Common Alerts.
    • An alert count shows the current value. It reads zero only while the Thanos ruler, and every Loki rule group where a Loki ruler runs, are being evaluated and written back. Otherwise it’s empty.
    • The recommended metric tier adds loki_prometheus_rule_group_last_evaluation_timestamp_seconds, loki_prometheus_rule_group_interval_seconds, loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds, and loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds.
  • alerts: count memory-limiter exits as replica oom kills
    • materialize-monitoring#555
    • cluster-replica-oomkilled now also fires when clusterd’s memory limiter kills a user cluster replica (exit code 167), not only on a kernel OOM kill. Expect it to fire more often on clusters with swap enabled.
  • Truncate oversized log lines and add a development alert tier
    • materialize-monitoring#549
    • Oversized log lines are cut rather than dropped, and the cut line ends in a suffix that names the rule: [TRUNCATED:SAFETYCLAMP], [TRUNCATED:LINE], or [TRUNCATED:FIELDS].
      • The agent and the gateway clamp every line to PIPELINE_LOG_LINE_SAFETY_CLAMP (default 1MiB). The gateway then cuts lines to GATEWAY_LOG_LINE_SIZE_LIMIT (default 240KiB), with a fixed 250KiB backstop. Set either variable in pipeline.env, with a unit such as 512KiB; a bare number stops Alloy from loading its configuration.
      • The gateway cuts label values to 2KiB and structured metadata to a per-entry budget: 512B for each key it sets, 16KiB for msg and for keys a Loki push client sends, and 16KiB held back for a push client’s own structured metadata. Entries then fit Loki’s default limits rather than being refused, as long as a push client’s own structured metadata stays under 16KiB.
      • A clamped JSON or logfmt line no longer parses, so it reaches Loki without its extracted fields.
    • loki_process_dropped_lines_total no longer reports the too large reason. Cuts count in loki_process_truncated_fields_total, labeled by field, which the default metric tier keeps.
    • Alloy Meta Monitoring: the Truncated Lines panel counts those cuts by field, beside the agents’ CRI reassembly cuts.
    • Alerts can carry audience: development, for conditions the stack has already contained that the people who write the software can remove. An extra route on audience="development" sends them to a development channel, and rules.overrides accepts the value.
      • oversized-log-lines (severity: notice, not in the default set) fires when Alloy cut a log line in the past hour. Select it with rules.selected.
  • Add a postgres_exporter subchart for the metadata and Grafana databases
    • materialize-monitoring#540
    • A postgres-exporter subchart (prometheus-postgres-exporter 8.2.0, postgres_exporter v0.20.1) watches PostgreSQL databases from the inside. It is off by default: enable it with tags.postgres-exporter: true and list the databases under postgres-exporter.targets. Credentials come from a postgres-exporter Secret you create. See the PostgreSQL exporter page for the Secret’s shape and the least-privilege pg_monitor grant.
    • externalDependencies.consensus accepts flavor: postgres, with an exporter target’s name as resourceId. The new postgres adapter records ext:consensus_up, ext:consensus_connections_used_ratio, ext:consensus_xid_used_ratio, ext:consensus_database_bytes (with a database label) and ext:consensus_version_info (with a version label) for that target.
    • rules.capabilities gains a derived capability, postgres-exporter, present when the exporter runs.
    • The Materialize Consensus (Metadata) dashboard’s Database Internals tab has a PostgreSQL section, rendered where the exporter is scraped, with a Database picker.

Dependencies#

materialize-monitoring (Helm chart + Terraform module) v1.3.0#

  • Terraform: run the postgres-exporter and declare metadata databases
    • materialize-monitoring#542
    • The Terraform module runs the postgres-exporter with enable_postgres_exporter = true. It watches Grafana’s database whenever the module configures one with a password, plus any database in postgres_exporter_targets, with passwords from postgres_exporter_passwords. The module creates the exporter’s postgres-exporter Secret and rolls the exporter when a password changes.
    • The Terraform module takes metadata_databases (flavor and resource_id) for the metadata databases a provider pull watches, and a metadata_database flag on an exporter target. Both become externalDependencies.consensus entries.
    • node_selector and tolerations reach the postgres-exporter.
  • Add a postgres_exporter subchart for the metadata and Grafana databases
    • materialize-monitoring#540
    • A postgres-exporter subchart (prometheus-postgres-exporter 8.2.0, postgres_exporter v0.20.1) watches PostgreSQL databases from the inside. It is off by default: enable it with tags.postgres-exporter: true and list the databases under postgres-exporter.targets. Credentials come from a postgres-exporter Secret you create. See the PostgreSQL exporter page for the Secret’s shape and the least-privilege pg_monitor grant.
    • externalDependencies.consensus accepts flavor: postgres, with an exporter target’s name as resourceId. The new postgres adapter records ext:consensus_up, ext:consensus_connections_used_ratio, ext:consensus_xid_used_ratio, ext:consensus_database_bytes (with a database label) and ext:consensus_version_info (with a version label) for that target.
    • rules.capabilities gains a derived capability, postgres-exporter, present when the exporter runs.
    • The Materialize Consensus (Metadata) dashboard’s Database Internals tab has a PostgreSQL section, rendered where the exporter is scraped, with a Database picker.
  • Move kubectl and the kind node image to Kubernetes 1.36
    • materialize-monitoring#537
    • The kubectl images move to Kubernetes 1.36: thanos.ruler.autoImportPrometheusRules.sidecar.image.tag is now 1.36.4, and cleanup.grafanaOperator.image.tag is now v1.36.5. Both clear most of the HIGH and CRITICAL CVEs the 1.35 images carried.
      • Kubernetes supports kubectl within one minor of the API server, so 1.34 clusters are now outside the supported range. kubectl doesn’t enforce that range, and these images only list, get, and delete custom resources, which work against a 1.34 API server. To stay inside the supported range on 1.34, set both tags to a 1.35 release through values.
  • Update the Grafana, node-exporter and Alertmanager charts, and access-log-exporter
  • Update ghcr.io/materializeinc/mzmon-alloy Docker tag to v1.20.1
  • Update Loki to 3.7.8 and its Helm chart to 18.15.1

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.13.0..v0.14.0

mzmon-lib (shared library) v0.14.0#

  • Name the Datadog dashboard files datadog-.json, unblocking the dashboards release
    • materialize-monitoring#633
    • The Datadog dashboard downloads are named datadog-<stem>.json, such as datadog-env-top.json, so they no longer share a file name with the Grafana render of the same dashboard.
  • Datadog dashboards: port every env-top panel, with a parity ledger
    • materialize-monitoring#630
    • Datadog: env-top, the Materialize Environment Overview, holds the Grafana dashboard’s panels, downloadable as JSON from the Available Dashboards page
      • Environment, namespace, cluster, and replica template variables. Pick one environment and the namespace it runs in: environments reuse cluster and replica ids, and the Kubernetes panels read the namespace.
      • The peek and Iceberg commit latency panels need percentiles enabled for their metrics in Datadog.
      • The Datadog dashboards page lists the panels it draws differently from Grafana or leaves out, such as the alert panels.
  • Build dev dependencies without debug info
  • Update Rust crate jsonschema to 0.60.0
  • Update Rust crate zip to v9 [SECURITY]
  • Add a GCS section to the persist dashboard, and drop its always-zero storage series
    • materialize-monitoring#625
    • The Materialize Persist dashboard’s Object Store Internals tab has a GCS section when the GCS Cloud Monitoring pull is on
      • Live data, objects, and average object size per bucket, their daily change, and the live, noncurrent, and soft-deleted split
    • The Materialize Persist dashboard’s Storage tab no longer shows orphaned data or uploads in flight, which Materialize always reported as zero
      • mz_persist_shard_usage_leaked_bytes and mz_persist_shard_usage_not_leaked_not_referenced_bytes leave the extended metric tier
  • Datadog dashboards: split widget queries into bare queries and formulas
  • Lock file maintenance (cargo)
  • Datadog dashboards: the build machinery and an env-top preview
    • materialize-monitoring#587
    • Datadog: a preview of the Materialize Environment Overview (env-top), with its cluster and replica counts, downloadable as JSON from the Available Dashboards page. Import it through Datadog’s API or Terraform’s datadog_dashboard_json, and pick one environment, because the counts merge environments when several are selected.
  • Render canonical label names from labels.yaml in the Materialize monitors (DEP-421)
    • materialize-monitoring#582
    • The Materialize PodMonitors, and the classic and Google Cloud Managed Prometheus scrape configs, add canonical label names beside the labels Materialize emits: cluster_id, replica_id, cluster_name, cluster_size, replica_name, replica_size, object_name, object_type, collection_id and shard_id. Each is copied only where it’s absent, so it adds no series. Canonical label names lists where each comes from.
    • Deprecated: the labels they’re copied from. Both names stay on every series for at least 30 days after this release; move queries and alert routes to the canonical names.
      • instance_id, compute_instance, compute_cluster_id and cluster_environmentd_materialize_cloud_cluster_id are cluster_id
      • compute_replica_id and cluster_environmentd_materialize_cloud_replica_id are replica_id
      • compute_cluster_name, and name on mz_cluster_info, are cluster_name; size on mz_cluster_info is cluster_size
      • name and size on mz_replica_info are replica_name and replica_size
      • name and type on mz_object_info are object_name and object_type
      • global_id is collection_id, and shard is shard_id
    • If you replace materialize.environmentd.serviceMonitor.metricEndpoints, materialize.environmentdSQL.serviceMonitor.metricEndpoints, materialize.clusterd.serviceMonitor.metricEndpoints or materialize-operator.serviceMonitor.metricEndpoints, restate the default metricRelabelings, or those series lack the canonical labels.
  • Split the dashboards chart into one chart per Grafana folder
    • materialize-monitoring#585
    • The dashboards ship in three charts, one per Grafana folder: materialize-monitoring-dashboards for the Materialize environment (env-*), materialize-monitoring-dashboards-infra for the platform underneath it, and materialize-monitoring-dashboards-meta for Alloy, Loki, and Thanos.
      • The Terraform module installs all three, with no variable changes.
      • If you install with Helm directly, upgrading materialize-monitoring-dashboards removes the infrastructure and meta-monitoring dashboards from it. To keep them, install materialize-monitoring-dashboards-infra and materialize-monitoring-dashboards-meta after that upgrade.
      • Each chart’s selected defaults to ["*"], and its grafana.folderUids default names only its own folder. Pass the same values file to all three; each ignores folder entries it doesn’t use.
      • Dashboard UIDs, titles, and links don’t change.
  • Lock file maintenance (pep621)
  • Update dependency uv_build to >=0.13,<0.14
  • Update Rust crate jsonschema to 0.59.0
  • Lock file maintenance (cargo)
  • Lock file maintenance (pep621)
  • Pull load balancer and NAT gateway metrics, and fill infra-net’s Cloud Networking tab
    • materialize-monitoring#552
    • Six new cloud provider pulls cover the load balancers in front of the cluster and the NAT gateways its traffic leaves through. Each is off until resources are listed.
      • pipeline.metrics.provider.cloudwatch.nlb.loadBalancers: Network Load Balancers, each as arn with its targetGroups, for open and new connections, bytes, resets by sender, and healthy and unhealthy targets. An ARN or its CloudWatch form, such as net/<name>/<id>, is accepted; Application Load Balancers are refused.
      • pipeline.metrics.provider.cloudwatch.natGateway.ids: NAT gateway IDs, for port allocation errors, dropped packets, idle timeouts, and connections.
      • pipeline.metrics.provider.gcp.loadBalancing.forwardingRules: passthrough load balancer forwarding rules, external or internal, for packets and bytes in each direction.
      • pipeline.metrics.provider.gcp.cloudNat.gateways: Cloud NAT gateway names, for allocation failures, per-VM port use, open connections, and dropped packets by reason.
      • pipeline.metrics.provider.azure.loadBalancer.clusters: AKS cluster names, whose kubernetes and kubernetes-internal load balancers are pulled for data path availability, backend health probes, bytes, and new connections.
      • pipeline.metrics.provider.azure.natGateway.names: NAT gateway names, for data path availability, SNAT connections attempted and failed, and dropped packets.
    • The AWS pulls need only cloudwatch:GetMetricStatistics, and the GCP pulls only roles/monitoring.viewer, which the existing pulls already use. On Azure, the identity needs Monitoring Reader on the AKS cluster’s node resource group and on each NAT gateway.
    • Series from the new pulls carry a provider_pull label, up included. Series from the existing pulls are unchanged.
    • The new families are tiered by each provider’s metricImportance, like the existing provider families.
    • Infrastructure Networking’s Cloud Networking tab draws the new pulls: target and backend health, connections, resets, traffic, and NAT port exhaustion. Each row renders only where its pull runs, with a note on how to turn on the rest. The two placeholder rows are gone.
  • Update Rust crate rustls to v0.23.46
  • Update Rust crate tokio to v1.53.3
  • Lead the Loki, Alloy and environment overviews with firing alerts
    • materialize-monitoring#551
    • Loki Meta Monitoring, Alloy Meta Monitoring, and the Summary tab of Materialize Environment Overview now count the alerts firing about their subject, and list every alert firing or pending in an Alerts row. Each alert’s name links to its entry under Common Alerts.
    • An alert count shows the current value. It reads zero only while the Thanos ruler, and every Loki rule group where a Loki ruler runs, are being evaluated and written back. Otherwise it’s empty.
    • The recommended metric tier adds loki_prometheus_rule_group_last_evaluation_timestamp_seconds, loki_prometheus_rule_group_interval_seconds, loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds, and loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds.
  • Update Rust crate serde-saphyr to v2
  • Truncate oversized log lines and add a development alert tier
    • materialize-monitoring#549
    • Oversized log lines are cut rather than dropped, and the cut line ends in a suffix that names the rule: [TRUNCATED:SAFETYCLAMP], [TRUNCATED:LINE], or [TRUNCATED:FIELDS].
      • The agent and the gateway clamp every line to PIPELINE_LOG_LINE_SAFETY_CLAMP (default 1MiB). The gateway then cuts lines to GATEWAY_LOG_LINE_SIZE_LIMIT (default 240KiB), with a fixed 250KiB backstop. Set either variable in pipeline.env, with a unit such as 512KiB; a bare number stops Alloy from loading its configuration.
      • The gateway cuts label values to 2KiB and structured metadata to a per-entry budget: 512B for each key it sets, 16KiB for msg and for keys a Loki push client sends, and 16KiB held back for a push client’s own structured metadata. Entries then fit Loki’s default limits rather than being refused, as long as a push client’s own structured metadata stays under 16KiB.
      • A clamped JSON or logfmt line no longer parses, so it reaches Loki without its extracted fields.
    • loki_process_dropped_lines_total no longer reports the too large reason. Cuts count in loki_process_truncated_fields_total, labeled by field, which the default metric tier keeps.
    • Alloy Meta Monitoring: the Truncated Lines panel counts those cuts by field, beside the agents’ CRI reassembly cuts.
    • Alerts can carry audience: development, for conditions the stack has already contained that the people who write the software can remove. An extra route on audience="development" sends them to a development channel, and rules.overrides accepts the value.
      • oversized-log-lines (severity: notice, not in the default set) fires when Alloy cut a log line in the past hour. Select it with rules.selected.
  • Terraform: run the postgres-exporter and declare metadata databases
    • materialize-monitoring#542
    • The Terraform module runs the postgres-exporter with enable_postgres_exporter = true. It watches Grafana’s database whenever the module configures one with a password, plus any database in postgres_exporter_targets, with passwords from postgres_exporter_passwords. The module creates the exporter’s postgres-exporter Secret and rolls the exporter when a password changes.
    • The Terraform module takes metadata_databases (flavor and resource_id) for the metadata databases a provider pull watches, and a metadata_database flag on an exporter target. Both become externalDependencies.consensus entries.
    • node_selector and tolerations reach the postgres-exporter.
  • Add a postgres_exporter subchart for the metadata and Grafana databases
    • materialize-monitoring#540
    • A postgres-exporter subchart (prometheus-postgres-exporter 8.2.0, postgres_exporter v0.20.1) watches PostgreSQL databases from the inside. It is off by default: enable it with tags.postgres-exporter: true and list the databases under postgres-exporter.targets. Credentials come from a postgres-exporter Secret you create. See the PostgreSQL exporter page for the Secret’s shape and the least-privilege pg_monitor grant.
    • externalDependencies.consensus accepts flavor: postgres, with an exporter target’s name as resourceId. The new postgres adapter records ext:consensus_up, ext:consensus_connections_used_ratio, ext:consensus_xid_used_ratio, ext:consensus_database_bytes (with a database label) and ext:consensus_version_info (with a version label) for that target.
    • rules.capabilities gains a derived capability, postgres-exporter, present when the exporter runs.
    • The Materialize Consensus (Metadata) dashboard’s Database Internals tab has a PostgreSQL section, rendered where the exporter is scraped, with a Database picker.
  • Add a Thanos meta-monitoring dashboard, with firing alerts on its Overview

materialize-monitoring (Helm chart + Terraform module) v1.2.0#

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
    • Add a Thanos meta-monitoring dashboard, with firing alerts on its Overview
    • Replace serde_yaml_ng with serde-saphyr
      • materialize-monitoring#529
      • Pre-rendered dashboards and the rules _index.yaml now use double quotes where a string needs quoting. The parsed content is unchanged, but a byte-level diff of vendored files will show the requoting.
    • Add CNPG and Rook Ceph tabs to the Consensus and Persist dashboards
      • materialize-monitoring#509
      • Materialize Consensus has a new Database Internals tab for metadata databases on CloudNativePG, and Materialize Persist a new Object Store Internals tab for persist buckets on Ceph run by Rook. Each appears only when the dependency is scraped — a PodMonitor for the CNPG cluster, or Rook’s manager and rook-ceph-exporter ServiceMonitors — and has a picker, shown only inside that tab’s section, for which cluster serves the environment.

  • Included Prometheus Scrapers @ v0.4.0..v0.5.0

  • Included mzmon-lib (shared library) @ v0.13.0..v0.14.0

materialize-monitoring (Helm chart + Terraform module) v1.1.0#

  • Size the gateway for the series it holds, and keep its memory limiter behind the GC
    • materialize-monitoring#496
    • The alloy-gateway’s default memory is now 2Gi, request and limit, up from 768Mi. The old size was set when the gateway carried logs only. A gateway holds about 200MiB plus 2.6KiB per series it scrapes, and below this it ran its garbage collector constantly. Installs that set alloy-gateway.alloy.resources keep their own values; check them against the sizing note in values.yaml.
    • The gateway’s GOMEMLIMIT is now 80% of its memory limit, set with AUTOMEMLIMIT: "0.8" in alloy-gateway.alloy.extraEnv, instead of a fixed 600MiB. An explicit GOMEMLIMIT there still takes precedence and does not follow the limit. A list that sets neither falls back to Alloy’s own 90%.
    • The gateway’s memory limiter now refuses data only above 85% of its memory limit, instead of about 60%. The garbage collector now acts before scrapes are refused.
    • The gateway’s autoscaler now scales down only after an hour of low CPU, one pod per ten minutes.
  • Alert on a gateway refusing its scrapes, and stop reading the gaps as a stall
    • materialize-monitoring#493
    • New default alert alloy-gateway-refusing-scrapes (warning). It fires when an alloy-gateway pod’s memory limiter has been discarding its scrapes for 15 minutes. While it fires, every target that pod scrapes is missing from the metrics store. It is a LogQL rule, so it needs the bundled Loki ruler.
    • clusterd-not-receiving-commands no longer fires on a healthy replica whose scrapes are missing. It now needs at least two scraped samples in its window.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0

materialize-monitoring (Helm chart + Terraform module) v1.0.0#

The first stable release of the Helm chart and the Terraform module, which share this version. From here, a breaking change to the committed surface owes a deprecation cycle under the stability guarantees. Before 1.0, a breaking change could ride a minor release.

On top of v0.33.0, this release carries only the dependency updates below. An install on an older 0.x release should read Upgrading first, which covers the StatefulSet fields an upgrade cannot patch.

What 1.0 ships#

AreaAt 1.0
DeliveryThe umbrella chart, published to GHCR as an OCI artifact, and the Terraform module that installs it. The per-cloud wrappers in materialize-terraform-self-managed turn observability on by default. Profiles cover sizing, scheduling, storage class, private registries, Grafana ingress and persistence, and mTLS. The optional CRDs chart and the dashboards chart keep version streams of their own
CollectionAn Alloy agent for pod logs and the node journal, and an Alloy gateway for metrics. The gateway scrapes Materialize, kube-state-metrics, node-exporter, every kubelet’s cAdvisor, the AWS VPC CNI, Cilium and kube-proxy. It also pulls provider metrics from CloudWatch, Cloud Monitoring and Azure Monitor. A multi-line Rust panic arrives as one log entry
DestinationsThe bundled Loki and Thanos, any number of Prometheus remote-write destinations, OTLP, Datadog and Google Cloud’s Telemetry API, each filtered by importance tier
DashboardsThrough the dashboards chart: env-top, env-logs, env-upgrade, env-persist and env-consensus for Materialize, and infra-logs, infra-nodes, infra-net, infra-autoscaling, infra-karpenter, infra-cloud, infra-loki and infra-alloy for the platform underneath. See All Dashboards
AlertingThanos and Loki rulers notifying a highly available Alertmanager, with severity-to-receiver presets, capability-tagged rules, notification templates that link to Grafana, and a Terraform surface. Thirty-four alerts are in the default set, three of them log-derived, beside the ext:consensus_* recording rules. Alert and recording-rule names are part of the committed surface. See Alert Architecture
SecurityA NetworkPolicy on every component, opt-in mTLS through cert-manager in three phases, and Trivy scanning of the rendered chart and the published images
QualificationRender checks against every Terraform example, two kind end-to-end tiers, and an end-to-end suite run against real cloud installs
ReleaseA SemVer stream per artifact, a changelog carrying the notes each pull request wrote for consumers, and a committed-surface check at release time

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0

  • Included Prometheus Scrapers @ v0.4.0..v0.5.0

  • Included mzmon-lib (shared library) @ v0.12.0..v0.13.0
    • Update Rust crate tokio to v1.53.2
    • Write the chart README badge in release PRs, and collapse dependency rollups
      • materialize-monitoring#489
      • Dependency rollups in CHANGELOG.md (Included <component> @ vPREV..vNEW) collapse the PRs nested under them in a <details>.

materialize-monitoring (Helm chart + Terraform module) v0.33.0#

  • Ship default notification templates that link to Grafana
    • materialize-monitoring#488
    • The chart ships default notification templates and makes them Alertmanager’s built-in defaults (alerting.defaultTemplates, on by default). Receivers that set no title, text or link of their own now get:
      • A subject naming the alert, cluster/namespace and severity. This changes Slack titles, email subjects and PagerDuty/Opsgenie incident titles.
      • A Slack body with the summary, description, one line per alert, and links to the runbook, the alert group in Grafana and a pre-filled Grafana silence. Slack colour follows severity.
      • Links into Grafana instead of Alertmanager’s pod address. Set alerting.defaultTemplates: false to keep Alertmanager’s built-ins.
    • New alerting.grafanaURL: the address people open Grafana at, for those links. Empty derives it from grafana.ini.server.root_url, or from connections.grafana.external.url for an external Grafana. With no address, notifications carry no Grafana links, and the render warns once a receiver exists.
    • alerting.templates files now load after the chart’s mzmon.gotmpl, so redefining a name there (including Alertmanager’s built-ins such as slack.default.text) overrides the chart’s. The mzmon. template prefix is reserved for the chart.
    • Terraform: alerting.default_templates and alerting.grafana_url.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.12.0..v0.13.0

mzmon-lib (shared library) v0.13.0#

  • Replace serde_yaml_ng with serde-saphyr
    • materialize-monitoring#529
    • Pre-rendered dashboards and the rules _index.yaml now use double quotes where a string needs quoting. The parsed content is unchanged, but a byte-level diff of vendored files will show the requoting.
  • Port the Terraform render check to Rust
  • Add CNPG and Rook Ceph tabs to the Consensus and Persist dashboards
    • materialize-monitoring#509
    • Materialize Consensus has a new Database Internals tab for metadata databases on CloudNativePG, and Materialize Persist a new Object Store Internals tab for persist buckets on Ceph run by Rook. Each appears only when the dependency is scraped — a PodMonitor for the CNPG cluster, or Rook’s manager and rook-ceph-exporter ServiceMonitors — and has a picker, shown only inside that tab’s section, for which cluster serves the environment.
  • Lock file maintenance (cargo)
  • Lock file maintenance (pep621)
  • Update Rust crate hyper to v1.12.0
  • Update Rust crate jsonschema to v0.58.6
  • Rename docs reference/internal to reference/development
  • Update Rust crate tokio to v1.53.2
  • Write the chart README badge in release PRs, and collapse dependency rollups
    • materialize-monitoring#489
    • Dependency rollups in CHANGELOG.md (Included <component> @ vPREV..vNEW) collapse the PRs nested under them in a <details>.
  • Update Rust crate jsonschema to v0.58.5
  • Let alerts read recorded series, and split persist-failures by dependency
    • materialize-monitoring#483
    • Alerts can read the normalized ext:* recorded series. Such an alert installs wherever any recording rule producing what it reads installs, and Common Alerts lists what each one reads.
    • persist-failures is split. Three new alerts join the default set:
      • consensus-unreachable (critical) fires when no call to the metadata database has succeeded for five minutes, or a cloud provider reports it down.
      • consensus-failures (warning) fires on ten minutes of indeterminate metadata-database failures.
      • blob-failures (warning) fires on ten minutes of failed object-storage calls, by operation.
    • persist-failures now covers only persist’s own failures: compaction, read leases, state updates, columnar validation and statistics. It no longer fires on a failing metadata database or object store, or on mz_persist_cmd_failed_count. It remains outside the default set.
    • A metric destination filtering by minMetricImportance now receives ext:consensus_up, and no longer receives mz_persist_blob_failures or mz_persist_cmd_failed_count, which no bundled rule reads any more.

materialize-monitoring (Helm chart + Terraform module) v0.32.0#

  • Merge Receive’s restart overlaps in the Thanos Compactor, and add a metrics architecture page
    • materialize-monitoring#482
    • The Thanos Compactor now runs with vertical compaction and deduplicates Receive replicas.
      • Changed: thanos.compactor.extraArgs has a chart default: --log.level=info, --log.format=logfmt, --consistency-delay=30m, --compact.enable-vertical-compaction and --deduplication.replica-label=receive_replica. An override replaces the whole list and has to restate both flags. The other three match Thanos’s own defaults.
      • A Compactor halted with overlaps found while gathering blocks recovers on its next start with no manual repair. Expect the first run to work through the backlog before downsampling catches up.
      • Compacted blocks lose the receive_replica external label, and the bucket keeps one copy of each series instead of one per Receive replica. This is irreversible for the blocks it touches. Queries are unchanged, because Thanos Query already deduplicates on receive_replica.
      • Downsampled blocks written before the upgrade are marked no-compact and keep their per-replica copies until retention expires them.
      • Upgrading with helm upgrade --reuse-values keeps the previous chart’s defaults and does not apply this change. Use --reset-then-reuse-values.
    • New docs page: Metrics > Architecture.
  • Let alerts read recorded series, and split persist-failures by dependency
    • materialize-monitoring#483
    • Alerts can read the normalized ext:* recorded series. Such an alert installs wherever any recording rule producing what it reads installs, and Common Alerts lists what each one reads.
    • persist-failures is split. Three new alerts join the default set:
      • consensus-unreachable (critical) fires when no call to the metadata database has succeeded for five minutes, or a cloud provider reports it down.
      • consensus-failures (warning) fires on ten minutes of indeterminate metadata-database failures.
      • blob-failures (warning) fires on ten minutes of failed object-storage calls, by operation.
    • persist-failures now covers only persist’s own failures: compaction, read leases, state updates, columnar validation and statistics. It no longer fires on a failing metadata database or object store, or on mz_persist_cmd_failed_count. It remains outside the default set.
    • A metric destination filtering by minMetricImportance now receives ext:consensus_up, and no longer receives mz_persist_blob_failures or mz_persist_cmd_failed_count, which no bundled rule reads any more.
  • Record the normalized ext:consensus_* series, with a producer for the registry’s rules: branch
    • materialize-monitoring#475
    • New recording rules record the metadata (consensus) database as normalized ext:consensus_* series, listed on the new Recorded Series reference page. They install wherever rules.enabled is true and their source is present; no selection is needed.
      • ext:consensus_up and ext:consensus_commit_latency_seconds:p99 come from Materialize’s own calls, carry flavor="persist" and namespace, and need no configuration.
      • ext:consensus_up, ext:consensus_storage_used_ratio and ext:consensus_xid_used_ratio from the CloudWatch, Cloud Monitoring and Azure Monitor pulls carry flavor (rds, cloudsql, azure-postgres) and resource, where the provider publishes each.
    • New value externalDependencies.consensus, a list of {flavor, resourceId} naming which databases a provider pull watches are a metadata database. The provider-sourced ext:consensus_* series record only these, so a pull that also watches Grafana’s database no longer needs to be told apart by hand. The render warns when a pull watches databases and none is named.
    • rules.capabilities gains three derived capabilities, cloudwatch, cloud-monitoring and azure-monitor, present when the matching pipeline.metrics.provider.* pull is enabled.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.12.0..v0.13.0

materialize-monitoring (Helm chart + Terraform module) v0.31.0#

  • Expose Materialize pod labels on kube_pod_labels
    • materialize-monitoring#479
    • kube_pod_labels now names the Materialize cluster and replica a pod runs, as cluster_id, replica_id and replica_size, and environment_id once the Materialize operator sets the materialize.cloud/environment-id pod label.
      • Published by a new pods entry in kube-state-metrics.metricLabelsAllowlist and renamed from the label_* form by kube-state-metrics.prometheus.monitor.http.metricRelabelings. Helm replaces lists, so an override of either restates the chart’s entries.
      • Join a kube_pod_* family onto it on namespace and pod, and wrap kube_pod_labels in group by first, or the join fails under more than one kube-state-metrics replica.
    • New Terraform input kube_state_metrics_pod_labels adds pod labels, such as an ownership label for cost-center allocation, to that allowlist entry without restating it. Each arrives as label_<key>.
  • Google Cloud metrics: export over OTLP to the Telemetry API, typed
    • materialize-monitoring#474
    • Changed: the Google Cloud metrics destination writes somewhere else. googleCloudExporter (Terraform google_cloud_metrics) now sends OTLP to Google’s Telemetry API instead of using otelcol.exporter.googlecloud, and metrics land as prometheus.googleapis.com/<name>/<kind> instead of workload.googleapis.com/mzmon/<name>. Cloud Monitoring dashboards, alerting policies and anything else reading the old metric types stop receiving data and have to be repointed. The old types are not deleted.
      • Enable the telemetry.googleapis.com API on the project before upgrading with this destination on. Without it every export is refused, and nothing else fails. roles/monitoring.metricWriter is still the only role needed.
      • Metrics are billed per sample ingested instead of per byte, about $75 a month at recommended on a test install where the old export cost about $2,400.
      • Series carry the prometheus_target labels (project_id, location, cluster, namespace, job, instance) and collected_by="materialize-monitoring".
      • Counter values start from zero at the gateway’s first scrape, so they differ from Thanos; rate() and increase() agree.
      • New values googleCloudExporter.project and googleCloudExporter.location are needed only off GKE.
    • Changed: there is no metric prefix any more. googleCloudExporter.prefix is removed from the chart values, along with instrumentation_library_labels, skip_create_descriptor and service_resource_labels; setting any of them renders a warning and does nothing.
    • Deprecated: Terraform google_cloud_metrics.prefix is ignored and warns at plan time; remove it. There is no replacement, because the Telemetry API has no prefix to choose. Everything else in google_cloud_metrics is unchanged.
    • Changed: the gateway’s scrapes honor metric metadata, so every OTLP destination (Google Cloud, Datadog, generic OTLP) receives typed metrics: counters as cumulative sums, histograms as histograms. Metric names in Datadog change accordingly. Thanos and other remote-write destinations are unchanged.
    • alloy-gateway.alloy.stabilityLevel now defaults to experimental, which the scrapes require; the chart refuses any other level.
    • The agent and gateway now run the Alloy v1.20.0 image their image.tag names. Since the bump to v1.20.0-mz3, a stale image.digest had kept both on Alloy v1.19.2.
    • denyMetrics entries match per Prometheus series against typed histograms: denying foo_bucket still keeps foo_count and foo_sum.
  • Update grafana-operator to v5.25.0
  • Update docker.io/kiwigrid/k8s-sidecar Docker tag to v2.11.2
  • Update kube-state-metrics Helm Chart to v8.6.0
  • Update ghcr.io/materializeinc/mzmon-alloy Docker tag to v1.20.0
  • Update quay.io/prometheus-operator/prometheus-config-reloader Docker tag to v0.94.1
  • Update quay.io/prometheus/alertmanager Docker tag to v0.34.1

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0

materialize-monitoring (Helm chart + Terraform module) v0.30.0#

  • Shard the Loki ruler’s rule groups across its two replicas
    • materialize-monitoring#444
    • The Loki ruler’s two replicas now divide the rule groups between them (loki.loki.rulerConfig.enable_sharding: true), so each rule is evaluated once rather than once per replica. Before this, every Loki alert reached Alertmanager twice.
      • A ruler that stops cleanly hands its groups over within one evaluation. One lost with its node hands them over after two minutes, and they are not evaluated until then.
      • Recording-rule samples from the Loki ruler carry instance="loki-ruler" rather than the pod name, and POD_NAME is removed from loki.ruler.extraEnv. An override of write_relabel_configs that still names ${POD_NAME} should restate the env var or switch to the constant.
      • The upgrade rolls every Loki component, since they share one configuration.
  • Update grafana Docker tag to v12.11.2
  • Thanos: datasource scrape interval, distinct SQL endpoint series, ruler on subchart remoteWrite
    • materialize-monitoring#447
    • The Thanos Grafana datasource now sets jsonData.timeInterval to the slower of pipeline.metrics.kubelet.scrapeInterval and pipeline.metrics.kubeProxy.scrapeInterval (60s by default), so rate() panels over cAdvisor and kube-proxy metrics draw continuously instead of as scattered points. Panels over faster-scraped metrics get a 60s minimum step. connections.datasources.thanos.jsonData.timeInterval overrides it, and the render warns when an override is shorter than the chart’s value.
    • Series from the environmentd SQL PodMonitor (/metrics/mz_compute, mz_frontier, mz_storage, mz_usage) carry a new metrics_path label, so that job’s up and scrape_* have one series per path rather than one shared series.
    • The Thanos ruler’s stateless mode is now the subchart’s thanos.ruler.remoteWrite, and the ruler no longer passes --objstore.config-file or runs a block shipper.
      • The remote-write configuration is a Secret, thanos-ruler-remote-write-v2 or -v2-tls, named by thanos.ruler.remoteWrite.secretName; it was a ConfigMap mounted through thanos.ruler.extraVolumes. profiles/mtls.values.yaml now sets secretName.
      • thanos.ruler.extraArgs no longer carries --remote-write.config-file, and extraVolumes / extraVolumeMounts no longer carry a remote-write entry. A values override that still restates either now fails the render with what to remove. Terraform installs are unaffected; Helm installs upgrading with --reuse-values or a copied mTLS override need the entries dropped.
      • To write rule results elsewhere, name another Secret in secretName, or set secretName: "" with createSecret: true and the configuration in config.
  • Renovate: unblock pending updates, automerge crates, split lock files; DEP-324 pin subchart images
    • materialize-monitoring#446
    • Every subchart image the chart renders under its shipped profiles is now pinned in values.yaml (loki.loki.image, loki.lokiCanary.image, loki.memcached.image, loki.memcachedExporter.image, loki.sidecar.image, thanos.global.image, grafana-operator.image, kube-state-metrics.image, metrics-server.image, grafana.initChownData.image). Rendered images are unchanged. These are the subcharts’ own value paths, so existing overrides keep applying.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0

Dashboards (Helm chart) v0.19.0#

  • Add a Thanos meta-monitoring dashboard, with firing alerts on its Overview
  • Replace serde_yaml_ng with serde-saphyr
    • materialize-monitoring#529
    • Pre-rendered dashboards and the rules _index.yaml now use double quotes where a string needs quoting. The parsed content is unchanged, but a byte-level diff of vendored files will show the requoting.
  • Add CNPG and Rook Ceph tabs to the Consensus and Persist dashboards
    • materialize-monitoring#509
    • Materialize Consensus has a new Database Internals tab for metadata databases on CloudNativePG, and Materialize Persist a new Object Store Internals tab for persist buckets on Ceph run by Rook. Each appears only when the dependency is scraped — a PodMonitor for the CNPG cluster, or Rook’s manager and rook-ceph-exporter ServiceMonitors — and has a picker, shown only inside that tab’s section, for which cluster serves the environment.
  • Rename docs reference/internal to reference/development
  • Size the gateway for the series it holds, and keep its memory limiter behind the GC
    • materialize-monitoring#496
    • The alloy-gateway’s default memory is now 2Gi, request and limit, up from 768Mi. The old size was set when the gateway carried logs only. A gateway holds about 200MiB plus 2.6KiB per series it scrapes, and below this it ran its garbage collector constantly. Installs that set alloy-gateway.alloy.resources keep their own values; check them against the sizing note in values.yaml.
    • The gateway’s GOMEMLIMIT is now 80% of its memory limit, set with AUTOMEMLIMIT: "0.8" in alloy-gateway.alloy.extraEnv, instead of a fixed 600MiB. An explicit GOMEMLIMIT there still takes precedence and does not follow the limit. A list that sets neither falls back to Alloy’s own 90%.
    • The gateway’s memory limiter now refuses data only above 85% of its memory limit, instead of about 60%. The garbage collector now acts before scrapes are refused.
    • The gateway’s autoscaler now scales down only after an hour of low CPU, one pod per ten minutes.
  • Alert on a gateway refusing its scrapes, and stop reading the gaps as a stall
    • materialize-monitoring#493
    • New default alert alloy-gateway-refusing-scrapes (warning). It fires when an alloy-gateway pod’s memory limiter has been discarding its scrapes for 15 minutes. While it fires, every target that pod scrapes is missing from the metrics store. It is a LogQL rule, so it needs the bundled Loki ruler.
    • clusterd-not-receiving-commands no longer fires on a healthy replica whose scrapes are missing. It now needs at least two scraped samples in its window.
  • Let alerts read recorded series, and split persist-failures by dependency
    • materialize-monitoring#483
    • Alerts can read the normalized ext:* recorded series. Such an alert installs wherever any recording rule producing what it reads installs, and Common Alerts lists what each one reads.
    • persist-failures is split. Three new alerts join the default set:
      • consensus-unreachable (critical) fires when no call to the metadata database has succeeded for five minutes, or a cloud provider reports it down.
      • consensus-failures (warning) fires on ten minutes of indeterminate metadata-database failures.
      • blob-failures (warning) fires on ten minutes of failed object-storage calls, by operation.
    • persist-failures now covers only persist’s own failures: compaction, read leases, state updates, columnar validation and statistics. It no longer fires on a failing metadata database or object store, or on mz_persist_cmd_failed_count. It remains outside the default set.
    • A metric destination filtering by minMetricImportance now receives ext:consensus_up, and no longer receives mz_persist_blob_failures or mz_persist_cmd_failed_count, which no bundled rule reads any more.
  • Record the normalized ext:consensus_* series, with a producer for the registry’s rules: branch
    • materialize-monitoring#475
    • New recording rules record the metadata (consensus) database as normalized ext:consensus_* series, listed on the new Recorded Series reference page. They install wherever rules.enabled is true and their source is present; no selection is needed.
      • ext:consensus_up and ext:consensus_commit_latency_seconds:p99 come from Materialize’s own calls, carry flavor="persist" and namespace, and need no configuration.
      • ext:consensus_up, ext:consensus_storage_used_ratio and ext:consensus_xid_used_ratio from the CloudWatch, Cloud Monitoring and Azure Monitor pulls carry flavor (rds, cloudsql, azure-postgres) and resource, where the provider publishes each.
    • New value externalDependencies.consensus, a list of {flavor, resourceId} naming which databases a provider pull watches are a metadata database. The provider-sourced ext:consensus_* series record only these, so a pull that also watches Grafana’s database no longer needs to be told apart by hand. The render warns when a pull watches databases and none is named.
    • rules.capabilities gains three derived capabilities, cloudwatch, cloud-monitoring and azure-monitor, present when the matching pipeline.metrics.provider.* pull is enabled.
  • dashboards: say object, freshness, and orphaned instead of collection, lag, and leaked
    • materialize-monitoring#435
    • Dashboard panels now use Materialize’s product terms: object for collection, freshness for lag, and orphaned for leaked persist data. Several panel titles on env-top, env-upgrade, env-persist, and env-consensus changed accordingly; dashboard UIDs and queries did not.

Dependencies#

  • Included mzmon-lib (shared library) @ v0.13.0..v0.14.0
    • Release mzmon-lib (shared library) v0.13.0
    • Port the Terraform render check to Rust
    • Lock file maintenance (cargo)
    • Lock file maintenance (pep621)
    • Update Rust crate hyper to v1.12.0
    • Update Rust crate jsonschema to v0.58.6
    • Update Rust crate tokio to v1.53.2
    • Write the chart README badge in release PRs, and collapse dependency rollups
      • materialize-monitoring#489
      • Dependency rollups in CHANGELOG.md (Included <component> @ vPREV..vNEW) collapse the PRs nested under them in a <details>.
    • Update Rust crate jsonschema to v0.58.5
    • Release mzmon-lib (shared library) v0.12.0
    • Google Cloud metrics: export over OTLP to the Telemetry API, typed
      • materialize-monitoring#474
      • Changed: the Google Cloud metrics destination writes somewhere else. googleCloudExporter (Terraform google_cloud_metrics) now sends OTLP to Google’s Telemetry API instead of using otelcol.exporter.googlecloud, and metrics land as prometheus.googleapis.com/<name>/<kind> instead of workload.googleapis.com/mzmon/<name>. Cloud Monitoring dashboards, alerting policies and anything else reading the old metric types stop receiving data and have to be repointed. The old types are not deleted.
        • Enable the telemetry.googleapis.com API on the project before upgrading with this destination on. Without it every export is refused, and nothing else fails. roles/monitoring.metricWriter is still the only role needed.
        • Metrics are billed per sample ingested instead of per byte, about $75 a month at recommended on a test install where the old export cost about $2,400.
        • Series carry the prometheus_target labels (project_id, location, cluster, namespace, job, instance) and collected_by="materialize-monitoring".
        • Counter values start from zero at the gateway’s first scrape, so they differ from Thanos; rate() and increase() agree.
        • New values googleCloudExporter.project and googleCloudExporter.location are needed only off GKE.
      • Changed: there is no metric prefix any more. googleCloudExporter.prefix is removed from the chart values, along with instrumentation_library_labels, skip_create_descriptor and service_resource_labels; setting any of them renders a warning and does nothing.
      • Deprecated: Terraform google_cloud_metrics.prefix is ignored and warns at plan time; remove it. There is no replacement, because the Telemetry API has no prefix to choose. Everything else in google_cloud_metrics is unchanged.
      • Changed: the gateway’s scrapes honor metric metadata, so every OTLP destination (Google Cloud, Datadog, generic OTLP) receives typed metrics: counters as cumulative sums, histograms as histograms. Metric names in Datadog change accordingly. Thanos and other remote-write destinations are unchanged.
      • alloy-gateway.alloy.stabilityLevel now defaults to experimental, which the scrapes require; the chart refuses any other level.
      • The agent and gateway now run the Alloy v1.20.0 image their image.tag names. Since the bump to v1.20.0-mz3, a stale image.digest had kept both on Alloy v1.19.2.
      • denyMetrics entries match per Prometheus series against typed histograms: denying foo_bucket still keeps foo_count and foo_sum.
    • Update Rust crate hyper-util to v0.1.21
    • Update Rust crate jsonschema to 0.58.0
    • Renovate: unblock pending updates, automerge crates, split lock files; DEP-324 pin subchart images
      • materialize-monitoring#446
      • Every subchart image the chart renders under its shipped profiles is now pinned in values.yaml (loki.loki.image, loki.lokiCanary.image, loki.memcached.image, loki.memcachedExporter.image, loki.sidecar.image, thanos.global.image, grafana-operator.image, kube-state-metrics.image, metrics-server.image, grafana.initChownData.image). Rendered images are unchanged. These are the subcharts’ own value paths, so existing overrides keep applying.
    • docs: flatten reference/stable-metrics into reference, and reorder by use

materialize-monitoring (Helm chart + Terraform module) v0.29.0#

  • Update docker.io/grafana/grafana Docker tag to v13.2.3
  • dashboards: add Infrastructure Autoscaling and Karpenter
    • materialize-monitoring#425
    • Two new dashboards, installed by the default infra-* pattern: Infrastructure Autoscaling (infra-autoscaling), on every cloud, and Karpenter (infra-karpenter), which has data on EKS where Karpenter’s ServiceMonitor is applied.
    • kube-state-metrics now publishes kube_node_labels for each provisioner’s pool label, the instance type and the zone, set by the new default kube-state-metrics.metricLabelsAllowlist. An install that brings its own kube-state-metrics needs the same allowlist for the Node Pools tab.
  • Re-enable the nginx Loki gateway so Grafana lists Loki rules
    • materialize-monitoring#432
    • The Loki gateway (nginx) is enabled by default, and Grafana’s Loki datasource points at it: connections.datasources.loki.url defaults to http://loki-gateway.<namespace>.svc:8080. Grafana’s alerting UI now lists the Loki ruler’s rules and their state, read-only.
      • The gateway routes only Grafana’s reads and the ruler’s rule and alert state. Pushes, rule definitions, ring pages, flushes and deletes are refused.
      • A values file that sets loki.gateway.enabled: false now fails the render until connections.datasources.loki.url points at the query frontend (http://loki-query-frontend.<namespace>.svc:3100).
      • New loki.gateway.nginxConfig.tls: certificate paths for the gateway’s listener and its connections to Loki. profiles/mtls.values.yaml and mtls-phase2.values.yaml set them; phase 2 also has Grafana present its certificate on the Loki datasource.
      • loki-gateway is added to certificates.components.loki.services.
      • Under profiles/split-namespace.values.yaml, loki.networkPolicy.ingress.namespaceSelector must admit the grafana namespace: it now governs Grafana’s access to the gateway.
      • profiles/registry/chainguard.values.yaml pins the gateway’s nginx tag to 1.31.6. Under profiles/registry/docker-hardened-images.values.yaml, restate loki.gateway.image.tag to a variant your mirror holds.
    • The Terraform module’s logs_url output is now the Loki gateway (http://loki-gateway.<namespace>.svc.cluster.local:8080), and https whenever internal_tls is not off. It was http under every internal_tls before, which was wrong for the TLS stages.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0
    • dashboards: say object, freshness, and orphaned instead of collection, lag, and leaked
      • materialize-monitoring#435
      • Dashboard panels now use Materialize’s product terms: object for collection, freshness for lag, and orphaned for leaked persist data. Several panel titles on env-top, env-upgrade, env-persist, and env-consensus changed accordingly; dashboard UIDs and queries did not.

materialize-monitoring (Helm chart + Terraform module) v0.28.0#

  • Render LogQL alerts from the query registry and deliver them to the Loki ruler
    • materialize-monitoring#426
    • Bundled log-derived alerts, evaluated by the Loki ruler. materialize-panic, data-correctness-error and persist-filter-pushdown-violation are in the default set. trace-logging-enabled installs when selected.
    • Every bundled PrometheusRule now carries mzmon.materialize.cloud/flavor: promql or logql. The alloy-gateway writes logql ones into the Loki ruler through its API, including a deployment’s own. They install only where the ruler’s rule store accepts writes; a local store, which a filesystem-only Loki gets, leaves them out with a render warning.
    • thanos.ruler.autoImportPrometheusRules.labelSelector defaults to mzmon.materialize.cloud/flavor!: logql. A replacement selector has to keep that key, or select flavor: promql. A cluster running a Prometheus Operator admission webhook has to exclude flavor: logql from it.
    • New rules.logTenants: the Loki tenants the log-derived rules are written into. Empty means pipeline.logging.tenancy.staticTenant. List them under byEnvironment tenancy.
    • The generated rule index moved from pre-rendered/rules/prometheus/_index.yaml to pre-rendered/rules/_index.yaml, and records each rule’s engine.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0

Container Images v0.7.0#

materialize-monitoring (Helm chart + Terraform module) v0.27.0#

  • Expose alerting rules, routing and receiver credentials through Terraform
    • materialize-monitoring#424
    • The Terraform module takes the chart’s alerting configuration:
      • alert_rules sets which bundled rules install and their overrides, namespaces and infrastructure workload tiers.
      • alerting sets the routing preset, receivers, extra routes, inhibit rules, time intervals, templates and Alertmanager’s global block.
    • alerting_receiver_secrets is new: a sensitive map the module turns into the alertmanager-receivers Secret, so receiver credentials stay out of the Helm values.
      • Receivers read each key as /etc/alertmanager/secrets/alertmanager-receivers/<key>, and the plan fails on a key the map does not set.
      • Leave it empty to manage that Secret some other way.
    • alertmanager_namespace is new: where that Secret is created. Set it to alertmanager under the chart’s split-namespace profile.
  • DEP-233 Pull instance availability from each cloud’s monitoring API
    • materialize-monitoring#422
    • Pull what each cloud publishes about instance availability, off by default:
      • pipeline.metrics.provider.cloudwatch.eks.clusters pulls EC2 status checks for every node of the listed EKS clusters, their managed node groups’ sizes, and the region’s On-Demand vCPU usage. It needs cloudwatch:GetMetricData, cloudwatch:ListMetrics, tag:GetResources and autoscaling:DescribeAutoScalingGroups.
      • pipeline.metrics.provider.gcp.compute.regions pulls per-family CPU and local-SSD quota, usage against limit, and refusals. roles/monitoring.viewer already covers it.
      • pipeline.metrics.provider.azure.aks.clusters pulls the AKS cluster autoscaler’s gauges and each node VM’s availability. It needs Monitoring Reader on each cluster and its node resource group.
      • The new families (aws_ec2_*, aws_autoscaling_*, aws_usage_*, stackdriver_compute_googleapis_com_location_*, azure_microsoft_containerservice_managedclusters_* and azure_microsoft_compute_virtualmachinescalesets_*) take each provider’s metricImportance.
  • Run Grafana on a read-only root, and give every pod a securityContext
    • materialize-monitoring#423
    • Grafana runs on a read-only root filesystem. The chart mounts an emptyDir at /tmp through grafana.extraEmptyDirMounts. A values file that sets that list replaces the chart’s, so it has to keep the /tmp entry; the render fails if it does not.
    • Grafana’s bundled datasource plugins stay at the image’s versions. grafana.ini.plugins.preinstall_auto_update defaults to false. Previously Grafana replaced Prometheus, Loki and the other bundled datasources with the newest release on grafana.com at every start.
    • New render errors when Grafana has a read-only root and nothing is mounted at /tmp, when preinstall auto-update is back on, or when grafana.env.GF_AWS_PROFILES is set without a mount at /usr/share/grafana/.aws. A warning fires when grafana.containerSecurityContext.readOnlyRootFilesystem is turned off.
    • RuntimeDefault seccomp and a pod-level securityContext on grafana-operator, every Thanos component, metrics-server, node-exporter and the pre-validate job, and seccomp on the Alloy containers. node-exporter drops every capability and forbids privilege escalation.
    • grafana-operator requests resources (10m CPU, 64Mi memory, 512Mi memory limit). It was BestEffort.
  • dashboards: add Persist, Consensus and Cloud Provider dashboards
    • materialize-monitoring#419
    • Three new dashboards, installed by default: Materialize Persist (Storage) (env-persist) and Materialize Consensus (Metadata) (env-consensus), Materialize’s own view of object storage and the metadata database, and Infrastructure Cloud Provider (infra-cloud), which draws the metrics collected by pipeline.metrics.provider.
    • Infrastructure Networking (infra-net) now shows its CNI and Security vendor rows. They were hidden on every cluster, which read as “No Dataplane Metrics”.
    • About 50 persist and timestamp-oracle metric families now ship to destinations at minMetricImportance: recommended, and the cloud provider families named by infra-cloud join the diagnostic tier. pipeline.metrics.provider.*.metricImportance still decides every tier above diagnostic.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0

materialize-monitoring (Helm chart + Terraform module) v0.26.0#

  • DEP-301 Pull Azure Monitor metrics into the gateway
    • materialize-monitoring#417
    • New pipeline.metrics.provider.azure, off by default.
      • It pulls a fixed, minimal set of PostgreSQL Flexible Server and Blob Storage metrics from Azure Monitor, for the servers and storage accounts listed under it. Nothing is discovered.
      • Join on resourceName. instance is postgres, blob_capacity or blob_requests.
      • See Cloud Provider Metrics.
    • Credentials come from the gateway pod’s identity, never from values. The identity needs Monitoring Reader on each named resource.
      • Workload identity needs both the azure.workload.identity/client-id annotation on alloy-gateway.serviceAccount and the azure.workload.identity/use: "true" label in alloy-gateway.controller.podLabels.
      • The Terraform module sets the label whenever the annotation is present.
    • Azure families default to the extended tier, like the other providers.
  • Alert on freshness, hydration and replica health, split by who acts on it
    • materialize-monitoring#418
    • The default set gains ten alerting rules: environmentd-down, environmentd-not-scraped, system-cluster-falling-behind, system-cluster-stale, system-cluster-hydration-stuck, system-cluster-memory-near-limit, cluster-replica-not-ready, cluster-hydration-stuck, cluster-memory-near-limit and cluster-replica-oomkilled.
      • cluster-falling-behind, cluster-stale, cluster-memory-high, cluster-cpu-high and source-disconnected are new and install only when selected.
      • The user-cluster rules install as a third PrometheusRule, <release>-materialize-workload-alerts.
    • Every bundled alert carries an audience label, platform or workload, for routing with alerting.routes.extra. Alerts about a cluster also carry cluster_name.
    • rules.overrides.<alert> is new: it sets a rule’s for and adds or replaces its labels. A deployment whose clusters take longer than an hour to hydrate should lengthen cluster-hydration-stuck’s for.
  • Install the bundled alerting rules as PrometheusRules, gated by capability
    • materialize-monitoring#408
    • The chart now installs alerting rules. rules.enabled, default true, renders the bundled alerts as PrometheusRule resources, which the Thanos ruler evaluates.
      • Eighteen rules are in the default set; the rest install only when named in rules.selected.
      • A rule installs only where every capability it requires is present. Capabilities for what the chart deploys are derived; others, such as crdb-dedicated or cilium, are listed in rules.capabilities.
      • The names of the rules in the default set are now part of the committed surface.
    • rules.capabilities, rules.selected, rules.disabled, rules.namespaces.{environment,operator,exclude} and rules.infraWorkloads.{core,important,nonessential,daemonset} are new.
      • Environment namespaces default to materialize.namespaces, then to materialize-system.namespace. A deployment whose environments live elsewhere needs to set one of them, or environment-scoped rules match nothing.
      • rules.infraWorkloads says which infrastructure workloads the rules treat as core, important, non-essential, or on every node. The defaults cover common EKS and GKE add-ons and this chart’s own collectors; list your cluster’s own add-ons in the matching tier.
    • materialize.deploymentMode now decides the metric prefix the rules read on SQL-backed metrics: mz_, or v2_mz_ for cloud. Any other value fails the render.
    • Removed: config.rules.prometheus.enabled, config.rules.loki.enabled, config.rules.thanos.enabled and config.alerts.enabled, which no template ever read. Use rules.enabled. A values file that still sets them renders with a warning.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0

Dashboards (Helm chart) v0.18.0#

  • dashboards: add Infrastructure Autoscaling and Karpenter
    • materialize-monitoring#425
    • Two new dashboards, installed by the default infra-* pattern: Infrastructure Autoscaling (infra-autoscaling), on every cloud, and Karpenter (infra-karpenter), which has data on EKS where Karpenter’s ServiceMonitor is applied.
    • kube-state-metrics now publishes kube_node_labels for each provisioner’s pool label, the instance type and the zone, set by the new default kube-state-metrics.metricLabelsAllowlist. An install that brings its own kube-state-metrics needs the same allowlist for the Node Pools tab.
  • Render LogQL alerts from the query registry and deliver them to the Loki ruler
    • materialize-monitoring#426
    • Bundled log-derived alerts, evaluated by the Loki ruler. materialize-panic, data-correctness-error and persist-filter-pushdown-violation are in the default set. trace-logging-enabled installs when selected.
    • Every bundled PrometheusRule now carries mzmon.materialize.cloud/flavor: promql or logql. The alloy-gateway writes logql ones into the Loki ruler through its API, including a deployment’s own. They install only where the ruler’s rule store accepts writes; a local store, which a filesystem-only Loki gets, leaves them out with a render warning.
    • thanos.ruler.autoImportPrometheusRules.labelSelector defaults to mzmon.materialize.cloud/flavor!: logql. A replacement selector has to keep that key, or select flavor: promql. A cluster running a Prometheus Operator admission webhook has to exclude flavor: logql from it.
    • New rules.logTenants: the Loki tenants the log-derived rules are written into. Empty means pipeline.logging.tenancy.staticTenant. List them under byEnvironment tenancy.
    • The generated rule index moved from pre-rendered/rules/prometheus/_index.yaml to pre-rendered/rules/_index.yaml, and records each rule’s engine.
  • dashboards: add Persist, Consensus and Cloud Provider dashboards
    • materialize-monitoring#419
    • Three new dashboards, installed by default: Materialize Persist (Storage) (env-persist) and Materialize Consensus (Metadata) (env-consensus), Materialize’s own view of object storage and the metadata database, and Infrastructure Cloud Provider (infra-cloud), which draws the metrics collected by pipeline.metrics.provider.
    • Infrastructure Networking (infra-net) now shows its CNI and Security vendor rows. They were hidden on every cluster, which read as “No Dataplane Metrics”.
    • About 50 persist and timestamp-oracle metric families now ship to destinations at minMetricImportance: recommended, and the cloud provider families named by infra-cloud join the diagnostic tier. pipeline.metrics.provider.*.metricImportance still decides every tier above diagnostic.
  • Alert on freshness, hydration and replica health, split by who acts on it
    • materialize-monitoring#418
    • The default set gains ten alerting rules: environmentd-down, environmentd-not-scraped, system-cluster-falling-behind, system-cluster-stale, system-cluster-hydration-stuck, system-cluster-memory-near-limit, cluster-replica-not-ready, cluster-hydration-stuck, cluster-memory-near-limit and cluster-replica-oomkilled.
      • cluster-falling-behind, cluster-stale, cluster-memory-high, cluster-cpu-high and source-disconnected are new and install only when selected.
      • The user-cluster rules install as a third PrometheusRule, <release>-materialize-workload-alerts.
    • Every bundled alert carries an audience label, platform or workload, for routing with alerting.routes.extra. Alerts about a cluster also carry cluster_name.
    • rules.overrides.<alert> is new: it sets a rule’s for and adds or replaces its labels. A deployment whose clusters take longer than an hour to hydrate should lengthen cluster-hydration-stuck’s for.
  • Install the bundled alerting rules as PrometheusRules, gated by capability
    • materialize-monitoring#408
    • The chart now installs alerting rules. rules.enabled, default true, renders the bundled alerts as PrometheusRule resources, which the Thanos ruler evaluates.
      • Eighteen rules are in the default set; the rest install only when named in rules.selected.
      • A rule installs only where every capability it requires is present. Capabilities for what the chart deploys are derived; others, such as crdb-dedicated or cilium, are listed in rules.capabilities.
      • The names of the rules in the default set are now part of the committed surface.
    • rules.capabilities, rules.selected, rules.disabled, rules.namespaces.{environment,operator,exclude} and rules.infraWorkloads.{core,important,nonessential,daemonset} are new.
      • Environment namespaces default to materialize.namespaces, then to materialize-system.namespace. A deployment whose environments live elsewhere needs to set one of them, or environment-scoped rules match nothing.
      • rules.infraWorkloads says which infrastructure workloads the rules treat as core, important, non-essential, or on every node. The defaults cover common EKS and GKE add-ons and this chart’s own collectors; list your cluster’s own add-ons in the matching tier.
    • materialize.deploymentMode now decides the metric prefix the rules read on SQL-backed metrics: mz_, or v2_mz_ for cloud. Any other value fails the render.
    • Removed: config.rules.prometheus.enabled, config.rules.loki.enabled, config.rules.thanos.enabled and config.alerts.enabled, which no template ever read. Use rules.enabled. A values file that still sets them renders with a warning.
  • Adopt the registry’s parameters in the alerts, fix what that exposed, and generate the rules

Dependencies#

  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0
    • Update Rust crate promql-parser to 0.11.0
    • chore(deps): update rust crate tokio-rustls to v0.26.6
    • DEP-233 Pull instance availability from each cloud’s monitoring API
      • materialize-monitoring#422
      • Pull what each cloud publishes about instance availability, off by default:
        • pipeline.metrics.provider.cloudwatch.eks.clusters pulls EC2 status checks for every node of the listed EKS clusters, their managed node groups’ sizes, and the region’s On-Demand vCPU usage. It needs cloudwatch:GetMetricData, cloudwatch:ListMetrics, tag:GetResources and autoscaling:DescribeAutoScalingGroups.
        • pipeline.metrics.provider.gcp.compute.regions pulls per-family CPU and local-SSD quota, usage against limit, and refusals. roles/monitoring.viewer already covers it.
        • pipeline.metrics.provider.azure.aks.clusters pulls the AKS cluster autoscaler’s gauges and each node VM’s availability. It needs Monitoring Reader on each cluster and its node resource group.
        • The new families (aws_ec2_*, aws_autoscaling_*, aws_usage_*, stackdriver_compute_googleapis_com_location_*, azure_microsoft_containerservice_managedclusters_* and azure_microsoft_compute_virtualmachinescalesets_*) take each provider’s metricImportance.
    • DEP-301 Pull Azure Monitor metrics into the gateway
      • materialize-monitoring#417
      • New pipeline.metrics.provider.azure, off by default.
        • It pulls a fixed, minimal set of PostgreSQL Flexible Server and Blob Storage metrics from Azure Monitor, for the servers and storage accounts listed under it. Nothing is discovered.
        • Join on resourceName. instance is postgres, blob_capacity or blob_requests.
        • See Cloud Provider Metrics.
      • Credentials come from the gateway pod’s identity, never from values. The identity needs Monitoring Reader on each named resource.
        • Workload identity needs both the azure.workload.identity/client-id annotation on alloy-gateway.serviceAccount and the azure.workload.identity/use: "true" label in alloy-gateway.controller.podLabels.
        • The Terraform module sets the label whenever the annotation is present.
      • Azure families default to the extended tier, like the other providers.
    • Add an alerting render context, capabilities and gen-rules to the query registry

materialize-monitoring (Helm chart + Terraform module) v0.25.1#

materialize-monitoring (Helm chart + Terraform module) v0.25.0#

  • DEP-229 Update thanos to v0.46.0 so its selectors stop matching Loki
    • materialize-monitoring#402
    • Fixed: Thanos no longer selects Loki’s pods. Thanos’s compactor, query-frontend and ruler selected on component and release name alone, which Loki’s pods of the same names also carry.
      • Node drains could not evict a Loki ruler. Each was covered by both the loki-ruler and thanos-ruler PodDisruptionBudgets, and the eviction API refuses a pod with two. The Loki query frontend had the same problem wherever thanos.queryFrontend.enabled was set.
      • Thanos’s NetworkPolicies, which allow all egress, also applied to the Loki ruler and compactor, so Loki’s own egress rules did not restrict them.
      • Thanos Query no longer dials the Loki rulers’ gRPC port, which logged a warning every five seconds per ruler.
    • Upgrading needs one manual step. Deployment and StatefulSet selectors are immutable, so the upgrade fails at the first Thanos workload unless those workloads are first deleted with --cascade=orphan. The pods keep running and are re-adopted without a restart, except in a workload whose template was changed outside Helm (usually by kubectl rollout restart), which rolls once. See Upgrading.
      • kubectl -n monitoring delete statefulset,deployment -l app.kubernetes.io/part-of=thanos,app.kubernetes.io/instance=mzmon --cascade=orphan
      • An upgrade that already failed on this recovers the same way, then runs again.
    • Resource names, pod labels, the Thanos image and the thanos-thanos ServiceAccount that workload identity bindings name are unchanged.
    • v0.46.0
    • v0.45.0
    • v0.44.0
    • v0.43.0
    • v0.42.0
    • v0.41.1
    • v0.41.0
    • v0.40.0
    • v0.39.0
    • v0.38.0
    • v0.37.0
    • v0.36.0
    • v0.35.0
    • v0.34.0
    • v0.33.1
    • v0.33.0
    • v0.32.0
    • v0.31.0
  • DEP-301 Pull CloudWatch and Cloud Monitoring metrics into the gateway
    • materialize-monitoring#396
    • New pipeline.metrics.provider.cloudwatch and pipeline.metrics.provider.gcp, both off by default.
      • They pull a fixed, minimal set of RDS and S3 metrics from CloudWatch, and of Cloud SQL and GCS metrics from Cloud Monitoring, for the resources listed under each. Nothing is discovered.
      • On CloudWatch series instance is the resource; join on dimension_DBInstanceIdentifier or dimension_BucketName.
      • See Cloud Provider Metrics.
    • Credentials come from the gateway pod’s identity, never from values.
      • CloudWatch needs cloudwatch:GetMetricStatistics, and iam:ListAccountAliases for the account_alias label, through IRSA, EKS Pod Identity, or static keys in the mzmon-alloy-gateway-env Secret.
      • GCP needs roles/monitoring.viewer through Workload Identity.
    • Outside Google Cloud, a GCP pull needs GOOGLE_APPLICATION_CREDENTIALS. With no credential at all, the exporter cannot start and the gateway fails to load.
    • Provider families default to the extended tier. They reach the bundled Thanos, and not a destination that filters at recommended or essential, unless metricImportance is raised.
    • Provider data is minutes old. Query it with last_over_time(...[15m]) or wider. Each gateway restart leaves a gap in Cloud Monitoring series as long as that lag.
  • Make both rulers’ remote-write to the gateway work under the mTLS profiles
    • materialize-monitoring#403
    • Both rulers’ remote-write to the gateway now works under profiles/mtls.values.yaml (Terraform: internal_tls), and the gateway now accepts rule results it had been rejecting for lacking job and instance. ALERTS and recording-rule results reach every metrics destination.
    • The Thanos ruler’s remote-write ConfigMaps are renamed thanos-ruler-remote-write-v2 and thanos-ruler-remote-write-v2-tls. A deployment that restates thanos.ruler.extraVolumes must use the new name; the render says which.
    • Installing with Helm directly: restart the Alloy gateway after upgrading, as for any pipeline change.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
    • Add an Alloy meta-monitoring dashboard, and fix infra-loki’s level picker
      • materialize-monitoring#397
      • New dashboard: Alloy Meta Monitoring (mz-mon-infra-alloy), in the Meta Observability folder. Collector health, both pipelines hop by hop, ingest over log push, remote write and OTLP, component and configuration state with uptime, gateway clustering, resource limits including GOMEMLIMIT, Kubernetes events including the pre-install validation Jobs, and Alloy’s own logs.
      • Fixed the Level picker on Loki Meta Monitoring, which offered only “All”.
      • Metric tiers now include the Alloy families the new dashboard reads. loki_source_file_read_bytes_total leaves the tiers, and the generic Go runtime families it adds are extended, so a metered destination on the default tier does not receive every target’s copy.
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0

materialize-monitoring (Helm chart + Terraform module) v0.24.0#

  • DEP-226 Serve Alertmanager’s API over TLS through the mTLS profiles
    • materialize-monitoring#394
    • Alertmanager can serve its API over TLS: alerting.server.tls, turned on by profiles/mtls.values.yaml (Terraform: internal_tls). Phase 2 verifies client certificates when given; the port stops there, like Loki’s HTTP port.
    • The mTLS profiles now move both rulers, the probes, the config reloader, the ServiceMonitor and the Grafana datasource to TLS with it. Turning it on rolls Alertmanager, both rulers and every Loki component.
    • amtool inside the Alertmanager pod no longer needs --alertmanager.url.
  • Make Alertmanager highly available, and configurable from alerting
    • materialize-monitoring#393
    • The bundled Alertmanager now runs two gossiping replicas, spread across zones and hardened. Clusters whose nodes carry no zone label need the no-zone-spread profile (Terraform: min_zones = 0).
    • Alertmanager’s resources are renamed to alertmanager and start on new volumes; the orphaned storage-<release>-alertmanager-0 PVC can be deleted.
    • New alerting key configures routing: receivers (Alertmanager’s own receiver config plus a class), preset/presets (severity to class), and routes.extra. Receiver credentials are read from Secrets through *_file fields, and an inline one fails the render.
    • New top-level clusterName labels every log line, metric and alert as cluster, replacing setting pipeline.env.CLUSTER_NAME directly (Terraform’s cluster_name now writes it). Give each cluster its own name.
    • Grafana gains an Alertmanager datasource, and Alertmanager’s own metrics are scraped.
  • DEP-216 Turn on both rule evaluators and wire them to Alertmanager
    • materialize-monitoring#384
    • Thanos Ruler is now deployed by default (thanos.ruler.enabled: true, 2 replicas). It runs stateless — no PersistentVolumeClaim — and remote-writes rule results to the alloy-gateway. Set thanos.ruler.enabled: false to keep the previous shape.
    • Every PrometheusRule in the cluster is now evaluated and notified through the bundled Alertmanager. This is the upstream import default. On a cluster running another rule owner, such as kube-prometheus-stack, set thanos.ruler.autoImportPrometheusRules.labelSelector to narrow the set.
    • A new image is pulled by default: docker.io/alpine/kubectl, for the rule-import sidecar. It must carry a shell and curl, so a distroless kubectl will crash-loop. The four profiles/registry/ overlays already address it.
    • The Loki ruler now notifies Alertmanager (loki.loki.rulerConfig.alertmanager_url) and remote-writes recording-rule samples to the alloy-gateway. It previously did neither.
    • split-namespace retargets both rulers and opens Alertmanager to the ruler namespaces. The Loki ruler’s own egress cannot be opened from values in Distributed mode — supply a NetworkPolicy for it, or set loki.networkPolicy.enabled: false.
    • The Terraform storage_class variable now also reaches thanos.ruler, so re-enabling its persistence picks up the class rather than silently missing it.
  • Fall back to CLUSTER_NAME for the gateway’s log cluster label
  • Add a Loki meta-monitoring dashboard, and split dashboards into their own chart
    • materialize-monitoring#383
    • Dashboards now install from a separate chart. materialize-monitoring-dashboards is a release of its own, installed beside materialize-monitoring in the same namespace. The umbrella chart no longer creates dashboards; a release that upgrades without installing the new chart will have its dashboards removed. Helm stores a release in a Kubernetes Secret and a Secret may not exceed 1 MiB, which the rendered set outgrew.
      • Terraform installs it automatically — set enable_dashboards = false to opt out.
      • dashboards.selected → the new chart’s selected.
      • dashboards.config.grafana.manifest.apiTarget → the new chart’s grafana.apiTarget.
      • dashboards.config.datadog is removed; it drove nothing.
      • The new chart cannot read the umbrella release’s values, so grafana.instanceSelector and grafana.folderUids must match it. The umbrella chart’s install notes print the folder UIDs.
      • Dashboard UIDs are unchanged, so saved links, playlists and alerts keep working.
    • New dashboard: Loki Meta Monitoring (mz-mon-infra-loki), in the Meta Observability folder. Ingest, queries, object storage, retention, and Loki’s own logs.
    • Fixed: the Loki canary and both memcached exporters were never scraped under profiles/mtls. The subchart’s single ServiceMonitor applied one scheme to every target, including three that only serve plaintext. They are now collected by a separate monitor, and carry prometheus.io/service-monitor: "false" plus monitoring.materialize.cloud/scrape-scheme: plaintext on their Services. Installs using mTLS gain loki_canary_* and memcached_* series that were previously absent.
    • New Terraform inputs: enable_dashboards, dashboards_chart_version, dashboards_selected, dashboards_instance_selector, dashboards_allow_cross_namespace_import.

Dependencies#

  • Included Pipelines @ v0.12.0..v0.13.0
    • Reassemble and classify Rust panics, and parse tracing’s plain text format
      • materialize-monitoring#376
      • Rust panics from Materialize services now arrive as a single log entry rather than one entry per line of the backtrace, carrying level=CRITICAL and panic_thread / panic_location as structured metadata. The msg names the source location and the panic message.
      • balancerd and materialize-operator logs now carry a parsed level and target. Both previously landed as level="UNKNOWN" for every line.
      • React to this if you filter or size on log level. Those services’ WARN and ERROR lines are no longer swept into the UNKNOWN rate-limit bucket, which drops, so they now reach Loki reliably and ingested volume from the operator namespace rises. A saved query or alert matching level="UNKNOWN" on these services will stop matching.
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0

Dashboards (Helm chart) v0.17.0#

  • Add an Alloy meta-monitoring dashboard, and fix infra-loki’s level picker
    • materialize-monitoring#397
    • New dashboard: Alloy Meta Monitoring (mz-mon-infra-alloy), in the Meta Observability folder. Collector health, both pipelines hop by hop, ingest over log push, remote write and OTLP, component and configuration state with uptime, gateway clustering, resource limits including GOMEMLIMIT, Kubernetes events including the pre-install validation Jobs, and Alloy’s own logs.
    • Fixed the Level picker on Loki Meta Monitoring, which offered only “All”.
    • Metric tiers now include the Alloy families the new dashboard reads. loki_source_file_read_bytes_total leaves the tiers, and the generic Go runtime families it adds are extended, so a metered destination on the default tier does not receive every target’s copy.
  • Add a Loki meta-monitoring dashboard, and split dashboards into their own chart
    • materialize-monitoring#383
    • Dashboards now install from a separate chart. materialize-monitoring-dashboards is a release of its own, installed beside materialize-monitoring in the same namespace. The umbrella chart no longer creates dashboards; a release that upgrades without installing the new chart will have its dashboards removed. Helm stores a release in a Kubernetes Secret and a Secret may not exceed 1 MiB, which the rendered set outgrew.
      • Terraform installs it automatically — set enable_dashboards = false to opt out.
      • dashboards.selected → the new chart’s selected.
      • dashboards.config.grafana.manifest.apiTarget → the new chart’s grafana.apiTarget.
      • dashboards.config.datadog is removed; it drove nothing.
      • The new chart cannot read the umbrella release’s values, so grafana.instanceSelector and grafana.folderUids must match it. The umbrella chart’s install notes print the folder UIDs.
      • Dashboard UIDs are unchanged, so saved links, playlists and alerts keep working.
    • New dashboard: Loki Meta Monitoring (mz-mon-infra-loki), in the Meta Observability folder. Ingest, queries, object storage, retention, and Loki’s own logs.
    • Fixed: the Loki canary and both memcached exporters were never scraped under profiles/mtls. The subchart’s single ServiceMonitor applied one scheme to every target, including three that only serve plaintext. They are now collected by a separate monitor, and carry prometheus.io/service-monitor: "false" plus monitoring.materialize.cloud/scrape-scheme: plaintext on their Services. Installs using mTLS gain loki_canary_* and memcached_* series that were previously absent.
    • New Terraform inputs: enable_dashboards, dashboards_chart_version, dashboards_selected, dashboards_instance_selector, dashboards_allow_cross_namespace_import.

Dependencies#

  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0
    • DEP-301 Pull CloudWatch and Cloud Monitoring metrics into the gateway
      • materialize-monitoring#396
      • New pipeline.metrics.provider.cloudwatch and pipeline.metrics.provider.gcp, both off by default.
        • They pull a fixed, minimal set of RDS and S3 metrics from CloudWatch, and of Cloud SQL and GCS metrics from Cloud Monitoring, for the resources listed under each. Nothing is discovered.
        • On CloudWatch series instance is the resource; join on dimension_DBInstanceIdentifier or dimension_BucketName.
        • See Cloud Provider Metrics.
      • Credentials come from the gateway pod’s identity, never from values.
        • CloudWatch needs cloudwatch:GetMetricStatistics, and iam:ListAccountAliases for the account_alias label, through IRSA, EKS Pod Identity, or static keys in the mzmon-alloy-gateway-env Secret.
        • GCP needs roles/monitoring.viewer through Workload Identity.
      • Outside Google Cloud, a GCP pull needs GOOGLE_APPLICATION_CREDENTIALS. With no credential at all, the exporter cannot start and the gateway fails to load.
      • Provider families default to the extended tier. They reach the bundled Thanos, and not a destination that filters at recommended or essential, unless metricImportance is raised.
      • Provider data is minutes old. Query it with last_over_time(...[15m]) or wider. Each gateway restart leaves a gap in Cloud Monitoring series as long as that lag.
    • Update Rust crate thiserror to v2.0.21
    • Update Rust crate jsonschema to 0.57.0
    • Fall back to CLUSTER_NAME for the gateway’s log cluster label
    • Reassemble and classify Rust panics, and parse tracing’s plain text format
      • materialize-monitoring#376
      • Rust panics from Materialize services now arrive as a single log entry rather than one entry per line of the backtrace, carrying level=CRITICAL and panic_thread / panic_location as structured metadata. The msg names the source location and the panic message.
      • balancerd and materialize-operator logs now carry a parsed level and target. Both previously landed as level="UNKNOWN" for every line.
      • React to this if you filter or size on log level. Those services’ WARN and ERROR lines are no longer swept into the UNKNOWN rate-limit bucket, which drops, so they now reach Loki reliably and ingested volume from the operator namespace rises. A saved query or alert matching level="UNKNOWN" on these services will stop matching.

materialize-monitoring (Helm chart + Terraform module) v0.23.0#

  • Clean up GrafanaFolder on uninstall
    • materialize-monitoring#375
    • Fixed an uninstall hang introduced in v0.22.0: the pre-delete cleanup hook did not delete GrafanaFolder resources, so their grafana-operator finalizers outlived the operator and left the release namespace stuck in Terminating. Installs that already hit this can recover by clearing the finalizers by hand — see Uninstalling.
  • Upgrade helm-unittest to v1.1.2 and pin it for Renovate
  • DEP-211 Add infra-net dashboard, and the CNI collection it needs

Dependencies#

  • Included Dashboards @ v0.16.0..v0.17.0
  • Included Pipelines @ v0.12.0..v0.13.0
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0

Container Images v0.6.0#

materialize-monitoring (Helm chart + Terraform module) v0.22.0#

  • Update docker.io/grafana/grafana Docker tag to v13.2.2
  • Update docker.io/grafana/grafana Docker tag to v13.2.1
  • Move the chart’s alloy image to v1.19.2-mz3
    • materialize-monitoring#348
    • The bundled Alloy image moves from v1.18.1-mz2 to v1.19.2-mz3, which is Alloy v1.19.2 on a rebased distroless base. This is an Alloy minor upgrade for anyone who has not overridden alloy-agent.image / alloy-gateway.image; see the Alloy v1.19 release notes.
  • Add Dashboard Folders; Update tags
    • materialize-monitoring#329
    • Added several GrafanaFolder resources (default enabled: mzmon-infra, mzmon-materialize, mzmon-meta-o11y)
    • Changed monitoring tag to mzmon within dashboards

Dependencies#

Dashboards (Helm chart) v0.16.0#

  • DEP-211 Add infra-net dashboard, and the CNI collection it needs
  • Add Dashboard Folders; Update tags
    • materialize-monitoring#329
    • Added several GrafanaFolder resources (default enabled: mzmon-infra, mzmon-materialize, mzmon-meta-o11y)
    • Changed monitoring tag to mzmon within dashboards
  • Show total lag in env-top / env-upgrade
    • materialize-monitoring#312
    • Add new queries around max lag (materialize.compute.freshness.lag_total_by_cluster, materialize.generations.lag.total)
    • Show max lag as new panels (including per-cluster breakdown) in env-top and env-upgrade dashboards

Dependencies#

Dashboards v0.15.0#

Dependencies#

materialize-monitoring (Helm chart + Terraform module) v0.21.0#

Dependencies#

Pipelines v0.13.0 (Unreleased)#

Changes Pending

Container Images v0.5.0#

Dashboards v0.14.0#

  • DEP-209 Add Infrastructure Logs & Events Dashboard

Dependencies#

Prometheus Scrapers v0.5.0#

  • Render canonical label names from labels.yaml in the Materialize monitors (DEP-421)
    • materialize-monitoring#582
    • The Materialize PodMonitors, and the classic and Google Cloud Managed Prometheus scrape configs, add canonical label names beside the labels Materialize emits: cluster_id, replica_id, cluster_name, cluster_size, replica_name, replica_size, object_name, object_type, collection_id and shard_id. Each is copied only where it’s absent, so it adds no series. Canonical label names lists where each comes from.
    • Deprecated: the labels they’re copied from. Both names stay on every series for at least 30 days after this release; move queries and alert routes to the canonical names.
      • instance_id, compute_instance, compute_cluster_id and cluster_environmentd_materialize_cloud_cluster_id are cluster_id
      • compute_replica_id and cluster_environmentd_materialize_cloud_replica_id are replica_id
      • compute_cluster_name, and name on mz_cluster_info, are cluster_name; size on mz_cluster_info is cluster_size
      • name and size on mz_replica_info are replica_name and replica_size
      • name and type on mz_object_info are object_name and object_type
      • global_id is collection_id, and shard is shard_id
    • If you replace materialize.environmentd.serviceMonitor.metricEndpoints, materialize.environmentdSQL.serviceMonitor.metricEndpoints, materialize.clusterd.serviceMonitor.metricEndpoints or materialize-operator.serviceMonitor.metricEndpoints, restate the default metricRelabelings, or those series lack the canonical labels.
  • Rename docs reference/internal to reference/development
  • Thanos: datasource scrape interval, distinct SQL endpoint series, ruler on subchart remoteWrite
    • materialize-monitoring#447
    • The Thanos Grafana datasource now sets jsonData.timeInterval to the slower of pipeline.metrics.kubelet.scrapeInterval and pipeline.metrics.kubeProxy.scrapeInterval (60s by default), so rate() panels over cAdvisor and kube-proxy metrics draw continuously instead of as scattered points. Panels over faster-scraped metrics get a 60s minimum step. connections.datasources.thanos.jsonData.timeInterval overrides it, and the render warns when an override is shorter than the chart’s value.
    • Series from the environmentd SQL PodMonitor (/metrics/mz_compute, mz_frontier, mz_storage, mz_usage) carry a new metrics_path label, so that job’s up and scrape_* have one series per path rather than one shared series.
    • The Thanos ruler’s stateless mode is now the subchart’s thanos.ruler.remoteWrite, and the ruler no longer passes --objstore.config-file or runs a block shipper.
      • The remote-write configuration is a Secret, thanos-ruler-remote-write-v2 or -v2-tls, named by thanos.ruler.remoteWrite.secretName; it was a ConfigMap mounted through thanos.ruler.extraVolumes. profiles/mtls.values.yaml now sets secretName.
      • thanos.ruler.extraArgs no longer carries --remote-write.config-file, and extraVolumes / extraVolumeMounts no longer carry a remote-write entry. A values override that still restates either now fails the render with what to remove. Terraform installs are unaffected; Helm installs upgrading with --reuse-values or a copied mTLS override need the entries dropped.
      • To write rule results elsewhere, name another Secret in secretName, or set secretName: "" with createSecret: true and the configuration in config.
  • Re-enable the nginx Loki gateway so Grafana lists Loki rules
    • materialize-monitoring#432
    • The Loki gateway (nginx) is enabled by default, and Grafana’s Loki datasource points at it: connections.datasources.loki.url defaults to http://loki-gateway.<namespace>.svc:8080. Grafana’s alerting UI now lists the Loki ruler’s rules and their state, read-only.
      • The gateway routes only Grafana’s reads and the ruler’s rule and alert state. Pushes, rule definitions, ring pages, flushes and deletes are refused.
      • A values file that sets loki.gateway.enabled: false now fails the render until connections.datasources.loki.url points at the query frontend (http://loki-query-frontend.<namespace>.svc:3100).
      • New loki.gateway.nginxConfig.tls: certificate paths for the gateway’s listener and its connections to Loki. profiles/mtls.values.yaml and mtls-phase2.values.yaml set them; phase 2 also has Grafana present its certificate on the Loki datasource.
      • loki-gateway is added to certificates.components.loki.services.
      • Under profiles/split-namespace.values.yaml, loki.networkPolicy.ingress.namespaceSelector must admit the grafana namespace: it now governs Grafana’s access to the gateway.
      • profiles/registry/chainguard.values.yaml pins the gateway’s nginx tag to 1.31.6. Under profiles/registry/docker-hardened-images.values.yaml, restate loki.gateway.image.tag to a variant your mirror holds.
    • The Terraform module’s logs_url output is now the Loki gateway (http://loki-gateway.<namespace>.svc.cluster.local:8080), and https whenever internal_tls is not off. It was http under every internal_tls before, which was wrong for the TLS stages.
  • Add a Loki meta-monitoring dashboard, and split dashboards into their own chart
    • materialize-monitoring#383
    • Dashboards now install from a separate chart. materialize-monitoring-dashboards is a release of its own, installed beside materialize-monitoring in the same namespace. The umbrella chart no longer creates dashboards; a release that upgrades without installing the new chart will have its dashboards removed. Helm stores a release in a Kubernetes Secret and a Secret may not exceed 1 MiB, which the rendered set outgrew.
      • Terraform installs it automatically — set enable_dashboards = false to opt out.
      • dashboards.selected → the new chart’s selected.
      • dashboards.config.grafana.manifest.apiTarget → the new chart’s grafana.apiTarget.
      • dashboards.config.datadog is removed; it drove nothing.
      • The new chart cannot read the umbrella release’s values, so grafana.instanceSelector and grafana.folderUids must match it. The umbrella chart’s install notes print the folder UIDs.
      • Dashboard UIDs are unchanged, so saved links, playlists and alerts keep working.
    • New dashboard: Loki Meta Monitoring (mz-mon-infra-loki), in the Meta Observability folder. Ingest, queries, object storage, retention, and Loki’s own logs.
    • Fixed: the Loki canary and both memcached exporters were never scraped under profiles/mtls. The subchart’s single ServiceMonitor applied one scheme to every target, including three that only serve plaintext. They are now collected by a separate monitor, and carry prometheus.io/service-monitor: "false" plus monitoring.materialize.cloud/scrape-scheme: plaintext on their Services. Installs using mTLS gain loki_canary_* and memcached_* series that were previously absent.
    • New Terraform inputs: enable_dashboards, dashboards_chart_version, dashboards_selected, dashboards_instance_selector, dashboards_allow_cross_namespace_import.
  • DEP-211 Add infra-net dashboard, and the CNI collection it needs

Dependencies#

  • Included mzmon-lib (shared library) @ v0.13.0..v0.14.0
    • Split the dashboards chart into one chart per Grafana folder
      • materialize-monitoring#585
      • The dashboards ship in three charts, one per Grafana folder: materialize-monitoring-dashboards for the Materialize environment (env-*), materialize-monitoring-dashboards-infra for the platform underneath it, and materialize-monitoring-dashboards-meta for Alloy, Loki, and Thanos.
        • The Terraform module installs all three, with no variable changes.
        • If you install with Helm directly, upgrading materialize-monitoring-dashboards removes the infrastructure and meta-monitoring dashboards from it. To keep them, install materialize-monitoring-dashboards-infra and materialize-monitoring-dashboards-meta after that upgrade.
        • Each chart’s selected defaults to ["*"], and its grafana.folderUids default names only its own folder. Pass the same values file to all three; each ignores folder entries it doesn’t use.
        • Dashboard UIDs, titles, and links don’t change.
    • Lock file maintenance (pep621)
    • Update dependency uv_build to >=0.13,<0.14
    • Update Rust crate jsonschema to 0.59.0
    • Lock file maintenance (cargo)
    • Lock file maintenance (pep621)
    • Pull load balancer and NAT gateway metrics, and fill infra-net’s Cloud Networking tab
      • materialize-monitoring#552
      • Six new cloud provider pulls cover the load balancers in front of the cluster and the NAT gateways its traffic leaves through. Each is off until resources are listed.
        • pipeline.metrics.provider.cloudwatch.nlb.loadBalancers: Network Load Balancers, each as arn with its targetGroups, for open and new connections, bytes, resets by sender, and healthy and unhealthy targets. An ARN or its CloudWatch form, such as net/<name>/<id>, is accepted; Application Load Balancers are refused.
        • pipeline.metrics.provider.cloudwatch.natGateway.ids: NAT gateway IDs, for port allocation errors, dropped packets, idle timeouts, and connections.
        • pipeline.metrics.provider.gcp.loadBalancing.forwardingRules: passthrough load balancer forwarding rules, external or internal, for packets and bytes in each direction.
        • pipeline.metrics.provider.gcp.cloudNat.gateways: Cloud NAT gateway names, for allocation failures, per-VM port use, open connections, and dropped packets by reason.
        • pipeline.metrics.provider.azure.loadBalancer.clusters: AKS cluster names, whose kubernetes and kubernetes-internal load balancers are pulled for data path availability, backend health probes, bytes, and new connections.
        • pipeline.metrics.provider.azure.natGateway.names: NAT gateway names, for data path availability, SNAT connections attempted and failed, and dropped packets.
      • The AWS pulls need only cloudwatch:GetMetricStatistics, and the GCP pulls only roles/monitoring.viewer, which the existing pulls already use. On Azure, the identity needs Monitoring Reader on the AKS cluster’s node resource group and on each NAT gateway.
      • Series from the new pulls carry a provider_pull label, up included. Series from the existing pulls are unchanged.
      • The new families are tiered by each provider’s metricImportance, like the existing provider families.
      • Infrastructure Networking’s Cloud Networking tab draws the new pulls: target and backend health, connections, resets, traffic, and NAT port exhaustion. Each row renders only where its pull runs, with a note on how to turn on the rest. The two placeholder rows are gone.
    • Update Rust crate rustls to v0.23.46
    • Update Rust crate tokio to v1.53.3
    • Lead the Loki, Alloy and environment overviews with firing alerts
      • materialize-monitoring#551
      • Loki Meta Monitoring, Alloy Meta Monitoring, and the Summary tab of Materialize Environment Overview now count the alerts firing about their subject, and list every alert firing or pending in an Alerts row. Each alert’s name links to its entry under Common Alerts.
      • An alert count shows the current value. It reads zero only while the Thanos ruler, and every Loki rule group where a Loki ruler runs, are being evaluated and written back. Otherwise it’s empty.
      • The recommended metric tier adds loki_prometheus_rule_group_last_evaluation_timestamp_seconds, loki_prometheus_rule_group_interval_seconds, loki_ruler_wal_prometheus_remote_storage_highest_timestamp_in_seconds, and loki_ruler_wal_prometheus_remote_storage_queue_highest_sent_timestamp_seconds.
    • Update Rust crate serde-saphyr to v2
    • Truncate oversized log lines and add a development alert tier
      • materialize-monitoring#549
      • Oversized log lines are cut rather than dropped, and the cut line ends in a suffix that names the rule: [TRUNCATED:SAFETYCLAMP], [TRUNCATED:LINE], or [TRUNCATED:FIELDS].
        • The agent and the gateway clamp every line to PIPELINE_LOG_LINE_SAFETY_CLAMP (default 1MiB). The gateway then cuts lines to GATEWAY_LOG_LINE_SIZE_LIMIT (default 240KiB), with a fixed 250KiB backstop. Set either variable in pipeline.env, with a unit such as 512KiB; a bare number stops Alloy from loading its configuration.
        • The gateway cuts label values to 2KiB and structured metadata to a per-entry budget: 512B for each key it sets, 16KiB for msg and for keys a Loki push client sends, and 16KiB held back for a push client’s own structured metadata. Entries then fit Loki’s default limits rather than being refused, as long as a push client’s own structured metadata stays under 16KiB.
        • A clamped JSON or logfmt line no longer parses, so it reaches Loki without its extracted fields.
      • loki_process_dropped_lines_total no longer reports the too large reason. Cuts count in loki_process_truncated_fields_total, labeled by field, which the default metric tier keeps.
      • Alloy Meta Monitoring: the Truncated Lines panel counts those cuts by field, beside the agents’ CRI reassembly cuts.
      • Alerts can carry audience: development, for conditions the stack has already contained that the people who write the software can remove. An extra route on audience="development" sends them to a development channel, and rules.overrides accepts the value.
        • oversized-log-lines (severity: notice, not in the default set) fires when Alloy cut a log line in the past hour. Select it with rules.selected.
    • Terraform: run the postgres-exporter and declare metadata databases
      • materialize-monitoring#542
      • The Terraform module runs the postgres-exporter with enable_postgres_exporter = true. It watches Grafana’s database whenever the module configures one with a password, plus any database in postgres_exporter_targets, with passwords from postgres_exporter_passwords. The module creates the exporter’s postgres-exporter Secret and rolls the exporter when a password changes.
      • The Terraform module takes metadata_databases (flavor and resource_id) for the metadata databases a provider pull watches, and a metadata_database flag on an exporter target. Both become externalDependencies.consensus entries.
      • node_selector and tolerations reach the postgres-exporter.
    • Add a postgres_exporter subchart for the metadata and Grafana databases
      • materialize-monitoring#540
      • A postgres-exporter subchart (prometheus-postgres-exporter 8.2.0, postgres_exporter v0.20.1) watches PostgreSQL databases from the inside. It is off by default: enable it with tags.postgres-exporter: true and list the databases under postgres-exporter.targets. Credentials come from a postgres-exporter Secret you create. See the PostgreSQL exporter page for the Secret’s shape and the least-privilege pg_monitor grant.
      • externalDependencies.consensus accepts flavor: postgres, with an exporter target’s name as resourceId. The new postgres adapter records ext:consensus_up, ext:consensus_connections_used_ratio, ext:consensus_xid_used_ratio, ext:consensus_database_bytes (with a database label) and ext:consensus_version_info (with a version label) for that target.
      • rules.capabilities gains a derived capability, postgres-exporter, present when the exporter runs.
      • The Materialize Consensus (Metadata) dashboard’s Database Internals tab has a PostgreSQL section, rendered where the exporter is scraped, with a Database picker.
    • Add a Thanos meta-monitoring dashboard, with firing alerts on its Overview
    • Release mzmon-lib (shared library) v0.13.0
    • Replace serde_yaml_ng with serde-saphyr
      • materialize-monitoring#529
      • Pre-rendered dashboards and the rules _index.yaml now use double quotes where a string needs quoting. The parsed content is unchanged, but a byte-level diff of vendored files will show the requoting.
    • Port the Terraform render check to Rust
    • Add CNPG and Rook Ceph tabs to the Consensus and Persist dashboards
      • materialize-monitoring#509
      • Materialize Consensus has a new Database Internals tab for metadata databases on CloudNativePG, and Materialize Persist a new Object Store Internals tab for persist buckets on Ceph run by Rook. Each appears only when the dependency is scraped — a PodMonitor for the CNPG cluster, or Rook’s manager and rook-ceph-exporter ServiceMonitors — and has a picker, shown only inside that tab’s section, for which cluster serves the environment.
    • Lock file maintenance (cargo)
    • Lock file maintenance (pep621)
    • Update Rust crate hyper to v1.12.0
    • Update Rust crate jsonschema to v0.58.6
    • Update Rust crate tokio to v1.53.2
    • Write the chart README badge in release PRs, and collapse dependency rollups
      • materialize-monitoring#489
      • Dependency rollups in CHANGELOG.md (Included <component> @ vPREV..vNEW) collapse the PRs nested under them in a <details>.
    • Update Rust crate jsonschema to v0.58.5
    • Let alerts read recorded series, and split persist-failures by dependency
      • materialize-monitoring#483
      • Alerts can read the normalized ext:* recorded series. Such an alert installs wherever any recording rule producing what it reads installs, and Common Alerts lists what each one reads.
      • persist-failures is split. Three new alerts join the default set:
        • consensus-unreachable (critical) fires when no call to the metadata database has succeeded for five minutes, or a cloud provider reports it down.
        • consensus-failures (warning) fires on ten minutes of indeterminate metadata-database failures.
        • blob-failures (warning) fires on ten minutes of failed object-storage calls, by operation.
      • persist-failures now covers only persist’s own failures: compaction, read leases, state updates, columnar validation and statistics. It no longer fires on a failing metadata database or object store, or on mz_persist_cmd_failed_count. It remains outside the default set.
      • A metric destination filtering by minMetricImportance now receives ext:consensus_up, and no longer receives mz_persist_blob_failures or mz_persist_cmd_failed_count, which no bundled rule reads any more.
    • Release mzmon-lib (shared library) v0.12.0
    • Record the normalized ext:consensus_* series, with a producer for the registry’s rules: branch
      • materialize-monitoring#475
      • New recording rules record the metadata (consensus) database as normalized ext:consensus_* series, listed on the new Recorded Series reference page. They install wherever rules.enabled is true and their source is present; no selection is needed.
        • ext:consensus_up and ext:consensus_commit_latency_seconds:p99 come from Materialize’s own calls, carry flavor="persist" and namespace, and need no configuration.
        • ext:consensus_up, ext:consensus_storage_used_ratio and ext:consensus_xid_used_ratio from the CloudWatch, Cloud Monitoring and Azure Monitor pulls carry flavor (rds, cloudsql, azure-postgres) and resource, where the provider publishes each.
      • New value externalDependencies.consensus, a list of {flavor, resourceId} naming which databases a provider pull watches are a metadata database. The provider-sourced ext:consensus_* series record only these, so a pull that also watches Grafana’s database no longer needs to be told apart by hand. The render warns when a pull watches databases and none is named.
      • rules.capabilities gains three derived capabilities, cloudwatch, cloud-monitoring and azure-monitor, present when the matching pipeline.metrics.provider.* pull is enabled.
    • Google Cloud metrics: export over OTLP to the Telemetry API, typed
      • materialize-monitoring#474
      • Changed: the Google Cloud metrics destination writes somewhere else. googleCloudExporter (Terraform google_cloud_metrics) now sends OTLP to Google’s Telemetry API instead of using otelcol.exporter.googlecloud, and metrics land as prometheus.googleapis.com/<name>/<kind> instead of workload.googleapis.com/mzmon/<name>. Cloud Monitoring dashboards, alerting policies and anything else reading the old metric types stop receiving data and have to be repointed. The old types are not deleted.
        • Enable the telemetry.googleapis.com API on the project before upgrading with this destination on. Without it every export is refused, and nothing else fails. roles/monitoring.metricWriter is still the only role needed.
        • Metrics are billed per sample ingested instead of per byte, about $75 a month at recommended on a test install where the old export cost about $2,400.
        • Series carry the prometheus_target labels (project_id, location, cluster, namespace, job, instance) and collected_by="materialize-monitoring".
        • Counter values start from zero at the gateway’s first scrape, so they differ from Thanos; rate() and increase() agree.
        • New values googleCloudExporter.project and googleCloudExporter.location are needed only off GKE.
      • Changed: there is no metric prefix any more. googleCloudExporter.prefix is removed from the chart values, along with instrumentation_library_labels, skip_create_descriptor and service_resource_labels; setting any of them renders a warning and does nothing.
      • Deprecated: Terraform google_cloud_metrics.prefix is ignored and warns at plan time; remove it. There is no replacement, because the Telemetry API has no prefix to choose. Everything else in google_cloud_metrics is unchanged.
      • Changed: the gateway’s scrapes honor metric metadata, so every OTLP destination (Google Cloud, Datadog, generic OTLP) receives typed metrics: counters as cumulative sums, histograms as histograms. Metric names in Datadog change accordingly. Thanos and other remote-write destinations are unchanged.
      • alloy-gateway.alloy.stabilityLevel now defaults to experimental, which the scrapes require; the chart refuses any other level.
      • The agent and gateway now run the Alloy v1.20.0 image their image.tag names. Since the bump to v1.20.0-mz3, a stale image.digest had kept both on Alloy v1.19.2.
      • denyMetrics entries match per Prometheus series against typed histograms: denying foo_bucket still keeps foo_count and foo_sum.
    • Update Rust crate hyper-util to v0.1.21
    • Update Rust crate jsonschema to 0.58.0
    • Renovate: unblock pending updates, automerge crates, split lock files; DEP-324 pin subchart images
      • materialize-monitoring#446
      • Every subchart image the chart renders under its shipped profiles is now pinned in values.yaml (loki.loki.image, loki.lokiCanary.image, loki.memcached.image, loki.memcachedExporter.image, loki.sidecar.image, thanos.global.image, grafana-operator.image, kube-state-metrics.image, metrics-server.image, grafana.initChownData.image). Rendered images are unchanged. These are the subcharts’ own value paths, so existing overrides keep applying.
    • docs: flatten reference/stable-metrics into reference, and reorder by use
    • dashboards: say object, freshness, and orphaned instead of collection, lag, and leaked
      • materialize-monitoring#435
      • Dashboard panels now use Materialize’s product terms: object for collection, freshness for lag, and orphaned for leaked persist data. Several panel titles on env-top, env-upgrade, env-persist, and env-consensus changed accordingly; dashboard UIDs and queries did not.
    • dashboards: add Infrastructure Autoscaling and Karpenter
      • materialize-monitoring#425
      • Two new dashboards, installed by the default infra-* pattern: Infrastructure Autoscaling (infra-autoscaling), on every cloud, and Karpenter (infra-karpenter), which has data on EKS where Karpenter’s ServiceMonitor is applied.
      • kube-state-metrics now publishes kube_node_labels for each provisioner’s pool label, the instance type and the zone, set by the new default kube-state-metrics.metricLabelsAllowlist. An install that brings its own kube-state-metrics needs the same allowlist for the Node Pools tab.
    • Update Rust crate promql-parser to 0.11.0
    • Render LogQL alerts from the query registry and deliver them to the Loki ruler
      • materialize-monitoring#426
      • Bundled log-derived alerts, evaluated by the Loki ruler. materialize-panic, data-correctness-error and persist-filter-pushdown-violation are in the default set. trace-logging-enabled installs when selected.
      • Every bundled PrometheusRule now carries mzmon.materialize.cloud/flavor: promql or logql. The alloy-gateway writes logql ones into the Loki ruler through its API, including a deployment’s own. They install only where the ruler’s rule store accepts writes; a local store, which a filesystem-only Loki gets, leaves them out with a render warning.
      • thanos.ruler.autoImportPrometheusRules.labelSelector defaults to mzmon.materialize.cloud/flavor!: logql. A replacement selector has to keep that key, or select flavor: promql. A cluster running a Prometheus Operator admission webhook has to exclude flavor: logql from it.
      • New rules.logTenants: the Loki tenants the log-derived rules are written into. Empty means pipeline.logging.tenancy.staticTenant. List them under byEnvironment tenancy.
      • The generated rule index moved from pre-rendered/rules/prometheus/_index.yaml to pre-rendered/rules/_index.yaml, and records each rule’s engine.
    • chore(deps): update rust crate tokio-rustls to v0.26.6
    • DEP-233 Pull instance availability from each cloud’s monitoring API
      • materialize-monitoring#422
      • Pull what each cloud publishes about instance availability, off by default:
        • pipeline.metrics.provider.cloudwatch.eks.clusters pulls EC2 status checks for every node of the listed EKS clusters, their managed node groups’ sizes, and the region’s On-Demand vCPU usage. It needs cloudwatch:GetMetricData, cloudwatch:ListMetrics, tag:GetResources and autoscaling:DescribeAutoScalingGroups.
        • pipeline.metrics.provider.gcp.compute.regions pulls per-family CPU and local-SSD quota, usage against limit, and refusals. roles/monitoring.viewer already covers it.
        • pipeline.metrics.provider.azure.aks.clusters pulls the AKS cluster autoscaler’s gauges and each node VM’s availability. It needs Monitoring Reader on each cluster and its node resource group.
        • The new families (aws_ec2_*, aws_autoscaling_*, aws_usage_*, stackdriver_compute_googleapis_com_location_*, azure_microsoft_containerservice_managedclusters_* and azure_microsoft_compute_virtualmachinescalesets_*) take each provider’s metricImportance.
    • dashboards: add Persist, Consensus and Cloud Provider dashboards
      • materialize-monitoring#419
      • Three new dashboards, installed by default: Materialize Persist (Storage) (env-persist) and Materialize Consensus (Metadata) (env-consensus), Materialize’s own view of object storage and the metadata database, and Infrastructure Cloud Provider (infra-cloud), which draws the metrics collected by pipeline.metrics.provider.
      • Infrastructure Networking (infra-net) now shows its CNI and Security vendor rows. They were hidden on every cluster, which read as “No Dataplane Metrics”.
      • About 50 persist and timestamp-oracle metric families now ship to destinations at minMetricImportance: recommended, and the cloud provider families named by infra-cloud join the diagnostic tier. pipeline.metrics.provider.*.metricImportance still decides every tier above diagnostic.
    • DEP-301 Pull Azure Monitor metrics into the gateway
      • materialize-monitoring#417
      • New pipeline.metrics.provider.azure, off by default.
        • It pulls a fixed, minimal set of PostgreSQL Flexible Server and Blob Storage metrics from Azure Monitor, for the servers and storage accounts listed under it. Nothing is discovered.
        • Join on resourceName. instance is postgres, blob_capacity or blob_requests.
        • See Cloud Provider Metrics.
      • Credentials come from the gateway pod’s identity, never from values. The identity needs Monitoring Reader on each named resource.
        • Workload identity needs both the azure.workload.identity/client-id annotation on alloy-gateway.serviceAccount and the azure.workload.identity/use: "true" label in alloy-gateway.controller.podLabels.
        • The Terraform module sets the label whenever the annotation is present.
      • Azure families default to the extended tier, like the other providers.
    • Alert on freshness, hydration and replica health, split by who acts on it
      • materialize-monitoring#418
      • The default set gains ten alerting rules: environmentd-down, environmentd-not-scraped, system-cluster-falling-behind, system-cluster-stale, system-cluster-hydration-stuck, system-cluster-memory-near-limit, cluster-replica-not-ready, cluster-hydration-stuck, cluster-memory-near-limit and cluster-replica-oomkilled.
        • cluster-falling-behind, cluster-stale, cluster-memory-high, cluster-cpu-high and source-disconnected are new and install only when selected.
        • The user-cluster rules install as a third PrometheusRule, <release>-materialize-workload-alerts.
      • Every bundled alert carries an audience label, platform or workload, for routing with alerting.routes.extra. Alerts about a cluster also carry cluster_name.
      • rules.overrides.<alert> is new: it sets a rule’s for and adds or replaces its labels. A deployment whose clusters take longer than an hour to hydrate should lengthen cluster-hydration-stuck’s for.
    • Adopt the registry’s parameters in the alerts, fix what that exposed, and generate the rules
    • Add an alerting render context, capabilities and gen-rules to the query registry
    • Add an Alloy meta-monitoring dashboard, and fix infra-loki’s level picker
      • materialize-monitoring#397
      • New dashboard: Alloy Meta Monitoring (mz-mon-infra-alloy), in the Meta Observability folder. Collector health, both pipelines hop by hop, ingest over log push, remote write and OTLP, component and configuration state with uptime, gateway clustering, resource limits including GOMEMLIMIT, Kubernetes events including the pre-install validation Jobs, and Alloy’s own logs.
      • Fixed the Level picker on Loki Meta Monitoring, which offered only “All”.
      • Metric tiers now include the Alloy families the new dashboard reads. loki_source_file_read_bytes_total leaves the tiers, and the generic Go runtime families it adds are extended, so a metered destination on the default tier does not receive every target’s copy.
    • DEP-301 Pull CloudWatch and Cloud Monitoring metrics into the gateway
      • materialize-monitoring#396
      • New pipeline.metrics.provider.cloudwatch and pipeline.metrics.provider.gcp, both off by default.
        • They pull a fixed, minimal set of RDS and S3 metrics from CloudWatch, and of Cloud SQL and GCS metrics from Cloud Monitoring, for the resources listed under each. Nothing is discovered.
        • On CloudWatch series instance is the resource; join on dimension_DBInstanceIdentifier or dimension_BucketName.
        • See Cloud Provider Metrics.
      • Credentials come from the gateway pod’s identity, never from values.
        • CloudWatch needs cloudwatch:GetMetricStatistics, and iam:ListAccountAliases for the account_alias label, through IRSA, EKS Pod Identity, or static keys in the mzmon-alloy-gateway-env Secret.
        • GCP needs roles/monitoring.viewer through Workload Identity.
      • Outside Google Cloud, a GCP pull needs GOOGLE_APPLICATION_CREDENTIALS. With no credential at all, the exporter cannot start and the gateway fails to load.
      • Provider families default to the extended tier. They reach the bundled Thanos, and not a destination that filters at recommended or essential, unless metricImportance is raised.
      • Provider data is minutes old. Query it with last_over_time(...[15m]) or wider. Each gateway restart leaves a gap in Cloud Monitoring series as long as that lag.
    • Update Rust crate thiserror to v2.0.21
    • Update Rust crate jsonschema to 0.57.0
    • Fall back to CLUSTER_NAME for the gateway’s log cluster label
    • Reassemble and classify Rust panics, and parse tracing’s plain text format
      • materialize-monitoring#376
      • Rust panics from Materialize services now arrive as a single log entry rather than one entry per line of the backtrace, carrying level=CRITICAL and panic_thread / panic_location as structured metadata. The msg names the source location and the panic message.
      • balancerd and materialize-operator logs now carry a parsed level and target. Both previously landed as level="UNKNOWN" for every line.
      • React to this if you filter or size on log level. Those services’ WARN and ERROR lines are no longer swept into the UNKNOWN rate-limit bucket, which drops, so they now reach Loki reliably and ingested volume from the operator namespace rises. A saved query or alert matching level="UNKNOWN" on these services will stop matching.
    • Update Rust crate clap to v4.6.7
    • Update Rust crate rustls to v0.23.45
    • Update Rust crate jsonschema to 0.56.0
    • Update Rust crate reqwest to v0.13.5
    • Update Rust crate rustls to v0.23.44
    • Update Rust crate tokio-rustls to v0.26.5
    • Add some sample prose in heather’s voice
    • Add Dashboard Folders; Update tags
      • materialize-monitoring#329
      • Added several GrafanaFolder resources (default enabled: mzmon-infra, mzmon-materialize, mzmon-meta-o11y)
      • Changed monitoring tag to mzmon within dashboards
    • Update Rust crate indexmap to v2.14.2
    • Show total lag in env-top / env-upgrade
      • materialize-monitoring#312
      • Add new queries around max lag (materialize.compute.freshness.lag_total_by_cluster, materialize.generations.lag.total)
      • Show max lag as new panels (including per-cluster breakdown) in env-top and env-upgrade dashboards
    • DEP-242 Add infra-nodes dashboard
    • Update Rust crate indexmap to v2.14.1
    • DEP-209 Add Infrastructure Logs & Events Dashboard
    • Update Rust crate hyper to v1.11.1

mzmon-lib (shared library) v0.12.0#

  • Record the normalized ext:consensus_* series, with a producer for the registry’s rules: branch
    • materialize-monitoring#475
    • New recording rules record the metadata (consensus) database as normalized ext:consensus_* series, listed on the new Recorded Series reference page. They install wherever rules.enabled is true and their source is present; no selection is needed.
      • ext:consensus_up and ext:consensus_commit_latency_seconds:p99 come from Materialize’s own calls, carry flavor="persist" and namespace, and need no configuration.
      • ext:consensus_up, ext:consensus_storage_used_ratio and ext:consensus_xid_used_ratio from the CloudWatch, Cloud Monitoring and Azure Monitor pulls carry flavor (rds, cloudsql, azure-postgres) and resource, where the provider publishes each.
    • New value externalDependencies.consensus, a list of {flavor, resourceId} naming which databases a provider pull watches are a metadata database. The provider-sourced ext:consensus_* series record only these, so a pull that also watches Grafana’s database no longer needs to be told apart by hand. The render warns when a pull watches databases and none is named.
    • rules.capabilities gains three derived capabilities, cloudwatch, cloud-monitoring and azure-monitor, present when the matching pipeline.metrics.provider.* pull is enabled.
  • Google Cloud metrics: export over OTLP to the Telemetry API, typed
    • materialize-monitoring#474
    • Changed: the Google Cloud metrics destination writes somewhere else. googleCloudExporter (Terraform google_cloud_metrics) now sends OTLP to Google’s Telemetry API instead of using otelcol.exporter.googlecloud, and metrics land as prometheus.googleapis.com/<name>/<kind> instead of workload.googleapis.com/mzmon/<name>. Cloud Monitoring dashboards, alerting policies and anything else reading the old metric types stop receiving data and have to be repointed. The old types are not deleted.
      • Enable the telemetry.googleapis.com API on the project before upgrading with this destination on. Without it every export is refused, and nothing else fails. roles/monitoring.metricWriter is still the only role needed.
      • Metrics are billed per sample ingested instead of per byte, about $75 a month at recommended on a test install where the old export cost about $2,400.
      • Series carry the prometheus_target labels (project_id, location, cluster, namespace, job, instance) and collected_by="materialize-monitoring".
      • Counter values start from zero at the gateway’s first scrape, so they differ from Thanos; rate() and increase() agree.
      • New values googleCloudExporter.project and googleCloudExporter.location are needed only off GKE.
    • Changed: there is no metric prefix any more. googleCloudExporter.prefix is removed from the chart values, along with instrumentation_library_labels, skip_create_descriptor and service_resource_labels; setting any of them renders a warning and does nothing.
    • Deprecated: Terraform google_cloud_metrics.prefix is ignored and warns at plan time; remove it. There is no replacement, because the Telemetry API has no prefix to choose. Everything else in google_cloud_metrics is unchanged.
    • Changed: the gateway’s scrapes honor metric metadata, so every OTLP destination (Google Cloud, Datadog, generic OTLP) receives typed metrics: counters as cumulative sums, histograms as histograms. Metric names in Datadog change accordingly. Thanos and other remote-write destinations are unchanged.
    • alloy-gateway.alloy.stabilityLevel now defaults to experimental, which the scrapes require; the chart refuses any other level.
    • The agent and gateway now run the Alloy v1.20.0 image their image.tag names. Since the bump to v1.20.0-mz3, a stale image.digest had kept both on Alloy v1.19.2.
    • denyMetrics entries match per Prometheus series against typed histograms: denying foo_bucket still keeps foo_count and foo_sum.
  • Update Rust crate hyper-util to v0.1.21
  • Update Rust crate jsonschema to 0.58.0
  • Renovate: unblock pending updates, automerge crates, split lock files; DEP-324 pin subchart images
    • materialize-monitoring#446
    • Every subchart image the chart renders under its shipped profiles is now pinned in values.yaml (loki.loki.image, loki.lokiCanary.image, loki.memcached.image, loki.memcachedExporter.image, loki.sidecar.image, thanos.global.image, grafana-operator.image, kube-state-metrics.image, metrics-server.image, grafana.initChownData.image). Rendered images are unchanged. These are the subcharts’ own value paths, so existing overrides keep applying.
  • docs: flatten reference/stable-metrics into reference, and reorder by use
  • dashboards: say object, freshness, and orphaned instead of collection, lag, and leaked
    • materialize-monitoring#435
    • Dashboard panels now use Materialize’s product terms: object for collection, freshness for lag, and orphaned for leaked persist data. Several panel titles on env-top, env-upgrade, env-persist, and env-consensus changed accordingly; dashboard UIDs and queries did not.
  • dashboards: add Infrastructure Autoscaling and Karpenter
    • materialize-monitoring#425
    • Two new dashboards, installed by the default infra-* pattern: Infrastructure Autoscaling (infra-autoscaling), on every cloud, and Karpenter (infra-karpenter), which has data on EKS where Karpenter’s ServiceMonitor is applied.
    • kube-state-metrics now publishes kube_node_labels for each provisioner’s pool label, the instance type and the zone, set by the new default kube-state-metrics.metricLabelsAllowlist. An install that brings its own kube-state-metrics needs the same allowlist for the Node Pools tab.
  • Update Rust crate promql-parser to 0.11.0
  • Render LogQL alerts from the query registry and deliver them to the Loki ruler
    • materialize-monitoring#426
    • Bundled log-derived alerts, evaluated by the Loki ruler. materialize-panic, data-correctness-error and persist-filter-pushdown-violation are in the default set. trace-logging-enabled installs when selected.
    • Every bundled PrometheusRule now carries mzmon.materialize.cloud/flavor: promql or logql. The alloy-gateway writes logql ones into the Loki ruler through its API, including a deployment’s own. They install only where the ruler’s rule store accepts writes; a local store, which a filesystem-only Loki gets, leaves them out with a render warning.
    • thanos.ruler.autoImportPrometheusRules.labelSelector defaults to mzmon.materialize.cloud/flavor!: logql. A replacement selector has to keep that key, or select flavor: promql. A cluster running a Prometheus Operator admission webhook has to exclude flavor: logql from it.
    • New rules.logTenants: the Loki tenants the log-derived rules are written into. Empty means pipeline.logging.tenancy.staticTenant. List them under byEnvironment tenancy.
    • The generated rule index moved from pre-rendered/rules/prometheus/_index.yaml to pre-rendered/rules/_index.yaml, and records each rule’s engine.
  • chore(deps): update rust crate tokio-rustls to v0.26.6
  • DEP-233 Pull instance availability from each cloud’s monitoring API
    • materialize-monitoring#422
    • Pull what each cloud publishes about instance availability, off by default:
      • pipeline.metrics.provider.cloudwatch.eks.clusters pulls EC2 status checks for every node of the listed EKS clusters, their managed node groups’ sizes, and the region’s On-Demand vCPU usage. It needs cloudwatch:GetMetricData, cloudwatch:ListMetrics, tag:GetResources and autoscaling:DescribeAutoScalingGroups.
      • pipeline.metrics.provider.gcp.compute.regions pulls per-family CPU and local-SSD quota, usage against limit, and refusals. roles/monitoring.viewer already covers it.
      • pipeline.metrics.provider.azure.aks.clusters pulls the AKS cluster autoscaler’s gauges and each node VM’s availability. It needs Monitoring Reader on each cluster and its node resource group.
      • The new families (aws_ec2_*, aws_autoscaling_*, aws_usage_*, stackdriver_compute_googleapis_com_location_*, azure_microsoft_containerservice_managedclusters_* and azure_microsoft_compute_virtualmachinescalesets_*) take each provider’s metricImportance.
  • dashboards: add Persist, Consensus and Cloud Provider dashboards
    • materialize-monitoring#419
    • Three new dashboards, installed by default: Materialize Persist (Storage) (env-persist) and Materialize Consensus (Metadata) (env-consensus), Materialize’s own view of object storage and the metadata database, and Infrastructure Cloud Provider (infra-cloud), which draws the metrics collected by pipeline.metrics.provider.
    • Infrastructure Networking (infra-net) now shows its CNI and Security vendor rows. They were hidden on every cluster, which read as “No Dataplane Metrics”.
    • About 50 persist and timestamp-oracle metric families now ship to destinations at minMetricImportance: recommended, and the cloud provider families named by infra-cloud join the diagnostic tier. pipeline.metrics.provider.*.metricImportance still decides every tier above diagnostic.
  • DEP-301 Pull Azure Monitor metrics into the gateway
    • materialize-monitoring#417
    • New pipeline.metrics.provider.azure, off by default.
      • It pulls a fixed, minimal set of PostgreSQL Flexible Server and Blob Storage metrics from Azure Monitor, for the servers and storage accounts listed under it. Nothing is discovered.
      • Join on resourceName. instance is postgres, blob_capacity or blob_requests.
      • See Cloud Provider Metrics.
    • Credentials come from the gateway pod’s identity, never from values. The identity needs Monitoring Reader on each named resource.
      • Workload identity needs both the azure.workload.identity/client-id annotation on alloy-gateway.serviceAccount and the azure.workload.identity/use: "true" label in alloy-gateway.controller.podLabels.
      • The Terraform module sets the label whenever the annotation is present.
    • Azure families default to the extended tier, like the other providers.
  • Alert on freshness, hydration and replica health, split by who acts on it
    • materialize-monitoring#418
    • The default set gains ten alerting rules: environmentd-down, environmentd-not-scraped, system-cluster-falling-behind, system-cluster-stale, system-cluster-hydration-stuck, system-cluster-memory-near-limit, cluster-replica-not-ready, cluster-hydration-stuck, cluster-memory-near-limit and cluster-replica-oomkilled.
      • cluster-falling-behind, cluster-stale, cluster-memory-high, cluster-cpu-high and source-disconnected are new and install only when selected.
      • The user-cluster rules install as a third PrometheusRule, <release>-materialize-workload-alerts.
    • Every bundled alert carries an audience label, platform or workload, for routing with alerting.routes.extra. Alerts about a cluster also carry cluster_name.
    • rules.overrides.<alert> is new: it sets a rule’s for and adds or replaces its labels. A deployment whose clusters take longer than an hour to hydrate should lengthen cluster-hydration-stuck’s for.
  • Adopt the registry’s parameters in the alerts, fix what that exposed, and generate the rules
  • Add an alerting render context, capabilities and gen-rules to the query registry
  • Add an Alloy meta-monitoring dashboard, and fix infra-loki’s level picker
    • materialize-monitoring#397
    • New dashboard: Alloy Meta Monitoring (mz-mon-infra-alloy), in the Meta Observability folder. Collector health, both pipelines hop by hop, ingest over log push, remote write and OTLP, component and configuration state with uptime, gateway clustering, resource limits including GOMEMLIMIT, Kubernetes events including the pre-install validation Jobs, and Alloy’s own logs.
    • Fixed the Level picker on Loki Meta Monitoring, which offered only “All”.
    • Metric tiers now include the Alloy families the new dashboard reads. loki_source_file_read_bytes_total leaves the tiers, and the generic Go runtime families it adds are extended, so a metered destination on the default tier does not receive every target’s copy.
  • DEP-301 Pull CloudWatch and Cloud Monitoring metrics into the gateway
    • materialize-monitoring#396
    • New pipeline.metrics.provider.cloudwatch and pipeline.metrics.provider.gcp, both off by default.
      • They pull a fixed, minimal set of RDS and S3 metrics from CloudWatch, and of Cloud SQL and GCS metrics from Cloud Monitoring, for the resources listed under each. Nothing is discovered.
      • On CloudWatch series instance is the resource; join on dimension_DBInstanceIdentifier or dimension_BucketName.
      • See Cloud Provider Metrics.
    • Credentials come from the gateway pod’s identity, never from values.
      • CloudWatch needs cloudwatch:GetMetricStatistics, and iam:ListAccountAliases for the account_alias label, through IRSA, EKS Pod Identity, or static keys in the mzmon-alloy-gateway-env Secret.
      • GCP needs roles/monitoring.viewer through Workload Identity.
    • Outside Google Cloud, a GCP pull needs GOOGLE_APPLICATION_CREDENTIALS. With no credential at all, the exporter cannot start and the gateway fails to load.
    • Provider families default to the extended tier. They reach the bundled Thanos, and not a destination that filters at recommended or essential, unless metricImportance is raised.
    • Provider data is minutes old. Query it with last_over_time(...[15m]) or wider. Each gateway restart leaves a gap in Cloud Monitoring series as long as that lag.
  • Update Rust crate thiserror to v2.0.21
  • Update Rust crate jsonschema to 0.57.0
  • Fall back to CLUSTER_NAME for the gateway’s log cluster label
  • Add a Loki meta-monitoring dashboard, and split dashboards into their own chart
    • materialize-monitoring#383
    • Dashboards now install from a separate chart. materialize-monitoring-dashboards is a release of its own, installed beside materialize-monitoring in the same namespace. The umbrella chart no longer creates dashboards; a release that upgrades without installing the new chart will have its dashboards removed. Helm stores a release in a Kubernetes Secret and a Secret may not exceed 1 MiB, which the rendered set outgrew.
      • Terraform installs it automatically — set enable_dashboards = false to opt out.
      • dashboards.selected → the new chart’s selected.
      • dashboards.config.grafana.manifest.apiTarget → the new chart’s grafana.apiTarget.
      • dashboards.config.datadog is removed; it drove nothing.
      • The new chart cannot read the umbrella release’s values, so grafana.instanceSelector and grafana.folderUids must match it. The umbrella chart’s install notes print the folder UIDs.
      • Dashboard UIDs are unchanged, so saved links, playlists and alerts keep working.
    • New dashboard: Loki Meta Monitoring (mz-mon-infra-loki), in the Meta Observability folder. Ingest, queries, object storage, retention, and Loki’s own logs.
    • Fixed: the Loki canary and both memcached exporters were never scraped under profiles/mtls. The subchart’s single ServiceMonitor applied one scheme to every target, including three that only serve plaintext. They are now collected by a separate monitor, and carry prometheus.io/service-monitor: "false" plus monitoring.materialize.cloud/scrape-scheme: plaintext on their Services. Installs using mTLS gain loki_canary_* and memcached_* series that were previously absent.
    • New Terraform inputs: enable_dashboards, dashboards_chart_version, dashboards_selected, dashboards_instance_selector, dashboards_allow_cross_namespace_import.
  • Reassemble and classify Rust panics, and parse tracing’s plain text format
    • materialize-monitoring#376
    • Rust panics from Materialize services now arrive as a single log entry rather than one entry per line of the backtrace, carrying level=CRITICAL and panic_thread / panic_location as structured metadata. The msg names the source location and the panic message.
    • balancerd and materialize-operator logs now carry a parsed level and target. Both previously landed as level="UNKNOWN" for every line.
    • React to this if you filter or size on log level. Those services’ WARN and ERROR lines are no longer swept into the UNKNOWN rate-limit bucket, which drops, so they now reach Loki reliably and ingested volume from the operator namespace rises. A saved query or alert matching level="UNKNOWN" on these services will stop matching.
  • DEP-211 Add infra-net dashboard, and the CNI collection it needs
  • Update Rust crate clap to v4.6.7
  • Update Rust crate rustls to v0.23.45
  • Update Rust crate jsonschema to 0.56.0
  • Update Rust crate reqwest to v0.13.5
  • Update Rust crate rustls to v0.23.44
  • Update Rust crate tokio-rustls to v0.26.5
  • Add some sample prose in heather’s voice
  • Add Dashboard Folders; Update tags
    • materialize-monitoring#329
    • Added several GrafanaFolder resources (default enabled: mzmon-infra, mzmon-materialize, mzmon-meta-o11y)
    • Changed monitoring tag to mzmon within dashboards
  • Update Rust crate indexmap to v2.14.2
  • Show total lag in env-top / env-upgrade
    • materialize-monitoring#312
    • Add new queries around max lag (materialize.compute.freshness.lag_total_by_cluster, materialize.generations.lag.total)
    • Show max lag as new panels (including per-cluster breakdown) in env-top and env-upgrade dashboards
  • DEP-242 Add infra-nodes dashboard
  • Update Rust crate indexmap to v2.14.1
  • DEP-209 Add Infrastructure Logs & Events Dashboard
  • Update Rust crate hyper to v1.11.1
  • DEP-240 Ensure kube-state-metrics metrics do not have namespace overwritten
    • materialize-monitoring#297
    • regression: kube-state-metrics were namespaced as export_namespace=$target with namespace=monitoring on all because the scraper overwrote that label with the namespace KSM was running in
  • DEP-209 Create a dashboard for Materialize logs and events
  • DEP-210 Upgrade visibility dashboard
    • materialize-monitoring#294
    • Adds a new mz-env-upgrade dashboard, enabled by default
      • Requires materialize v26.41.0
    • Enable sql scrapers by default

Dashboards v0.13.0#

  • DEP-240 Ensure kube-state-metrics metrics do not have namespace overwritten
    • materialize-monitoring#297
    • regression: kube-state-metrics were namespaced as export_namespace=$target with namespace=monitoring on all because the scraper overwrote that label with the namespace KSM was running in
  • DEP-209 Create a dashboard for Materialize logs and events
  • DEP-210 Upgrade visibility dashboard
    • materialize-monitoring#294
    • Adds a new mz-env-upgrade dashboard, enabled by default
      • Requires materialize v26.41.0
    • Enable sql scrapers by default

Dependencies#

mzmon-lib (shared library) v0.11.0#

materialize-monitoring (Helm chart + Terraform module) v0.20.0#

  • Update Helm release alloy to v1.12.1
  • DEP-240 Ensure kube-state-metrics metrics do not have namespace overwritten
    • materialize-monitoring#297
    • regression: kube-state-metrics were namespaced as export_namespace=$target with namespace=monitoring on all because the scraper overwrote that label with the namespace KSM was running in
  • DEP-210 Upgrade visibility dashboard
    • materialize-monitoring#294
    • Adds a new mz-env-upgrade dashboard, enabled by default
      • Requires materialize v26.41.0
    • Enable sql scrapers by default

Dependencies#

  • Included Dashboards @ v0.13.0..v0.14.0
    • DEP-209 Create a dashboard for Materialize logs and events
    • DEP-222 Port Materialize Environment Overview to rust sdk
      • materialize-monitoring#285
      • Dashboards have been rewritten under a different dashboard framework
        • Dashboard queries have adopted queries from our query registry
    • DEP-222 Rust implementation of Grafana Dashboard framework
  • Included Pipelines @ v0.12.0..v0.13.0
    • DEP-241 Remove deprecated alias k8s_* labels from logging pipeline; drop pod to metadata only
      • materialize-monitoring#300
      • Removed: the k8s_namespace, k8s_app, and k8s_container Loki stream labels, deprecated to consumers in April, 2026. Use namespace, app and container respectively
      • Removed: the k8s_pod label has been fully removed with expectation to use the pod structured metadata. Prefer the namespace and app labels for label filters. Remember that you can always filter by structured metadata like {app="my-app"} | pod="my-app-pod-12345"
  • Included Prometheus Scrapers @ v0.4.0..v0.5.0
  • Included mzmon-lib (shared library) @ v0.11.0..v0.12.0

materialize-monitoring (Helm chart + Terraform module) v0.19.0#

  • DEP-232 Support multiple prometheus RemoteWrite destinations
    • materialize-monitoring#272
    • terraform: add new variables for tuning prometheus remote_write destinations (as a map)
    • breaking helm: move prometheusRemoteEntries destinations down one to a keyed map pipeline.metrics.gateway.destination.prometheusRemoteWrite -> pipeline.metrics.gateway.destination.prometheusRemoteWrite.thanos
  • Ensure auto-format on version-update PRs does not update Chart.lock
  • Update Helm release metrics-server to ^3.14.0

Dependencies#

  • Included Dashboards @ v0.11.0..v0.12.0
  • Included Pipelines @ v0.11.0..v0.12.0
  • Included Prometheus Scrapers @ v0.3.0..v0.4.0
  • Included mzmon-lib (shared library) @ v0.9.0..v0.10.0

materialize-monitoring (Helm chart + Terraform module) v0.18.1#

Pipelines v0.12.0#

  • DEP-241 Remove deprecated alias k8s_* labels from logging pipeline; drop pod to metadata only
    • materialize-monitoring#300
    • Removed: the k8s_namespace, k8s_app, and k8s_container Loki stream labels, deprecated to consumers in April, 2026. Use namespace, app and container respectively
    • Removed: the k8s_pod label has been fully removed with expectation to use the pod structured metadata. Prefer the namespace and app labels for label filters. Remember that you can always filter by structured metadata like {app="my-app"} | pod="my-app-pod-12345"
  • DEP-210 Upgrade visibility dashboard
    • materialize-monitoring#294
    • Adds a new mz-env-upgrade dashboard, enabled by default
      • Requires materialize v26.41.0
    • Enable sql scrapers by default
  • DEP-232 Support multiple prometheus RemoteWrite destinations
    • materialize-monitoring#272
    • terraform: add new variables for tuning prometheus remote_write destinations (as a map)
    • breaking helm: move prometheusRemoteEntries destinations down one to a keyed map pipeline.metrics.gateway.destination.prometheusRemoteWrite -> pipeline.metrics.gateway.destination.prometheusRemoteWrite.thanos
  • DEP-127 Policy for deprecations and breaking changes

Dependencies#

materialize-monitoring (Helm chart + Terraform module) v0.18.0#

Dependencies#

  • Included Dashboards @ v0.11.0..v0.12.0
  • Included Pipelines @ v0.11.0..v0.12.0
  • Included Prometheus Scrapers @ v0.3.0..v0.4.0
  • Included mzmon-lib (shared library) @ v0.9.0..v0.10.0

materialize-monitoring (Helm chart + Terraform module) v0.17.0#

Container Images v0.4.0#

materialize-monitoring (Helm chart + Terraform module) v0.16.2#

Dependencies#

  • Included Dashboards @ v0.11.0..v0.12.0
  • Included Pipelines @ v0.10.0..v0.11.0
  • Included Prometheus Scrapers @ v0.3.0..v0.4.0
  • Included mzmon-lib (shared library) @ v0.9.0..v0.10.0

materialize-monitoring (Helm chart + Terraform module) v0.16.1#

Pipelines v0.11.0#

Dependencies#

Container Images v0.3.0#

materialize-monitoring (Helm chart + Terraform module) v0.16.0#

Dependencies#

  • Included Dashboards @ v0.11.0..v0.12.0
  • Included Pipelines @ v0.10.0..v0.11.0
  • Included Prometheus Scrapers @ v0.3.0..v0.4.0
  • Included mzmon-lib (shared library) @ v0.9.0..v0.10.0

Pipelines v0.10.0#

Dependencies#

materialize-monitoring (Helm chart + Terraform module) v0.15.0#

Dependencies#

  • Included Dashboards @ v0.11.0..v0.12.0
  • Included Pipelines @ v0.9.0..v0.10.0
  • Included Prometheus Scrapers @ v0.3.0..v0.4.0
  • Included mzmon-lib (shared library) @ v0.9.0..v0.10.0

mzmon-lib (shared library) v0.10.0#

materialize-monitoring (Helm chart + Terraform module) v0.14.0#

Dependencies#

  • Included Dashboards @ v0.11.0..v0.12.0
  • Included Pipelines @ v0.8.0..v0.9.0
  • Included Prometheus Scrapers @ v0.3.0..v0.4.0
  • Included mzmon-lib (shared library) @ v0.9.0..v0.10.0

materialize-monitoring (Helm chart + Terraform module) v0.13.1#

materialize-monitoring (Helm chart + Terraform module) v0.13.0#

materialize-monitoring (Helm chart + Terraform module) v0.12.0#

materialize-monitoring (Helm chart + Terraform module) v0.11.1#

materialize-monitoring (Helm chart + Terraform module) v0.11.0#

materialize-monitoring (Helm chart + Terraform module) v0.10.0#

materialize-monitoring (Helm chart + Terraform module) v0.9.0#

Dependencies#

  • Included Dashboards @ v0.11.0..v0.12.0
  • Included Pipelines @ v0.8.0..v0.9.0
  • Included Prometheus Scrapers @ v0.3.0..v0.4.0
  • Included mzmon-lib (shared library) @ v0.8.0..v0.9.0

materialize-monitoring Optional CRDs v0.4.0 (Unreleased)#

Changes Pending

materialize-monitoring Helm Chart v0.8.0#

Pipelines v0.9.0#

Dependencies#

Pipelines v0.8.0#

Dependencies#

Prometheus Scrapers v0.4.0#

Dependencies#

materialize-monitoring Helm Chart v0.7.0#

Dependencies#

materialize-monitoring Helm Chart v0.6.0#

materialize-monitoring Helm Chart v0.5.0#

Pipelines v0.7.0#

Dependencies#

  • Included mzmon-lib (shared library) @ v0.8.0..v0.9.0

mzmon-lib (shared library) v0.9.0#

Pipelines v0.6.0#

Dependencies#

Pipelines v0.5.0#

Dependencies#

  • Included mzmon-lib (shared library) @ v0.7.0..v0.8.0

Prometheus Scrapers v0.3.0#

Dependencies#

Container Images v0.2.0#

mzmon-lib (shared library) v0.8.0#

materialize-monitoring Helm Chart v0.4.0#

Dependencies#

Dashboards v0.12.0#

Dependencies#

Dashboards v0.11.0#

Dependencies#

  • Included mzmon-lib (shared library) @ v0.6.0..v0.7.0

Prometheus Scrapers v0.2.0#

Dependencies#

materialize-monitoring Optional CRDs v0.3.0#

Container Images v0.1.1#

Container Images v0.1.0#

  • Bootstrapped

Dashboards v0.10.0#

Dependencies#

Prometheus Scrapers v0.1.1#

Dependencies#

Prometheus Scrapers v0.1.0#

  • Bootstrapped

Dashboards v0.9.0#

Dependencies#

mzmon-lib (shared library) v0.7.0#

Pipelines v0.4.0#

Dependencies#

materialize-monitoring Helm Chart v0.3.0#

Dependencies#

materialize-monitoring Optional CRDs v0.2.0#

Dashboards v0.8.0#

Dependencies#

Pipelines v0.3.0#

Dependencies#

mzmon-lib (shared library) v0.6.0#

materialize-monitoring Helm Chart v0.2.0#

Dependencies#

Dashboards v0.7.0#

Dependencies#

Pipelines v0.2.0#

Dependencies#

  • Included mzmon-lib (shared library) @ v0.4.0..v0.5.0

mzmon-lib (shared library) v0.5.0#

Dashboards v0.6.0#

Pipelines v0.1.0#

mzmon-lib (shared library) v0.4.0#

materialize-monitoring Helm Chart v0.1.0#

materialize-monitoring Optional CRDs v0.1.0#

mzmon-lib (shared library) v0.3.0#

Dashboards v0.5.0#

Backward Alloy Contributing Forward