Recorded Series#

These are the series the chart’s recording rules write, generated from the query registry. A recorded series is a metric name this repository mints, so it reads the same whatever produced the measurement underneath it. The Thanos ruler evaluates the rules and writes the results back through the alloy-gateway, so a recorded series is queryable wherever the gateway’s other metrics are.

Every ext:* series is part of the normalized layer for an external dependency. One name is recorded by each adapter that can measure it, and the flavor label names the adapter.

LabelOnHolds
flavorEvery ext:* seriesThe adapter that recorded it: persist is Materialize’s own measurement, rds, cloudsql and azure-postgres are a cloud provider’s, and postgres is the postgres-exporter’s
namespaceSeries from persistThe Materialize environment’s namespace
resourceSeries from a cloud provider or the exporterThe database’s name at the provider, or the exporter target’s name, as declared in externalDependencies.consensus
databaseext:consensus_database_bytesThe name of one database on the instance
versionext:consensus_version_infoThe version the database server reports

An adapter that cannot measure a series records nothing for it, rather than a zero. The provider and exporter adapters record only the databases externalDependencies.consensus declares; see Configuring Alerting.

Recorded-series names are covered by the stability policy from the release that first ships them, like alert names.

ext:consensus_up #

flavor="persist" from ext-consensus, group ext_consensus_persist

1 while Materialize’s calls to the metadata database succeed, per environment namespace, and 0 when none has succeeded in five minutes. An environment makes about 80 calls a second while idle, so five minutes without a success is a database it cannot reach rather than one it had no reason to call. A call that hangs counts as not succeeding. Absent when the environment is not being scraped.

Recorded where: materialize.

sum by (namespace) (
  rate(mz_persist_external_succeeded_count{op=~"consensus_.*"}[5m])
) > bool 0

flavor="cloudsql" from ext-consensus, group ext_consensus_cloudsql

1 while Cloud SQL reports each metadata database instance up, and 0 when it reports it down.

Recorded where: cloud-monitoring.

max by (resource) (
  label_replace(
    last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_up{database_id=~"(?i).+:(${consensusCloudsqlResources})"}[15m]),
    "resource", "$1", "database_id", ".+:(.+)"
  )
)

flavor="azure-postgres" from ext-consensus, group ext_consensus_azure_postgres

1 while Azure reports each metadata database flexible server alive, and 0 when it reports it down.

Recorded where: azure-monitor.

max by (resource) (
  label_replace(
    last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_is_db_alive_minimum_count{resourceName=~"(?i)${consensusAzurePostgresResources}"}[15m]),
    "resource", "$1", "resourceName", "(.+)"
  )
)

flavor="postgres" from ext-consensus, group ext_consensus_postgres

1 while the postgres-exporter can connect to each metadata database, and 0 when it cannot. From pg_up. Absent while the exporter itself cannot be scraped, which is a fault in the exporter rather than in the database.

Recorded where: postgres-exporter.

max by (resource) (
  label_replace(
    pg_up{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"},
    "resource", "$1", "target", "(.+)"
  )
)

ext:consensus_commit_latency_seconds:p99 #

flavor="persist" from ext-consensus, group ext_consensus_persist

The 99th-percentile time a commit to the metadata database takes, over five minutes, per environment namespace. A commit is persist’s compare-and-set, op="consensus_cas", the only consensus operation with a latency histogram. Measured by the client, so it includes the network and any wait for a pooled connection. Absent when there was no commit in the window.

Recorded where: materialize.

histogram_quantile(0.99,
  sum by (namespace, le) (
    rate(mz_persist_external_op_latency_bucket{op="consensus_cas"}[5m])
  )
) >= 0

ext:consensus_xid_used_ratio #

flavor="rds" from ext-consensus, group ext_consensus_rds

The fraction of PostgreSQL’s transaction-ID space each RDS metadata database has used, against the 2^31 at which it stops accepting writes. From CloudWatch’s MaximumUsedTransactionIDs.

Recorded where: cloudwatch.

max by (resource) (
  label_replace(
    last_over_time(aws_rds_maximum_used_transaction_ids_maximum{dimension_DBInstanceIdentifier=~"(?i)${consensusRdsResources}"}[15m]),
    "resource", "$1", "dimension_DBInstanceIdentifier", "(.+)"
  )
) / 2147483648

flavor="cloudsql" from ext-consensus, group ext_consensus_cloudsql

The fraction of PostgreSQL’s transaction-ID space each Cloud SQL metadata database has used. Cloud SQL publishes this as a ratio of the same 2^31 limit the other adapters divide by.

Recorded where: cloud-monitoring.

max by (resource) (
  label_replace(
    last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_postgresql_transaction_id_utilization{database_id=~"(?i).+:(${consensusCloudsqlResources})"}[15m]),
    "resource", "$1", "database_id", ".+:(.+)"
  )
)

flavor="azure-postgres" from ext-consensus, group ext_consensus_azure_postgres

The fraction of PostgreSQL’s transaction-ID space each metadata database flexible server has used, against the 2^31 at which it stops accepting writes.

Recorded where: azure-monitor.

max by (resource) (
  label_replace(
    last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_maximum_used_transactionids_maximum_count{resourceName=~"(?i)${consensusAzurePostgresResources}"}[15m]),
    "resource", "$1", "resourceName", "(.+)"
  )
) / 2147483648

flavor="postgres" from ext-consensus, group ext_consensus_postgres

The fraction of PostgreSQL’s transaction-ID space each metadata database’s instance has used, against the 2^31 at which it stops accepting writes. The oldest age(datfrozenxid) of any database on the instance, since wraparound stops the whole instance whichever database reaches it.

Recorded where: postgres-exporter.

max by (resource) (
  label_replace(
    pg_database_wraparound_age_datfrozenxid_seconds{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"},
    "resource", "$1", "target", "(.+)"
  )
) / 2147483648

ext:consensus_storage_used_ratio #

flavor="cloudsql" from ext-consensus, group ext_consensus_cloudsql

The fraction of each Cloud SQL metadata database’s provisioned disk in use.

Recorded where: cloud-monitoring.

max by (resource) (
  label_replace(
    last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_disk_utilization{database_id=~"(?i).+:(${consensusCloudsqlResources})"}[15m]),
    "resource", "$1", "database_id", ".+:(.+)"
  )
)

flavor="azure-postgres" from ext-consensus, group ext_consensus_azure_postgres

The fraction of each metadata database flexible server’s provisioned storage in use.

Recorded where: azure-monitor.

max by (resource) (
  label_replace(
    last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_storage_percent_maximum_percent{resourceName=~"(?i)${consensusAzurePostgresResources}"}[15m]),
    "resource", "$1", "resourceName", "(.+)"
  )
) / 100

ext:consensus_connections_used_ratio #

flavor="postgres" from ext-consensus, group ext_consensus_postgres

Client connections open on each metadata database’s instance, as a fraction of the connections an ordinary role may open. The ceiling is max_connections less superuser_reserved_connections and, from PostgreSQL 16, reserved_connections, which is where an ordinary role is refused. Connections to every database on the instance count, not only Materialize’s, because they share the ceiling. While a rollout runs two exporter pods, each probes the database, and their copies are counted once.

Recorded where: postgres-exporter.

max by (resource) (
  label_replace(
    sum by (target) (max without (pod) (pg_stat_activity_count{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}", backend_type="client backend"}))
    / on (target) (
        max by (target) (pg_settings_max_connections{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"})
      - on (target) max by (target) (pg_settings_superuser_reserved_connections{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"})
      - on (target) (
          max by (target) (pg_settings_reserved_connections{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"})
          or 0 * max by (target) (pg_settings_max_connections{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"})
        )
    ),
    "resource", "$1", "target", "(.+)"
  )
)

ext:consensus_database_bytes #

flavor="postgres" from ext-consensus, group ext_consensus_postgres

The size of each database on each metadata database’s instance, with the database’s name as database. Every database on the instance, not only Materialize’s: a neighbour filling the volume is what this series exists to attribute, and the instance’s total cannot.

Recorded where: postgres-exporter.

max by (resource, database) (
  label_replace(
    label_replace(
      pg_database_size_bytes{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"},
      "resource", "$1", "target", "(.+)"
    ),
    "database", "$1", "datname", "(.+)"
  )
)

ext:consensus_version_info #

flavor="postgres" from ext-consensus, group ext_consensus_postgres

1, with the PostgreSQL version each metadata database runs as version. The server’s own version number, such as 16.4.0. Join it onto another ext:consensus_* series with group_left (version) on flavor and resource.

Recorded where: postgres-exporter.

max by (resource, version) (
  label_replace(
    label_replace(
      pg_static{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"},
      "resource", "$1", "target", "(.+)"
    ),
    "version", "$1", "short_version", "(.+)"
  )
)

mzmon:heartbeat #

chart_version="__mzmon_chart_version__" from mzmon-heartbeat, group mzmon_heartbeat

1 at every evaluation, labelled with the version of the materialize-monitoring chart that installed the rule. Reads no metric, so it stops only when the ruler, its query path, or its writes through the gateway stop. Each ruler replica writes its own, under its own ruler_replica.

Recorded where: thanos.

vector(1)

mzmon:cluster_created_timestamp_seconds #

from mzmon-heartbeat, group mzmon_heartbeat_kube_state_metrics

When the cluster’s kube-system namespace was created, in seconds since the epoch. Stands in for when the cluster was created. A cluster rebuilt from the same values reports a new value from then on, and a cloned one reports its own beside the original’s.

Recorded where: kube-state-metrics, thanos.

min(kube_namespace_created{namespace="kube-system"})

mzmon:environment_info #

from mzmon-heartbeat, group mzmon_heartbeat_kube_state_metrics

1 for each Materialize environment in the cluster, by environment_id. Read from the Materialize resources. Carries the environment’s UUID and not its name, which is the resource’s name.

Recorded where: kube-state-metrics, thanos.

group by (environment_id) (kube_mz_environment_info)

mzmon:build_info:count #

component="kubelet" from mzmon-heartbeat, group mzmon_heartbeat_kube_state_metrics

Nodes running each kubelet version. version is the kubelet’s, as the node reports it, which is the cluster’s Kubernetes version once an upgrade finishes. Each node counts once however many kube-state-metrics replicas report it.

Recorded where: kube-state-metrics, thanos.

count by (version) (
  label_replace(
    group by (node, kubelet_version) (kube_node_info),
    "version", "$1", "kubelet_version", "(.+)"
  )
)

component="environmentd" from mzmon-heartbeat, group mzmon_heartbeat_materialize

Running environmentd processes of each Materialize version.

Recorded where: materialize, thanos.

count by (version) (mz_start_time_environmentd)

component="alloy" from mzmon-heartbeat, group mzmon_heartbeat_alloy

Running Alloy collectors, agents and gateways together, of each version.

Recorded where: alloy, thanos.

count by (version) (alloy_build_info)

component="loki" from mzmon-heartbeat, group mzmon_heartbeat_loki

Running Loki processes of each version, every component together.

Recorded where: loki, thanos.

count by (version) (loki_build_info)

component="node-exporter" from mzmon-heartbeat, group mzmon_heartbeat_node_exporter

Running node-exporter processes of each version.

Recorded where: node-exporter, thanos.

count by (version) (node_exporter_build_info)

component="postgres" from mzmon-heartbeat, group mzmon_heartbeat_postgres

Metadata databases running each PostgreSQL version, read through the postgres-exporter. Covers each externalDependencies.consensus entry with flavor: postgres, whichever service runs the database: Amazon RDS, Cloud SQL, Azure Database for PostgreSQL, or a cluster of your own. Absent where none is declared, or the exporter doesn’t run.

Recorded where: thanos, postgres-exporter.

count by (version) (
  label_replace(
    group by (target, short_version) (
      pg_static{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"}
    ),
    "version", "$1", "short_version", "(.+)"
  )
)

component="thanos" from mzmon-heartbeat, group mzmon_heartbeat_thanos

Running Thanos processes of each version, every component together.

Recorded where: thanos.

count by (version) (thanos_build_info)

mzmon:node_uname_info:count #

from mzmon-heartbeat, group mzmon_heartbeat_node_exporter

Nodes with each CPU architecture and kernel release. machine and release are uname -m and uname -r on the node, such as aarch64 and 6.12.40-64.114.amzn2023.aarch64. The node’s host name stays behind.

Recorded where: node-exporter, thanos.

count by (machine, release) (
  group by (nodename, machine, release) (node_uname_info)
)

mzmon:rules:count #

from mzmon-heartbeat, group mzmon_heartbeat_thanos

The rules the Thanos ruler evaluates, alerting and recording, on the replica that loaded the most. Includes these heartbeat rules. A count well below the install’s usual one is a rule file the ruler refused to load.

Recorded where: thanos.

max(
  sum by (pod) (prometheus_rule_group_rules{app="thanos", container="ruler"})
)

mzmon:rule_evaluation_failures:rate5m #

from mzmon-heartbeat, group mzmon_heartbeat_thanos

Failed rule evaluations per second over five minutes, on the Thanos ruler replica with the most. 0 while every rule evaluates. A failing rule records nothing and never fires, so its alert goes missing from ALERTS without a sign anywhere else.

Recorded where: thanos.

max(
  sum by (pod) (
    rate(prometheus_rule_evaluation_failures_total{app="thanos", container="ruler"}[5m])
  )
)