Recorded Series#
These are the series the chart’s recording rules write, generated from the query registry. A recorded series is a metric name this repository mints, so it reads the same whatever produced the measurement underneath it. The Thanos ruler evaluates the rules and writes the results back through the alloy-gateway, so a recorded series is queryable wherever the gateway’s other metrics are.
Every ext:* series is part of the normalized layer for an external dependency.
One name is recorded by each adapter that can measure it, and the flavor label names the adapter.
| Label | On | Holds |
|---|---|---|
flavor | Every ext:* series | The adapter that recorded it: persist is Materialize’s own measurement, rds, cloudsql and azure-postgres are a cloud provider’s, and postgres is the postgres-exporter’s |
namespace | Series from persist | The Materialize environment’s namespace |
resource | Series from a cloud provider or the exporter | The database’s name at the provider, or the exporter target’s name, as declared in externalDependencies.consensus |
database | ext:consensus_database_bytes | The name of one database on the instance |
version | ext:consensus_version_info | The version the database server reports |
An adapter that cannot measure a series records nothing for it, rather than a zero.
The provider and exporter adapters record only the databases externalDependencies.consensus declares; see Configuring Alerting.
Recorded-series names are covered by the stability policy from the release that first ships them, like alert names.
ext:consensus_up
#
flavor="persist" from ext-consensus, group ext_consensus_persist
1 while Materialize’s calls to the metadata database succeed, per
environment namespace, and 0 when none has succeeded in five minutes.
An environment makes about 80 calls a second while idle, so five
minutes without a success is a database it cannot reach rather than one
it had no reason to call. A call that hangs counts as not succeeding.
Absent when the environment is not being scraped.Recorded where: materialize.
sum by (namespace) (
rate(mz_persist_external_succeeded_count{op=~"consensus_.*"}[5m])
) > bool 0
flavor="cloudsql" from ext-consensus, group ext_consensus_cloudsql
1 while Cloud SQL reports each metadata database instance up, and 0
when it reports it down.Recorded where: cloud-monitoring.
max by (resource) (
label_replace(
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_up{database_id=~"(?i).+:(${consensusCloudsqlResources})"}[15m]),
"resource", "$1", "database_id", ".+:(.+)"
)
)
flavor="azure-postgres" from ext-consensus, group ext_consensus_azure_postgres
1 while Azure reports each metadata database flexible server alive, and
0 when it reports it down.Recorded where: azure-monitor.
max by (resource) (
label_replace(
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_is_db_alive_minimum_count{resourceName=~"(?i)${consensusAzurePostgresResources}"}[15m]),
"resource", "$1", "resourceName", "(.+)"
)
)
flavor="postgres" from ext-consensus, group ext_consensus_postgres
1 while the postgres-exporter can connect to each metadata database,
and 0 when it cannot.
From pg_up. Absent while the exporter itself cannot be scraped,
which is a fault in the exporter rather than in the database.Recorded where: postgres-exporter.
max by (resource) (
label_replace(
pg_up{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"},
"resource", "$1", "target", "(.+)"
)
)
ext:consensus_commit_latency_seconds:p99
#
flavor="persist" from ext-consensus, group ext_consensus_persist
The 99th-percentile time a commit to the metadata database takes, over
five minutes, per environment namespace.
A commit is persist’s compare-and-set, op="consensus_cas", the only
consensus operation with a latency histogram. Measured by the client,
so it includes the network and any wait for a pooled connection.
Absent when there was no commit in the window.Recorded where: materialize.
histogram_quantile(0.99,
sum by (namespace, le) (
rate(mz_persist_external_op_latency_bucket{op="consensus_cas"}[5m])
)
) >= 0
ext:consensus_xid_used_ratio
#
flavor="rds" from ext-consensus, group ext_consensus_rds
The fraction of PostgreSQL’s transaction-ID space each RDS metadata
database has used, against the 2^31 at which it stops accepting writes.
From CloudWatch’s MaximumUsedTransactionIDs.Recorded where: cloudwatch.
max by (resource) (
label_replace(
last_over_time(aws_rds_maximum_used_transaction_ids_maximum{dimension_DBInstanceIdentifier=~"(?i)${consensusRdsResources}"}[15m]),
"resource", "$1", "dimension_DBInstanceIdentifier", "(.+)"
)
) / 2147483648
flavor="cloudsql" from ext-consensus, group ext_consensus_cloudsql
The fraction of PostgreSQL’s transaction-ID space each Cloud SQL
metadata database has used.
Cloud SQL publishes this as a ratio of the same 2^31 limit the other
adapters divide by.Recorded where: cloud-monitoring.
max by (resource) (
label_replace(
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_postgresql_transaction_id_utilization{database_id=~"(?i).+:(${consensusCloudsqlResources})"}[15m]),
"resource", "$1", "database_id", ".+:(.+)"
)
)
flavor="azure-postgres" from ext-consensus, group ext_consensus_azure_postgres
The fraction of PostgreSQL’s transaction-ID space each metadata database
flexible server has used, against the 2^31 at which it stops accepting
writes.Recorded where: azure-monitor.
max by (resource) (
label_replace(
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_maximum_used_transactionids_maximum_count{resourceName=~"(?i)${consensusAzurePostgresResources}"}[15m]),
"resource", "$1", "resourceName", "(.+)"
)
) / 2147483648
flavor="postgres" from ext-consensus, group ext_consensus_postgres
The fraction of PostgreSQL’s transaction-ID space each metadata
database’s instance has used, against the 2^31 at which it stops
accepting writes.
The oldest age(datfrozenxid) of any database on the instance, since
wraparound stops the whole instance whichever database reaches it.Recorded where: postgres-exporter.
max by (resource) (
label_replace(
pg_database_wraparound_age_datfrozenxid_seconds{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"},
"resource", "$1", "target", "(.+)"
)
) / 2147483648
ext:consensus_storage_used_ratio
#
flavor="cloudsql" from ext-consensus, group ext_consensus_cloudsql
The fraction of each Cloud SQL metadata database’s provisioned disk in
use.Recorded where: cloud-monitoring.
max by (resource) (
label_replace(
last_over_time(stackdriver_cloudsql_database_cloudsql_googleapis_com_database_disk_utilization{database_id=~"(?i).+:(${consensusCloudsqlResources})"}[15m]),
"resource", "$1", "database_id", ".+:(.+)"
)
)
flavor="azure-postgres" from ext-consensus, group ext_consensus_azure_postgres
The fraction of each metadata database flexible server’s provisioned
storage in use.Recorded where: azure-monitor.
max by (resource) (
label_replace(
last_over_time(azure_microsoft_dbforpostgresql_flexibleservers_storage_percent_maximum_percent{resourceName=~"(?i)${consensusAzurePostgresResources}"}[15m]),
"resource", "$1", "resourceName", "(.+)"
)
) / 100
ext:consensus_connections_used_ratio
#
flavor="postgres" from ext-consensus, group ext_consensus_postgres
Client connections open on each metadata database’s instance, as a
fraction of the connections an ordinary role may open.
The ceiling is max_connections less superuser_reserved_connections
and, from PostgreSQL 16, reserved_connections, which is where an
ordinary role is refused. Connections to every database on the
instance count, not only Materialize’s, because they share the
ceiling.
While a rollout runs two exporter pods, each probes the database, and
their copies are counted once.Recorded where: postgres-exporter.
max by (resource) (
label_replace(
sum by (target) (max without (pod) (pg_stat_activity_count{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}", backend_type="client backend"}))
/ on (target) (
max by (target) (pg_settings_max_connections{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"})
- on (target) max by (target) (pg_settings_superuser_reserved_connections{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"})
- on (target) (
max by (target) (pg_settings_reserved_connections{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"})
or 0 * max by (target) (pg_settings_max_connections{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"})
)
),
"resource", "$1", "target", "(.+)"
)
)
ext:consensus_database_bytes
#
flavor="postgres" from ext-consensus, group ext_consensus_postgres
The size of each database on each metadata database’s instance, with
the database’s name as database.
Every database on the instance, not only Materialize’s: a neighbour
filling the volume is what this series exists to attribute, and the
instance’s total cannot.Recorded where: postgres-exporter.
max by (resource, database) (
label_replace(
label_replace(
pg_database_size_bytes{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"},
"resource", "$1", "target", "(.+)"
),
"database", "$1", "datname", "(.+)"
)
)
ext:consensus_version_info
#
flavor="postgres" from ext-consensus, group ext_consensus_postgres
1, with the PostgreSQL version each metadata database runs as
version.
The server’s own version number, such as 16.4.0. Join it onto
another ext:consensus_* series with group_left (version) on
flavor and resource.Recorded where: postgres-exporter.
max by (resource, version) (
label_replace(
label_replace(
pg_static{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"},
"resource", "$1", "target", "(.+)"
),
"version", "$1", "short_version", "(.+)"
)
)
mzmon:heartbeat
#
chart_version="__mzmon_chart_version__" from mzmon-heartbeat, group mzmon_heartbeat
1 at every evaluation, labelled with the version of the
materialize-monitoring chart that installed the rule.
Reads no metric, so it stops only when the ruler, its query path, or
its writes through the gateway stop. Each ruler replica writes its
own, under its own ruler_replica.Recorded where: thanos.
vector(1)mzmon:cluster_created_timestamp_seconds
#
from mzmon-heartbeat, group mzmon_heartbeat_kube_state_metrics
When the cluster’s kube-system namespace was created, in seconds
since the epoch.
Stands in for when the cluster was created. A cluster rebuilt from the
same values reports a new value from then on, and a cloned one
reports its own beside the original’s.Recorded where: kube-state-metrics, thanos.
min(kube_namespace_created{namespace="kube-system"})mzmon:environment_info
#
from mzmon-heartbeat, group mzmon_heartbeat_kube_state_metrics
1 for each Materialize environment in the cluster, by
environment_id.
Read from the Materialize resources. Carries the environment’s UUID
and not its name, which is the resource’s name.Recorded where: kube-state-metrics, thanos.
group by (environment_id) (kube_mz_environment_info)mzmon:build_info:count
#
component="kubelet" from mzmon-heartbeat, group mzmon_heartbeat_kube_state_metrics
Nodes running each kubelet version.
version is the kubelet’s, as the node reports it, which is the
cluster’s Kubernetes version once an upgrade finishes. Each node
counts once however many kube-state-metrics replicas report it.Recorded where: kube-state-metrics, thanos.
count by (version) (
label_replace(
group by (node, kubelet_version) (kube_node_info),
"version", "$1", "kubelet_version", "(.+)"
)
)
component="environmentd" from mzmon-heartbeat, group mzmon_heartbeat_materialize
Running environmentd processes of each Materialize version.Recorded where: materialize, thanos.
count by (version) (mz_start_time_environmentd)component="alloy" from mzmon-heartbeat, group mzmon_heartbeat_alloy
Running Alloy collectors, agents and gateways together, of each version.Recorded where: alloy, thanos.
count by (version) (alloy_build_info)component="loki" from mzmon-heartbeat, group mzmon_heartbeat_loki
Running Loki processes of each version, every component together.Recorded where: loki, thanos.
count by (version) (loki_build_info)component="node-exporter" from mzmon-heartbeat, group mzmon_heartbeat_node_exporter
Running node-exporter processes of each version.Recorded where: node-exporter, thanos.
count by (version) (node_exporter_build_info)component="postgres" from mzmon-heartbeat, group mzmon_heartbeat_postgres
Metadata databases running each PostgreSQL version, read through the
postgres-exporter.
Covers each externalDependencies.consensus entry with
flavor: postgres, whichever service runs the database: Amazon RDS,
Cloud SQL, Azure Database for PostgreSQL, or a cluster of your own.
Absent where none is declared, or the exporter doesn’t run.Recorded where: thanos, postgres-exporter.
count by (version) (
label_replace(
group by (target, short_version) (
pg_static{job="postgres-exporter", target=~"(?i)${consensusPostgresResources}"}
),
"version", "$1", "short_version", "(.+)"
)
)
component="thanos" from mzmon-heartbeat, group mzmon_heartbeat_thanos
Running Thanos processes of each version, every component together.Recorded where: thanos.
count by (version) (thanos_build_info)mzmon:node_uname_info:count
#
from mzmon-heartbeat, group mzmon_heartbeat_node_exporter
Nodes with each CPU architecture and kernel release.
machine and release are uname -m and uname -r on the node,
such as aarch64 and 6.12.40-64.114.amzn2023.aarch64. The node’s
host name stays behind.Recorded where: node-exporter, thanos.
count by (machine, release) (
group by (nodename, machine, release) (node_uname_info)
)
mzmon:rules:count
#
from mzmon-heartbeat, group mzmon_heartbeat_thanos
The rules the Thanos ruler evaluates, alerting and recording, on the
replica that loaded the most.
Includes these heartbeat rules. A count well below the install’s
usual one is a rule file the ruler refused to load.Recorded where: thanos.
max(
sum by (pod) (prometheus_rule_group_rules{app="thanos", container="ruler"})
)
mzmon:rule_evaluation_failures:rate5m
#
from mzmon-heartbeat, group mzmon_heartbeat_thanos
Failed rule evaluations per second over five minutes, on the Thanos
ruler replica with the most.
0 while every rule evaluates. A failing rule records nothing and
never fires, so its alert goes missing from ALERTS without a sign
anywhere else.Recorded where: thanos.
max(
sum by (pod) (
rate(prometheus_rule_evaluation_failures_total{app="thanos", container="ruler"}[5m])
)
)