A Label Vocabulary for Materialize Metrics#

This doc proposes one vocabulary of label names for the metrics Materialize emits, a list of reserved names those metrics must not carry, and a two-pass path to both. The first pass is relabeling in this repository’s shipped scrapers, on this repository’s release cadence. The second pass is adoption in MaterializeInc/materialize, where the metrics are registered, until the relabeling has nothing left to do. It owes the label-family harmonization item in DEP-207. It also writes down the relabeling contract that DEP-109 planned and was canceled before writing.

The central claim is that relabeling is how this repository makes a promise about labels it does not emit. Upstream metric labels are an unstable interface by policy, so nothing a consumer writes against them can be promised today. A translation layer owned here turns that into a surface this repository can commit to. Consumers are written once, against the canonical names. The second pass changes who produces those names and nothing about what they are, so a consumer cannot observe it happening.

A second claim decides what moves upstream. The identity of the thing measured belongs to the emitter, and the identity of the process doing the emitting belongs to the pipeline. A Materialize cluster id means the same thing in every deployment, so Materialize should emit it under the canonical name. A Kubernetes cluster, a namespace, and an environment’s name differ by deployment, so they stay target and external labels permanently. This is DEP-109’s principle restated: a label that is the same across deployment modes is emitted correctly at the source, and one that varies is normalized in the pipeline.

The doc also settles how every consumer keys a Materialize environment, which several consumers are waiting on. The key is environment_id, the spec.environmentId UUID of the Materialize resource, and it is carried on every series. The display name is environment_name, carried on mz_environment_info and joined in rather than stamped everywhere. Both are read from the Materialize resource, which is the only place either is declared. materialize_cloud_organization_name is retired: it is the resource’s metadata.name, copied through a pod label whose name describes Materialize Cloud’s billing model rather than the value it holds.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

The keywords carry the obligations on emitters, on the shipped monitors, and on the vocabulary file. They appear on the reserved list, on the form of each relabeling rule, and on the order of the upstream renames.

Goals#

Functional requirements, framed as value-first user stories. Priority tags (Must / Should / Could) are relative to the first shipped version.

The stakeholders are operators reading the shipped dashboards, customers who bring their own metrics backend, the authors of dashboards and alerts in this repository, Materialize engineers adding metrics upstream, and Materialize Cloud as it adopts this stack.

• [Must] As any consumer of this stack’s telemetry, whether Console, the tenant-scoped read path, Cloud or a customer’s own dashboards, I want one authoritative key for a Materialize environment, so that every consumer scopes to an environment the same way.
• [Must] As an operator, I want that key to be unique within my Kubernetes cluster and never to change, so that an environment’s history stays one set of series for its whole life.
• [Must] As an operator, I want one label name for a Materialize cluster’s id on every metric that describes a cluster, so that a query written against one family works against the next without label_replace.
• [Must] As an operator running more than one Kubernetes cluster, I want cluster on every series to name the Kubernetes cluster, so that alerts from two clusters never collapse into one incident.
• [Must] As a customer with my own metrics backend, I want Materialize’s metrics to carry no label my backend claims for itself, so that ingest neither renames them nor lets them overwrite its own identity labels.
• [Must] As a dashboard or alert author, I want the canonical names in the scrape output before upstream changes anything, so that dashboards and rules migrate once rather than once per Materialize release.
• [Must] As a customer with dashboards of my own, I want a renamed label to keep its old name for a deprecation window, so that my dashboards move on my schedule rather than on upgrade.
• [Must] As a customer routing alerts in my own Alertmanager, I want the labels on shipped alerts to change only under the deprecation cycle, so that a routing tree written against them does not silently stop matching.
• [Should] As a customer scraping Materialize with my own Prometheus, I want the canonical names to come from the shipped monitors themselves, so that I get them without running the bundled pipeline.
• [Should] As an operator, I want an _info join to add a label that says what it names, so that the cluster and replica info metrics can be joined in one expression.
• [Should] As an operator, I want to rename an environment’s display name without splitting its series, so that a name is a label for humans rather than part of every series’ identity.
• [Should] As an operator diagnosing an outage, I want an environment’s name to resolve while its environmentd is down, so that the dashboard for the broken environment still says which one it is.
• [Should] As a Materialize engineer adding a metric, I want CI to refuse a reserved label name and to flag a non-canonical spelling of a canonical concept, so that the vocabulary does not erode one metric at a time.
• [Should] As a customer routing alerts, I want one documented list of the labels on shipped metrics and alerts, so that routing has a contract to be written against.
• [Could] As a Materialize Cloud engineer, I want Cloud’s scrape path to produce the same vocabulary, so that a query written for self-managed answers the same question in Cloud.
• [Could] As an operator, I want the build version to carry one label name across environmentd, clusterd, balancerd and persist, so that a version-skew panel needs no per-component cases.

Technical BLUF#

• One cluster id, five label names. instance_id on environmentd’s controller metrics, compute_instance on the adapter’s, cluster_id on the _info family, compute_cluster_id from the SQL exporter, and cluster_environmentd_materialize_cloud_cluster_id on everything clusterd emits. Replica id has three names, cluster name three, and a catalog object at least six.
• The long form is named for its transport, not its meaning. cluster_environmentd_materialize_cloud_cluster_id is the Kubernetes pod-label key cluster.environmentd.materialize.cloud/cluster-id passed through the scraper’s sanitizer by podTargetLabels. Nobody upstream chose that name.
• Upstream has already chosen the canonical spelling three times. mz_cluster_info and mz_replica_info, the federated /metrics/public endpoint, and the upstream cluster-metrics design doc all use cluster_id, replica_id, cluster_name and replica_name. This design adopts that vocabulary and finishes it.
• cluster is emitted today and it is destructive. clusterd stamps cluster="compute" as a const label on mz_timely_step_duration_seconds and mz_cluster_handle_command_duration_seconds. Remote-write external_labels and both rulers fill cluster only where it is absent, so those series lose the Kubernetes-cluster identity the stack puts on everything else. No query reads either family, so the fix is free today.
• Alias rules copy, never rename, and they retire themselves. A rule fills cluster_id from instance_id only where cluster_id is empty and instance_id is not. When upstream emits cluster_id, the rule matches nothing. A 1:1 copy adds no series.
• The rules belong in the shipped monitors, not the gateway. The Prometheus Scrapers component is consumed directly by customers’ own Prometheus and by the GMP flavor. A rule in the Alloy gateway reaches only the bundled path.
• The transpiler has to learn to rename a pod label first. podTargetLabels can only copy a pod label under its sanitized key. The operator flavor needs relabelings, GMP needs targetLabels.fromPod[].to, and the GMP flavor drops target relabelings silently today.
• Process identity stays in the pipeline. cluster, namespace, pod, the environment label, and clusterd’s own cluster and replica ids are target or external labels, and remain so after the second pass.
• The upstream hook already exists. bin/gen-metrics-catalog walks every metric! invocation, records its label keys, and CI diffs the result. A reserved-name and canonical-spelling check is a lint over a file that is already generated.
• The federated endpoint has a latent duplicate-label bug. add_replica_labels appends cluster_id and replica_id without checking whether the series already has them. It is harmless only because no clusterd metric does yet. The second pass fixes it before any emitter adopts a canonical name.
• Alert labels get the strictest treatment. Rules pass instance_id and the long-form ids through to alerts, and an external routing tree may match on them. Rules aggregate by both names through the overlap, so an alert never loses a label inside the window.
• An environment is keyed by environment_id, read from Materialize.spec.environmentId. It is a UUID that orchestratord already resolves on first reconcile and already refuses to duplicate within a Kubernetes cluster, and the CRD forbids changing it because persist’s layout depends on it. Every series from an environment’s processes carries it.
• An environment is named by environment_name, on mz_environment_info only. The name is metadata.name today and could become a mutable spec.environmentName. Names are not unique and may change, so they are joined in, not stamped on every series, which is the same rule this doc applies to cluster and replica names.
• The environment key needs one small upstream change before this repo can relabel it. No pod label carries spec.environmentId today. orchestratord adding materialize.cloud/environment-id to the labels it puts on every pod, and emitting mz_environment_info and mz_environment_active_generation, is a single change and the first item of upstream work.
• %%{mzEnvironmentFilter} is the switch. The query registry names the fragment 250 times and writes the environment label out 24 times, 22 of them in the two alert files. Moving the fragment from materialize_cloud_organization_name to environment_id is one change, made once the supported operator floor carries the pod label.
• Cloud is the documented exception to global uniqueness. Cloud sets spec.environmentId to the organization’s UUID, which repeats in every region the organization uses. The globally unique key is therefore (cluster, environment_id), which is one more reason cluster is reserved.

Non-goals#

• Cardinality. Series counts, per-shard label bounding (PER-101), and dropping families belong to another team. Every change here is a rename or a 1:1 copy, and none adds or removes a series.
• Metric names. The v2_mz_ prefix, units and suffixes are the same class of problem with a different mechanism, and %%{mzSqlPrefix} already handles the prefix.
• Label values. status="ok" beside status="success" is real and is left to a later pass. See open questions.
• The log vocabulary. Loki stream labels are a committed surface with rules of their own. The environment key reaches logs as structured metadata; the rest of the log pipeline’s names are a follow-up.
• OpenTelemetry-native emission. If Materialize emits OTLP itself one day, this vocabulary maps onto a materialize.* attribute namespace. Designing that mapping is not this doc.
• Organization labels. An organization is a Materialize Cloud billing concept with no counterpart in self-managed. Cloud MAY stamp one as a pipeline label of its own; the vocabulary does not define it.
• Cloud’s scrape configuration. This doc says what Cloud would need in order to converge, not how Cloud’s Pulumi gets there.
• Tenancy classification. Which families are environment-scoped is DEP-272’s question. This doc names the labels that classification reads.

What exists today#

One concept, many names#

The live install has 77 distinct label names across environmentd, clusterd and the SQL exporter, not counting target labels. Most are dimensions of a single measurement and are fine. The identity labels are the problem, because identity labels are join keys.

The consumers in this repository carry the cost. The cluster picker’s value comes from compute_cluster_id and is matched against three different label names, chosen per query: the long form 24 times, instance_id 18 times and compute_cluster_id 6 times. The query author has to know which family a metric belongs to, and the style guide exists largely to tell them. The mzClusterName and mzObjectName helpers exist because a name join needs a label_replace first. The rendered env-top dashboard spells instance_id 254 times and the long form 156 times, and the rendered rules 86 and 113 times.

The long form is named for its transport#

The Materialize orchestrator labels every clusterd pod cluster.environmentd.materialize.cloud/cluster-id. The monitors copy that pod label with podTargetLabels, and prometheus-operator sanitizes the key into a label name. cluster_environmentd_materialize_cloud_cluster_id is therefore the label’s route into Prometheus, spelled out. It says nothing about what the value is, and it cannot be shortened without a rule that renames it, which no shipped monitor has.

Materialize Cloud reaches the same names by another route: a blanket labelmap over every pod label and every pod annotation, which also produces the _size, _scale, _workers, _replica_role, _cluster_name and _replica_name variants. The shipped monitors copy only the id pair, which is why those variants are absent on self-managed.

name means four things#

name is the shard’s name on 28 persist families, the cluster’s name on mz_cluster_info, the replica’s name on mz_replica_info, and the object’s name on mz_object_info. On an _info metric a label is the payload of a group_left, so an unqualified one collides with the next join:

# Even with the ids aligned, the second join's `name` overwrites the first's:
# the result carries the replica's name and has lost the cluster's.
mz_compute_replica_history_dataflow_count
  * on (cluster_id) group_left (name) mz_cluster_info
  * on (cluster_id, replica_id) group_left (name) mz_replica_info

type and size have the same problem. mz_cluster_info and mz_replica_info both carry size, meaning the cluster’s managed size on one and the replica’s size on the other.

Architecture#

flowchart LR
  subgraph upstream["MaterializeInc/materialize"]
    reg["metric! registrations<br/>environmentd · clusterd · balancerd"]
    orch["orchestratord<br/>pod labels incl. materialize.cloud/environment-id<br/>mz_environment_info"]
    lint["gen-metrics-catalog lint<br/>second pass"]
  end

  subgraph here["materialize-monitoring · Prometheus Scrapers component"]
    vocab[("labels.yaml<br/>canonical · aliases · reserved")]
    mon["Shipped monitors<br/>relabelings + metricRelabelings<br/>operator · GMP · classic"]
  end

  subgraph pipe["Pipeline"]
    scr["Any scraper<br/>bundled Alloy · customer Prometheus · GMP"]
    ext["Identity stamps<br/>cluster · tenant · replica labels"]
  end

  consumers["Dashboards · alert rules · customer queries and routing"]

  reg -->|"/metrics"| scr
  orch -.->|"meta labels · /metrics"| scr
  vocab -->|"generated rules"| mon
  mon -->|"consumed by"| scr
  scr --> ext --> consumers
  vocab -.->|"published; checked against"| lint
  lint -.-> reg

Three properties of this diagram carry the design.

The vocabulary is one file, and everything else is generated from it. The monitors’ rules in all three flavors, the docs table, the query-registry check and the upstream lint all read labels.yaml. A canonical name added in one of those places and not the others is the drift this design exists to remove.

The rules sit in the monitors, upstream of every scraper. A customer running their own Prometheus or GMP consumes the monitors directly, and the bundled gateway is one consumer among several. Identity stamps are added after the rules, so nothing a rule does can touch cluster.

The arrow into the upstream registrations is dotted. The second pass changes who produces the canonical names, not what they are. Nothing to the right of the monitors changes when it happens.

Reserved labels#

A reserved label is a label name that some system on the path from emitter to reader assigns a meaning of its own. Materialize metrics MUST NOT carry a reserved label. Where one is emitted today, the shipped monitors MUST move its value to a canonical name and drop the reserved label, with no overlap. A collision is a defect, and the colliding name has no consumer worth preserving.

The list is closed by the vocabulary file and open to additions. A name added to it is added for every emitter at once, which is the reason it lives in one place.

cluster is live today#

src/compute/src/metrics.rs registers two families with const_labels: {"cluster" => "compute"}. On the live install every other series carries cluster="default", from clusterName, and these two carry cluster="compute". The stamp fills cluster only where it is absent, in remote-write external_labels, in the Thanos ruler’s --label, and in the Loki ruler’s alert_relabel_configs. That rule is correct, because it lets a destination’s own externalLabels override the default. It is also exactly why an emitted cluster wins.

The consequences depend on the destination:

No query in the registry reads either family, so moving the label costs nothing today. The canonical name is server_name, which the CTP transport metrics in src/service/src/transport/metrics.rs already use for the same compute-or-storage distinction.

Names to avoid#

A second tier is not reserved by any one system and is still ambiguous wherever Materialize’s metrics share a store with anything else. New metrics SHOULD NOT use these names, and existing uses SHOULD move to a qualified name through the alias mechanism.

The canonical vocabulary#

Five rules generate the table below.

1. A label naming a Materialize entity MUST be <entity>_<property>, where the entity is a catalog noun: cluster_id, replica_name, shard_id.
2. Every label on an _info metric MUST be entity-qualified, because each one is the payload of a group_left.
3. An identity label SHOULD take the name of the catalog column a reader would join it to in SQL.
4. A dimension label, one that describes the measurement rather than the thing measured, MAY be unqualified unless its name is in names to avoid.
5. Canonical names carry no vendor or product prefix.

The fifth rule is a choice, and the alternative was considered. An mz_ prefix would make every identity label collision-proof by construction, and it reads naturally beside mz_* metric names. It loses on two counts. Materialize Cloud already stamps mz_cluster as the EKS cluster’s name, so mz_cluster_id would put a Materialize cluster one suffix away from a Kubernetes one in Cloud’s own store. And upstream has already chosen the unprefixed spelling in its three newest pieces of work, so a prefix would mean renaming the _info family, which was designed as the join target and is the newest part of the contract.

collection_id is preferred over global_id because it names an entity rather than a type, and because it is the spelling the controllers emit and the dashboards already group by. The catalog is not consistent here either, and the alternative is recorded as an open question.

Deliberately unchanged#

Ids on series, names on _info#

An id is part of a series’ identity, and a name is not. A label on a series is part of what makes it that series, so a label whose value changes ends one series and starts another, and every rate() across the change breaks. Ids never change, so they MUST be on the series that describe their entity. Names can change and are not unique, so a name SHOULD appear only on its entity’s _info metric, and consumers join it in.

The rule covers environments, clusters and replicas alike. clusterd pods carry cluster-name and replica-name annotations, and Cloud’s annotation labelmap turns them into labels on every series. The shipped monitors could copy them the same way, and this design deliberately does not. The _info family reads the catalog on every scrape, so a name taken from it is current by construction. An annotation is current only as of the pod’s last template change, and a renamed cluster is exactly the case where the two disagree. /metrics/public appending cluster_name and replica_name to every federated series is an existing departure from the rule, listed under upstream work.

The environment key#

An environment is one Materialize resource: one environmentd and the clusters, balancers and consoles it runs.

Why spec.environmentId#

It is the only field the resource declares as the environment’s identity, and its upstream guarantees already match what a key needs.

The alternatives each fail one of those. metadata.name is unique only within a namespace and is chosen by whoever writes the manifest. The namespace is a deployment choice and repeats across clusters. status.resourceId, which prefixes every pod name and reaches pods as materialize.cloud/mz-resource-id, is unique within the cluster but is derived status rather than declared spec, and nothing outside the cluster knows it.

The value is the UUID, not environmentd’s environment id#

environmentd is started with a composite --environment-id, <cloud_provider>-<region>-<uuid>-<ordinal>, which orchestratord builds with an ordinal of 0 and which mz_environment_id() returns in SQL. Upstream’s EnvironmentId type calls the UUID component organization_id. environment_id carries the UUID alone. The provider and region are deployment facts with labels of their own, and the ordinal is a constant. A consumer joining from SQL takes the UUID component of mz_environment_id().

Global uniqueness is a SHOULD, and Cloud is why#

The globally unique key is therefore (cluster, environment_id). A store holding more than one Kubernetes cluster MUST scope by both. That is already the stack’s behaviour, since cluster is stamped on every series, and it is one more reason cluster is reserved.

mz_environment_info#

mz_environment_info carries environment_id, environment_name and environment_namespace, the namespace the resource lives in, with a constant value of 1. It is the only series that names an environment.

orchestratord is the right emitter, rather than environmentd, for three reasons. It already holds the resource, and every field the info metric carries is on it. It emits one series per resource, where environmentd would emit one per generation and two during every blue/green rollout. And it keeps emitting while environmentd is down, which is when a dashboard most needs to say which environment it is looking at.

The resource’s namespace is environment_namespace and not namespace, because orchestratord’s own namespace target label names the operator’s namespace and an emitted namespace would be renamed exported_namespace. It replaces materialize_cloud_organization_namespace, which on an environment’s own series repeats the namespace target label.

Beside it, orchestratord emits mz_environment_active_generation, with the same labels. Its value is the generation the environment serves, from status.activeGeneration. Its generation label carries the same number, so a query can join it to the generation in a pod’s name.

kube-state-metrics publishes both from its customResourceState support, with no upstream change, by reading the Materialize resource. It is the fallback if the orchestratord change slips, and it covers operator versions that predate it. Its series are kube_mz_environment_info and kube_mz_environment_active_generation, with the same labels. The kube_ prefix keeps the two producers from ever writing the same series where both run, and says which one a query reads. Consumers read the bridge’s names until an operator release emits the upstream ones. Every join reads the info metric through group by (cluster, environment_id, environment_name), which drops the labels particular to each producer. A consumer that reads both producers during the overlap groups mz_environment_info or kube_mz_environment_info the same way, which gives one series per environment. The join is on (cluster, environment_id), never on environment_id alone, for the uniqueness reason above.

What consumers change#

The transition is gated on the operator, not on a relabel#

Every other alias in this design is a fill-if-empty copy of the same value under a new name. The environment key cannot be one, because environment_id and materialize_cloud_organization_name hold different values, and no relabeling rule can derive a UUID from a name. Its overlap is therefore “both labels present”, and it starts when the operator puts the new pod label on the pods.

1. orchestratord adds materialize.cloud/environment-id to default_labels(), which reaches environmentd, balancerd and console pods directly and clusterd pods through the service labels orchestratord passes environmentd. It emits mz_environment_info and mz_environment_active_generation. The label changes the pod template, so it reaches an environment’s pods at its next rollout.
2. The shipped monitors add a target alias from that pod label to environment_id. On an older operator the meta label is absent and the rule does nothing.
3. Compatibility declares the operator version that carries the label.
4. Once the supported operator floor includes that version, %%{mzEnvironmentFilter}, the picker, the namespace picker and mzEnvironmentName move to the new keys in one change. A **Deprecated:** note names materialize_cloud_organization_name, _namespace and _id.
5. After 30 days the old keys leave podTargetLabels.

Step 4 is a render-time switch, like %%{mzSqlPrefix}. If a fleet ever needs both keys at once, the fragment can render either, and the dashboards are built for one or the other.

The first pass: relabeling in the shipped monitors#

Where the rules live#

The per-target placement has one cost: a rule appears in every monitor. Every alias renders on every Materialize monitor, whether or not that monitor’s process emits it. A rule whose alias is absent does nothing, and the rule keeps working if Materialize moves a metric from one process to another. Generating the rules from one file makes the replication free, and keeps a cross-cutting rule from becoming the one monitor that forgot it.

Four rule forms#

Every rule the vocabulary generates takes one of four forms.

A metric alias MUST be fill-if-empty, and it MUST NOT overwrite a canonical label that is already present:

# Fill cluster_id from instance_id only where cluster_id is empty and instance_id is not.
# Relabel regexes are fully anchored, so "u1;u1" and ";" both fail to match.
- sourceLabels: [cluster_id, instance_id]
  separator: ";"
  regex: ";(.+)"
  targetLabel: cluster_id
  replacement: "$1"
  action: replace

That form is what makes the second pass invisible. Once upstream emits cluster_id, the joined value is u1;u1 or u1;, the rule matches nothing, and the series is unchanged. A plain replace would also produce the right value today, and would silently overwrite upstream’s value tomorrow if the two ever disagreed.

An alias scoped to some families adds __name__ to the source labels:

# `name` on mz_cluster_info is the cluster's name. Elsewhere it means something else.
- sourceLabels: [__name__, cluster_name, name]
  separator: ";"
  regex: "mz_cluster_info;;(.+)"
  targetLabel: cluster_name
  replacement: "$1"
  action: replace

The target alias reads a meta label, which exists only during target relabeling:

- sourceLabels: [__meta_kubernetes_pod_label_cluster_environmentd_materialize_cloud_cluster_id]
  regex: "(.+)"
  targetLabel: cluster_id
  replacement: "$1"
  action: replace

A copy of a label into another label with the same value cannot create a series, since the set of distinct label sets is unchanged. It adds index entries and bytes per series, which is the whole of the cost.

One vocabulary file#

The vocabulary lives in packages/prometheus-scrapers/labels.yaml, owned by the Prometheus Scrapers component, because that is the component whose output changes. Its schema is packages/mzmon-lib/schemas/scrape/mzmon-labels.schema.yaml. The file holds only what something reads, in the following shape:

monitors: [environmentd, clusterd, materialize-operator]
canonical:
  - name: cluster_id
    values: "[su][0-9]+"
    aliases:
      - label: instance_id
        deprecatedIn: "1.6.0"
      - label: cluster_environmentd_materialize_cloud_cluster_id
        deprecatedIn: "1.6.0"
  - name: cluster_name
    aliases:
      - label: name
        families: [mz_cluster_info]
        deprecatedIn: "1.6.0"

gen-scrape-configs renders every alias into every endpoint of the monitors that monitors names, in every flavor. The docsite renders the list of canonical labels from it.

The following fields wait for their first reader:

• meaning and joinsTo, for the customer-facing label contract in DEP-356.
• reserved and drop, until more than clusterd’s cluster move and honeycomb drop in DEP-347 need rendering.
• fromPodLabel, with target-alias rendering in DEP-349.

The query-registry check in DEP-356 reads the same file to reject a query matching on a retired alias or on a reserved label. The upstream lint in DEP-352 takes a published copy rather than reading this repository.

The three flavors#

The GMP flavor MUST fail the render on a rule it cannot express, rather than dropping it. A rule dropped silently is a dashboard that is empty on one flavor and fine on the others, which no render test can see. Whether GMP accepts a labeldrop of cluster is untested. If it does not, the GMP flavor cannot correct the cluster collision at scrape, and the upstream fix is the only one available to it, which is one more reason to land that fix first.

The overlap, and what retires it#

A metric label rename is graded as visible and non-paging by the deprecation policy, and the policy asks for a dual-publish shim where this repository’s layer allows one. The alias rules are that shim.

Step 3 also decouples the dashboards from the Materialize version. The canonical names are produced by the monitors against every Materialize release, so a dashboard on cluster_id works on a release that emits instance_id and on one that emits cluster_id. Today a rename upstream would need a min-mz-version bump or a second arm in every affected query.

1.0 shipped before the first pass, so the pre-1.0 budget that listed label-family harmonization is spent. The migration still fits a minor release. Upstream label names are on the coordinated surface, which asks for disclosure and for a dual-publish shim where this repository’s layer allows one, and sets no major-version requirement. The alias overlap is that shim.

Alerts keep both labels through the window#

Alert labels are the strictest surface in the policy, because a routing tree that stops matching is silent. Of the shipped rules, platform alerts pass instance_id through three times and the long-form ids five times, and workload alerts pass them seven times. An external Alertmanager may route on any of them.

During the overlap, a rule that aggregates by (instance_id) MUST aggregate by (cluster_id, instance_id) instead. Both labels are present and equal on every series, so the aggregation produces the same series and the alert carries both names. At step 5 the alias drops out of the by clause with the rest. The promtool tests that assert by (…) (ALERTS) on the old names change in the same pull request, which is the check that no alert lost a label early.

The second pass: adoption upstream#

The order keeps it invisible#

The second pass is a sequence, and the order matters more than the content.

1. The lint lands first. It refuses reserved names outright and flags a non-canonical spelling of a canonical concept on any new metric, with the existing ones on an exception list that only shrinks.
2. The federated endpoint stops appending blindly. add_replica_labels in src/environmentd/src/http/metrics_public.rs pushes cluster_id, replica_id, process, cluster_name and replica_name without checking for an existing label of the same name. A series carrying two values for one label name is invalid exposition.
3. Reserved names go. The two cluster const labels become server_name, and honeycomb is removed. These need no coordination, since the monitors already move them.
4. Renames land, one family at a time, after the matching alias has shipped here. Upstream MAY rename at any point after step 2 of the overlap. The fill-if-empty rule absorbs the change, and the release note here has already told consumers the new name.
5. The exception list empties. At that point every alias rule in the monitors is a no-op, and the next minor removes them.

The environment key sits outside this sequence. orchestratord’s pod label and mz_environment_info add names and rename nothing, so they can land first and in parallel, and the environment transition is gated on the operator version rather than on an alias.

The constraint on step 4 is one-directional. Upstream renaming early is safe, because the alias covers it. Upstream renaming a family that has no alias here yet is the one ordering that produces a visible break, and the lint’s exception list is what makes that ordering visible before it merges.

The hook already exists#

src/metrics-catalog/src/main.rs, run as bin/gen-metrics-catalog, parses every metric! invocation with syn and writes each family’s label keys into doc/user/data/metrics.yml. ci/test/lint-metrics-catalog.sh fails CI when that file is stale. A label lint is therefore a pass over a file that CI already guarantees is current, and needs no new extraction.

The lint reads the reserved and canonical sections of labels.yaml. Where that file lives, and which repository is its source of truth, is an open question.

Two label sources sit outside metric! and need the check separately:

What stays in the pipeline#

Some labels never move upstream, and the second pass is done when everything else has.

The last row is the subtle one. cluster_id is canonical everywhere, and the second pass makes environmentd emit it on the families that describe clusters from outside. clusterd describes itself, and its identity arrives the way every other process’s does.

Materialize Cloud#

Cloud’s scrape configuration (internal) reaches the long forms through labelmap and adds three external labels of its own: mz_cluster (the EKS cluster), mz_aws_region and mz_cloud_stack_name. Its alerting reads cluster_environmentd_materialize_cloud_cluster_id and mz_cluster directly. Cloud sets spec.environmentId to the organization’s UUID, and names each resource environment-<organization-id>-<ordinal>. Cloud’s usage pipeline (internal) calls that resource name the environment_id and calls spec.environmentId the organization_id, which is the reverse of this vocabulary. Cloud also has a design (internal) proposing to collapse its organization and environment labels into one environment_id through relabeling, without fixing which of the two values it carries. This doc fixes it: environment_id is the resource’s spec.environmentId, in Cloud as everywhere, and the globally unique key is (cluster, environment_id) because Cloud repeats the UUID across regions.

Cloud converges by adopting the shipped monitors, which is already the Cloud adoption direction, and gets the aliases and the overlap with them. Two decisions are Cloud’s alone. The first is mz_cluster against cluster: this stack stamps cluster, the ecosystem expects cluster, and Cloud’s alerting would move with the adoption. The second is whether Cloud sets spec.environmentName to the name a customer sees, since its metadata.name is an internal identifier and environment_name would otherwise show it.

Upstream work#

Work in MaterializeInc/materialize, in the order above. Tracked in the Metric label standardization milestone of the CLOM project, with the upstream review in SQL-754. The metrics owners are the SQL team’s Improve Prometheus Metrics project, with persist’s families coordinated with the persist team.

Chart-side prerequisites#

Work in this repo. Ordered roughly by dependency. Tracked in the same milestone, gated on design acceptance in DEP-346; DEP-207 is the umbrella.

Testing#

The render tests prove a rule exists, and the important assertions are about what it does to real series.

• Rule semantics, table-driven. Every generated rule is applied to sample label sets: old name only, canonical only, both equal, neither. The canonical-only and both-equal cases MUST come out unchanged, which is the property that keeps the second pass invisible.
• No cardinality change. Against a live tier, the series count per Materialize job is equal before and after the aliases ship, within scrape noise. A 1:1 copy adds none, and a test that asserts it catches a rule that is accidentally not 1:1.
• Kubernetes-cluster identity everywhere. count({job=~".*materialize.*", cluster!="<clusterName>"}) is zero. This is the assertion the cluster collision fails today.
• Every alias has its canonical partner. For each alias, count({<alias>!=""} unless on (…) {<canonical>!=""}) is zero.
• Reserved names absent. No Materialize series carries a reserved label, read from labels.yaml rather than restated in the test.
• GMP admission. The rendered PodMonitoring resources are applied to a GKE cluster with managed collection on, because GMP’s admission webhook is the only authority on which rules it accepts. The cluster drop is the case to watch.
• Flavor parity. For one Materialize release, the bundled path and a plain prometheus-operator Prometheus fed the same monitors produce the same label names on the same families.
• Alert label sets. The promtool tests’ by (…) (ALERTS) assertions carry both names through the window and only the canonical name after.
• Join correctness. A query joining mz_cluster_info and mz_replica_info in one expression returns both cluster_name and replica_name, which fails today.
• Environment key everywhere. On an operator that sets the pod label, every series from environmentd, clusterd, balancerd and console carries environment_id, and its value equals the resource’s spec.environmentId.
• One info series per environment. With orchestratord and the bridge both running, mz_environment_info or kube_mz_environment_info grouped by (cluster, environment_id, environment_name) has one series for every environment, so the two producers agree on its name.
• A rename splits nothing. Changing spec.environmentName changes mz_environment_info and no other series’ label set.
• The name survives an outage. With environmentd scaled to zero, the environment picker still lists the environment by name.
• Upstream lint. A test fixture in MaterializeInc/materialize registers a metric with a reserved label and asserts the lint refuses it.

Documentation to update#

• A customer-facing label contract page, generated from labels.yaml: every canonical label, its meaning and join target, every alias with its retirement date, and the reserved list. It absorbs the alerting design’s owed label contract, and it is the artifact a customer writes queries and routes against.
• reference/development/dashboard/style-guidelines.md — the label-family section becomes a pointer to the contract page once the aliases ship, and the “query author writes the label” paragraph goes with it.
• reference/development/pipelines/metrics.md — the three-phase table names inputMetricProcessor as the home for dashboard-contract normalization. It moves to the monitors, and the table should say why.
• reference/stability.md — the canonical vocabulary becomes a public interface (“documented metric labels”), while raw upstream labels stay unstable. That is the central claim, written as policy.
• reference/development/versioning.md — the same surface, in the policy of record.
• reference/development/queries/alerts.md — its scoping guidance names the long-form label and a query-time label_replace from pod.
• packages/prometheus-scrapers/README.md — the allowlist rationale gains the aliases and the vocabulary file.
• metrics/collecting/ — the GMP page states which rules the GMP flavor cannot express, once that is measured.
• reference/development/design-docs/20260916-tenant-query-api.md — its What exists today rows name materialize_cloud_organization_name and organization_name as the environment labels the proxy enforces on. The key becomes environment_id, with cluster where a store spans clusters.
• logs-and-events/ — the structured-metadata reference gains environment_id.
• compatibility.md — the operator version that sets materialize.cloud/environment-id.
• reference/development/roadmap.md — ✅ done. Its Metric label standardization section and its Designs in flight row link here.

Open questions#

• Which repository owns labels.yaml? The DEP-109 position was that the source-of-truth contract lives with the producer and this repository records what it consumes. The reserved list is this stack’s concern and the aliases are this repository’s mechanism, which argues for here. The canonical names are upstream’s to emit, which argues for there. A split, with upstream owning canonical and this repository owning reserved and aliases, is the likely answer.
• Is cluster_id safe enough? It is not reserved by anything verified, and it sits next to cluster in every store. Multi-cluster cost tooling keys on a configurable cluster label whose documented examples include cluster_id. This design accepts that residual risk in exchange for upstream’s existing spelling, and the contract page should name it.
• collection_id or global_id? The catalog uses global_id for the column and the controllers use collection_id for the label. This doc picks the controllers’ name; the persist and compute owners may see it differently.
• Who owns spec.environmentName, and is it validated? It is a CRD change upstream. A display name wants a length bound and nothing else, since uniqueness is explicitly not promised.
• When does the operator floor move? The environment switch waits on every supported operator setting the pod label, and the support window is set in Compatibility, not here. If that is long, the render-time switch is the way to ship both keys meanwhile.
• Does the tenant claim become environment_id? The tenant-scoped read path left the claim’s shape open pending control-plane identity. The label it enforces on is now settled; whether the claim carries the same UUID, or something the proxy maps to it, is still that design’s question.
• Should SQL expose the bare UUID? mz_environment_id() returns the composite, so joining a metric to SQL means parsing it. A function or column returning spec.environmentId alone would make the join direct.
• Should server_name exist at all on the two compute families? The value is the constant compute on families only compute registers, so the label carries no information. Moving it is safe and dropping it is simpler; dropping is safe only while no storage-side family registers the same name.
• Does the vocabulary extend to label values? status="ok", status="success" and outcome="success" coexist. A value vocabulary for the handful of dimensions alerts read would make thresholds portable across families, and it is a larger change than this one.
• Do the other log structured-metadata names follow? The environment key is settled for both engines. Whether cluster_id and replica_id replace the log pipeline’s own spellings is the one-picker-across-two-engines problem with a vocabulary behind it, and Loki stream labels move only under the deprecation cycle.
• Should queries select Materialize components by app rather than by job value? job is reserved and its value encodes the monitor’s namespace and name, which a customer’s own install will not match. Several shipped queries and alerts match on it.
• Does Cloud keep mz_cluster? Cloud’s alerting reads it, and this stack stamps cluster. Adoption is the natural moment to converge.
• Does GMP accept a labeldrop of cluster? If not, GMP consumers wait for the upstream rename, and the GMP flavor’s documentation says so.