Cloud Provider Metrics#

The gateway can pull what a cloud provider’s monitoring API publishes about the metadata database and the buckets a Materialize deployment depends on, and about whether the provider can supply the nodes its cluster asks for. The result is written beside every other metric, with the same retention, the same PromQL, and the same destinations. A provider series is therefore joinable with mz_persist_* in a single expression. A Grafana CloudWatch, Cloud Monitoring or Azure Monitor datasource cannot offer that.

The Infrastructure Cloud Provider dashboard (infra-cloud) draws the database and bucket pulls, and Infrastructure Autoscaling (infra-autoscaling) the node and quota pulls on its Cloud Capacity tab, each with rows for whichever provider it finds; see Available Dashboards.

Provider collection is off by default and adds to what the clients already report about the same dependencies. Persist, Loki and Thanos measure every request they make against the database and the bucket, at full resolution and at no cost. The provider adds what no client can see: CPU, memory and storage headroom, the burst credits that throttle a volume, transaction-ID consumption, and bucket growth. For the cluster’s nodes it adds the provider’s own view: host and instance health, node groups that cannot reach their desired size, compute quota, and on AKS the managed cluster autoscaler, which runs where nothing in the cluster can scrape it. The external-dependency design records the reasoning.

What is pulled#

The metric sets are fixed, and values only name the resources. Each pull is a custom component in the gateway’s gateway-provider pipeline, so changing what is pulled is a change to that pipeline. CloudWatch takes one instance of it per listed resource; GCP and Azure take one per service, whose filter names every listed resource. The Cloud SQL entries are metric-type prefixes, so postgresql/num_backends also pulls num_backends_by_state and num_backends_by_application, and up also pulls uptime.

Azure applies one aggregation list to every metric in a call. The Flexible Server pull therefore asks for all four and keeps the one or two each metric needs. Those are the statistics the RDS pull asks CloudWatch for: average and maximum CPU, minimum credits, maximum connections, and so on. That is 11 series per server, and 12 on a Burstable tier, which also publishes cpu_credits_remaining.

The nodes and the quota behind them#

A node that cannot be had looks the same from inside the cluster whatever the reason: pods stay pending, and the autoscaler reports a failed launch. These pulls say which reason it is, as far as the provider publishes one.

EKS nodes are the one resource found by tag rather than named, because Karpenter and the node groups replace them. EKS and Karpenter tag every node they launch with aws:eks:cluster-name, and each managed node group’s Auto Scaling group with eks:cluster-name, and the pull matches those tags to the listed cluster names exactly. Karpenter’s node pools have no group of their own, so the group metrics cover only managed node groups; Karpenter’s own metrics cover the rest. The vCPU usage counts every On-Demand instance in the account and region, not only the cluster’s, because that is what the quota counts. Its limit is the Service Quotas console’s L-1216C47A.

The Compute Engine quotas are per machine family: C4 and C4A each have their own, and the regional CPUS quota covers only older families. A family appears only once something in the project uses it, so a project running only C4 and C4A has a handful of series per region. Zonal limits read 2^63 − 1 where no zonal quota applies. A region is matched with its zones and nothing else, so europe-west1 does not also pull europe-west10.

AKS runs its cluster autoscaler inside the managed control plane. Its metrics reach no scrape, and Azure Monitor publishes these four instead. Each keeps the aggregation that shows the worst of its five minutes: the most pods waiting and the most unneeded nodes, any cooldown, and any moment it was unsafe to scale. The node pools are scale sets in the cluster’s node resource group, which the pull finds through the cluster.

Metric names#

The node and quota pulls follow the same shapes.

A node joins to kube_node_info on its provider ID: the instance ID on EKS, and the scale set and index on AKS. The discovery pull also writes an aws_<service>_info series carrying every tag on the resource, which the pull drops, since tags are free text; the two tags it keeps are set by EKS and Karpenter.

CloudWatch series also carry region, account_id and account_alias, and name, which is rds or s3 on every series. Join on the dimension_* label.

Azure series also carry resourceGroup, subscriptionID and subscriptionName. They also carry the interval and timespan each pull reads, as ISO 8601 durations. The Blob Storage families are named for the blob service, as azure_microsoft_storage_storageaccounts_blobservices_*. No resource tag is copied onto a series; the exporter copies owner by default, and the pull turns that off.

Every provider series carries job="integrations/cloudwatch", job="integrations/gcp" or job="integrations/azure". instance is the resource on RDS and S3 series, since each resource has its own pull, and eks on the EKS pull, which covers every listed cluster. On Cloud Monitoring series it is cloudsql, gcs, compute_quota or compute_quota_exceeded, since one pull covers every listed resource of a service. Azure series are the same, with postgres, blob_capacity, blob_requests, aks_autoscaler or aks_nodes.

Configuring it#

Each provider lists the resources to watch. Nothing else is pulled, and a provider that is enabled with no resources fails at render.

pipeline:
  metrics:
    provider:
      cloudwatch:
        enabled: true
        region: us-east-1
        rds:
          instances: [mz-prod-db]
        s3:
          buckets: [mz-prod-storage-a1b2]
        eks:
          clusters: [mz-prod-eks]
      gcp:
        enabled: true
        projectId: my-project
        cloudSql:
          instances: [mz-prod-pg]
        gcs:
          buckets: [mz-prod-storage]
        compute:
          regions: [us-east1]
      azure:
        enabled: true
        subscriptionId: 00000000-0000-0000-0000-000000000000
        postgres:
          servers: [mz-prod-pg]
        blob:
          storageAccounts: [mzprodstorage]
        aks:
          clusters: [mz-prod-aks]

A Cloud SQL entry is the instance name, not the project:region:instance connection name. A Flexible Server entry is the server name, not its FQDN, and a storage account entry is the account name, not its endpoint. Both kinds of Azure name are unique across Azure, so no resource group is needed. An EKS or AKS entry is the cluster’s name, not its ARN or resource ID. AKS names are unique only within a resource group, so two clusters of one name in the subscription are both pulled, and told apart by resourceGroup. A Compute Engine entry is a region, such as us-east1; its zones are included, and a zone is refused. The chart refuses anything else at render, since the names sit inside the Resource Graph query’s filter. subscriptionId is the subscription’s GUID. On Azure Government or Azure China, set cloudEnvironment to azureusgovernmentcloud or azurechinacloud.

The chart refuses, at render, the values that would otherwise stop the gateway starting: a scrape timeout longer than the interval, and two buckets whose names differ only in . and -, which would become the same component label. Alloy accepts both in alloy validate and exits on them at load, which would stop logs and metrics along with the pull.

The full set of keys is in the values reference, under pipeline.metrics.provider.

Identity and grants#

Credentials never travel through values. The pull runs as the gateway pod’s own cloud identity, bound through alloy-gateway.serviceAccount.annotations.

Azure workload identity needs two things on the gateway, not one. The annotation names the identity. The Entra webhook injects it only into pods labelled azure.workload.identity/use: "true", set through alloy-gateway.controller.podLabels. The chart’s Terraform module sets the label whenever the gateway’s annotations include azure.workload.identity/client-id. On Azure the grant can be scoped to each resource, unlike on the other two clouds. Resource Graph returns only the resources the identity can read, so the grant is also what decides which named resources are pulled.

On GCP, the service account the Terraform module creates for the Google Cloud Metrics exporter holds roles/monitoring.metricWriter, which writes metrics and cannot read them. Reading needs roles/monitoring.viewer added to the same account.

The chart warns at render when a provider is enabled and the gateway’s service account carries no matching annotation. On Azure it also warns when the annotation has no pod label beside it. It cannot tell whether EKS Pod Identity, a static key, or a direct Workload Identity principal is in use instead, so the warning is advisory.

The providers fail differently without a credential. CloudWatch resolves its credential on each pull, so a missing one produces empty pulls and the gateway keeps running. The GCP exporter resolves its credential when it starts. On GKE the metadata server always supplies one, so the result is again empty pulls. Outside Google Cloud there is no metadata server, so GOOGLE_APPLICATION_CREDENTIALS has to point at a mounted key or a Workload Identity Federation configuration. Without one the exporter cannot start, and the gateway fails to load along with every log and metric it carries. Azure, like CloudWatch, resolves its credential on the first pull, so the gateway keeps running. On AKS without the pod label, the pull falls through to the node’s managed identity, which Resource Graph refuses with a 403. up is 0, and the gateway logs service discovery failed.

Lag#

Provider data is minutes old when it arrives, and that decides how it can be used.

CloudWatch and Azure samples are stamped at scrape time. Stamping a daily S3 datapoint with its own time would put it a day in the past, and the bundled Thanos Receive, which accepts no out-of-order samples, refuses samples that old. Compute Engine quota usage and limits are stamped at scrape time for the same reason, since each pull reads back a day to find a point that may be hours old. A usage point therefore reads as current until the next one lands, and a change can take hours to show.

Cloud Monitoring samples keep their own timestamps. An instant query at “now” looks back five minutes by default and finds no GCS sample at all. Queries on provider families need last_over_time(<series>[15m]) or a wider window.

Every gateway restart leaves a gap as long as the lag. Remote write forwards only samples stamped after it started, so that a restart does not resend what was already written. A Cloud Monitoring sample stamped before the restart is therefore never sent. After each restart, Cloud SQL series resume a few minutes later and GCS series over ten minutes later. CloudWatch and Azure samples carry the scrape’s time, so they have no gap.

No provider signal backs a fast page. Provider alerts cover the slow-moving conditions — storage headroom, burst-credit exhaustion, connection ceilings — with for: windows well above the publication delay.

Cost#

The provider bills each pull, and the bill depends on configuration, not on how many dashboards are open.

scrapeInterval is the main lever, and defaults to five minutes. Rates change, so current provider pricing is the reference for what a call costs.

The gateway runs several replicas, and every replica runs the exporter. The scrape of it is clustered, so one replica owns it and each provider is called once per interval.

Which destinations receive it#

Each provider assigns its families a tier through metricImportance, which defaults to extended. The families infra-cloud and infra-autoscaling draw are also named in the query registry, at diagnostic, the lowest tier. A registry tier admits a metric at that tier and above, so metricImportance decides for every destination floor above diagnostic.

A destination that bills per series, such as Datadog or a BYOC fan-out, therefore does not receive them unless its floor or the provider’s tier is changed.

The Google Cloud destination deserves the same care on an install that also pulls from GCP. Its default floor is recommended, which keeps the pulled families out. Raising either would write each series back into Cloud Monitoring as a prometheus.googleapis.com/ metric: a billed second copy of data Cloud Monitoring already holds. The pull does not read those types back, so it does not loop.

What is deliberately not pulled#

Checking it works#

On CloudWatch and GCP, up does not say whether a pull succeeded. An exporter whose provider call fails still answers its scrape, with no provider series in it. Measured with credentials missing — for CloudWatch anywhere, and for GCP where a metadata server exists — both exporters return HTTP 200 and report healthy, so up stays 1. up only says the exporter exists and is being scraped.

On Azure it says half of it. A Resource Graph query that fails — no identity, or one with no access to the subscription — fails the scrape, and up is 0. A metrics call that fails for one resource only drops that resource’s series and logs a warning. A resource the identity cannot read is never found at all. up stays 1 in both cases.

CloudWatch publishes no equivalent of the GCP error series. Its yace_cloudwatch_getmetricstatistics_requests_total counts billed calls, but it is one counter per gateway replica, and every CloudWatch target a replica owns reports it. Measured with five targets across two replicas, the five series read 12, 14, 26, 28 and 2: running totals of 28 and 2 calls, not 82. It cannot be summed across instance, so the cost is best read from the configuration: 12 calls per RDS instance and 2 per bucket, each interval. A CloudWatch pull that fails for want of an identity or a grant is visible only as missing series and as errors in the gateway’s logs, such as Couldn't get account Id. An EKS pull missing tag:GetResources finds no nodes, and one missing autoscaling:DescribeAutoScalingGroups finds no node groups, while the vCPU usage still arrives.

The Collection tab of the Infrastructure Cloud Provider dashboard (infra-cloud) draws these checks for every provider it finds.