Scraping Metrics with materialize-monitoring#

By default, materialize-monitoring is configured to scrape metrics from any ServiceMonitor or PodMonitor resources in the cluster. This allows you to easily add new metrics to your monitoring stack by simply creating a new ServiceMonitor resources.

Scrape Architecture with Grafana Alloy#

materialize-monitoring runs Grafana Alloy with prometheus.operator components on alloy-gateway instances (Deployment) which read ServiceMonitors and PodMonitors in order to determine what targets to scrape.

alloy-gateway runs in clustering mode by default, which means that scraping is distributed across all replicas of alloy-gateway and the scrape state is shared between them.

Manually Configured Scraping#

If you are not using default materialize-monitoring setup, you can use the following scrape configuration files as a starting point for your own Prometheus setup.

Authenticating the SQL metrics endpoint#

The materialize-sql scrapers collect SQL-derived metrics from the environmentd /metrics/mz_compute, /metrics/mz_frontier, /metrics/mz_storage, and /metrics/mz_usage endpoints. Scrape it as the built-in mz_support role.

The Classic and Google Cloud Managed Prometheus configs carry username: mz_support inline, so they need no extra setup. Prometheus Operator basicAuth can only reference a Kubernetes Secret — it has no inline fields — so the materialize-sql PodMonitor reads its credentials from a Secret named materialize-sql-monitor. It needs a username key (mz_support) and an empty password key. mz_support is password-less, but alloy’s prometheus.operator.* scrapeconfig generation rejects an absent password reference with resource name may not be empty; an empty password produces the same mz_support: Basic header the username-only configs already send. Create it in the namespace the scrapers run in (for example, materialize):

kubectl create secret generic materialize-sql-monitor \
  --namespace materialize \
  --from-literal=username=mz_support \
  --from-literal=password=

Which Prometheus Distribution Am I Using?#

An easy way to check if you are using Prometheus Operator is to see if you have PodMonitor or ServiceMonitor CRDs in your cluster:

# Get the CRD resource directly:
kubectl get crd podmonitors.monitoring.coreos.com servicemonitors.monitoring.coreos.com
# This may fail with NotFound if you you do not have Prometheus Operator.
# If you get an kind of permission error, you should try the api-resources command:
kubectl api-resources | grep monitoring.coreos.com
# This is empty if you do not have Prometheus Operator

monitoring.coreos.com is the group for Prometheus Operator CRDs.

Alternatively, you can check this table of common Prometheus Distributions if you remember how you installed it:

If you are using an OpenTelemetry Database (Honeycomb, Datadog, etc.), you should not use these scrape configurations. Native OTLP collection will be provided in a future release.

Prometheus Operator Scrape Downloads#

These individual files can be used with a manual prometheus-operator (including kube-prometheus-stack and kube-prometheus) setup or a less-common manual Grafana Alloy prometheus.operator setup.

Installing Prometheus Operator Scrape Configurations#

The above files are meant to be used as manifests that can be passed to kubectl apply directly. The namespace isn’t generally too important, but you may elect to put them alongside your materialize-operator resource.

If your materialize-operator is in the materialize namespace, you can download each into a direcotory and apply like:

kubectl apply -f scrapers/ -n materialize

Classic ScrapeConfig Downloads#

These are classic scrape_configs for non-operator Prometheus setups. These are placed into your Prometheus configuration (prometheus.yml) as a single scrape_config.

These are provided as best-effort convenience. Prometheus Operator Monitors are the preferred implementation.

See Prometheus Configuration for information on how to configure scrape_configs in your Prometheus setup.

Google Cloud Managed Service for Prometheus PodMonitoring#

These are PodMonitoring resources specifically for Google Cloud Managed Service for Prometheus (GMP).

These are provided as best-effort convenience.

cAdvisor metrics are collected by Google Cloud Managed Service for Prometheus by default.