Pipelines on materialize-monitoring Documentation

Overview

Mon, 01 Jan 0001 00:00:00 +0000

Pipelines as Code#

Instead of hand-maintained alloy config files, we author pipeline definitions as YAML, validate them against a project-owned JSONSchema, and render them to config.alloy via the mz-monitoring-build gen-pipelines subcommand. Sources live under packages/alloy-pipelines/; schemas under packages/mzmon-lib/schemas/alloy/; the Rust renderer and validator live under packages/mzmon-lib/.

The audience for this section is repo contributors — SRE, Field Engineering, CloudOps, Database Engineers — and AI agents reading the corresponding pipelines-as-code skill. The pipelines themselves target machines (alloy), not end users, so the conventions below favor contributor ergonomics: IDE autocomplete, schema-driven docs, escape hatches when reality outpaces the typed model.

Authoring

Mon, 01 Jan 0001 00:00:00 +0000

Authoring Pipelines#

This page covers the model and conventions for authoring alloy pipelines as YAML under packages/alloy-pipelines/. The runtime aspects (label families, retention) live in Logging and Metrics.

Pipeline model#

A pipeline is a YAML document that maps roughly 1:1 to an alloy config file:

description: |
 What this pipeline does, why it exists, what it forwards to.
logging:
 level: info
 format: logfmt
blocks:
 - loki.process:
 label: inputProcessor
 forward_to: [{ref: "loki.write.gateway.receiver"}]
 blocks:
 - stage.drop:
 older_than: "12h"
 drop_counter_reason: "backlog > 12hr"
 - stage.match:
 selector: '{app="alloy"}'
 blocks:
 - stage.logfmt:
 mapping: { msg: msg, level: level }

Each entry in blocks is a single-key object whose key is either:

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

Metrics Pipelines#

Metrics are processed primarily in otelcol: Prometheus ingest and the final write use the prometheus.* family, but everything between is converted to OTLP and shaped by otelcol.processor.* components (we found little value in doing the processing with prometheus.* blocks). They are authored the same way as log pipelines — see Authoring for the pipeline model, the strict-attributes policy, and the raw: escape hatch.

Gateway topology#

The gateway carries metrics alongside logs (packages/alloy-pipelines/gateway.yaml). Prometheus ingest is bridged into OTLP, processed in otelcol, then converted back to Prometheus for the write.

Logging

Mon, 01 Jan 0001 00:00:00 +0000

Logging Pipeline#

This is the authoritative reference for the log-processing pipelines that run on alloy-agent and alloy-gateway. The customer-facing Logs & Events section links here for pipeline detail; this page is for repo contributors.

Where the pipelines live#

The pipelines are authored as code (see Authoring) and rendered to config.alloy by mz-monitoring-build gen-pipelines: