# Authoring




# Authoring Pipelines

This page covers the model and conventions for authoring alloy pipelines as YAML under `packages/alloy-pipelines/`. The runtime aspects (label families, retention) live in [Logging](/materialize-monitoring/reference/development/pipelines/logging/) and [Metrics](/materialize-monitoring/reference/development/pipelines/metrics/).

## Pipeline model

A pipeline is a YAML document that maps roughly 1:1 to an alloy config file:

```yaml
description: |
  What this pipeline does, why it exists, what it forwards to.
logging:
  level: info
  format: logfmt
blocks:
  - loki.process:
      label: inputProcessor
      forward_to: [{ref: "loki.write.gateway.receiver"}]
      blocks:
        - stage.drop:
            older_than: "12h"
            drop_counter_reason: "backlog > 12hr"
        - stage.match:
            selector: '{app="alloy"}'
            blocks:
              - stage.logfmt:
                  mapping: { msg: msg, level: level }
```

Each entry in `blocks` is a **single-key object** whose key is either:

- a typed component name (`loki.process`, `loki.echo`, `discovery.kubernetes`, ...), validated by the per-component schema;
- `declare` or `custom`, which define and instantiate a custom component (see [Custom components](#custom-components-declare)); or
- the literal key `raw`, validated by `raw.schema.yaml` and used for anything the typed schemas don't cover.

The same rule applies recursively to nested sub-blocks (e.g. stages inside `loki.process`, rules inside `discovery.relabel`).

**Typed blocks are flat** — attributes live directly under the discriminator key alongside `label` and (where applicable) `blocks`. **Raw blocks keep the `component / label / attributes / blocks` partition** because raw needs `component` as its own discriminator and serves as a generic container.

## The strict-attributes / raw-escape policy

Each typed component declares `additionalProperties: false` on the block body. **Undocumented attributes are rejected by design.**

When the validator says

```
schema violation at `/blocks/0/loki.process`:
  Additional properties are not allowed ('drop_malformed' was unexpected)
  hint: this key isn't typed in the schema. Either use a `raw:` block
        for one-off usage, or extend the relevant schema $def to add it.
```

…you have two choices, depending on whether the attribute is reusable:

1. **Extend the schema** — preferred when the attribute is something the team will plausibly use again. Edit the relevant `$def` in `packages/mzmon-lib/schemas/alloy/loki.schema.yaml` (or `discovery.schema.yaml`), add a `properties` entry with a `description`, an optional `examples`, and a `type`. The validator picks it up at build time (the schema is `include_str!`-embedded), and the attribute now shows up in IDE autocomplete and in reference docs.

2. **Use a `raw:` block** — preferred for one-off or experimental usage. Replace the typed block entirely with the raw form:

   ```yaml
   - raw:
       component: stage.json
       attributes:
         expressions: { msg: message }
         drop_malformed: true   # undocumented in our schema; raw bypasses validation
   ```

   `raw:` is **all-or-nothing per block** — you can't mix typed attributes with `raw:` attributes inside the same block. If you only need one undocumented attribute, you have to write the whole block raw.

When in doubt, prefer `raw:` and graduate to the schema if it becomes a pattern. The schemas are meant to grow with usage, not to mirror alloy upstream exhaustively.

### Why strict?

The trade-off is deliberate and worth understanding:

| Strict (current) | Loose |
|---|---|
| Catches typos (`older_then`) immediately | Typos slip through; caught only by `alloy validate` later, if at all |
| Schema is a contract: documented == accepted | Schema is a hint: documented ⊂ accepted |
| Forces explicit `raw:` for anything undocumented | Lets undocumented attributes pass silently |
| Schema is the source of truth for reference docs | Reference docs may not reflect what's actually accepted |

We chose strict because the schemas double as reference documentation. Loose would let the docs drift from reality silently; strict makes the gap explicit at validation time.

## Schema layout

```
packages/mzmon-lib/schemas/alloy/
├── mzmon-alloy.schema.yaml    # entry point ($ref top + future imports)
├── top.schema.yaml            # description / logging / livedebugging / blocks
├── raw.schema.yaml            # the {raw: <block>} escape hatch + AST primitives
├── loki.schema.yaml           # loki.*, plus stage.* $defs and the shared `rule` $def
└── discovery.schema.yaml      # discovery.*, cross-references `rule` from loki
```

Each component file declares its own `$id` URL. The validator (`packages/mzmon-lib/src/alloy/validate.rs`) registers all schemas in a `referencing::Registry` so cross-file `$ref`s resolve at runtime against the embedded copies, with no network or filesystem dependency.

### Cross-file `$ref`s

Cross-file references use a relative path (`./loki.schema.yaml#/$defs/ruleBlock`) which the registry resolves through `$id`. The same path also works in the IDE — `yaml-language-server` follows relative paths directly. **Don't use absolute URLs in `$ref`s**; they only work if `$id`s match exactly and break the local IDE experience.

A relative `$ref` also resolves against the referring document's **`$id`**, not its filename — so a file's `$id` tail **must match its path** (e.g. `common/raw.schema.yaml` must declare `.../common/raw.schema.yaml`, not `.../common/block.schema.yaml`). A mismatch makes every `$ref` to that file point at a URI no schema claims; the Rust validator papers over it (it registers each resource under an explicit key in `validate.rs`), but `yaml-language-server` keys on the in-file `$id` and breaks. Keep the filename, the `$id` tail, and the `ID_*` constant in `validate.rs` in lockstep.

### Known tooling issue: yaml-language-server "Maximum call stack size exceeded"

If your editor reports `Request textDocument/hover failed with message: Maximum call stack size exceeded` (or the same on completion/validation) while editing a pipeline YAML, **it is not your YAML or our schemas** — it's an upstream `yaml-language-server` bug, fixed by [redhat-developer/yaml-language-server#1269](https://github.com/redhat-developer/yaml-language-server/pull/1269).

The server meta-validates the *bound JSON Schema itself* against its draft meta-schema (`additionalProperties: {$ref: "#"}`), and that meta-validation has no termination guard, so it overflows on a sufficiently recursive/self-referential schema. Ours are recursive **by necessity** — `stageBlock → stage.match → stageBlock`, and the `attributeValue`/`expression`/`block` cluster — so they trip it.

- **Don't flatten the recursion to appease the IDE.** It's inherent to alloy's nesting; the schema is correct.
- **The CLI is the source of truth.** The Rust validator resolves the same recursion fine, so `make pipelines` + `cargo test -p mzmon-lib` passing means the schema is good regardless of what the editor says.
- **Fix:** update `yaml-language-server` past the patch, or locally patch the AJV meta-validation call to degrade gracefully.
- **Debugging tip** (if you ever need to confirm it's this and not your content): the overflow is invisible to a debugger — V8 can't pause with no stack room — and the server rebinds `console.*` to the LSP channel, so logs vanish. Set `Error.stackTraceLimit = Infinity` and `.catch` the failing request to write `e.stack` to `process.stderr`, which bypasses the rebind.

## Extending the schema

Three recipes, in increasing scope: add an attribute, add a typed sub-block, add a whole component. All three follow the same pattern: schema first (so validation + IDE hover get the change), Rust struct + `ToBlock` if it's a typed sugar variant, then a round-trip test colocated with the impl.

### Adding an attribute to an existing component

Say `drop_malformed` on `stage.json`:

1. Open `packages/mzmon-lib/schemas/alloy/loki.schema.yaml`.
2. Find the `stage.json` `$def` under `$defs`.
3. Add the attribute alongside the existing properties (typed schemas are flat — no `attributes:` nesting):

   ```yaml
   stage.json:
     # ...
     type: object
     properties:
       expressions: { ... }
       source: { ... }
       drop_malformed:                  # ← new
         description: |
           When true, entries with unparseable JSON are dropped.
         type: boolean
         examples: [true]
     required: [expressions]
     additionalProperties: false
   ```

4. If the component has a Rust sugar struct (e.g. `LokiSourceJournalBlock`), add the corresponding field with `#[serde(default, skip_serializing_if = "Option::is_none")]` and extend `to_block` to emit the attribute when set. (For schema-only stages without Rust sugar, you can stop here — the schema documents the attribute and validation enforces the type.)
5. Add or extend a round-trip test (preferably colocated with the Rust impl in `components/*.rs`, or in `pipeline.rs` for top-level concerns).
6. Run `cargo test -p mzmon-lib`, `cargo fmt`, and `make pipelines`.

### Adding a typed sub-block

The `discovery.kubernetes`'s `selectors` and `attach_metadata` blocks are the canonical example. Pattern:

1. **Schema** — in `discovery.schema.yaml`, the component's `blocks.items` refs a `<componentName>SubBlock` `$def` (a `oneOf` over typed branches + a final `raw` escape):
   ```yaml
   kubernetesSubBlock:
     oneOf:
       - type: object
         properties:
           selectors:       { $ref: "#/$defs/selectors" }
         required: [selectors]
         additionalProperties: false
       - type: object
         properties:
           attach_metadata: { $ref: "#/$defs/attach_metadata" }
         required: [attach_metadata]
         additionalProperties: false
       - $ref: "./raw.schema.yaml"     # ← always the last branch
   ```
2. **`$def` for each sub-block** — typed properties, `required`, `additionalProperties: false`, a `See:` link to canonical alloy docs.
3. **Rust struct** — flat, `#[serde(deny_unknown_fields)]`. One per typed sub-block (e.g. `DiscoveryKubernetesSelector`).
4. **`impl ToBlock`** — populate an `IndexMap<String, AttributeValue>`, return `Block { component: "<sub-block name>", label: None, attributes, blocks: Vec::new() }`.
5. **Sub-block enum** — extend the component's existing `<Component>SubBlock` enum with a new variant, `#[serde(rename = "<yaml-key>")]`:
   ```rust
   pub enum KubernetesSubBlock {
       #[serde(rename = "selectors")]
       Selectors(DiscoveryKubernetesSelector),
       #[serde(rename = "attach_metadata")]
       AttachMetadata(DiscoveryKubernetesAttachMetadata),
       #[serde(rename = "raw")]
       Raw(Block),
   }
   ```
6. **Dispatch in the enum's `ToBlock`** — add a match arm.
7. **Test** — a round-trip test through `Pipeline::from_yaml_str` that exercises the new sub-block, colocated in the same `components/*.rs` file.

Sub-block enums *always* keep `Raw(Block)` as a fallback so unsupported sub-blocks don't require a schema change.

### Adding a typed component

Adding a whole new typed component (e.g. a `prometheus.*` family) is the same shape one level up:

1. **New schema file** `prometheus.schema.yaml`, registered in `validate.rs` (a fresh `ID_PROMETHEUS` constant + an entry in the `Registry::extend` list).
2. Add a branch to `top.schema.yaml`'s `blocks.items.oneOf` pointing at the new file.
3. **Rust struct + `ToBlock`** in `components/prometheus.rs`.
4. **Variant in `ComponentBlock`** (`pipeline.rs`) with `#[serde(rename = "<exact-component-name>")]`.
5. **Dispatch in `ComponentBlock::to_block`**.
6. Tests colocated in `components/prometheus.rs`.

Mirror the `loki.schema.yaml` / `components/loki.rs` structure as a template.

## Custom components (`declare`)

A custom component is how a pre-rendered pipeline holds the shape of something whose count is only known at install.
The pipeline defines the component once with `declare`, and the chart renders one instance of it per resource named in values.
The first user is the cloud provider pull in `packages/alloy-pipelines/gateway-provider.yaml`; [Metrics](/materialize-monitoring/reference/development/pipelines/metrics/#cloud-provider-pulls) covers what it does.

The alternative is Helm rendering the components themselves, as the destination still does.
That output never passes through the schema, and a build-time stub standing in for it drifts from the template it imitates.
A custom component moves the component bodies into the pipeline, where they are typed and validated, and leaves the chart with a flat block of arguments per instance.
`alloy validate` checks those arguments against the `declare`: a misspelled or missing argument fails the pre-validate job with the line that names it.

```yaml
blocks:
  - declare:
      label: provider_cloudwatch_rds
      blocks:
        - argument: {label: instance_id}
        - argument: {label: region}
        - prometheus.exporter.cloudwatch:
            label: rds
            sts_region: {ref: argument.region.value}
            blocks:
              - static:
                  label: rds
                  regions: [{ref: argument.region.value}]
                  namespace: AWS/RDS
                  dimensions:
                    DBInstanceIdentifier: {ref: argument.instance_id.value}
                  blocks:
                    - metric: {name: CPUUtilization, statistics: [Average], period: 5m, length: 10m}
        - custom:
            component: provider_scrape
            label: rds
            attributes:
              targets: {ref: prometheus.exporter.cloudwatch.rds.targets}
              instance: {ref: argument.instance_id.value}
```

| Node | Renders | Rules |
|---|---|---|
| `declare` | `declare "<label>" { ... }` | The label is an identifier with no dots. The body is ordinary blocks plus `argument` blocks, and may instantiate other custom components declared in the same file |
| `argument` | `argument "<label>" { optional, default, comment }` | Only valid inside a `declare` body; the schema refuses it anywhere else. Read in the body as `{ref: argument.<label>.value}` |
| `custom` | `<component> "<label>" { <attributes> }` | Instantiates a custom component. `component` has no dots, so this cannot emit an untyped built-in the way `raw:` can. Whether the component exists, and whether its arguments match, is left to `alloy validate` |

What a custom component can and cannot vary:

- **An argument is only ever a value.** A `declare` body cannot repeat, add or omit a block because of an argument. Anything that varies in block structure stays with the caller, which is why the provider pull takes one CloudWatch instance per resource and fixes its metric set in the body.
- **Fields a body passes arguments into have to accept expressions.** Typed fields are widened as bodies need them: a scalar becomes `Expressable`, a list becomes `ExpressableList` (whose `Items` variant holds a literal list with expression members, `[{ref: argument.region.value}]`), and a map's values become `Expressable<String>`. Widen the schema in the same change, per the lockstep invariant.
- **A receiver passes as a single value, not a list.** The body writes `forward_to: ["argument.forward_to.value"]`, which renders as a one-element list of the argument, and the caller passes one receiver. A caller passing a list would give the scrape a list of lists.
- **A body can process its own output before the caller's receiver.**
  `provider_azure_postgres` passes a nested `provider_scrape` the receiver of a `prometheus.relabel` declared in the same body.
  Only that relabel forwards to `argument.forward_to.value`.
  Order does not matter inside a body, any more than at top level.
- **Instances are top-level components, so their labels are identifiers and unique.** Whatever derives a label from a resource name maps other characters to `_` and prefixes a letter, and something has to refuse two names that collide: Alloy refuses a duplicate label at load, which `alloy validate` does not catch.
- **`alloy validate` checks the body's shape, not its values.** Component validation runs only when Alloy builds the component, so a body can pass validation and still fail to load when an instance supplies, say, a scrape timeout longer than its interval. Whatever renders the instances validates those values itself.

A deployed pipeline of custom components is inert until something instantiates it, so it can be included in every install unconditionally.
The instances the chart renders need a build-time stand-in only for their argument names: a stub that instantiates each component once, validated joined with the module, as `gateway-provider-stub.yaml` does.

## Reference-valued attributes

A real foot-gun worth its own section. Some component attributes are *references to other components' exports*, NOT string literals:

- `loki.process.forward_to` — array of `loki.write.*` receivers
- `loki.source.journal.forward_to` — same
- `loki.relabel.forward_to` — same
- `discovery.relabel.targets` — refs to another `discovery.*` component's `.targets`
- `loki.source.journal.relabel_rules` — ref to a `discovery.relabel` rule set

These render in alloy as **bare identifiers**:

```alloy
forward_to = [loki.write.gateway.receiver]   // ← bare ref, NOT "loki.write.gateway.receiver"
targets    = [discovery.kubernetes.pods.targets]
```

Wrapping a `TargetRef` (a `String` alias in Rust) as `AttributeValue::String` would render it *quoted* — which is syntactically valid alloy but semantically wrong (alloy would treat it as a string literal, not a component reference). The pipeline would silently misbehave at runtime.

**The fix in `to_block`**: wrap each `TargetRef` as an `Expression::ref_name`:

```rust
AttributeValue::Array(
    self.forward_to
        .iter()
        .map(|s| AttributeValue::Expression(Expression {
            ref_name: Some(s.clone()),
            ..Default::default()
        }))
        .collect(),
)
```

When a component has even one ref-valued attribute, walk every `to_block` site for that struct and make sure the wrap is in place. The renderer produces bare refs only for `AttributeValue::Expression { ref_name: Some(_) }`.

## Load-bearing invariants

These are *non-obvious things that must stay true* — flagged here so future refactors don't silently break them. Each has a regression test in `ast.rs` that pins the behavior.

### `AttributeValue` variant order

The `AttributeValue` enum in `ast.rs` uses `#[serde(untagged)]`. Serde tries variants top-to-bottom and picks the first that deserializes. **The current order is load-bearing:**

```rust
#[serde(untagged)]
pub enum AttributeValue {
    Null,
    Bool(bool),
    Number(f64),
    String(String),
    Array(Vec<AttributeValue>),
    Expression(Expression),
    Object(IndexMap<Identifier, AttributeValue>),
}
```

Two specific rules:

- **`String` and `Array` MUST come before `Expression`.** Serde's struct deserializer accepts a sequence by *positional field assignment* by default. Without this order, `["a", "b"]` would silently deserialize as `Expression { raw: Some("a"), env: Some("b"), ... }`, then fail at render time with "Too many expressions" (or worse, succeed and render wrong).
- **`Expression` must come before `Object`.** Otherwise the catch-all map would swallow expression-shaped objects (`{ref: "..."}`, `{env: "..."}`) before they can be recognized.

The `deny_unknown_fields` attribute on `Expression` is the third piece: without it, a generic object like `{mapping: ...}` would silently match `Expression` (all heads `None`, unknown field ignored) instead of falling through to `Object`.

### `raw:` is always the last `oneOf` branch

Every sub-block `oneOf` ends with `$ref: "./raw.schema.yaml"`. New typed branches go *before* the raw branch, never after. The raw escape is the contract — without it, undocumented sub-blocks have nowhere to go.

### Schemas-as-docs

The schemas double as reference documentation; `description` text renders in IDE hover and (eventually) in the published reference site. Keep descriptions user-facing, concise (first line ≤80 chars), and link to canonical alloy upstream via a trailing `See:` line. When alloy upstream renames a field, update both the description and the `properties` entry — leave the `See:` link stable so future contributors can audit drift.

### Renderer alignment quirk

The renderer's block-attribute alignment rule is *currently* "any multi-line value disables `=`-alignment for the surrounding attribute group." This is more aggressive than `alloy fmt`'s actual rule, which aligns in more cases. Tests sometimes need to work around this — e.g. by writing rule blocks with one attribute each — until the rule is refined. The investigation log lives on the renderer-alignment task. If your test's `assert_renders` fails on a single attribute getting unexpectedly padded (or unpadded), suspect this quirk before suspecting the rendered output itself.

## Sub-block recursion and the `raw` escape

Sub-block lists (a `loki.process` body, a `stage.match` body, a `loki.relabel`'s `rule` list, a `discovery.kubernetes` body) are typed as a `oneOf` whose **last branch is always `$ref: "./raw.schema.yaml"`** (or the cross-file equivalent). This is non-negotiable in the design: every sub-block context has an escape hatch. New typed branches are added before the `raw` branch so the escape stays as the final fallback.

`stage.match` is recursive: its body refs `#/$defs/stageBlock`, which itself includes `stage.match`. JSONSchema handles the cycle natively via `$ref`.

## Linking to canonical alloy docs

Each `$def` description ends with a `See:` line linking to the canonical Grafana alloy documentation:

```yaml
description: |
  Drops log entries matching the configured condition.

  See: https://grafana.com/docs/alloy/latest/reference/components/loki/loki.process/#stagedrop-block
```

When alloy upstream renames a field or changes semantics, **update the schema's `description` and `properties` to match**, leaving the `See:` link stable. The schema is a snapshot, not a mirror; explicitly noting the drift on the link's destination is part of the contributor workflow.

## How validation interacts with rendering

The `Pipeline::from_yaml_str` entry point does three things in order:

1. Parses the YAML into a generic `serde_json::Value`.
2. Validates the value against the embedded JSONSchema, collecting *all* violations into `Error::Multiple`.
3. Deserializes the value into the typed `Pipeline` struct (serde, externally-tagged enums).

This means schema errors fire **before** any serde decoding errors, and any single document can report multiple problems in one pass. The renderer (`Pipeline::render`) then produces canonical `config.alloy` output verified by the `alloy fmt` oracle in tests; see `mzmon-lib/src/alloy/test_support.rs`.

## CLI

`mz-monitoring-build gen-pipelines` is the entry point:

```
mz-monitoring-build gen-pipelines \
    --output-dir charts/materialize-monitoring/pre-rendered/pipelines \
    --target gateway        # optional; defaults to all *.yaml in --input-dir
```

The Makefile target `make pipelines` invokes this once per target and then runs `alloy validate` on the rendered output as a second-layer sanity check. Both layers should be green before merging.

### Schemas are embedded at compile time

The schemas are baked into the binary with `include_str!` (see `validate.rs`). **After editing any `schemas/alloy/*.yaml`, rebuild the binary (`cargo build --bin mz-monitoring-build`, or just use `make pipelines`, which depends on it) before a manual `gen-pipelines`.** Otherwise the renderer validates against the *stale* embedded schema and reports confusing "doesn't match any typed schema" errors for blocks that are actually valid. Note the asymmetry: `cargo test` recompiles the library, so unit tests pick up schema edits immediately — meaning **tests can pass while a manual `gen-pipelines` of the same construct fails** purely because the binary is stale. If a render rejects something your tests accept, rebuild first.

A brand-new schema *file* additionally needs registering in `validate.rs`: a `SCHEMA_*` `include_str!`, an `ID_*` constant whose value matches the file's `$id` exactly, and an entry in the `Registry` list. Keep the filename, the `$id` tail, the relative `$ref`s that target it, and the `ID_*` constant all in agreement (see [Cross-file `$ref`s](#cross-file-refs) above).