Internal Development on materialize-monitoring Documentation

Repo Layout

Mon, 01 Jan 0001 00:00:00 +0000

This page is a convenience cache of where things live in the repo. The repository is under active development, so this layout goes stale easily. If it disagrees with what is actually in the tree, trust the repo and update this page.

The repo is a polyglot monorepo: a Rust workspace, a uv-managed Python workspace, and a Go-tooled Hugo docsite all share the root.

materialize-monitoring/
- Cargo.toml / Cargo.lock: top-level Rust workspace (members under packages/)
- pyproject.toml / uv.lock / .python-version: Python workspace, managed by uv
- go.mod / go.sum: Go module that pins hugo and helm-docs via tool directives
- Makefile: top-level entrypoint (make all, charts, dashboards, helm-docs, serve-docs)
- .pre-commit-config.yaml: contributor-experience hooks (see Internal Development)
- bin/: bash dev/CI entrypoints (flat; no subdirectories)
  - check-lfs.sh: verify/repair Git LFS state
  - dashboard-sync.sh: regenerate and sync dashboards
  - grafonnet-render: grafonnet rendering helper
  - mz-monitoring-build / mz-monitoring-check: thin wrappers over the Rust binaries
- packages/: hand-authored, contributor-facing inputs
  - grafana-dashboards/: Grafana dashboards-as-code (Python + grafana-foundation-sdk); sources under dashboards/ (e.g. mz_environment/, render.py, palette.py)
  - py-mzmon-lib/: Python helper library imported by the dashboard packages; not consumed by customers
  - alloy-pipelines/: Alloy pipeline YAML inputs (agent.yaml, gateway.yaml)
  - ref-alloy-pipelines/: Python reference pipelines being ported into the typed Rust path (agent_config.py, gateway_config.py, processor.alloy, alloy/)
  - mzmon-lib/: Rust library — typed Alloy model plus embedded JSONSchemas under schemas/alloy/; not consumed by customers
  - mz-monitoring-build/: Rust CLI for artifact generation (gen_pipelines.rs, main.rs)
  - mz-monitoring-check/: Rust schema/consistency checks
- charts/
  - materialize-monitoring/: umbrella chart
    - Chart.yaml / Chart.lock: chart metadata; lock pins subchart versions
    - values.yaml / README.md / README.md.gotmpl: profile-driven defaults; README generated from the template via helm-docs
    - charts/: vendored subchart tarballs (LFS) — alloy, loki, thanos, alertmanager, grafana, grafana-operator, kube-state-metrics, metrics-server
    - pre-rendered/: generated artifacts loaded via {{ .Files.Get }}; never hand-edited
      - dashboards/: grafana/ and datadog/ JSON
      - pipelines/: rendered Alloy (agent.alloy, gateway.alloy)
      - rules/: prometheus/, loki/, thanos/
    - templates/: provided resources — alerts/, dashboards/, pipelines/, scrapers/, plus grafana-grafana.yaml and _helpers.tpl
    - examples/: example values overlays (e.g. existing-grafana, IRSA/S3)
    - profiles/: profile value sets (still being defined)
  - materialize-monitoring-crds/: CRDs chart (Chart.yaml, values.yaml, README.md)
- docs/: Hugo docsite (the source of this page)
  - hugo.toml: site config
  - content/: authored Markdown
    - top-level sections: getting-started/, metrics/ (incl. collecting/), logs-and-events/, dashboards/ (incl. grafana/), alerting/, operating/
    - reference/: helm/, stable-metrics/, and internal/ (this section — dashboard/, pipelines/, design-docs/, plus repo-layout.md, roadmap.md, releasing.md, skills.md, helm.md)
  - layouts/, static/, assets/, data/, i18n/, archetypes/, themes/: Hugo machinery
  - public/, resources/: generated output (not checked in)
- legacy/: preserved field-engineering assets — sql_exporter/, prometheus/, grafana/, datadog/, tests/, docker-compose.yml
- tools/: ancillary ecosystems kept out of bin/
  - chartlib/: helm-docs templates
  - grafonnet/: grafonnet/jsonnet vendoring (jsonnetfile.json + lock)
  - shlib/: shared bash helpers
- .claude/skills/: authoring conventions consumed by both contributors and AI agents
- .github/: GitHub Actions workflows

Skills

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Helm

Mon, 01 Jan 0001 00:00:00 +0000

TODO

Roadmap

Mon, 01 Jan 0001 00:00:00 +0000

Roadmap#

The goal of materialize-monitoring is first-class, opt-in observability for self-managed Materialize — logs, metrics, events, and alerts — for customers who want a one-stop-shop, without forcing our stack on customers who already run their own. This page is the current source of truth for what is built, what is in flight, and what is planned next. It supersedes the original Linear project (internal), which captured an earlier architecture that has since diverged (see below).

Releasing

Mon, 01 Jan 0001 00:00:00 +0000

Releasing#

Each artifact releases on its own version stream (see Versioning).

Changes-pending placeholders#

The top of CHANGELOG.md holds one unreleased placeholder per component that has changes, with the body _Changes Pending_. Placeholders are not populated on main — population happens in a version-update PR (below). A component with no changes since its last release has no placeholder, so the unreleased placeholders are not necessarily the very top entries and rarely-changed components are not churned every cycle.

Versioning

Mon, 01 Jan 0001 00:00:00 +0000

Versioning in materialize-monitoring#

All artifacts of this repo have separate versioning streams. They generally follow semantic versioning (SemVer).

Components#

Components are defined in packages/components.yaml. Each component declares:

changelog — whether the component maintains its own changelog stream.
title — the human-readable name used in CHANGELOG.md headings.
version_paths — files whose version field is rewritten when the component is bumped (the write targets).
content_paths — the paths whose changes are attributed to this component (the attribution inputs).
content_exclude — paths to subtract from content_paths, typically generated outputs that belong to a dependency.
dependencies — other components whose bumps cascade into this one.
artifacts — glob patterns (repo-root-relative) for files attached as GitHub Release assets when the component is published; resolved file names must be unique.

Each changed file is attributed to the component with the longest matching content_paths entry, after dropping any component that excludes it. Generated outputs route to their source: the chart excludes its pre-rendered/ tree, and pre-rendered/dashboards and pre-rendered/pipelines are claimed by the dashboards and pipelines components, so a dashboard change appears under Dashboards (and rolls up into the chart via cascade) rather than as a first-class chart change.