https://materializeinc.github.io/materialize-monitoring/reference/internal/pipelines/Recent content in Pipelines on materialize-monitoring DocumentationHugoen-ushttps://materializeinc.github.io/materialize-monitoring/reference/internal/pipelines/authoring/Mon, 01 Jan 0001 00:00:00 +0000https://materializeinc.github.io/materialize-monitoring/reference/internal/pipelines/authoring/<h1 id="authoring-pipelines">Authoring Pipelines<a class="anchor" href="#authoring-pipelines">#</a></h1> <p>This page covers the model and conventions for authoring alloy pipelines as YAML under <code>packages/alloy-pipelines/</code>. The runtime aspects (label families, retention) live in <a href="https://materializeinc.github.io/materialize-monitoring/reference/internal/pipelines/logging/">Logging</a> and <a href="https://materializeinc.github.io/materialize-monitoring/reference/internal/pipelines/metrics/">Metrics</a>.</p> <h2 id="pipeline-model">Pipeline model<a class="anchor" href="#pipeline-model">#</a></h2> <p>A pipeline is a YAML document that maps roughly 1:1 to an alloy config file:</p> <div class="highlight"><pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;-webkit-text-size-adjust:none;"><code class="language-yaml" data-lang="yaml"><span style="display:flex;"><span><span style="color:#f92672">description</span>: |<span style="color:#e6db74"> </span></span></span><span style="display:flex;"><span><span style="color:#e6db74"> What this pipeline does, why it exists, what it forwards to.</span> </span></span><span style="display:flex;"><span><span style="color:#f92672">logging</span>: </span></span><span style="display:flex;"><span> <span style="color:#f92672">level</span>: <span style="color:#ae81ff">info</span> </span></span><span style="display:flex;"><span> <span style="color:#f92672">format</span>: <span style="color:#ae81ff">logfmt</span> </span></span><span style="display:flex;"><span><span style="color:#f92672">blocks</span>: </span></span><span style="display:flex;"><span> - <span style="color:#f92672">loki.process</span>: </span></span><span style="display:flex;"><span> <span style="color:#f92672">label</span>: <span style="color:#ae81ff">input_processor</span> </span></span><span style="display:flex;"><span> <span style="color:#f92672">forward_to</span>: [{<span style="color:#f92672">ref</span>: <span style="color:#e6db74">&#34;loki.write.gateway.receiver&#34;</span>}] </span></span><span style="display:flex;"><span> <span style="color:#f92672">blocks</span>: </span></span><span style="display:flex;"><span> - <span style="color:#f92672">stage.drop</span>: </span></span><span style="display:flex;"><span> <span style="color:#f92672">older_than</span>: <span style="color:#e6db74">&#34;12h&#34;</span> </span></span><span style="display:flex;"><span> <span style="color:#f92672">drop_counter_reason</span>: <span style="color:#e6db74">&#34;backlog &gt; 12hr&#34;</span> </span></span><span style="display:flex;"><span> - <span style="color:#f92672">stage.match</span>: </span></span><span style="display:flex;"><span> <span style="color:#f92672">selector</span>: <span style="color:#e6db74">&#39;{app=&#34;alloy&#34;}&#39;</span> </span></span><span style="display:flex;"><span> <span style="color:#f92672">blocks</span>: </span></span><span style="display:flex;"><span> - <span style="color:#f92672">stage.logfmt</span>: </span></span><span style="display:flex;"><span> <span style="color:#f92672">mapping</span>: { <span style="color:#f92672">msg: msg, level</span>: <span style="color:#ae81ff">level }</span></span></span></code></pre></div><p>Each entry in <code>blocks</code> is a <strong>single-key object</strong> whose key is either:</p>https://materializeinc.github.io/materialize-monitoring/reference/internal/pipelines/metrics/Mon, 01 Jan 0001 00:00:00 +0000https://materializeinc.github.io/materialize-monitoring/reference/internal/pipelines/metrics/<p>TODO</p>https://materializeinc.github.io/materialize-monitoring/reference/internal/pipelines/logging/Mon, 01 Jan 0001 00:00:00 +0000https://materializeinc.github.io/materialize-monitoring/reference/internal/pipelines/logging/<p>TODO</p>