Secret Filtering: A Last Line of Defense for Stored and Egressed Logs#

This doc proposes a last line of defense against credentials in logs. Every log line the gateway receives passes a secret filter before it is parsed or stored. Every line bound for a Materialize control plane, over BYOC or call-home, passes a second and stricter filter on its own branch, immediately before the writer. It is the credentials half of DEP-220. The BYOC design applies pattern redaction “as defense in depth rather than as the primary control” and leaves it unspecified; this doc specifies it.

Alloy promoted loki.secretfilter to general availability in v1.20.0. The component wraps the Gitleaks detector and its default rule set, so the mechanism is a dependency rather than a project. What this doc settles is where the filter runs, which rules it runs, what it deliberately ignores, how anyone can tell that it is working, and which log levels may leave the cluster at all.

The central claim is that the filter has to run before the pipeline reads the line, and it has to prove continuously that it ran. loki.secretfilter scans the log line and nothing else. The gateway’s first act of parsing is to copy substrings of that line into structured metadata and labels. A filter placed after parsing therefore redacts the line and leaves the copies. A filter that a refactor has routed around looks exactly like a filter with nothing to redact, which is why the design carries a canary.

A second claim shapes the rules. A false negative and a false positive cost different amounts on each side of the boundary, so the two sides run different rule sets. A credential missed in the customer’s own Loki is readable by everyone with Loki access for the retention period. A credential missed on the egress branch is in another company’s systems, where the customer cannot recall it. The customer-side filter is tuned so that redactions stay rare enough to be read, and the egress filter trades more false positives for fewer misses.

The rule choices below are measured rather than argued. A corpus of Materialize-shaped log lines was run through the Gitleaks v8.30.1 detector that Alloy v1.20.0 embeds, with decoding off as Alloy leaves it. The default rules miss the metadata-database URL, Authorization headers, presigned-URL signatures, a truncated private key, and any credential inside an escaped JSON string, which is where a Rust Debug dump lands in a JSON log.

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119.

The keywords carry the obligations an implementation takes on. They appear on the placement of each filter, on its configuration, and on the handful of settings whose obvious value fails open.

Goals#

Functional requirements, framed as value-first user stories. Each describes what a user needs and why it matters; the Technical BLUF and the sections below describe how it is delivered. Priority tags (Must / Should / Could) are relative to the first shipped version.

Five stakeholder classes consume this:

• Customers’ security and compliance reviewers, who decide whether logs may be stored centrally and whether any may leave.

• Operators of an install, who debug with the same logs the filter edits.

• Materialize support and SRE, who read the egressed copy during an escalation.

• Materialize security, who are accountable for what the control plane holds.

• Maintainers of this repo, who upgrade Alloy on Renovate’s schedule rather than on the rule set’s.

• [Must] As a customer’s security reviewer, I want every log line to pass a credential filter before it is stored, so that a token a component logs by mistake is not searchable in Loki for the full retention.

• [Must] As a customer’s security reviewer, I want the copy that leaves for Materialize to pass a stricter filter than the copy that stays, so that the disclosure I agreed to is not widened by a logging bug.

• [Must] As a customer’s security reviewer, I want to be told when the filter redacted something, so that the component is fixed at the source and the credential is rotated.

• [Must] As an operator, I want continuous proof that the filter is in the path, so that a change that routes around it is noticed in minutes rather than at an audit.

• [Must] As an operator, I want the filter never to be the reason logs queue, so that protecting some lines never costs me all of them.

• [Must] As an operator, I want the identifiers Materialize logs constantly left intact, so that redaction does not destroy the correlation I debug with.

• [Must] As Materialize support, I want the placeholder to name what was redacted, so that “a credential was here” is still evidence.

• [Must] As a maintainer, I want an Alloy upgrade that changes the default rules to fail CI when it loses a detection or gains a false positive on our corpus, so that detection does not drift on a dependency bump.

• [Should] As Materialize security, I want a backstop at the control-plane ingress that Materialize can update without a customer upgrade, so that version skew does not decide what Materialize holds.

• [Should] As Materialize support on an escalation, I want DEBUG and TRACE from named components for a bounded window, so that a hard case is tractable without a standing increase in what crosses.

• [Should] As a customer’s security reviewer, I want to read the rules that run, so that “secrets are redacted” is a file I can review rather than a sentence.

• [Could] As an operator, I want to add rules for my own credential formats, so that secrets specific to my environment are covered by the same mechanism.

Technical BLUF#

• loki.secretfilter is GA in Alloy v1.20.0, at release candidate as of this writing, and public preview in the pinned v1.19.2. Public-preview components are acceptable here, and the chart sets stabilityLevel to the highest level its components allow. v1.20.0 is expected to ship before this design is implemented, so the gateway is expected to stay at generally-available. If it does not, the gateway runs at public-preview until it does.
• It scans entry.Line and nothing else. Labels and structured metadata are never scanned. The gateway copies msg, error, path, query, flow and more out of the line into structured metadata, so the ingest filter runs after the multi-line merge and before any parsing, and everything downstream is derived from redacted text.
• Credential filtering is the one redaction that attaches to the pipeline. BYOC attaches redaction to the destination so that the customer’s copy is never degraded to protect Materialize’s. A credential in the customer’s own Loki is the customer’s exposure, so removing it protects their copy rather than degrading it.
• Four layers, three of them here. Redaction at the source is upstream’s. The ingest filter covers every stored line and is on by default. The egress filter runs on each control-plane branch, stricter and fail-closed, and cannot be disabled while that branch exists. The backstop runs at the control-plane ingress on rules Materialize updates on its own schedule.
• The default rules miss the credentials Materialize deployments actually carry. The metadata-database and persist URLs, Authorization headers, presigned-URL signatures, truncated PEM keys, and anything inside an escaped JSON string all pass the Gitleaks defaults. Seven Materialize rules close those gaps on the corpus.
• Identifiers are allowlisted by field name, never by value shape. A bare UUID, an environment ID or a resource-name prefix matches no keyword rule. Where a UUID follows a credential-named key, the name decides. The Gitleaks defaults themselves carry rules for UUID-shaped credentials from Heroku, HubSpot, Snyk and others.
• The egress filter drops the entropy floor on credential-named keys. That floor is the upstream generic rule’s false-positive guard, and it is exactly what lets a human-chosen password through.
• Log queueing is a larger risk than an unscanned DEBUG line. A filter slower than its input back-pressures the whole pipeline and turns into loss on every stream. So DEBUG and TRACE form a bulk tier that MAY be sampled below rate = 1.0, after cheaper levers run out. Nothing sampled ever crosses.
• Payment card numbers are in scope, through stage.luhn. It is a checksum over digit runs and costs little. Left at its defaults it redacts one in ten of the millisecond timestamps Materialize logs as frontiers, so it ships with a skip_regex for epoch timestamps.
• A canary per layer proves each filter runs. Each is a fixed, published string in the spirit of the EICAR test file, matched by a rule of our own and by no third-party scanner, free of every rule keyword, and emitted on a schedule by a purpose-built mz-monitoring-canary through the same path as Materialize’s logs. Detected means the filter is in the path. Found verbatim in a store means a path bypassed it. One canary cannot serve both layers, because the ingest filter redacts it before the egress filter sees it.
• A redaction is a notice alert, and it means the source leaked. The filter protects the stores, not the node. The raw line remains in the container log file, in kubectl logs, and in any other agent reading those files, so the runbook is fix-at-source and rotate.
• The placeholder names the rule and carries no hash. $SECRET_HASH is an unsalted SHA-1 of the secret, which confirms any guess. The default redact_percent leaves the first fifth of the secret in the line.
• TRACE and DEBUG never cross in a standing configuration. They cross inside an elevated window scoped to named components, and the elevation MUST NOT relax the secret filter. This revises the call-home design’s elevated row, which lists “relaxed redaction”.
• Two paths print raw lines today. The gateway’s sampled debug tap is fed raw input in parallel with the processor, and live debugging on the filter itself prints each line before and after redaction.

Non-goals#

• Customer data that is not a credential or a card number. Object names, query text, source and sink names, and other personal data belong to DEP-220’s allowlist and identifier hashing in the BYOC design. Alloy’s own documentation states that PII is out of scope for loki.secretfilter. Card numbers are the exception, because a separate stage redacts them for almost nothing.
• Replacing redaction at the source. Materialize components not logging credentials is upstream work in the materialize repo. This design catches what escapes it and reports the escape.
• The node’s copy. Container runtimes write log files before any collector reads them. Nothing in a log pipeline can redact those files, kubectl logs, or a third-party agent reading them.
• Metrics. A credential-shaped label value on a metric is a different surface, governed by the metric allowlist.
• Recognizing every credential format. A pattern detector cannot find a format it has no rule for. On the egress branch the allowlist bounds exposure to unknown formats; the filter bounds it inside what the allowlist admits.
• Verifying findings. Some scanners test a found credential against its provider. Presenting a leaked credential to a provider is a use of it, and nothing here does that.

What exists today#

Two paths print raw lines today#

The gateway’s loki.source.api forwards every entry to two receivers in parallel: loki.process.inputProcessor and loki.process.sampleDebug. The tap samples about 1% of what it receives and prints it through loki.echo to the gateway’s own stdout. Those echoed lines come back through the agent and are dropped by the alloy loki.echo stage, so they never reach Loki. They do reach the node’s disk and kubectl logs for the gateway pod. Readers of the monitoring namespace’s pod logs are a different RBAC audience from readers of the environment namespaces, so the tap moves a sample of every namespace’s raw lines to a place with its own access policy. The agent defines the same tap, and nothing forwards to it.

Live debugging is the second path. loki.secretfilter publishes each entry to the live-debugging stream as original => redacted, so the component’s own debugging view is a plaintext copy of everything it caught. The option is off by default, and port 12345 is closed by NetworkPolicy to everything but the gateway’s own pods. A kubectl port-forward reaches it anyway, which is one more reason Securing asks operators to restrict pods/portforward in the monitoring namespace.

Today the tap’s cost is the change of audience, and live debugging shows nothing that reading the logs would not. Both become bypasses of the filter the moment one exists, and both are cheaper to close in the same change than to discover afterwards.

What loki.secretfilter does#

Read from the component source at Alloy v1.20.0-rc.0.

What the default rules miss#

The corpus is 38 lines shaped like what this stack and Materialize actually log: tracing JSON from environmentd, logfmt from Go components, header maps from spans, and the identifiers that fill every line. Each line ran through the Gitleaks v8.30.1 defaults and through the draft ingest rules, with decoding off.

The misses share two causes. The generic rule needs a credential-named key, a separator, and an unquoted or plainly quoted value of ten or more high-entropy characters, and each of those conditions excludes a real case. The provider rules recognize provider formats, and a URL, a header and a PEM fragment are not provider formats.

The last three rows are the false positives the ingest rules keep or add. They are the measurement shadow mode exists to take on real logs before any rule enforces.

Four defects found by running the fixes#

Running the fixes back through the detector found problems that review had not.

The third is the one to remember. A rule file that silently loses the default set is the purest false negative this design can produce, because it loads, runs, reports healthy, and redacts only what the custom rules name.

Where each filter runs#

flowchart LR
  subgraph cust["Customer cluster"]
    node[("container log files<br/>not reachable by any filter")]
    agent["alloy-agent<br/>format-agnostic, unchanged"]
    src["gateway sources<br/>loki.source.api · OTLP bridge · k8s events"]
    pre["loki.process.inputPrefilter<br/>multi-line merge · size and age drops · global limit<br/>stage.luhn · level sniff"]
    ing["loki.secretfilter.ingest<br/>full tier · rate 1.0"]
    bulk["loki.secretfilter.ingestBulk<br/>DEBUG and TRACE · MAY sample"]
    ip["loki.process.inputProcessor<br/>parse · structured metadata · labels · level · tenancy"]
    local["egress:local<br/>customer Loki"]
    gate["egress:controlPlane<br/>level allowlist · drop bulk and timed-out<br/>class and body allowlist"]
    egr["loki.secretfilter.egress<br/>strict · fail-closed"]
    attr["metadata allowlist · byte ceiling · writer"]
    node --> agent --> src --> pre
    pre -->|"full tier"| ing --> ip
    pre -->|"bulk tier"| bulk --> ip
    ip --> local
    ip --> gate --> egr --> attr
  end
  subgraph mz["Materialize control plane"]
    back["loki.secretfilter.backstop<br/>Materialize-owned rules"]
    store[("control-plane Loki")]
    back --> store
  end
  attr ==>|"OTLP · mTLS or token"| back

Ingest: after the merge, before the parse#

The ingest filter MUST run before any stage that copies text out of the line. That is the whole reason for its position, and it is structural rather than conventional: the component cannot see structured metadata, so the only way metadata derived from the line is clean is to derive it from a clean line. Placed there, msg, error, path, query and every later template read redacted text, and nothing downstream needs to know a filter exists.

The current inputProcessor is one component, and the filter is a component of its own, so the processor splits in two at its global stage.limit.

Every source MUST forward to inputPrefilter. loki.source.api is rendered by the chart’s source helper and mirrored in the source stub; the OTLP bridge and loki.source.kubernetes_events are in gateway.yaml. The debug tap MUST move behind the filter or be removed. A tap that samples 1% of lines adds little beside the live-debugging view, and removing it also removes a stdout copy that the gateway then has to drop.

Sampling the bulk tier#

Log queueing is the failure to design against, and a slow filter causes it. The filter runs inline in the gateway’s processing path. processing_timeout bounds the time it spends on one line, not the rate at which it clears lines. When scanning falls behind arrival, entries back up into the gateway’s receivers, agent pushes slow down and retry, the agents’ tailing falls behind, and container log rotation eventually removes files nobody has read. When the backlog does drain, the older_than: "2h" drops discard what is left of it. That loss lands on every stream, including the ERROR lines that matter most.

So the ingest filter runs as two instances, and the cheaper levers come first.

The bulk tier is DEBUG and TRACE, which are the most voluminous levels and the ones that do not cross in a standing configuration. The level label does not exist before parsing, so the prefilter sniffs it by matching the level field of the three shapes this stack emits: tracing JSON, tracing’s plain text, and logfmt. The sniff errs toward the full tier. A line it cannot classify is scanned at full rate, so a miss costs CPU rather than coverage.

A line from the bulk tier MUST NOT cross. The component counts the entries it skips without marking them, so the bulk splitter stamps secretfilter="bulk" on every line in the tier, and step 2 of the egress branch drops that label the way it drops timed-out. The egress filter would scan such a line, but its msg copy was derived before the fork from text that may never have been scanned. An elevated window routes its named components to the full tier for its duration, so their DEBUG and TRACE are scanned in full before they can cross.

The bulk instance’s rate defaults to 1.0. Lowering it is a decision for one install, taken from processing_duration_seconds against the arrival rate. entries_bypassed_total on the dashboard row shows what the decision costs.

Why not the agent#

The agent is the earliest point a collector sees a line, and it is still the wrong place.

Egress: last, strict, and fail-closed#

The egress branch is the BYOC per-destination chain, and the filter is one step in it. Order matters, and each step is placed for a reason.

Step 5 uses otelcol.processor.transform, which is GA, rather than otelcol.processor.redaction. The redaction processor’s allowed_keys is exactly the fail-closed semantics this step wants, and it is experimental, a level below the public preview this chart accepts. If it reaches GA it is the better fit, and it adds keyed HMAC hashing that the secret filter lacks.

The egress filter MUST be present on every branch whose destination is outside the customer’s control, and there MUST be no values key that removes it. A validator enforces that, alongside the settings in the table below. The ingest filter MUST be enabled whenever such a branch exists. The egress filter scans only the line, and msg is a copy of part of the line made before the branch forks. A metadata allowlist that keeps the message therefore keeps an unredacted copy of it whenever the ingest filter is off, however strict the egress filter is.

Backstop: the control-plane ingress#

The control-plane gateway runs this repo’s pipeline, so it can run the egress rules too. Its copy is the one Materialize can update without waiting for a customer to upgrade, which answers the version-skew half of BYOC’s open question about which side redacts: both, with the control plane as the backstop.

A detection at the backstop means something different from a detection anywhere else. The credential has already crossed the boundary. That makes it an incident for Materialize security rather than a notice: the line is deleted from the control-plane store and the customer is told. The timeline for telling them belongs beside the support obligation the call-home design already requires. Aggregated across tenants and grouped by the customer-side chart version, the same detections are the fleet’s measured false-negative rate.

Configuration per layer#

The ingest filter forwards a timed-out line rather than dropping it. The raw line is still on the node’s disk, so dropping it at ingest removes the customer’s ability to read it without removing the exposure. The label marks it for the alert and for step 2 of the egress branch.

The placeholder is fixed as <redacted:$SECRET_NAME> and becomes part of the alert contract, since queries match on it. It contains no quote, backslash, space or =, so a redacted value leaves JSON and logfmt parseable. It matches no rule, which the global placeholder allowlist guarantees and a test asserts. It carries no hash, and a keyed one would be welcome if upstream adds it: correlation across lines is useful, and an unsalted SHA-1 of a password is a lookup table away from the password.

A values sketch, for shape rather than as the final surface:

pipeline:
  logging:
    secretFilter:
      ingest:
        # Off is permitted only when no control-plane destination is configured.
        enabled: true
        # Set from the measured p99 of loki_secretfilter_processing_duration_seconds.
        processingTimeout: ""
        # Gitleaks TOML. Rendered with [extend] path = the shipped ingest rules.
        extraConfig: ""
        bulk:
          # DEBUG and TRACE. Below 1.0 only when measured latency would queue the
          # pipeline; nothing in this tier ever crosses.
          rate: 1.0

# The mz-monitoring-canary workload.
canary:
  enabled: true
  interval: 5m

The egress settings have no values surface here. They live in the per-destination processing block BYOC defines, and a control-plane destination gets the strict filter without being asked.

The rules#

Materialize rules on top of the default set#

The ingest file extends the Gitleaks defaults with useDefault = true and adds the following. Every rule declares keywords, so a line containing none of them costs only the shared keyword prefilter.

Two upstream rules are adjusted rather than added. jwt is overridden so that its match cannot end on a backslash. generic-api-key gains a name allowlist, described next.

Whether the license key is covered depends on its encoding. jwt covers a JWT anywhere, and generic-api-key covers the license_key field only for values of 150 characters or fewer. The corpus MUST carry a line in the real license-key format, synthesized, and a license_key field rule joins the set if neither rule catches it.

Identifiers are allowlisted by name, not by shape#

Materialize logs are full of values that look random and are not secret.

The UUID problem is narrower than it looks. The generic rule needs a credential-named key immediately before a separator and the value, and a bare UUID has no key. Almost every identifier above either lives in a label the filter never scans or follows a key like environment_id that names no credential. Under the default rules, three of the corpus’s identifier lines were redacted: a UUID after token_id, a UUID after cache key: in prose, and a pagination cursor.

The Alloy documentation’s example suppresses these by allowlisting UUID-shaped values for generic-api-key. This design rejects value-shape allowlists on any keyword rule. A v4 UUID carries 122 random bits, which makes it a strong bearer credential. The Gitleaks default set carries rules for UUID-shaped credentials from Heroku, HubSpot, Snyk, Squarespace and others. A shape allowlist turns "api_key": "<uuid>" into a miss on the layer whose priority is misses.

Allowlists instead use regexTarget = "match" and name the field. The match text runs from the key through the value, so an entry like token_id exempts a UUID after token_id and nothing else. Each entry MUST name a field whose meaning Materialize knows, and the list MUST be built from shadow-mode measurement rather than written ahead of it.

Two limits of that mechanism shape what can be allowlisted.

• The match text starts at the key. In cache key: <uuid> the match is key: <uuid>, and cache is outside it. An entry cannot distinguish that from key: <uuid> in a credential context, so the finding stays redacted. Where context is ambiguous the rule redacts.
• regexTarget = "line" MUST NOT be used. A line-targeted allowlist suppresses every finding of the rule on any line it matches, including a real credential sitting beside the allowlisted identifier.

The negative corpus MUST carry every identifier shape above, in the contexts it appears in, and assert it passes the ingest filter untouched. The positive corpus MUST carry a UUID after a credential-named key and assert it is redacted, which is the test that stops a future shape allowlist.

What the egress file adds#

The egress file carries the ingest rules and two more. mzmon-egress-canary matches the egress canary. mzmon-egress-keyword-value is the generic rule’s pattern with the entropy floor removed, the minimum length lowered to eight, and the escaped-string handling of mzmon-password-field. It uses the same name allowlist.

It is kept off the ingest side because its false positives land in the customer’s own debugging. Pagination cursors, cache keys and short identifiers after words like key are the expected cost, and on the egress side a support engineer loses one value with the rule’s name in its place.

Rule files, extension and versioning#

The rule files are authored under packages/alloy-pipelines/secretfilter/ and rendered by mz-monitoring-build gen-pipelines into pre-rendered/secretfilter/, from which a ConfigMap mounts them into the gateway. Each shipped file MUST be flat: a single [extend] useDefault = true layer, with the egress file generated as the ingest rules plus its own rather than as an [extend] path onto the ingest file. That leaves exactly one level for the customer.

A customer’s extraConfig renders to a file that extends the shipped ingest rules by path, and the ingest filter points at it. It MAY add rules, add allowlists, and disable upstream rules, because the store it governs is the customer’s. It does not reach the egress filter, and it does not need to: a customer rule redacts at ingest, before the egress branch forks, so the customer’s own formats never reach the egress branch in the first place.

The default rules change between Alloy versions, and the design accepts that. Pinning a copy of the Gitleaks default would freeze out every rule upstream adds, which trades misses for stability on a design that optimizes for misses. The corpus is what makes accepting upstream safe: every Alloy bump runs it, a lost detection fails CI, and so does a new false positive on the identifier set.

The rendered rule files are the customer-readable artifact BYOC asks for. They SHOULD be published on the docsite beside the metric tiers, as a table generated from the files rather than written beside them.

Payment card numbers#

stage.luhn redacts digit runs that pass the Luhn checksum, which every payment card number does. It is a loki.process stage rather than a detector, and its cost is a digit scan and a checksum per line. Card numbers reach logs the way other customer data does: through an error message that quotes the value it rejected, such as an input-syntax error or a source decode error. That makes it a cheap improvement in posture, and it runs at ingest beside the filter.

At its defaults it would damage the logs Materialize is debugged with. The stage considers every run of 13 or more digits, and a random number passes the Luhn check one time in ten. Materialize logs frontiers, as_of and upper among them, as millisecond epoch timestamps, which are 13 digits. A reimplementation of the stage’s scan redacted 10.0% of 100,000 millisecond timestamps and 10.1% of 100,000 microsecond ones. The loss is inconsistent as well as frequent: in a line carrying an as_of and an upper one millisecond apart, the first was redacted and the second was not.

The stage ships with a skip_regex for epoch timestamps. skip_regex exempts matching substrings rather than whole lines, so a card number on the same line is still redacted. The expression exempts 13-, 16- and 19-digit runs that begin with 1, which covers epoch timestamps in milliseconds, microseconds and nanoseconds. The ISO/IEC 7812 industry identifier 1 belongs to airlines, whose UATP cards are 15 digits, so they stay redacted.

skip_regex = "\\b1(?:\\d{12}|\\d{15}|\\d{18})\\b"

The expression is dated. Millisecond timestamps pass 2000000000000 in May 2033, and the leading digit it keys on changes then. A unit test SHOULD start failing a year before that date, so the expression is revisited on a schedule rather than discovered.

The stage runs before the tier split, so the bulk tier is never sampled for card numbers. It exports no counter of its own. A stage.match on its placeholder followed by stage.metrics counts its redactions, for the dashboard row and for the notice alert. The runtime canary carries no card number, for the reason in its table. The corpus proves the stage in CI instead, with Visa, American Express and UATP numbers and with each timestamp width.

Calibrating misses against false alarms#

The two error types cost different amounts on each side, so each side is tuned separately.

The four cases#

A positive is a value the filter redacted, and a negative is a value it passed untouched. Every value the filter sees lands in one of four cells.

The egress rules deliberately put more lines in the false-positive cell, because that is how they take lines out of the false-negative one.

The matrix describes what is stored, not what the component returned. A component can redact the line and still leave a false negative behind, when a copy of the credential was made before it ran. That is the placement argument, restated as a cell.

A missing rule is only one way into the false-negative cell.

The canaries are the one place the false-negative cell becomes observable. Each is a known positive, so a canary that passes a filter undetected is a false negative produced on demand, which is what the missing-canary alert reports.

The ingest side’s false-positive rate matters for a reason beyond lost values. A redaction raises a notice alert, and an alert that fires daily on pagination cursors is an alert nobody reads, which hides the true positive among the false ones. So the ingest side is tuned until its alerts are rare enough to be investigated every time.

Shadow mode comes first#

Before the ingest filter enforces anywhere, it runs in shadow. loki.secretfilter.shadow receives a copy of the prefilter’s output and forwards to a loki.process that keeps only lines containing a placeholder, under job="secretfilter-shadow", to the local Loki. The main stream is unchanged.

Shadow mode is safe to run on a customer install, because what it stores has already been redacted. The per-rule, per-origin counters give the rate, and the shadow stream gives the context needed to tell a true positive from a false one. It SHOULD run for a week on the tier 2 substrate and on a live install before the ingest filter enforces by default, and the name allowlist SHOULD come out of that week.

Measuring misses#

A false negative is unobservable directly, since nothing flags what nobody detected. Three measurements stand in for it.

A numeric false-positive budget is deliberately not set here, and is an open question until shadow data exists.

A canary, in the spirit of EICAR#

The EICAR test file is a fixed string every antivirus engine agrees to detect, so a deployment can test its scanner without handling malware. The equivalent here is a pair of published strings, one per layer:

mzmon-canary-v1-7f3a9c2e4b6d8015
mzmon-egress-canary-v1-3b8e1d6a9c4f2e07

The first is matched by mzmon-canary, which both rule files carry. The second is matched by mzmon-egress-canary, which only the egress file carries. Neither is matched by any upstream rule. The version is in each string so that a future format can run beside the current one during a transition.

Each filter needs a canary of its own, because the ingest filter redacts the ingest canary before the egress branch forks. A single canary would reach the egress filter already replaced by its placeholder, the egress filter’s counter would never rise, and a missing egress filter would look identical to a working one. The egress canary passes the ingest filter untouched and is stored locally verbatim, which is harmless for a string that is published.

The producer is mz-monitoring-canary, a purpose-built binary in this repository that the chart runs as one long-running pod, on by default. It emits the canary line on its interval and does nothing else. Forking loki-canary is unnecessary. That tool’s value is its write-then-read round trip, which the bundled Loki canary already provides, and this producer only needs to write. A long-running pod rather than a CronJob avoids a pod and its Kubernetes events every interval. Its image is built and published beside mzmon-alloy, and joins the image scan and the registry profiles. A small stage.match on {container="mz-monitoring-canary"} in inputProcessor parses the line so the canary fields reach structured metadata.

The pair answers four questions at once.

The last doubles as a log-egress heartbeat per install, which the call-home heartbeat cannot be, since it carries no log lines. An egress canary that reaches the backstop verbatim is detected there by the same rule. That is a bug report against the install’s egress branch rather than an exposure, because the canary is published and the backstop still filters what follows it.

Two traps in watching the canary#

Loki logs the text of every query it evaluates, and the gateway stores Loki’s logs. A leak query written as |= "mzmon-canary-v1-7f3a9c2e4b6d8015" puts the full canary into Loki’s query log on every evaluation. The ingest filter then detects it there, and the canary counter keeps rising from origin="loki" after the producer has died. So the leak query MUST NOT contain a full canary. It searches {container="mz-monitoring-canary"} for the ingest canary’s suffix alone, which no rule matches. The missing-canary alert MUST also scope to the producer’s origin, because a human pasting a canary into Grafana produces the same detection.

The runtime canary proves one path. It follows the producer’s route from its pod to storage, so it cannot show that some other route skips the filter. That is a property of the rendered pipeline graph rather than of any one line, and a render-time assertion owns it.

Provider-shaped samples belong in CI#

The corpus carries AWS-, JWT- and PEM-shaped samples, because they are what proves the default rules still work after a bump. They MUST NOT run in a customer’s cluster, for the DLP reason above. Test fixtures SHOULD assemble them from fragments at test time, so that the repository holds no string that GitHub’s push protection or a contributor’s own scanner flags.

Alerts#

Severity follows the registry’s scale: critical over warning over notice. Nothing here is critical. Each alert reports either a finding to act on or a bug in the pipeline, and none describes an exposure that paging someone would stop. The metric alerts read Alloy’s own metrics, which the gateway already scrapes, and ride the query registry’s alert path. The leak alert is LogQL and waits on log-derived alert definitions.

Sampling the bulk tier is deliberate and raises no alert. Its bypassed count belongs on the dashboard row, where it reads as the fraction of DEBUG and TRACE stored unscanned.

Sketches of the two that need care:

# secret-filter-redacted: by rule and origin, so the notification names both.
sum by (component_id, rule, origin) (
  increase(loki_secretfilter_secrets_redacted_by_category_total{
    component_id="loki.secretfilter.ingest",
    rule!~"mzmon-(egress-)?canary"
  }[1h])
) > 0

# secret-filter-canary-missing, for the ingest filter; the egress form swaps in
# its component and its canary rule. Summed across gateway replicas, since the
# producer's line reaches one replica per interval. A filter that never saw its
# canary exports no series, so the absent() arm is what catches a new install.
sum by (component_id) (
  increase(loki_secretfilter_secrets_redacted_by_category_total{
    component_id="loki.secretfilter.ingest",
    rule="mzmon-canary", origin="mz-monitoring-canary"
  }[15m])
) == 0
or absent(loki_secretfilter_secrets_redacted_by_category_total{
  component_id="loki.secretfilter.ingest",
  rule="mzmon-canary", origin="mz-monitoring-canary"
})

The runbook for secret-filter-redacted starts from the fact that the source leaked. The placeholder in the stored line names the rule, and the origin names the container. If the finding is real, the credential MUST be treated as exposed and rotated, because it is still in the container log file, in kubectl logs, and in any other agent the customer runs over those files. The component that logged it gets a bug at its source. If the finding is false, it gets a name allowlist entry. A redaction is never the end of the story, which is why the alert exists at a severity that routes somewhere.

Backstop detections are Materialize-internal and are not in this table. They are an incident process in the control plane, described above.

TRACE and DEBUG do not cross in a standing configuration#

The question of whether trace-level logs should leave at all has good arguments on both sides, and they turn out to be about different time horizons.

Decision: TRACE and DEBUG MUST NOT cross in any standing configuration, and cross only inside an elevated window. The case for them is a case for having them during an escalation, and nobody argues for them the rest of the time. The call-home ladder and the BYOC triggered window already define that construct: customer-approved, self-expiring, audited.

The window is scoped by component, the same way Materialize scopes verbose logging by target, so a window for one controller does not raise the level for every environment. It expires on its own.

Elevation MUST NOT relax the secret filter. A support engineer never needs a credential’s value, only the fact that one was there, and the placeholder records that fact. What elevation relaxes is reduction: the level gate, the class allowlist and the body allowlist.

Support bundles#

Keeping TRACE locally is what makes a support bundle possible later, and nothing about that requires it to stream. A bundle drawn from the local store through the egress chain, and reviewed by the customer before it is transferred, is the offline path call-home defers. It inherits every filter above without a separate one.

Positions this revises#

The call-home design lists elevated as “a higher tier and relaxed redaction”, and the BYOC design describes its triggered window as one that “temporarily raise[s] the tier and lift[s] redaction”. Both use redaction for two mechanisms that deserve different answers.

Both docs SHOULD adopt that wording when they are next revised.

The BYOC design also holds that redaction attaches to the destination, not to the pipeline, because a global stage “would destroy the customer’s own copy to protect ours”. That holds for reduction and for body redaction, which remove information the customer wants. It does not hold for credentials. A credential in the customer’s own Loki is readable by everyone with Loki access and sits in object storage for the retention period, which is the customer’s exposure rather than Materialize’s. Removing it protects the customer’s copy. The ingest filter is therefore the one redaction that attaches to the pipeline, and where nothing leaves the cluster it stays the customer’s to turn off. Everything else about the BYOC principle stands, including every egress-only rule in this design.

What the filter cannot protect#

Chart-side prerequisites#

Work in this repo, ordered roughly by dependency.

The ingest half is not blocked on BYOC. It covers the customer’s own Loki and the third-party OTLP log destination that already ships. Nor does it wait on the image bump, since a public-preview component is acceptable in the meantime.

Testing#

The per-destination test revises the one the BYOC and call-home designs specify, which expects a sensitive line to be absent from the egress copy and present locally. With the ingest filter on, a credential is absent from both copies, which is the intent. The per-destination property is still worth asserting, and an egress-only rule is now what exercises it.

Documentation to update#

• Securing: the “keep secrets out of logs at the source” item keeps its advice and gains what the pipeline now does and does not do; the “Redaction in the pipeline” row moves off ❌.
• A customer-facing page under logs-and-events/: what is redacted, card numbers included, the placeholder, the alerts and their runbooks, the canaries and how to test against them, the bulk tier’s rate, and how to add rules.
• Logging pipeline reference: the prefilter split and the filter’s position.
• The BYOC and call-home designs: the wording revision, and a pointer from BYOC’s Redaction section to this doc.
• Roadmap: the design-doc row and the follow-up-documentation entry.
• Runbooks under operating/runbooks/ for each alert above.

Open questions#

• What is the processing_timeout? It should be a multiple of the measured p99, and nothing has been measured. A value too low drops real lines on the egress branch; a value too high lets one pathological line stall the gateway.
• What false-positive rate is acceptable at ingest? A budget per million lines, per rule, would make the tuning loop mechanical. It needs shadow data first.
• What sets the bulk tier’s rate on an install? The levers are ordered, and the trigger for the last one is not. Candidates are a documented latency-to-arrival ratio an operator applies by hand, or a value the Terraform module derives from sizing.
• Should mz-monitoring-canary carry more than the two canaries? A stable, low-volume synthetic stream is useful to other checks, such as the log-egress heartbeat. The risk is that it accretes content until its line is no longer obviously harmless.
• Should mzmon-password-field enforce at ingest? Its false positives include ordinary prose such as password: must be at least 8 characters. Shadow mode decides; the egress file carries it either way.
• Should a verbatim egress canary pause that tenant’s log ingestion? It is a bug in the install’s egress branch rather than an exposure, and the backstop still filters what follows it. Pausing trades support visibility for a margin the backstop already provides; filing the bug and telling the customer may be enough.
• Should upstream scan structured metadata? An option to do so would remove the ingest placement constraint for metadata that did not come from the line. Worth raising with the Alloy maintainers.
• Should upstream offer a keyed hash? An HMAC placeholder would give correlation without a confirmation oracle.
• Should the ingest filter be on by default where nothing leaves the cluster? The recommendation is yes, since the goal is no credentials at rest. The measured CPU cost may argue for a slower rollout.
• Is INFO in the standing egress set? That is BYOC’s open question about the default level, and this design only requires that the gate be an allowlist.

Appendix: draft ingest rules#

title = "mzmon ingest"

[extend]
useDefault = true

[[rules]]
id = "mzmon-canary"
description = "The published ingest canary"
regex = '''mzmon-canary-v1-[0-9a-f]{16}'''
keywords = ["mzmon-canary"]

[[rules]]
id = "mzmon-materialize-app-password"
description = "Materialize app password"
regex = '''\b(mzp_[A-Za-z0-9_-]{16,})'''
keywords = ["mzp_"]

[[rules]]
id = "mzmon-url-userinfo"
description = "Password in the userinfo of a URL"
regex = '''(?i)\b[a-z][a-z0-9+.-]{1,20}://[^/\s:@"'\\]{1,256}:([^/\s@"'\\]{1,256})@'''
keywords = ["://"]
  [[rules.allowlists]]
  regexes = ['''^(?:\*+|x+|(?i:redacted)|\$\{?[A-Za-z_]+\}?)$''']

[[rules]]
id = "mzmon-authorization-header"
description = "Credential in an Authorization header, including inside an escaped JSON string"
regex = '''(?i)\b(?:proxy-)?authorization\\?["']?\s*[:=]\s*\\?["']?\s*(?:bearer|basic|token|negotiate|aws4-hmac-sha256)\s+([A-Za-z0-9._~+/=,:-]{8,})'''
keywords = ["authorization"]

[[rules]]
id = "mzmon-url-query-credential"
description = "Credential in a URL query string"
regex = '''(?i)[?&](?:x-amz-signature|x-amz-security-token|x-amz-credential|x-goog-signature|x-goog-credential|sig|signature|access_token|id_token|refresh_token|token|api_key|apikey|client_secret|password)=([^&\s"'\\#]{8,})'''
keywords = ["x-amz-", "x-goog-", "sig=", "signature=", "token=", "api_key=", "apikey=", "client_secret=", "password="]

[[rules]]
id = "mzmon-private-key-header"
description = "PEM private key header, with or without its END line"
regex = '''-----BEGIN[ A-Z0-9_-]{0,100}PRIVATE KEY(?: BLOCK)?-----(?:[A-Za-z0-9+/=\s]|\\[nr])*'''
keywords = ["-----begin"]

[[rules]]
id = "mzmon-password-field"
description = "A password-named field, any entropy, including inside escaped JSON or Rust Debug output"
regex = '''(?i)\b(?:passw(?:or)?d|pwd|client[_-]?secret|secret[_-]?access[_-]?key)\\?["']?\s*[:=]\s*(?:Some\()?\\?["']?([^\s"'\\,;)}]{4,})'''
keywords = ["passw", "pwd", "secret"]
  [[rules.allowlists]]
  regexes = ['''^(?:None|Some|null|nil|true|false|\*+|(?i:<?redacted.*)|\$\{?[A-Za-z_]+\}?)$''']

[[rules]]
id = "mzmon-base64-postgres-url"
description = "Base64 of a postgres:// URL, as in a logged Kubernetes Secret"
regex = '''\b((?:cG9zdGdyZXM6Ly9|cG9zdGdyZXNxbDovL)[A-Za-z0-9+/]{8,}={0,2})'''
keywords = ["cg9zdgdyzxm6ly9", "cg9zdgdyzxnxbdovl"]

# Upstream jwt admits a backslash in its last segment, so it can consume the
# escape of a closing \" and leave the redacted line unparseable.
[[rules]]
id = "jwt"
regex = '''\b(ey[a-zA-Z0-9]{17,}\.ey[a-zA-Z0-9\/\\_-]{17,}\.(?:[a-zA-Z0-9\/\\_-]{9,}[a-zA-Z0-9\/_-]={0,2})?)(?:[\x60'"\s;]|\\[nr"]|$)'''

# Identifier fields whose name says they are not credentials. Built from
# shadow-mode data; this entry is the one the corpus measured.
[[rules]]
id = "generic-api-key"
  [[rules.allowlists]]
  regexTarget = "match"
  regexes = ['''(?i)^[\w.-]{0,50}?\b(?:token|access|api|key|credential)[_.-]?(?:id|name|type|version|count)\b''']

# A finding whose secret is already a placeholder is not a finding.
[[allowlists]]
description = "Placeholders written by an earlier pass"
regexes = ['''^<redacted:[a-z0-9-]+>$''']