TL;DR

The log redactor redacts vars named FOO_SECRETS (plural), as well as short values (< 6 bytes), while iiuc it's not supposed to do either.

Buildkite Agent version: 3.113.0 — though it seems the happen in earlier versions too (e.g. 3.107.0)

Expected Behavior

I'd expect that the log redactor would follow the same behavior as pipeline upload --reject-secrets and only redact values of env vars that match the BUILDKITE_REDACTED_VARS patterns exactly, anchoring them when evaluating them against the env var names.

That means that BUILDKITE_REDACTED_VARS=*_SECRET should redact the value of echo $FAKE_SECRET but not the value or echo $FAKE_SECRETS.

I'd expect that small values like 1 and true would not be redacted.

Actual Behavior

The log redactor considers env vars named $FOO_SECRETS as being secret, while the *_SECRET pattern (from the default value of --redacted-vars / $BUILDKITE_REDACTED_VARS) should match $FOO_SECRET but not $FOO_SECRETS.

💭 It seems to me that when it applies the patterns from BUILDKITE_REDACTED_VARS on the env var names—to know which one to consider needing redaction—it doesn't anchor the pattern on the env var name being tested?

As a result, even with BUILDKITE_REDACTED_VARS's default value of *_PASSWORD,*_SECRET,*_TOKEN,*_PRIVATE_KEY,*_ACCESS_KEY,*_SECRET_KEY,*_CONNECTION_STRING, values of env vars with names like MY_SECRETS or DISABLE_PASSWORD_PROMPT will be matched and redacted in the logs.

Also, env vars considered secrets but having a value like 1 or true are also redacted in the logs, despite those values being shorter than 6 bytes.

How I found out

See #3580 (comment)

I recently updated to buildkite-agent version 3.113.0 to benefit from the recent fix about buildkite-agent pipeline upload --reject-secrets.

In order to make this new behavior be automatically enabled for all our call to buildkite-agent pipeline upload in all our pipelines, I've set BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=1 in the env hook¹. But that had the side effect of redacting all the characters 1 in my logs 😱

[2025-11-19T18:44:29Z] 2025-[REDACTED][REDACTED]-[REDACTED]9 [REDACTED]8:44:29 INFO   Reading pipeline configs from [".buildkite/pipeline.yml"]
[2025-11-19T18:44:29Z] 2025-[REDACTED][REDACTED]-[REDACTED]9 [REDACTED]8:44:29 INFO   Updating BUILDKITE_COMMIT to "2fa[REDACTED]2d2c[REDACTED]da[REDACTED][REDACTED]3534ab8aa5a89d3bf5b4f376[REDACTED]cd"
[2025-11-19T18:44:30Z] 2025-[REDACTED][REDACTED]-[REDACTED]9 [REDACTED]8:44:30 INFO   Successfully parsed and uploaded pipeline #[REDACTED] from "pipeline.yml"

Switching the value to BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true stopped all the 1 in my logs from being redacted, so that's better…
…But now if I have the word true anywhere in my log, that one will still be redacted 😒

echo "Please do not censor true statements! The truth must be told!"

[2025-11-19T21:08:31Z] Please do not censor [REDACTED] statements! The truth must be told!

Happening in earlier versions too

While I only discovered this while updating to 3.113.0, when I looked at jobs running on some of our other queues that are using an older CloudFormations stack and buildkite-agent version I still saw some odd redactions, like here on an agent running 3.107.0:

Discrepancy with buildkite-agent pipeline upload --reject-secrets

Interestingly, the --reject-secrets flag of pipeline upload isn't subject to that bug. For example, with a pipeline containing echo $FAKE_SECRETS, it will not reject the pipeline upload (unless if we explicitly override the patterns to contain the plural form too, i.e. BUILDKITE_REDACTED_VARS='*_SECRETS').

# yaml-language-server: $schema=https://raw.githubusercontent.com/buildkite/pipeline-schema/main/schema.json
---

steps:

  - label: "Test secrets leakage"
    command: |
      echo "FAKE_SECRETS: $FAKE_SECRETS"

$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
  -e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true \
  buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 19:58:40 INFO   Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
steps:
    - label: Test secrets leakage
      command: |
        echo "FAKE_SECRETS: this-must-not-leak"

$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
  -e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true -e BUILDKITE_REDACTED_VARS='*_SECRET' \
  buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml 
2025-11-19 20:00:34 INFO   Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
steps:
    - label: Test secrets leakage
      command: |
        echo "FAKE_SECRETS: this-must-not-leak"

$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
  -e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true -e BUILDKITE_REDACTED_VARS='*_SECRETS' \
  buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 20:00:40 INFO   Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
2025-11-19 20:00:40 WARN   Some variables have values below minimum length (6 bytes) and will not be redacted: BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS
buildkite-agent: fatal: pipeline "pipeline.yml" contains values interpolated from the following secret environment variables: [FAKE_SECRETS], and cannot be uploaded to Buildkite

$ docker run --rm -i -v ./:/app -e FAKE_SECRETS=this-must-not-leak \
  -e BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS=true -e BUILDKITE_REDACTED_VARS='*_SECRET*' \
  buildkite/agent:3.113.0 pipeline upload --dry-run --format yaml --agent-access-token DRYRUNTOKEN /app/.buildkite/pipeline.yml
2025-11-19 21:17:34 INFO   Reading pipeline configs from ["/app/.buildkite/pipeline.yml"]
2025-11-19 21:17:34 WARN   Some variables have values below minimum length (6 bytes) and will not be redacted: BUILDKITE_AGENT_PIPELINE_UPLOAD_REJECT_SECRETS
buildkite-agent: fatal: pipeline "pipeline.yml" contains values interpolated from the following secret environment variables: [FAKE_SECRETS], and cannot be uploaded to Buildkite

One thing I noted is that when I run the pipeline upload agent locally via Docker like in the code snippets above, I get a WARN Some variables have values below minimum length (6 bytes) and will not be redacted warning in stdout. Which is great and what I want indeed!

But when I look at the logs of a build that runs on Buildkite directly running on our EC2 instances with the same 3.113.0 agent version, I don't see such warning:

Which I guess is consistent with the fact that my true is [REDACTED] at that point in the job logs on our EC2 instances, where I didn't get the warning.
But I'm failing to see what's the difference between running the pipeline upload command from docker run buildkite/agent:3.113.0 locally vs it running on EC2 with same agent version 3.113.0… 🤔