bug: Conditional Processing Not Supported with Filter as Processor

[c-neto]

Bug Report

Describe the bug

The Conditional Processing feature is not supported when using Filter as Processor. This limitation is not clearly stated in the documentation, which may cause confusion for users who expect conditional rules to work with Filters (such as Lua and Modify) when used as processors.

To Reproduce

• Steps to reproduce the problem:

Docker-Compose with example: https://github.com/c-neto/my-devops-labs/tree/main/fluentbit/bug-filter-as-processor-conditionally-ignoring

service:
  log_level: trace

pipeline:
  inputs:
    - name: dummy
      tag: dummy
      dummy: |
        {"stream": "stdout"}
      processors:
        logs:
          # this field is added, because the condition rule met
          - name: content_modifier
            action: insert
            key: field_add_only_if_stream_is_stdout
            value: foobar
            condition:
              op: and
              rules:
                - field: $stream
                  op: eq
                  value: stdout

          # the field is not added, because the condition rule is not met
          - name: content_modifier
            action: insert
            key: field_add_only_if_stream_is_stderr
            value: foobar
            condition:
              op: and
              rules:
                - field: $stream
                  op: eq
                  value: stderr

          # >>> BUG: the field is always added, ignoring the condition rule
          - name: lua
            call: main
            code: |
              function main(tag, timestamp, record)
                record["field_add_by_lua_if_stream_is_stderr"] = "BUG_DETECTED"
                return 1, timestamp, record
              end
            condition:
              op: and
              rules:
                - field: $stream
                  op: eq
                  value: stderr

          # >>> BUG: the field is always added, ignoring the condition rule
          - name: modify
            match: '*'
            add:
              - field_add_by_modify_filter_if_stream_is_stderr BUG_DETECTED
            condition:
              op: and
              rules:
                - field: $stream
                  op: eq
                  value: stderr

  outputs:
    - name: stdout
      format: json_lines
      match: "*"

• The result is:

{
  "date": 1751412593.244093,
  "stream": "stdout",
  "field_add_only_if_stream_is_stdout": "foobar",
  "field_add_by_lua_if_stream_is_stderr": "BUG_DETECTED",
  "field_add_by_modify_filter_if_stream_is_stderr": "BUG_DETECTED"
}

When using the built-in Processors plugin, conditional rules work as expected. In the provided example, the Content Modifier processor correctly respects the condition, so field_add_only_if_stream_is_stderr is not added when the condition is not met.

However, when using Filter as Processor, conditional rules are ignored. This issue is demonstrated with the Lua and Modify filters, where the fields are always added regardless of the specified conditions.

Expected behavior

Implement support for Conditional Processing when using Filter as Processor.

Your Environment

• Version used: 4.0.3

---

Note

I created a PR in the fluent-bit-docs repo to address this gap in the documentation while this feature is not yet implemented. More details:

• pipeline: add note to explicit the limitation of conditional processing by using filter as processor fluent-bit-docs#1828

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.