[Q&A] Question about using relabel output as secondary and encrypting failed messages

[stagegrowth]

What is a problem?

I want to securely store failed messages when delivery to the primary destination fails.

In our production environment, Fluentd receives HTTP requests and forwards them to an internal HTTP endpoint.
When delivery succeeds, logs are already encrypted using AWS KMS and stored according to our security policy.

However, when delivery fails, records are handled by a output and written to a file in plain text.
To avoid storing sensitive payloads in plain text, I attempted to route failed records through a separate encrypted processing pipeline.

Specifically, I tried to use @type relabel as a secondary output so that failed records could be routed to another label and encrypted before being written to a file.
This approach was ineffective, as Fluentd fails to start when relabel is configured as a secondary output.

Based on the error message, my understanding is that only supports buffered outputs and that relabel is non-buffered.
I would like to confirm whether this limitation is intentional and if there is any supported way to securely process failed records without storing them in plain text.

Describe the configuration of Fluentd

Fluentd is deployed on Kubernetes using a Helm chart and receives requests via an HTTP input.

High-level configuration:

• An HTTP input (@type http) receives incoming requests.
• Records are routed using @type route and labels.
• Valid requests are transformed and forwarded to an internal HTTP endpoint using @type http.
• The primary HTTP output uses a file-based buffer.
• When delivery to the primary endpoint fails, a output is used to handle failed records.
• Successful records and re-ingested error records are encrypted using AWS KMS and stored through separate pipelines.

To encrypt failed records before writing them to a file, I attempted to route secondary records using @type relabel and apply encryption logic in a dedicated label.

Relevant part of the configuration (simplified):

# On delivery failure: attempt to route failed records via relabel
<secondary>
  @type relabel
  @label @SA_FAMILY_ERR_TMP
</secondary>

<label @SA_FAMILY_ERR_TMP>
  <filter **>
    @type record_transformer
    enable_ruby true
    <record>
      content ${ encrypted_content }
    </record>
  </filter>

  <match **>
    @type file
    ...
  </match>
</label>

Encryption for successful deliveries and re-ingested error logs is already implemented using record_transformer with Ruby and AWS KMS, and works as expected outside of the secondary output path.

Describe the logs of Fluentd

When Fluentd is started with @type relabel configured as a secondary output, it fails during startup with the following error:

secondary plugin 'Fluent::Plugin::RelabelOutput' must support buffering, but doesn't

Environment

- Fluentd version: 1.17.1
- TD Agent version:
- Fluent Package version:
- Docker image (tag): fluent/fluentd:v1.17-1
- Operating system: Alpine Linux v3.19
- Kernel version: 5.10.236-227.928.amzn2.x86_64