aws: Update compression support for AWS plugins

[ShelbyZ]

• Enable snappy compression support for AWS plugins (S3, Kinesis Firehose/Streams)
• Add compression support to Kinesis Streams

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change
• Debug log output from testing the change

• Attached Valgrind output that shows no leaks or memory corruption was found

Summary view
https://gist.github.com/ShelbyZ/6a325762003785ba1804faeff958a1c7

Debug Logs
https://gist.github.com/ShelbyZ/9d0036c46600b44cb9a4cea99795230a

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [NA] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

fluent/fluent-bit-docs#2359

Backporting

• [N/A] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • Snappy compression added to AWS compression support.
  • Per-record compression option added to Kinesis Streams and S3 outputs (supported: gzip, zstd, snappy).

• Documentation

  • Firehose compression docs updated to list supported formats and note Arrow availability when enabled.
  • S3 now sets Content-Encoding for snappy uploads.

• Tests

  • Expanded runtime and unit tests for gzip, zstd, and snappy (including aggregation scenarios).

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds SNAPPY compression support across AWS outputs: new compression constant, snappy wiring in AWS compression dispatch, Kinesis config + runtime compression paths, S3 header handling, Firehose docs update, and expanded unit/runtime tests for snappy (and other compressions).

Changes

Cohort / File(s)	Summary
Compression Type Definition include/fluent-bit/aws/flb_aws_compress.h	Added FLB_AWS_COMPRESS_SNAPPY constant.
AWS Compression Dispatch src/aws/flb_aws_compress.c	Included flb_snappy.h, added flb_snappy_compress_wrapper, and registered "snappy" in the compression options.
Kinesis Streams Config & Context plugins/out_kinesis_streams/kinesis.h, plugins/out_kinesis_streams/kinesis.c	Added int compression to struct flb_kinesis and config parsing/validation for compression (supports gzip, zstd, snappy, default none).
Kinesis Record Processing plugins/out_kinesis_streams/kinesis_api.c, plugins/out_kinesis_streams/kinesis_api.h	Added compression-aware branches for per-record and aggregated-record flows using flb_aws_compression_b64_truncate_compress; added MAX_B64_EVENT_SIZE macro for sizing.
S3 Content-Encoding / Config plugins/out_s3/s3.c	Added Content-Encoding: snappy handling in header construction and updated config descriptions to include snappy.
Firehose Docs plugins/out_kinesis_firehose/firehose.c	Updated compression config description to list gzip, zstd, and snappy (retains arrow note).
Tests — Internal Compression tests/internal/aws_compress.c	Added Snappy include, uncompress wrapper, and Snappy-specific test cases wired into the compression test harness.
Tests — Runtime Kinesis tests/runtime/out_kinesis.c	Added runtime tests for compression_gzip, compression_zstd, compression_snappy, and aggregation variant.
Tests — Runtime Firehose tests/runtime/out_firehose.c	Added runtime tests for compression_zstd, compression_snappy, and snappy with aggregation.
Tests — Runtime S3 tests/runtime/out_s3.c	Added runtime tests for gzip, zstd, snappy (standard and putobject variants).

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant KinesisPlugin as Kinesis Output
    participant CompressDispatch as AWS Compression Dispatch
    participant EventBuf as Record/Event Buffer
    Client->>KinesisPlugin: Submit record(s)
    KinesisPlugin->>KinesisPlugin: Check ctx->compression
    alt compression != NONE
        KinesisPlugin->>CompressDispatch: flb_aws_compression_b64_truncate_compress(data)
        CompressDispatch-->>KinesisPlugin: compressed_b64_data + size
        KinesisPlugin->>EventBuf: Replace buffer with compressed data
    else compression == NONE
        KinesisPlugin->>EventBuf: Base64-encode data (existing path)
        EventBuf-->>KinesisPlugin: base64_data + size
    end
    KinesisPlugin->>Client: Send record(s) to AWS

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• out_s3: Add ZSTD Compression Support #10794: Overlaps in adding compression types and S3/Kinesis/Firehose wiring.
• zstd: aws: compress: fix incompatible pointer glitches #10833: Related edits to the AWS compression dispatch and function-pointer handling.
• out_s3: Add parquet compression type with pure C #10691: Similar extension of AWS compression subsystem and plugin integration.

Suggested reviewers

• edsiper
• cosmo0920
• fujimotos
• koleini

Poem

🐰 Snappy hopped into the code,
A tiny pack to lighten the load.
Headers set and tests in queue,
Records shrink and zip on through.
Hooray — small hops, big throughput! 🥕✨

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 16.22% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'aws: Update compression support for AWS plugins' directly relates to the main changeset, which adds snappy compression support and extends compression capabilities across S3, Kinesis Firehose, and Kinesis Streams plugins.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

🤖 Fix all issues with AI agents

In `@plugins/out_kinesis_streams/kinesis_api.h`:
- Line 26: The macro MAX_B64_EVENT_SIZE currently defines 1398075 but the
comment formula ceil(1048556 / 3) * 4 evaluates to 1398076; update the macro or
the comment to be consistent: either change the macro MAX_B64_EVENT_SIZE to
1398076 to match the formula, or if the -1 is intentional (e.g., to
reserve/exclude a NUL), update the comment next to MAX_B64_EVENT_SIZE to clearly
state that the value intentionally subtracts one (explain reason). Ensure you
edit the definition of MAX_B64_EVENT_SIZE and its inline comment together so the
code and comment match.

In `@src/aws/flb_aws_compress.c`:
- Around line 35-41: The wrapper flb_snappy_compress_wrapper currently
propagates negative error codes from flb_snappy_compress; change it to call
flb_snappy_compress into an int variable (e.g., ret) and then return -1 if ret <
0, otherwise return ret, so all negative error codes are normalized to -1 to
match the AWS compression interface contract.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f5e4ce37e5

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[cosmo0920]

It looks good to me. Thanks for your contribution.