feat: add production Kubernetes examples and fix GLIBC compatibility

- Build plugins inside manylinux_2_28 containers for portable binaries
- Add production/ directory with init container examples that download
  plugins from GitHub Releases at pod startup
- Auto-detect architecture (amd64/arm64) at runtime
- Update README with GitHub Releases as primary download source
- Document both init container and pre-built Docker image approaches

The init container approach may still have GLIBC issues until the next
release is built with the updated workflow.

Author: jackluo923

.github/workflows/build.yaml

@@ -15,27 +15,35 @@ jobs:
         include:
           - arch: "amd64"
             runner: "ubuntu-24.04"
+            manylinux: "quay.io/pypa/manylinux_2_28_x86_64"
           - arch: "arm64"
             runner: "ubuntu-24.04-arm"
+            manylinux: "quay.io/pypa/manylinux_2_28_aarch64"
     steps:
       - uses: "actions/checkout@v4"
         with:
           submodules: "recursive"
 
-      - uses: "actions/setup-go@v5"
-        with:
-          go-version: "1.24"
+      # Build inside manylinux container for glibc compatibility
+      - name: "Build plugins"
+        run: |
+          docker run --rm -v "${{ github.workspace }}:/src" -w /src ${{ matrix.manylinux }} bash -c '
+            set -e
 
-      - name: "Install task"
-        run: "npm install -g @go-task/cli"
+            # Install Go
+            curl -fsSL "https://go.dev/dl/go1.24.3.linux-${{ matrix.arch }}.tar.gz" | tar -C /usr/local -xz
+            export PATH="/usr/local/go/bin:$PATH"
 
-      - name: "Download clp-ffi-go libraries"
-        run: "bash third-party/clp-ffi-go/scripts/download-libs.sh"
+            # Install task
+            curl -fsSL https://taskfile.dev/install.sh | sh -s -- -d -b /usr/local/bin
 
-      - name: "Build plugins"
-        run: |
-          task build:s3
-          task build:s3v2
+            # Download clp-ffi-go libraries
+            bash third-party/clp-ffi-go/scripts/download-libs.sh
+
+            # Build plugins
+            task build:s3
+            task build:s3v2
+          '
 
       - name: "Upload artifacts"
         uses: "actions/upload-artifact@v4"
@@ -103,7 +111,7 @@ jobs:
         include:
           - plugin: "out_clp_s3"
             image: "fluent-bit-clp-s3"
-            platforms: "linux/amd64"
+            platforms: "linux/amd64,linux/arm64"
           - plugin: "out_clp_s3_v2"
             image: "fluent-bit-clp-s3-v2"
             platforms: "linux/amd64,linux/arm64"

README.md

@@ -1,17 +1,7 @@
 # Fluent Bit CLP Plugins
 
 [Fluent Bit][fluent-bit] output plugins that compress logs using [CLP][clp-blog] for efficient
-storage and search on AWS S3.
-
-## Table of Contents
-
-- [Overview](#overview)
-- [Which Plugin Should I Use?](#which-plugin-should-i-use)
-- [Quick Start](#quick-start)
-- [Docker Images](#docker-images)
-- [Pre-built Binaries](#pre-built-binaries)
-- [Documentation](#documentation)
-- [Development](#development)
+storage and search on S3.
 
 ## Overview
 
@@ -23,126 +13,96 @@ flowchart LR
     D --> E[YScope Log Viewer]
 ```
 
-**[CLP][clp-blog]** (Compressed Log Processor) is a log compression tool that achieves 10-100x
-better compression than gzip while enabling fast search. These plugins compress logs into CLP's
-Intermediate Representation (IR) format with [Zstd][zstd] compression, then upload to S3.
+**[CLP][clp-blog]** (Compressed Log Processor) achieves 10-100x better compression than gzip while
+enabling fast search. These plugins compress logs into CLP's Intermediate Representation (IR)
+format with [Zstd][zstd] compression, then upload to S3.
 
-Use [YScope Log Viewer][log-viewer] to view compressed logs directly in the browser, or ingest
-into CLP for search at scale.
+View compressed logs directly in the browser with [YScope Log Viewer][log-viewer], or ingest into
+CLP for search at scale.
 
-## Which Plugin Should I Use?
+## Plugins
 
-| Plugin | Upload Trigger | Best For |
-|--------|---------------|----------|
-| **[out_clp_s3_v2](plugins/out_clp_s3_v2/README.md)** | Time-based (per log level) | Kubernetes, latency-sensitive workloads |
-| **[out_clp_s3](plugins/out_clp_s3/README.md)** | Size-based (MB threshold) | Batch processing, size optimization |
+Two plugins with different log delivery strategies:
 
-**Recommended: Start with `out_clp_s3_v2`** - it offers more control over upload latency and
-works well with Kubernetes deployments.
+| Plugin | Strategy | Description |
+|--------|----------|-------------|
+| **[out_clp_s3_v2](plugins/out_clp_s3_v2/README.md)** | Continuous sync | Compressed logs continuously synced to S3; sync frequency tunable to balance freshness vs cost |
+| **[out_clp_s3](plugins/out_clp_s3/README.md)** | Batch upload | Logs buffered locally until size threshold reached, then uploaded as batch |
 
-### out_clp_s3_v2 (Recommended)
+### out_clp_s3_v2 — Continuous Sync
 
-Time-based flushing with per-log-level control:
-- ERROR logs uploaded in seconds, DEBUG logs batched for minutes
-- Dual-timer strategy (hard + soft) prevents upload storms
+Log rotation is delegated to the log appender. The plugin continuously syncs compressed log files
+to S3.
+
+- Higher severity logs (e.g., ERROR) trigger faster sync of the entire log file
+- Dual-timer strategy (hard + soft deadlines) prevents upload storms
 - Designed for Kubernetes (sidecar and DaemonSet patterns)
-- MinIO compatible for local development
 
-### out_clp_s3
+### out_clp_s3 — Batch Upload
+
+Logs are buffered locally and uploaded when a size threshold is reached.
 
-Size-based uploads with crash recovery:
 - Upload when buffer reaches size threshold (default 16 MB)
 - Disk buffering with crash recovery
 - IAM role assumption for cross-account access
-- Single key extraction from log records
 
 ## Quick Start
 
-### Docker Compose (Easiest)
-
-The fastest way to try the plugins locally:
+The fastest way to try locally with Docker Compose:
 
 ```shell
-# out_clp_s3_v2 (recommended)
-cd plugins/out_clp_s3_v2/examples/docker-compose
-docker compose up
+git clone --recursive https://github.com/y-scope/fluent-bit-clp.git
+cd fluent-bit-clp
+
+# Choose one:
+cd plugins/out_clp_s3_v2/examples/docker-compose  # Continuous sync
+cd plugins/out_clp_s3/examples/docker-compose     # Batch upload
 
-# out_clp_s3
-cd plugins/out_clp_s3/examples/docker-compose
 docker compose up
 ```
 
-This starts MinIO, Fluent Bit with the plugin, and a log generator. View logs at
-http://localhost:9001 (minioadmin/minioadmin).
-
-### Kubernetes (k3d)
+This starts MinIO (S3-compatible storage), Fluent Bit with the plugin, and a log generator.
+View logs at http://localhost:9001 (minioadmin/minioadmin).
 
-```shell
-# Download plugins from GitHub Actions (see Pre-built Binaries section)
-# Extract to a local directory, e.g., ./plugins/
-
-# Create k3d cluster with plugin mounted
-k3d cluster create yscope --servers 1 --agents 1 \
-  -v $(pwd)/plugins:/fluent-bit/plugins \
-  -p 9000:30000@agent:0 -p 9001:30001@agent:0
-
-# Deploy MinIO and Fluent Bit
-cd plugins/out_clp_s3_v2/examples/kubernetes
-kubectl apply -f minio.yaml -f aws-credentials.yaml
-kubectl apply -f logs-bucket-creation.yaml
-kubectl apply -f sidecar/fluent-bit-sidecar.yaml -f sidecar/fluent-bit-sidecar-config.yaml
-```
+## Installation
 
-See [Kubernetes Examples](plugins/out_clp_s3_v2/examples/kubernetes/README.md) for sidecar and
-DaemonSet deployment patterns.
+### Docker Images
 
-### Docker (Build from Source)
+Pre-built Fluent Bit images with CLP plugins:
 
 ```shell
-# out_clp_s3_v2
-cd plugins/out_clp_s3_v2
-docker build -t fluent-bit-clp-v2 -f Dockerfile ../../
-docker run -v ~/.aws/credentials:/root/.aws/credentials fluent-bit-clp-v2
-
-# out_clp_s3
-cd plugins/out_clp_s3
-docker build -t fluent-bit-clp -f Dockerfile ../../
-docker run -v ~/.aws/credentials:/root/.aws/credentials fluent-bit-clp
+docker pull ghcr.io/y-scope/fluent-bit-clp-s3-v2:latest  # Continuous sync
+docker pull ghcr.io/y-scope/fluent-bit-clp-s3:latest     # Batch upload
 ```
 
-## Docker Images
-
-Pre-built Fluent Bit images with CLP plugins are published to GitHub Container Registry:
-
-```shell
-# Time-based plugin (recommended)
-docker pull ghcr.io/y-scope/fluent-bit-clp-s3-v2:latest
+Images are tagged with `latest` (main branch), branch names, and commit SHAs.
 
-# Size-based plugin
-docker pull ghcr.io/y-scope/fluent-bit-clp-s3:latest
-```
+### Pre-built Binaries
 
-Images are tagged with `latest` (main branch), branch names, and commit SHAs.
+Standalone `.so` plugin files available from [GitHub Releases](../../releases):
 
-## Pre-built Binaries
+| File | Architecture |
+|------|--------------|
+| `out_clp_s3_v2_linux_amd64.so` | x86_64 |
+| `out_clp_s3_v2_linux_arm64.so` | ARM64 |
+| `out_clp_s3_linux_amd64.so` | x86_64 |
+| `out_clp_s3_linux_arm64.so` | ARM64 |
 
-Standalone plugin binaries (`.so` files) are also available as GitHub Actions artifacts.
+Also available as zip from [Actions → build](../../actions/workflows/build.yaml).
 
-**Download:** Go to [Actions → build](../../actions/workflows/build.yaml), select a workflow run,
-and download the `fluent-bit-clp-plugins-linux-amd64` artifact.
+## Kubernetes
 
-| File | Plugin |
-|------|--------|
-| `out_clp_s3_v2_linux_amd64.so` | Time-based plugin (recommended) |
-| `out_clp_s3_linux_amd64.so` | Size-based plugin |
+See [Kubernetes Examples](plugins/out_clp_s3_v2/examples/kubernetes/README.md) for:
+- Local development with k3d
+- Production deployment with init containers
+- Sidecar and DaemonSet patterns
 
 ## Documentation
 
 | Document | Description |
 |----------|-------------|
-| [out_clp_s3_v2 README](plugins/out_clp_s3_v2/README.md) | Configuration, flush strategy, Kubernetes deployment |
-| [out_clp_s3 README](plugins/out_clp_s3/README.md) | Configuration, disk buffering, crash recovery |
-| [Kubernetes Examples](plugins/out_clp_s3_v2/examples/kubernetes/README.md) | Sidecar and DaemonSet deployments |
+| [out_clp_s3_v2 README](plugins/out_clp_s3_v2/README.md) | Configuration and flush strategy |
+| [out_clp_s3 README](plugins/out_clp_s3/README.md) | Configuration and disk buffering |
 
 ## Development
 
@@ -151,22 +111,13 @@ and download the `fluent-bit-clp-plugins-linux-amd64` artifact.
 - Go 1.24+
 - [Task](https://taskfile.dev/installation/)
 
-### Cloning
-
-This project uses git submodules for the [clp-ffi-go](https://github.com/y-scope/clp-ffi-go)
-dependency:
+### Building
 
 ```shell
-# New clone (recommended)
+# Clone with submodules
 git clone --recursive https://github.com/y-scope/fluent-bit-clp.git
+cd fluent-bit-clp
 
-# Existing clone
-git submodule update --init --recursive
-```
-
-### Building
-
-```shell
 # Download clp-ffi-go native libraries
 bash third-party/clp-ffi-go/scripts/download-libs.sh
 
@@ -177,19 +128,11 @@ task build
 ### Linting
 
 ```shell
-# Install prerequisites
-# - Task: https://taskfile.dev/installation/
-# - uv: https://docs.astral.sh/uv/getting-started/installation/
-
-# Run checks
-task lint:check
-
-# Auto-fix
-task lint:fix
+task lint:check  # Run checks
+task lint:fix    # Auto-fix
 ```
 
-> **Note:** Linting for the v1 plugin (`out_clp_s3`) is currently disabled and will be addressed in
-> a future update.
+> **Note:** Linting for `out_clp_s3` is temporarily disabled.
 
 [clp-blog]: https://www.uber.com/blog/reducing-logging-cost-by-two-orders-of-magnitude-using-clp
 [fluent-bit]: https://fluentbit.io/

plugins/out_clp_s3/Dockerfile

@@ -1,12 +1,16 @@
 # Build with manylinux for portable binaries compatible with Fluent Bit's runtime.
 # manylinux_2_28 uses glibc 2.28, ensuring compatibility with most Linux distributions.
-#
-# Note: This plugin only supports amd64 architecture.
-FROM quay.io/pypa/manylinux_2_28_x86_64 AS builder
+FROM quay.io/pypa/manylinux_2_28_x86_64 AS builder-amd64
+FROM quay.io/pypa/manylinux_2_28_aarch64 AS builder-arm64
+
+# Select the right base image based on target platform
+ARG TARGETARCH
+FROM builder-${TARGETARCH} AS builder
 
 # Install Go
+ARG TARGETARCH
 ARG GO_VERSION=1.24.3
-RUN curl -fsSL "https://go.dev/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar -C /usr/local -xz
+RUN curl -fsSL "https://go.dev/dl/go${GO_VERSION}.linux-${TARGETARCH}.tar.gz" | tar -C /usr/local -xz
 ENV PATH="/usr/local/go/bin:${PATH}"
 
 # Install task
@@ -27,7 +31,7 @@ RUN task build:s3
 
 FROM fluent/fluent-bit:3.2.7
 
-COPY --from=builder /src/pre-built/out_clp_s3_linux_amd64.so /fluent-bit/lib/out_clp_s3.so
+COPY --from=builder /src/pre-built/out_clp_s3_linux_*.so /fluent-bit/lib/out_clp_s3.so
 COPY --from=builder /src/plugins/out_clp_s3/*.conf /fluent-bit/etc/
 
 EXPOSE 2020