out_es: custom http headers

[AndrewChubatiuk]

Added ability to set custom HTTP headers for Elasticsearch output

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

• Example configuration file for the change

[INPUT]
    name              tail
    path              /var/lib/docker/containers/**/*.log
    path_key         path
    multiline.parser  docker, cri
    Parser docker
    Docker_Mode  On

[INPUT]
    Name     syslog
    Listen   0.0.0.0
    Port     5140
    Parser   syslog-rfc3164
    Mode     tcp

[SERVICE]
    Flush        1
    Parsers_File parsers.conf

[Output]
    Name es
    Match *
    host victorialogs
    port 9428
    compress gzip
    path /insert/elasticsearch
    header AccountID 0
    header ProjectID 0
    header VL-Stream-Fields path
    header VL-Msg-Field log
    header VL-Time-Field @timestamp

• [N/A] Debug log output from testing the change

• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• Documentation required for this feature

fluent/fluent-bit-docs#1390

Backporting

• Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features
  • Added support for custom HTTP headers in the Elasticsearch output.
  • Introduced a new configuration option: header, allowing multiple key/value pairs (can be specified multiple times).
  • Configured headers are automatically included on all outgoing HTTP requests, including during flush operations and initial setup.
  • Default behavior is unchanged unless headers are configured.

[nunopintocicd]

@PettitWesley @edsiper any news on this ?

[AndrewChubatiuk]

hey @edsiper
could this feature be also included in release 3.2?

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[strawgate]

Is there anything we can do to get this included in a future release?

[github-actions]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[coderabbitai]

Walkthrough

Adds support for user-defined HTTP headers in the Elasticsearch output plugin. Introduces a headers field in the context, config map entry for multiple header pairs, and iterates configured headers to inject them into HTTP requests during flush and initial request setup.

Changes

Cohort / File(s)	Summary of Changes
Header configuration plumbing plugins/out_es/es.h, plugins/out_es/es.c	Adds ctx->headers field to struct flb_elasticsearch and exposes a new FLB_CONFIG_MAP_SLIST_1 config entry "header" (MULT true) to accept multiple header key/value pairs.
HTTP request header injection plugins/out_es/es.c	Iterates ctx->headers at flush/request setup, extracting key/value pairs and adding them to outgoing HTTP requests. Introduces traversal vars (head, mv, key, val).

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant FB as Fluent Bit
    participant ES as ES Output Plugin
    participant HC as HTTP Client
    participant ELS as Elasticsearch

    FB->>ES: flush(cb_es_flush)
    Note over ES: Load config (ctx), incl. headers list
    ES->>HC: build request (URI, payload)
    loop For each configured header
        ES->>HC: addHeader(key, value)
    end
    HC->>ELS: HTTP request with headers
    ELS-->>HC: HTTP response
    HC-->>ES: deliver response
    ES-->>FB: flush result

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

I tuck new keys in my satchel tight,
Hop-hop I go, by moonlit byte—
Headers stitched to every quest,
Whispered names on each request.
Elasticsearch hears my tap-tap-tap,
A rabbit courier with an HTTP cap.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title clearly and concisely summarizes the primary feature—adding custom HTTP headers support for the Elasticsearch output plugin—using the established “out_es” prefix and specific terminology without unnecessary detail or ambiguity.

✨ Finishing touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🧪 Early access (Sonnet 4.5): enabled

We are currently testing the Sonnet 4.5 model, which is expected to improve code review quality. However, this model may lead to increased noise levels in the review comments. Please disable the early access features if the noise level causes any inconvenience.

Note:

• Public repositories are always opted into early access features.
• You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 0

🧹 Nitpick comments (1)

plugins/out_es/es.c (1)

890-898: Validate header entries and handle flb_http_add_header failures

• Check the return value of flb_http_add_header and log or abort on -1 to avoid silent header‐addition errors.
• Before calling flb_http_add_header, verify mv->val.list contains exactly two entries (key + value); reject or warn on single‐token configs to prevent “X: X” headers.

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between d985e8e and 4653a2d.

📒 Files selected for processing (2)

• plugins/out_es/es.c (3 hunks)
• plugins/out_es/es.h (1 hunks)

🧰 Additional context used

🧬 Code graph analysis (1)

plugins/out_es/es.c (1)

src/flb_http_client.c (1)

• flb_http_add_header (963-995)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (29)

• GitHub Check: pr-windows-build / call-build-windows-package (Windows 32bit, x86, x86-windows-static, 3.31.6)
• GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit, x64, x64-windows-static, 3.31.6)
• GitHub Check: pr-windows-build / call-build-windows-package (Windows 64bit (Arm64), amd64_arm64, -DCMAKE_SYSTEM_NAME=Windows -DCMA...
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_COVERAGE=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_THREAD=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_ARROW=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SIMD=Off, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SANITIZE_MEMORY=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_SMALL=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_UNDEFINED=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, clang, clang++)
• GitHub Check: run-ubuntu-unit-tests (-DSANITIZE_ADDRESS=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=On, 3.31.6, gcc, g++)
• GitHub Check: run-ubuntu-unit-tests (-DFLB_JEMALLOC=Off, 3.31.6, gcc, g++)
• GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-22.04, clang-12)
• GitHub Check: PR - fuzzing test
• GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, gcc, g++, ubuntu-24.04, clang-14)
• GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-24.04, clang-14)
• GitHub Check: pr-compile-system-libs (-DFLB_PREFER_SYSTEM_LIBS=On, 3.31.6, clang, clang++, ubuntu-22.04, clang-12)
• GitHub Check: pr-compile-centos-7

🔇 Additional comments (3)

plugins/out_es/es.h (1)

54-56: LGTM! Headers field added correctly.

The headers field is properly declared as a pointer to struct mk_list and is appropriately positioned with other HTTP-related configuration fields. This aligns with the config map entry in es.c that uses FLB_CONFIG_MAP_SLIST_1.

plugins/out_es/es.c (2)

828-832: LGTM! Variable declarations are appropriate.

The new variables (head, mv, key, val) are correctly typed for iterating over the headers configuration using flb_config_map_foreach.

---

1143-1148: LGTM! Configuration entry properly defined.

The config map entry correctly uses FLB_CONFIG_MAP_SLIST_1 for key/value pairs with FLB_CONFIG_MAP_MULT to allow multiple headers. The description clearly communicates the functionality.

[eschabell]

@patrick-stephens can you review this PR?

[patrick-stephens]

Looks fine to me although I was expecting us to just use the HTTP plugin configuration for this, plus we probably want to replicate it to Opensearch if not there already.