fluentbit in and out to support hec_token passthrough

[CrashLaker]

Is your feature request related to a problem? Please describe.

Hi all,

i'd like to ask if it's possible to have the same behavior as fluentbit version prior to 4.0.2 where this change was made #10329

in a multi-tenant central logging platform i want to be able to passthrough the splunk hec token from the sources to splunk. to keep their respective indexes.

diagram of what I need.

Describe the solution you'd like

I need to be able to passthrough the hec token set by the sources
example:

server1 fluentbit.conf

[OUTPUT]
  name splunk
  host server3
  Splunk_Token AAA

server2 fluentbit.conf

[OUTPUT]
  name splunk
  host server3
  Splunk_Token BBB

server3 fluentbit.conf

[INPUT]
  name splunk
  store_token_in_metadata true

[OUTPUT]
  name splunk
  host splunk_server
  Splunk_Token ZZZ

behavior in fluentbit 4.0.1
server3 sends logs from server1 and server2 to splunk_server using their respective index in splunk

behavior in fluentbit 4.2.1
server3 overwrites hec_token to ZZZ therefore all logs goes to one index

Describe alternatives you've considered
For now 4.0.1 version solves this problem but not 100% my need (addressed in additional context).

one suggestion would be to allow Splunk_Token to be blank optional - now it's obligatory
if Splunk_Token is blank it then uses the one from metadata hec_token

Additional context
I'm assessing this solution to enrich data before sending to datadog for example.
if I have the origin hec_token I can enrich with team, account_id tag before sending to datadog

problem is that to access the log metadata in lua i need at least version 4.0.4
https://docs.fluentbit.io/manual/data-pipeline/filters/lua#lua-extended-callback-with-groups-and-metadata-support

currently this is the solution that would work

• use 4.0.1 to pass hec_token forward
• to 4.2.1 to enrich using lua filter

thanks,
regards,c

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Possible Duplicates

• Splunk_token from in_splunk overwrites splunk_token from out_splunk #10208

Related Issues

• in_splunk with http2 enabled fails to accept correct token #10483
• Splunk output requires token as plain text #10834
• Add support for AWS Enable_Identity #10340
• Add source IP to Forward input plugin #10748

👤 Suggested Assignees

• edinel
• lecaros
• BartekCipirski
• max-allan-cgr
• kam193

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!

[CrashLaker]

I was studying the code and found out that if i force http_passwd to NULL I can then get the desired result

in line 964 below

fluent-bit/plugins/out_splunk/splunk.c

Lines 964 to 975 in ddfef36

	if (ctx->http_user && ctx->http_passwd) {
	flb_http_basic_auth(c, ctx->http_user, ctx->http_passwd);
	}
	else if (ctx->auth_header) {
	flb_http_add_header(c, "Authorization", 13,
	ctx->auth_header, flb_sds_len(ctx->auth_header));
	}
	else if (metadata_auth_header) {
	flb_http_add_header(c, "Authorization", 13,
	metadata_auth_header,
	flb_sds_len(metadata_auth_header));
	}

since it's default value is being set to "" here

fluent-bit/plugins/out_splunk/splunk.c

Lines 1075 to 1076 in ddfef36

	FLB_CONFIG_MAP_STR, "http_passwd", "",
	0, FLB_TRUE, offsetof(struct flb_splunk, http_passwd),

just tested it. if i change it to FLB_CONFIG_MAP_STR, "http_passwd", NULL, then the if case goes to hec from input

because if pass in C.

#include <stdio.h>

void main(){
    char * c = "";
    if (c){
        printf("pass\n");
    }
}

output: pass

#include <stdio.h>

void main(){
    char * c = NULL;
    if (c){
        printf("pass\n");
    }
}

output:

can I do a PR?
this would be my first time opening it on a big oss project. should I just push it to the master branch?

regards,c