Feature Request - Adding Logged filename to the metadata

[OfirYaron]

Is your feature request related to a problem? Please describe.
I'm currently working on a logging setup where my Kubernetes cluster, running on AWS EKS, generates logs across approximately 20 files. The challenge I'm encountering revolves around the static nature of the Fluent Bit configuration. Each new log file necessitates a manual addition of output/input configurations, and this has become a bit cumbersome as the number of files grows.

Describe the solution you'd like

Describe alternatives you've considered
I'm envisioning a feature that introduces dynamic index routing based on the filename. Ideally, a syntax akin to ${@metadata__filename} could be incorporated into the Elasticsearch output plugin configuration. This would allow Fluent Bit to dynamically generate indices based on the source filename, providing a seamless solution for ingesting logs from multiple files without the need for constant manual intervention.

A sample configuration might look like this:

[OUTPUT]
    Name         es
    Match        kube.*
    Host         your-elasticsearch-host
    Port         9200
    Logstash_Format On
    Index        ${@metadata__filename}

Additional context
I understand that feature development involves careful consideration, and I appreciate your time and efforts in evaluating this request. Your continuous dedication to refining Fluent Bit is truly commendable.
Thank you for your attention, and I look forward to any insights.

[patrick-stephens]

If you're using tail you can already include the filename via the path key.
https://docs.fluentbit.io/manual/pipeline/inputs/tail
Also, if you use a wildcard in the tag it will then inject the full filename into the tag as well. The K8S filter uses this in fact.

I've used this in the past to then easily use it to simplify filters in Grafana for example: https://www.couchbase.com/blog/fluent-bit-tips-tricks-log-forwarding-couchbase/#aliases

I think in your case, the issue is that ES output does not support record-accessor syntax for the index but Opensearch does. However there is a PR open I believe to add this. Then you could just look up the index from a key and you can set that key however you like (e.g. a custom filter or just the path key from tail).

EDIT:

• PR: out_es: Add target_index variable using record accessor syntax. #7716

[mp3monster]

The key value in having the input include the file name is how to know which file was the source of the log event when the input path has wildcards in it.

having the path would allow the pipeline to extract context from it. For example if my J2EE app server has multiple apps logging to a common logging location in the file system. I either need to have a tail defined for each app - and use the tag to derive context - messy and error prone if we forget the config needing to be added to Fluent Bit. Or we have Fluent Bit use a wildcard in the path, and then with the full file path add a filter take the name/infer the app from the source file

[patrick-stephens]

Yeah so the path key and tag (when using wildcards) both include the full filename at the moment.
I agree it might be nice to have some formal metadata around it now as well particularly to help with misconfiguration in the K8S filter but I wanted to cover the options available now.

[DanielOved93]

I am also trying to grasp how to reference the filename in the output, especially based on the path.
Let me provide an example input configuration:

[INPUT]
    Name   tail
    Path   /var/log/myapp/*.log
    Tag    myapp.logs

Assuming I want to dynamically generate indices in the Elasticsearch output based on the filenames (e.g., mylog1.log, mylog2.log), how would I reference the filename in the output configuration?
For instance, in the output configuration, would it look something like this:

[OUTPUT]
    Name         es
    Match        myapp.logs
    Host         your-elasticsearch-host
    Port         9200
    Logstash_Format On
    Index        ${?filename_variable}

I used ${?filename_variable} as a placeholder since I'm not sure about the actual syntax. Additionally, I'm not certain I fully understood the solution provided below.
Could you kindly clarify or provide an example based on the given input configuration?
Any guidance on how to achieve this dynamic referencing would be highly appreciated!

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.