Loki Output structured_metadata from Map-like data structure

[0x006EA1E5]

Is your feature request related to a problem? Please describe.

We want to use the Loki structure_metadata feature to store data alongside a given log message. This is currently possible with the loki output, but we need to explicitly define key/value pairs. This obviously requires us to know all key names ahead of runtime, so they can be defined in the config.

However. in our use case, we want to be able to handle arbitrary keys detected at runtime.

Describe the solution you'd like
We are open minded about how to solve this, but the most obvious idea would be to be able to configure the loki output with a Map-like data structure.

Imagine we input data like this "__attributes": {"test_1": "hello", "test_2": "World!"}

We can already configure the 'loki' output like this:

  outputs:
    - name: loki
      sructured_metadata: test_1=$attributes['test_1'], test_2=$attributes['test_2']

The proposal is to be able to do the following, to acheive the same result with this input data described above:

  outputs:
    - name: loki
      sructured_metadata_map: $attributes

...and indeed, if more keys were present in the attributes object, then they would also be added.

Describe alternatives you've considered
I wondered if it was possible to extend the Record Accessor syntax to have some kind of "varargs".

For example, if we could do the following to say the argument should be used as a set of keys (note trailing ... on $attributes)

  outputs:
    - name: loki
      sructured_metadata_map: $attributes...

Additional context

Within our organisation, we are an infrastructure team providing logging systems to application teams. Application teams are simply expected to output logs as JSON to the console.

We want to support as many logging use cases as possible, and essentially want application teams to be able to determine what keys go into structured metadata themselves, without us having to hard code every possibility into the fluent-bit config

[0x006EA1E5]

Hi all, I have created a PR to address this, but as I mentioned in the Fluent Community Meeting on ≈3rd October:

• It has been many years since I actively wrote C, so I probably need hand-holding, and am very happy to be told how to do it.
• I want to submit some kind of proof of concept first, to get a steer from the maintainers if this is the right direction and/or acceptable feature, as early as possible, and before sinking too much effort into it.

I am happy to add more to this, once I get some feedback.

[0x006EA1E5]

I created a PR for the docs here fluent/fluent-bit-docs#1527

[0x006EA1E5]

I creates a PR to backport this to 3.2.x #10012