Decoder don't support new timestamp + metadata log record format

[edsiper]

In Fluent Bit we introduced a new format for log records:

[[timestamp, {METADATA}], {EVENT}]

Fluent Bit code base has been updated to work with the old format and the new one. But the golang decoder is not updated and needs to be fixed.

[cosmo0920]

I tested this for in_dummy and gstdout that is built by this Golang interface.
But, it works.
Is there a fluent-bit configuration any containing for this issue? Or, minimal reproducible steps?
IIRC, there are a few cases to add metadata on fluent-bit. One of the main scenarios is using multiline feature.

[edsiper]

How do you extract metadata by using the golang output decoder?

…

On Fri, Jul 14, 2023, 04:51 Hiroshi Hatake ***@***.***> wrote: I tested this for in_dummy and gstdout that is built by this Golang interface. But, it works. Is there a fluent-bit configuration any containing for this issue? Or, minimal reproducible steps? — Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC2INVW76O5YFPX6V46ZX3XQEQCNANCNFSM6AAAAAA2JGA2KY> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[cosmo0920]

Ah, sorry for confusing. I misunderstood for this issue and I wrote down a patch for handling this issue.

[randomizedcoder]

Tried to add a little more to the tests, but it fails.
#69

[rasturic]

I'm looking forward to the fix as our company depends on https://github.com/logzio/fluent-bit-logzio-output which uses this.

[randomizedcoder]

I realized today that Fluentbit can send an array messages, rather than single message at a time. This is probably good for performance (batching), but there are no tests with this type. I only noticed because I increased the message rate of my test.

I just added a few more test data files here
https://github.com/randomizedcoder/fluent-bit-go/tree/tweaks/output/testdata

Decoding the msgpack to yaml, you can see the data1 file turns into data1.yaml. The second record starts here, for example
https://github.com/randomizedcoder/fluent-bit-go/blob/tweaks/output/testdata/data1.yaml#L527