JSon parser with decoder

[cfeltz34]

Reference : https://docs.fluentbit.io/manual/pipeline/parsers/decoders

I have a json parser like this :

[PARSER]
    Name         json
    Format       json
    Decode_Field_As    json    log

# Try parsing log as json and lift its keys to the first-level
[FILTER]
    Name parser
    Match *
    Parser json
    Key_Name log
    Reserve_Data On
    Preserve_Key On

My input is from Fluentd logging driver (docker) :

[INPUT]
    Name forward
    Tag logger
    Listen 0.0.0.0
    port 24224

My forward input is :

2024-03-14 09:26:00 [0] logger: [[1710404038.000000000, {}], {"source"=>"stdout", "log"=>"{"level":"Warning","message":"Connection Failed","exceptions":["Message":"ex1","StackTraceString":"   at source1\n   at source2\n   at source3"]}]

My output is :

2024-03-14 09:26:07 [0] logger: [[1710404766.000000000, {}], {"level"=>"Warning", "message"=>"Connection Failed", "exceptions"=>["Message"=>"ex1", "StackTraceString"=>"   at source1
2024-03-14 09:26:07    at source2
2024-03-14 09:26:07    at source3]}]

Expected result : only one line like this :

2024-03-14 09:26:07 [0] logger: [[1710404766.000000000, {}], {"level"=>"Warning", "message"=>"Connection Failed", "exceptions"=>["Message"=>"ex1", "StackTraceString"=>"   at source1\n   at source2\n   at source3]}]

Can you help me please

[patrick-stephens]

What output are you talking about there? It looks like it is interpreting it as a newline within the string but you'll notice Fluent Bit is not adding any further timestamps so I think this is just an output issue - the actual record is a single one with embedded new lines.

[cfeltz34]

I have two outputs, the first is the FluentBit stdout output from where I copied the result.
The second, which is the one that really interests me and which has the same problem (it creates as many records as lines in my stdout), is ElasticSearch.

# send logs to stdout
[OUTPUT]
    Name stdout
    Match logger

# send logs to elasticsearch
[OUTPUT]
    Name es
    Match logger
    Host ${ELASTICSEARCH_HOST}
    Path ${ELASTICSEARCH_PATH}
    Port ${ELASTICSEARCH_PORT}
    Logstash_Format On
    Logstash_Prefix logs
    Generate_ID On
    Suppress_Type_Name On
    Trace_Error    On
    Replace_Dots On
    #Mem_Buf_Limit 512M