Rename field mapper_parsing_exception

[mibaboo]

Hi

I'm running fluentbit to push logs to opensearch from kubernetes. I have multiple application which have a field "message" and the type is string. There is a new 3rd party application deployed and it too has a field "message" but the type is an Object and I would like to keep it that way. What I would like to know , is it possible to rename the field for the new app from "message" to "newmessage" this way I wont get any mapper_parsing_exception

[patrick-stephens]

I would look at the docs for the modify/record_modifier filters, these can add/update/delete fields.
Mapping type issues sounds like this too: https://docs.fluentbit.io/manual/pipeline/outputs/opensearch#action-metadata-contains-an-unknown-parameter-type

[patrick-stephens]

What's the issue? It would help to have input/output and any logs.

I'd recommend trying this all locally with simple dummy input and stdout output, you can easily test and insert filters then with sample data plus people can see what you're doing and reproduce it themselves to help.

https://calyptia.com/blog/fluent-bit-tips-tricks

[isatfg]

The issue is that I'm getting the following error in the fluentbit logs

{"took":43,"errors":true,"items":[{"create":{"_index":"k8s-staging-2023.06.30","_type":"_doc","_id":"0d7QGokBwF80o6YY2fN3","status":400,"error":{"type":"mapper_parsing_exception","reason":"failed to parse field [On.message] of type [text] in document with id '0d7QGokBwF80o6YY2fN3'. Preview of field's value: '{CPU Usage % for process=1.10%, pid=74, Memory Stats for process={heapUsed=283.317608 MB, rss=536.403968 MB, heapTotal=365.44512 MB, percentageHeapUsed=77.53%}, Memory Stats for container={freeSystemMemory=4104.282112 MB, percentageFreeMemory=49.9%, totalSystemMemory=8224.37888 MB}}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:131"}}}}]}

This is the config

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    multiline.parser docker, cri
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On
[FILTER]
    Name kubernetes
    Match kube.*
    Merge_Log On
    Merge_Log_Key On
    Keep_Log Off
    K8S-Logging.Parser On
    K8S-Logging.Exclude On
[FILTER]
    Name modify
    Match *
    Hard_rename app k8s-app
    Hard_rename kubernetes.labels.app kubernetes.labels.k8s-app
# Testing name field
[FILTER]
    Name modify
    Match *
    Condition Key_Value_Equals kubernetes.labels.app_kubernetes_io/name retool
    Rename message retoolmessage
# Testing name field
[FILTER]
    Name modify
    Match *
    Condition Key_Value_Equals kubernetes.labels.app_kubernetes_io/name retool-jobs-runner
    Rename message retoolmessage

[OUTPUT]
    Name es
    Match kube.*
    Host $HOST 
    Port 443
    tls on
    Index k8s
    Logstash_Format On
    Logstash_Prefix k8s-staging
    Retry_Limit False
    HTTP_User USER
    HTTP_Passwd PASS
    Trace_Error On
    Replace_Dots On

[OUTPUT]
    Name es
    Match host.*
    Host $HOST 
    Port 443
    tls on
    Index k8s
    Logstash_Format On
    Logstash_Prefix node
    Retry_Limit False
    HTTP_User USER 
    HTTP_Passwd PASS 
    Trace_Error On
    Replace_Dots On
[PARSER]
    # https://rubular.com/r/IhIbCAIs7ImOkc
    Name        k8s-nginx-ingress
    Format      regex
    Regex       ^(?<host>[^ ]*) - (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*) "(?<referer>[^\"]*)" "(?<agent>[^\"]*)" (?<request_length>[^ ]*) (?<request_time>[^ ]*) \[(?<proxy_upstream_name>[^ ]*)\] (\[(?<proxy_alternative_upstream_name>[^ ]*)\] )?(?<upstream_addr>[^ ]*) (?<upstream_response_length>[^ ]*) (?<upstream_response_time>[^ ]*) (?<upstream_status>[^ ]*) (?<reg_id>[^ ]*).*$
    Time_Key    time
    Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
    Name docker_no_time
    Format json
    Time_Keep Off
    Time_Key time
    Time_Format %Y-%m-%dT%H:%M:%S.%L

[patrick-stephens]

So https://fluent-all.slack.com/archives/C0CTQGHKJ/p1688131288739689

However that error is not related to what you've asked above. Renaming an object is trivial using the filters indicated above.

You then said So I tried modify and I think I'm close but still not working. What is not working? The rename? How is it failing and what did you do/try?

If you mean you are getting the same error then it sounds like it is not an issue with rename but it is rejected by Opensearch. Aravind gave some guidance on how to fix this: https://fluent-all.slack.com/archives/C0CTQGHKJ/p1688133494816959?thread_ts=1688131288.739689&cid=C0CTQGHKJ and https://discuss.elastic.co/t/mapper-parsing-exception/94617

[isatfg]

Thanks for the help thus far @patrick-stephens.

Ok so I have tested locally as you suggested and I am able to rename a field. I changed my config in my k8s cluster to print to stdout just to check the fields and found a field to match in my condition. It's seems to be matching now how ever it's creating mutiple fields

My Filter is

        Name modify
        Match *
        Condition Key_Value_Equals app.kubernetes.io/instance bash-retool
        rename message retoolmessage

However in Opensearch I now see many message fields before I has 10 now I have 60
Eg. of a few

On.message.CPU Usage % for process
On.message.Memory Stats for process.heapUsed

I was hoping to see one new field
On.retoolmessage

[patrick-stephens]

Can you provide a sample of input and output with stdout?
I can't debug what you have in Opensearch I'm afraid.

All that filter should do is rename the existing field for the particular record that matches the condition. So, is it not doing that?