Skip to content

Allow message field as object #522

@kaisecheng

Description

@kaisecheng

The plugin transforms message field from object to string 1, 2.
In the case of an Agent with SailPoint Integration sending data directly to Elasticsearch, everything works fine. However, when Logstash is used as a passthrough between the Agent and Elasticsearch, the message field changes the data structure, causing Elasticsearch to return a 400 error.

The source events are in valid format. message field should be allowed to accept object type.

The solution need to be discussed.
One quick and dirty fix would be to add an option to configure the field, instead of hardcoding it to message

Reproducer

filebeat sends json through stdin
echo '{"message": {"HIHI": "from filebeat"}}' | ./filebeat -e -c filebeat.yml

filebeat.inputs:
- type: stdin
  enabled: true

output.logstash:
  hosts: ["localhost:5044"]

logstash receive data

input {
    beats {
        port => "5044"
        enrich => "none"
    }
}
output {
    stdout {}
}

Logstash v7.17 and v8 give different string formats when stringifying the message.

v7

 "message" => "{\"HIHI\"=>\"from filebeat\"}",

v8

 "message" => "{HIHI=from filebeat}",

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions