json keys that look like references get dereferenced #38
Description
- Version: logstash-filter-json (3.0.5) (as part of logstash 6.5.1)
- Operating System: osx 10.13.6
- Config File (if you have sensitive info, please remove it):
output {
stdout {
codec => rubydebug
}
}
input {
generator {
lines => [
'{"foo":"bar","nested":{"one":"two"},"not_nested[\"three\"]":"four","[what][is][this]":"five"}'
]
count => 1
}
}
filter {
json {
source => "message"
remove_field => ["message"]
}
}
- Sample Data: in conf above
- Steps to Reproduce:
logstash -f bug.conf
Expected:
{
"sequence" => 0,
"@timestamp" => 2018-12-05T18:28:29.623Z,
"[what][is][this]" => "five",
"@version" => "1",
"host" => "laptop.local",
"not_nested[\"three\"]" => "four",
"foo" => "bar",
"nested" => {
"one" => "two"
}
}
Actual:
{
"sequence" => 0,
"@timestamp" => 2018-12-05T18:28:29.623Z,
"what" => {
"is" => {
"this" => "five"
}
},
"@version" => "1",
"host" => "laptop.local",
"not_nested" => {
"\"three\"" => "four"
},
"foo" => "bar",
"nested" => {
"one" => "two"
}
}
We spotted this with log data where a query string had ref-like keys, and got the "Detected ambiguous Field Reference" warning. I was surprised that it's doing anything with this at all, and there doesn't seem to be an option to disable this behaviour - these are keys in our log data not conf for processing.