Don't override user-supplied data_stream fields #24683
Description
While adding support for data_stream.dataset and data_stream.namespace fields for ECS loggers (elastic/ecs-logging#38). I noticed that Filebeat overrides the fields with the static value from the integration policy rather than using the fields from the log events. That happens even though the input setting json.override_keys is set to true.
I do think we'd want users to define the dataset in their ECS logging configuration for several reasons:
- We don't want users having to create a dedicated log configuration per application. Ideally, they'd be able to set up the logging for all their applications with just one integration. For example, by providing several log file paths pointing to ECS log files of different applications.
- We already make
event.datasetconfigurable which is used for the log anomaly ML job. Couldn't we use that to set thedata_stream.dataset? No, we can't: Theevent.datasetanddata_stream.datasetfields should always have the same values. However, thedata_stream.datasetimposes more restrictions on the allowed characters (as it ends up as a part of the index name). Thus, we can't use the values ofevent.datasetfordata_stream.dataset.