Skip to content

Support to exclude fields from concatenate_all_fields #43

@ypid-geberit

Description

@ypid-geberit

I would like to use all fields in an event for fingerprint calculation except a few of them which are non-deterministic when re-importing logs using the file or unix socket input for example. This is useful when the list of fields varies and is not fully known. Example use case:

fingerprint {
  concatenate_all_fields => true
  exclude => [
    "#logstash_timestamp",

    "@version",
    "path",
    "port",
    "input_type",
    "type",
    "source",

    "host",
    "@timestamp"
  ]
  target => "[@metadata][_id]"
  method => "SHA512"
  key => "XXX"
  base64encode => true
}

Hint: The @timestamp is excluded because when you re-import logs, Logstash sets the @timestamp at input stage and it is only later overwritten by the timestamp extracted from the log line. Fingerprint calculation is done as early as possible to still generate the same fingerprint even when the processing/transformation of the event is changed.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions