Can't get dedup to work

[securitymonster]

Hi all, I'm new to Vector, only started experimenting two weeks ago. I have an issue where I would like to dedup some events.

My config looks like this at the moment:

  parse_saml_assertions_logs:
    type: "remap"
    inputs: [saml_assertions_logs]
    source: |
      parts = split(string!(.message), "|")
        .forwarded_for = parts[0]
        .user = parts[1]
        .timestamp = parse_timestamp!(parts[2], "%+")
        .host = parts[3]
        .user_agent = parts[4]
        .assertion_id = parts[5]
        .name_id = parts[6]
      del(.message)

  dedupe_saml_assertions_logs:
      type: "dedupe"
      inputs: [parse_saml_assertions_logs]
      fields:
        ignore: [".timestamp"]

So "sinking" parse_saml_assertions_logs works fine, however sinking dedupe_saml_assertions_logs does result in zero output. I tried with fields.match as well. What am I missing?

[pront]

Hi @securitymonster, can you give me an example of two events that want to be treated as duplicates? Your dedupe might be missing match fields. Also, you can inspect component_discarded_events_total metrics. For debugging tools you can read more here: https://vector.dev/guides/developer/debugging/#vector-tools