Possible to test dedupe transform?

[davidcpell]

Is it possible to test a dedupe transform? Say I have something like this:

transforms:
  dedupe-tfm-1:
    type: dedupe
    inputs: ["router.level_warn"]
    fields:
      match:
      - data.message.error

tests:
- name: dedupes warn logs based on error message
  inputs:
  - insert_at: dedupe-tfm-1
    type: log
    log_fields:
      data.message.level: warn
      data.message.error: "message 1"
  - insert_at: dedupe-tfm-1
    type: log
    log_fields:
      data.message.level: warn
      data.message.error: "message 2"
  - insert_at: dedupe-tfm-1
    type: log
    log_fields:
      data.message.level: warn
      data.message.error: "message 2"
  outputs:
  - extract_from: dedupe-tfm-1
    conditions:

Do these inputs work? What would the output conditions look like?

[pront]

This full example should help you:

transforms:
  dedupe-tfm-1:
    type: dedupe
    inputs: ["router.level_warn"]
    fields:
      match:
      - data.message.error
tests:
- name: dedupes warn logs based on error message
  inputs:
  - insert_at: dedupe-tfm-1
    type: log
    log_fields:
      data.message.level: warn
      data.message.error: "message 1"
  - insert_at: dedupe-tfm-1
    type: log
    log_fields:
      data.message.level: warn
      data.message.error: "message 2"
  - insert_at: dedupe-tfm-1
    type: log
    log_fields:
      data.message.level: warn
      data.message.error: "message 2"
  outputs:
  - extract_from: dedupe-tfm-1
    conditions:
      - type: vrl
        source: |
          assert_eq!(.data, { "message": { "error": "message 2", "level": "warn" }})

Note we cannot assert on the number of expected events. This is very useful enhancement tracked by #20278.

[davidcpell]

Thanks. As I understand it, this will ensure that "message 2" appears once but not only once. It also does not validate that "message 1" is also emitted?

[pront]

this will ensure that "message 2" appears once but not only once.

Correct. We need the enhancement described in #20278 to do this.

It also does not validate that "message 1" is also emitted?

I think you can add another assertion for this.