Is there any to do stateful dedupe (across restart)

[BhautikChudasama]

Hello there,
Is there any to do stateful dedupe (across restart)? Currently, the state is flush on restart.

[jszwedko]

Unfortunately not at the moment, but that would be a useful feature. It could leverage the existing scaffolding other components (like the file source) use for storing checkpoints.

[BhautikChudasama]

Hey @jszwedko thanks for reply,

How because I failed. Current flow is
Kafka -> dedupe -> transform -> clickhouse

I though but failed.
Kafka -> dedupe -> transform -> file -> dedupe -> clickhouse

In above case (Restart happen), again duplicate events write to file and dedupe fail.