Achieving exactly-once / "effectively-once" semantics

[alexandergunnarson]

Hey! Thanks so much for making what looks like an amazing piece of software.

So I'm new to Kafka and am trying to achieve exactly-once / "effectively-once" semantics with ParallelConsumer. I've set the following in producer config:

"acks" = "all"
"enable.idempotence" = "true"
// This doesn't effectuate zombie fencing
"transactional.id" = "<processor-name>-<container-id>"

and the following in consumer config:

// Not sure if this will break `ParallelConsumer`
"isolation.level" = "read_committed"

and am using ParallelConsumerOptions.CommitMode.PERIODIC_TRANSACTIONAL_PRODUCER. It seems I'm missing at very least zombie fencing — probably related is #144, since I see ProducerFencedException in your brainstorming — but other than that, is all this enough for achieve exactly-once / "effectively-once" semantics?

Thanks!

[astubbs]

Yes, that's right. But remember, Kafka's EoS only applies to messages in Kafka - not side effects of message processing. I.e. you may get duplicate calls to your web service or whatever you are doing in message processing. It's only useful when you want a poll and produce loop and you want to ensure exactly once existence of result messages in the output topics.

When reading, don't forget to use isolation.level=read_committed in your consumer.

[alexandergunnarson]

Thanks so much for your helpful reply @astubbs!