ProducerManager: Commit Offsets After Transaction Expired

[vladimirocorsi]

Hi,

I am new to Kafka and to this project as well but I am wondering what happens when ProducerManager commits offsets after a sufficiently long period of inactivity (> transaction.timeout.ms ?) which causes the transaction to timeout. Does the parallel consumer just stop?

Thanks in advance

Vladimiro

[astubbs]

Great question! Sorry for missing it!

No, the PC will initially not stop, it will simply fail committing the transaction, and retry.

If a tx fails to commit after 200 attempts, it may crash. In which case the partition will be handed off and retried. If a consumer in your group crashes, your process monitoring should restart a new instance of it.

Currently however, incorrectly, If the transaction cannot be committed because it has been aborted by the broker, it will be still be retried. This is probably not correct, and we should instead proactively handle different types of failures correctly. See issue #144. This will cause a terminal failure to just take longer to resolve.

Note that most people don't use the Producer commit mode, they use the async consumer mode: CommitMode#PERIODIC_CONSUMER_ASYNCHRONOUS (the default).