Default for max.in.flight.requests.per.connection breaks idempotency.

[maowerner]

According to the Kafka documentation of enable.idempotence

enabling idempotence requires max.in.flight.requests.per.connection to be less than or equal to 5 (with message ordering preserved for any allowable value), retries to be greater than 0, and acks must be 'all'.

This is because the leader keeps updating the five last sequence IDs every time a new message is produced. However if more than five messages are inflight, it is possible that messages are duplicated because their sequence ID ran out of cache.

Since apache/kafka#11788 which landed in Kafka 3.2 and was backported to 3.0 and 3.1 a max.in.fight.request > 5 idempotence is disabled implicitly in the Java client itself:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L545

The librdkafka default is 1.000.000. Looking through old issues I found #831 (comment)

max.inflight..=1 is a performance killer since the thoughput rate will be directly limited by producer roundtrip time

which is older than idempotence and does not appear to have changed since. In a "fire-and-forget" scenario I would arguably prefer setting acks=0 if data loss is okay or tuning batch.size before changing max.inflight.requests, but I can agree with the setting. Since idempotence is a thing though, would it be possible to either change the default to 5 or through some kind of error?

[edenhill]

max.in.flight is automatically adjusted to 5 when transactions or the idempotent producer is enabled.

[maowerner]

Could you point me to where this is set? I cannot find it for the life of me and our librdkafka app also logs

  "enable.idempotence": "true",
  ...
  "max.in.flight.requests.per.connection": "1000000",

at startup. As it is a producer config, this has to be somewhere in librdkafka itself; is this line of reasoning correct?

[edenhill]

https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_conf.c#L3833-L3849