Questions About Moving From Sarama To Confluent Kafka Go

[ianjhoffman]

My team is currently using the sarama Go Kafka library as our Kafka client but are thinking of qualifying & switching to confluent-kafka-go due to issues with how sarama handles offline partitions and how it handles partition choices (with a random partitioner not requiring consistency) when a write fails. I had a few questions on how to replicate our current client's behavior and also about how confluent-kafka-go handles offline partitions.

1. The documentation for librdkafka configuration states that for request.required.acks,

   This field indicates the number of acknowledgements the leader broker must receive from ISR brokers before responding to the request: 0=Broker does not send any response/ack to client

   The default behavior for sarama is to consider a commit to be a success once a single ack from the leader is received. Is that equivalent to 0 in request.required.acks? The phrase Broker does not send any response/ack to client is a bit ominous and almost sounds like even the leader won't send an ack.

2. If we use the random partitioner, will retries (as configured by message.send.max.retries) pick a new partition each time or is the random partition picked only once per message?

3. The sarama SyncProducer allows us to have several goroutines all writing messages to one producer and learning about their writes' successes/failures synchronously. It wraps an async producer much like the producer in confluent-kafka-go and creates a new channel per write for success/failure delivery. Is this an acceptable pattern to use in a synchronous wrapper for the confluent-kafka-go producer?

4. Is there any information I can read about how confluent-kafka-go/librdkafka handles offline partitions? We've noticed that sarama does not do a great job of detecting/avoiding partitions that cannot be written to. In combination with pinning each message to the first partition picked by the producer's partitioner, this means that some messages are just doomed even with retries.

Thank you! I look forward to hearing more about how we can make our Kafka client more resilient through the use of confluent-kafka-go.

[edenhill]

1. The Sarama default seems to be equivalent to acks=1, e.g., only require ack from the partition leader. We strongly recommend acks=all (librdkafka default) and properly replicated topics (3 replicas) to avoid silent data loss.
2. ..
   a. First off we recommend using the Java compatible murmur2_random.
   b. Secondly, any random partitioner in librdkafka will (by default) use sticky partitioning, that is, key-less messages will be randomly distributed across available partitions but will stick with a single partition for sticky.partitioning.linger.ms (10ms) to improve batch accumulation (and thus throughput).
   c. Thirdly, the partitioner is only run once per message. Typically inside produce() but sometimes later if the topic metadata is not known. But regardless, a message will never change partition internally.
3. confluent-kafka-go's Produce() is also asynchronous and delivery reports will be emitted on the producer.Events() channel once the message has been successfully delivered or permanently fails (temporary errors will be retried automatically). You can also pass a specific channel to Produce() if you want a message's delivery report event on your custom channel.
4. The random partitioners (all with random in their name) will try to avoid partitioning messages for partitions that do not have a leader.

Also consider enable idempotency, which provides great message delivery guarantees at very little extra cost. See https://github.com/edenhill/librdkafka/blob/master/INTRODUCTION.md#idempotent-producer

[ianjhoffman]

Thanks for the quick reply @edenhill ! For a long time we've been qualifying AWS SQS as a replacement for our use of Kafka but the SendMessage latency was a major turn-off for us. It sounds like we may have to eat a similar amount of latency anyways if we use acks=all (I believe we are using properly replicated topics), but it will also let us use the idempotent producer guarantees. Thanks for the clarity on sticky partitioning. Having looked at Shopify/sarama and segmentio/kafka-go it appears that this is standard behavior across all Go Kafka clients, so we'll just have to trust that librdkafka does a better job at keeping its metadata on offline partitions up to date than sarama (I think this is likely).

[edenhill]

I'm no expert in SQS, but Kafka should easily outperform SQS both in latency and thruput.
A well tuned Kafka cluster has produce latencies with acks=all in the single digit millisecond space.

[ianjhoffman]

Yeah the p50 write latency for SQS is 8ms in our experience. Right now with acks=1 with our cluster and producer settings we're seeing a p50 Kafka write latency of around 920µs. We'll have to see how that timing scales when we bump up our acks, it'll be interesting to see how cross-AZ broker communication contributes to it.