max_buffer_bytesize for async_producer

[alan-sapaad]

I am able to set max_buffer_bytesize for a synchronous producer and as expected I am receiving the error Kafka::BufferOverflow: Cannot produce to topic, max buffer bytesize (960000 bytes) reached

But I need to use an async_producer with delivery_interval set to 50ms (0.05). I've set max_buffer_bytesize when initializing the producer. But I am able to push as many messages as possible to the Queue and all I can see is the queue size getting bigger every second (even if I don't produce any message). I am guessing it must be the [:deliver_message, nil] packets that are being pushed into the queue every 50ms.

I am not able to receive any errors from the underlying sync producer. How do I make sure that the package size is under 1MB? I am using Heroku Kafka and they will not let any packages over 1MB. I want the async_producer.produce to error out if the buffer size will be more then 950kb with the current message added.

Thanks!

[dasch]

You need to configure the queue in addition to the buffer – but there should already be an upper bound on the queue size. The async producer runs in a different thread than the calling code, so you cannot rely on knowing the current buffer size, since the producer will be trying to flush that buffer concurrently with your calling code.