Per-topic ProducerBatchMaxBytes

[ct1n]

We're using a single franz-go Client to produce to multiple topics. Until recently they all had the same default size limit (both in the producer and on the broker) but we encountered large records on one topic so we decided we need to increase the topic record size limit on the broker side.

The problem is that we realized that we can't find any way to increase it safely on the producer side: the ProducerBatchMaxBytes limit applies to all topics and the topics with smaller records might be affected: with a larger ProducerBatchMaxBytes value large batches with many small records can be created on those topics that will be rejected by the broker.

We considered using MaxBufferedBytes, but that has two downsides: it applies to the total across topics and (more importantly) it puts a hard limit on the maximum record size, so it can't be used as a soft target at which to flush.

That would leave one solution (if constrained to using one Client): manually check BufferedProduceBytes and call Flush. But this has pretty severe performance implications.

Is there anything we're missing? What would be the recommended solution in this case?
Should we use a Client per topic? How much overhead would this cause? Are connection pools reused across Clients?

As an aside, it seems unfortunate that this config needs to be manually synchronized between broker and producer and can't be auto-detected.

[twmb]

As an aside, it seems unfortunate that this config needs to be manually synchronized between broker and producer and can't be auto-detected.

It technically can be by querying configs via kadm in your application beforehand, but doing it automatically within the library is too much magic and would penalize the startup of all clients even if they don't need this.

connection pools

Connections aren't pooled, it's difficult to do, I've tried a few times. They aren't pooled across clients. Maybe I'll try again with Claude because a large part of the hangup is how tedious the wiring is and how careful one has to be.

The two options are to either use separate clients where you segregate producing to one or the other based off the topic max-batch-size. The other option is I add a different config option, ProducerBatchMaxBytesFn(func(string) int32) ProducerOpt: you can only use one of ProducerBatchMaxBytes or the fn. Internally I'd replace the int32 config opt with the function. If you use the non-function option, I'd just wrap it in a function that always returns the constant. To satisfy OptValues, I'd call with ("") if you ask for ProducerBatchMaxBytes (not the fn).

Writing that^ so I can point claude at this comment.

[twmb]

#1269
should do it. PTAL. I'm not aiming to release for a bit (order: weeks) while I'm spin looping tests.