Small Batch Inserts Into ClickHouse — Need Way to Delay Flush

[Ilanlido]

Hi,

I’ve noticed that when the Kafka topic being consumed has little to no lag, the connector reads and inserts very small batches (sometimes just 1–10 records) into ClickHouse. Even when there is some lag, the batches tend to be quite small—around 100–200 records—processed in a very short time. This behavior isn’t ideal for ClickHouse, as it performs best with larger batch inserts.

I came across the buffer.flush.timeout.ms configuration

 // Flush timeout(in milliseconds) if max records is not reached.
    BUFFER_FLUSH_TIMEOUT( "buffer.flush.timeout.ms"),

which sounds like it might help—setting a timeout to allow the buffer to accumulate more records before flushing. However, I couldn’t find any part of the codebase that actually makes use of this setting.

Is this configuration currently supported? If not, is there another way to control batch flush timing to ensure larger inserts into ClickHouse?

Thanks!

[subkanthi]

Hi @Ilanlido , please check this https://github.com/Altinity/clickhouse-sink-connector/blob/develop/doc/production_setup.md
buffer.max.records, should control the batch size that gets persited to clickhouse

[Ilanlido]

From what ive seen the default value of this is like 100k and i didnt change the default value, the problem is that the put function receives low amounts of records each time