Windowing / batching of inserts

[kahole]

Is there any way to increase the size of the window of time used to batch together upserts?

I get a lot of "1 records sent to..." very close in time, and I'd like to give the sink a bit more time to batch them together.
Thanks

[ryancrawcour]

that's a great request. not that i know of. i mean there isn't a batching configuration in the connector specifically.
do you know how other Sink connectors handle this?

[kahole]

Ah, understood. So only the «x records inserted…». statement logging is «batched» as of now?

Would be really nice to do batch inserts from the sink to cosmos :)

The jdbc sink has some options to adjust the batch and rate of flow.

batch.size=500

# based on 500*3000 byte message size
consumer.fetch.min.bytes=1500000
consumer.fetch.wait.max.ms=1500
consumer.max.poll.records=4000

Here’s a discussion about it’s use:
confluentinc/kafka-connect-jdbc#290

If you’d like I can create an issue / feature request for this.

Thank you

[ryancrawcour]

Yes, please create a feature request for this so we can track it.

[ryancrawcour]

Tracking this with issue #398