Configurable batch size #389

emasab · 2025-10-16T11:35:38Z

Closes #286.

batch size is now configurable through the 'js.max.batch.size` property (-1 for unlimited batch size). Even when unlimited batch sizes are actually limited by librdkafka consumer buffer.

Checklist

Contains customer facing changes? The new js property.
Did you add sufficient unit test and/or integration test coverage for this PR? TODO

Test & Review

automatic tests and running the performance example

rebalances in case of long processing time

milindl

Checked the code and documentation, checking tests and perf scripts next.

milindl · 2025-10-21T04:14:38Z

CHANGELOG.md

@@ -1,3 +1,12 @@
+# confluent-kafka-javascript 1.6.1
+
+v1.6.1 is a maintenance release. It is supported for all usage.


nit: Let's make it 1.7.0

milindl · 2025-10-21T04:17:42Z

MIGRATION.md

+  - An API compatible version of `eachBatch` is available, maximum batch size
+    can be configured through the `js.consumer.max.batch.size` configuration property
+    and defaults to 32.


Suggested change

- An API compatible version of `eachBatch` is available, maximum batch size

can be configured through the `js.consumer.max.batch.size` configuration property

and defaults to 32.

- An API compatible version of `eachBatch` is available, maximum batch size

can be configured through the `js.consumer.max.batch.size` configuration property

and defaults to 32. It is not dependent on the size of the batches present on the broker, as the batches are constructed client-side.

Also, I think we should add that while we start attempting to fill batches with js.consumer.max.batch.size, it may be scaled down based on the actual messages received.

milindl · 2025-10-21T04:19:44Z

lib/kafkajs/_consumer.js

    this.#cacheExpirationTimeoutMs = this.#maxPollIntervalMs;
    rdKafkaConfig['max.poll.interval.ms'] = this.#maxPollIntervalMs * 2;

+    if (rdKafkaConfig['js.consumer.max.batch.size'] !== undefined) {


Suggested change

if (rdKafkaConfig['js.consumer.max.batch.size'] !== undefined) {

if (Object.hasOwn(rdKafkaConfig, 'js.consumer.max.batch.size')) {

…ncurrency

… one partition and there are messages being fetched for other partitions

…me before first assignment

sonarqube-confluent · 2025-10-29T09:08:08Z

79.60% Coverage on New Code (is less than 80.00%)

Analysis Details

14 Issues

0 Bugs
4 Vulnerabilities
10 Code Smells

Coverage and Duplications

79.60% Coverage (48.50% Estimated after merge)
No duplication information (2.00% Estimated after merge)

Project ID: confluent-kafka-javascript

View in SonarQube

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from e881b8f to 6900c8c Compare October 16, 2025 11:37

emasab changed the base branch from master to dev_performance_test_improvements October 16, 2025 11:37

airlock-confluentinc bot force-pushed the dev_performance_test_improvements branch from 9e73424 to ed0fb03 Compare October 16, 2025 11:38

emasab changed the base branch from dev_performance_test_improvements to master October 16, 2025 11:41

emasab changed the base branch from master to dev_performance_test_improvements October 16, 2025 11:42

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 6900c8c to 9b3c43a Compare October 16, 2025 12:07

airlock-confluentinc bot force-pushed the dev_performance_test_improvements branch 2 times, most recently from 15928f1 to 4209743 Compare October 17, 2025 10:29

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 9b3c43a to 25ec968 Compare October 17, 2025 10:32

airlock-confluentinc bot force-pushed the dev_performance_test_improvements branch from 4209743 to 9f4461f Compare October 17, 2025 11:02

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 25ec968 to 7d20ea4 Compare October 17, 2025 11:04

Configurable batch size

e8bd34f

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 7d20ea4 to e8bd34f Compare October 17, 2025 12:26

Keep 1.5s of data in cache

4ad7f02

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 8bb5221 to 4ad7f02 Compare October 19, 2025 12:39

Remove minimum 1024 cache size for faster

2f86688

rebalances in case of long processing time

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch 3 times, most recently from f776f0c to 0a91e33 Compare October 19, 2025 17:34

Fix CI performance run

e62aee1

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 0a91e33 to e62aee1 Compare October 19, 2025 17:48

Test higher message size

cfa7321

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 92368f1 to cfa7321 Compare October 19, 2025 18:27

emasab added 2 commits October 19, 2025 21:05

Remove E2E latency metrics for serial run

866e94e

Two partitions consumed concurrently

4f22357

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 94e66ba to 4f22357 Compare October 19, 2025 19:30

emasab added 2 commits October 20, 2025 12:26

Fix for test flakyness

c7bffdb

1.6.1-alpha.0

2432d04

This comment has been minimized.

Sign in to view

Changed property name to js.consumer.max.batch.size

9b556d2

This comment has been minimized.

Sign in to view

milindl reviewed Oct 21, 2025

View reviewed changes

This comment has been minimized.

Sign in to view

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch 2 times, most recently from cefcf3f to 38aa2dc Compare October 22, 2025 15:39

Use only 1.5 seconds cache size estimation aligned at batch size * co…

99a906c

…ncurrency

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 38aa2dc to 99a906c Compare October 22, 2025 20:16

emasab added 3 commits October 23, 2025 11:06

Fix for at-least-once guarantee not ensured in case a seek happens on…

33d1370

… one partition and there are messages being fetched for other partitions

Make is cleared before rebalance less flaky in case of increased ti…

5f9be45

…me before first assignment

More accurate latency and throughput measurement

3816d53

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from b6266ca to 3816d53 Compare October 23, 2025 22:07

milindl mentioned this pull request Oct 24, 2025

feat: Add configurable maxBatchSize to consumer.run() #388

Closed

2 tasks

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 57b16eb to 8c112e4 Compare October 24, 2025 16:31

This comment has been minimized.

Sign in to view

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 8c112e4 to a7e6023 Compare October 24, 2025 18:32

This comment has been minimized.

Sign in to view

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch 3 times, most recently from ea61499 to 72c3bed Compare October 25, 2025 12:46

Add latency percentiles calculation

68cc1e4

airlock-confluentinc bot force-pushed the dev_fix_configurable_batch_size branch from 72c3bed to 68cc1e4 Compare October 27, 2025 12:36

USE_KEYS and timestamp in headers

52c7f65

This comment has been minimized.

Sign in to view

Don't modify passed messages in sendOptions

c98b2e5

This comment has been minimized.

Sign in to view

emasab added 2 commits October 29, 2025 09:25

Rename IS_HIGHER_LATENCY_BROKER to IS_HIGHER_LATENCY_CLUSTER

23097b5

Reduce flakyness of 'times out if messages are pending'

ce77e18

This comment has been minimized.

Sign in to view

Configurable cache size in milliseconds

b570a7d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Configurable batch size #389

Configurable batch size #389

Uh oh!

emasab commented Oct 16, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

milindl left a comment

Uh oh!

milindl Oct 21, 2025

Uh oh!

milindl Oct 21, 2025

Uh oh!

milindl Oct 21, 2025

Uh oh!

milindl Oct 21, 2025

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sonarqube-confluent bot commented Oct 29, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -1,3 +1,12 @@
		# confluent-kafka-javascript 1.6.1

		v1.6.1 is a maintenance release. It is supported for all usage.

	if (rdKafkaConfig['js.consumer.max.batch.size'] !== undefined) {
	if (Object.hasOwn(rdKafkaConfig, 'js.consumer.max.batch.size')) {

Configurable batch size #389

Are you sure you want to change the base?

Configurable batch size #389

Uh oh!

Conversation

emasab commented Oct 16, 2025

Checklist

Test & Review

Uh oh!

This comment has been minimized.

This comment has been minimized.

milindl left a comment

Choose a reason for hiding this comment

Uh oh!

milindl Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

milindl Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

milindl Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

milindl Oct 21, 2025

Choose a reason for hiding this comment

Uh oh!

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

sonarqube-confluent bot commented Oct 29, 2025

Analysis Details

14 Issues

Coverage and Duplications

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants