Skip to content
Open
Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
f4b68b1
Configurable batch size
emasab Oct 29, 2025
09c6197
Keep 1.5s of data in cache
emasab Oct 19, 2025
df6c833
Remove minimum 1024 cache size for faster
emasab Oct 29, 2025
cf45a3c
Changed property name to `js.consumer.max.batch.size`
emasab Oct 20, 2025
28edb19
Use only 1.5 seconds cache size estimation aligned at batch size * co…
emasab Oct 22, 2025
e341232
Fix for at-least-once guarantee not ensured in case a seek happens on…
emasab Oct 23, 2025
82a4867
Configurable cache size in milliseconds
emasab Oct 29, 2025
03369f4
Add worker identifier to the payload for better debugging
emasab Oct 29, 2025
940bc20
Fix for test flakyness
emasab Oct 20, 2025
bd0537f
Make `is cleared before rebalance` less flaky in case of increased ti…
emasab Oct 23, 2025
dc21a54
Reduce flakyness of 'times out if messages are pending'
emasab Oct 29, 2025
9ae3f59
1.6.1-alpha.1
emasab Oct 20, 2025
d30ea6d
fixup CHANGELOG and spaces
emasab Oct 29, 2025
5a7b93d
1.6.1
emasab Oct 29, 2025
06e6bcb
FIXME about KIP-848 autocommit issue
emasab Oct 29, 2025
697a98f
Updated PR in changelog entry
emasab Oct 29, 2025
3634aef
Address comments
emasab Nov 3, 2025
1924126
v1.7.0
emasab Nov 3, 2025
6a68b49
Performance test improvements, measure eachBatch rate, time and lag, …
emasab Oct 8, 2025
d7d5b0f
Run performance tests with SASL_PLAINTEXT or SASL_SSL and plain authe…
emasab Oct 9, 2025
b62ebc2
Fixes
emasab Oct 9, 2025
03f3458
Improve KafkaJS producing, more reliable createTopics, configurable m…
emasab Oct 14, 2025
a9dd2c2
Log number of partitions
emasab Oct 14, 2025
b15bc1d
Fix skipCtpTest
emasab Oct 14, 2025
90966c2
Configurable partitions consumed concurrently
emasab Oct 14, 2025
a8bb201
Consume rate in msg/s, average batch size, different messages dependi…
emasab Oct 14, 2025
363d38a
fixup
emasab Oct 14, 2025
2cb7acd
Average and max E2E latency
emasab Oct 14, 2025
5a66af1
Log average and max size
emasab Oct 14, 2025
3243a75
Changes to run the test for a longer time
emasab Oct 15, 2025
417c571
Add broker lag monitoring
emasab Oct 15, 2025
fbff472
fixup
emasab Oct 16, 2025
d1a3e8c
Use same producer and different E2E latencies
emasab Oct 17, 2025
8335808
Auto commit on batch end
emasab Oct 17, 2025
07bdc11
fixup
emasab Oct 29, 2025
ec6f16c
Fix CI performance run
emasab Oct 19, 2025
0725638
Two partitions consumed concurrently
emasab Oct 29, 2025
a230484
Remove E2E latency metrics for serial run
emasab Oct 19, 2025
f4905b0
More accurate latency and throughput measurement
emasab Oct 23, 2025
89f4ea4
Add latency percentiles calculation
emasab Oct 24, 2025
195cf73
USE_KEYS and timestamp in headers
emasab Oct 29, 2025
f0fc298
Don't modify passed messages in sendOptions
emasab Oct 28, 2025
560d492
CONSUMER_MAX_BATCH_SIZE in performance test
emasab Oct 29, 2025
1f3db88
Changelog entry for `producer.send` fix
emasab Oct 29, 2025
58d611e
Fix to avoid counting the message size two times
emasab Oct 29, 2025
bd14669
Fix in cases for large batch sizes
emasab Nov 4, 2025
00187f4
Fix use warmup messages with headers in case messages from other part…
emasab Nov 4, 2025
c8d5c77
Stats callback
emasab Nov 4, 2025
5eaa4e2
fixup performance example
emasab Nov 5, 2025
d78d6e6
Logger and statistics callback
emasab Nov 5, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,5 @@ coverage
.nyc_output/
*lcov.info
**/lcov-report
examples/performance/*.json
*.log
2 changes: 1 addition & 1 deletion .semaphore/semaphore.yml
Original file line number Diff line number Diff line change
Expand Up @@ -191,7 +191,7 @@ blocks:
- export NODE_OPTIONS='--max-old-space-size=1536'
- cd examples/performance
- npm install
- bash -c '../../ci/tests/run_perf_test.sh'
- node '../../ci/tests/run_perf_test.js'
- rm -rf ./node_modules

- name: "Linux amd64: Release"
Expand Down
14 changes: 14 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,17 @@
# confluent-kafka-javascript 1.7.0

v1.7.0 is a feature release. It is supported for all usage.

### Enhancements

1. Configurable batch size through the `js.consumer.max.batch.size` property
and cache size through the `js.consumer.max.cache.size.per.worker.ms`
property (#393).
2. Fix for at-least-once guarantee not ensured in case a seek happens on one
partition and there are messages being fetched about other partitions (#393).
3. Messages passed to the producer `send` method are kept constant (#394).


# confluent-kafka-javascript 1.6.0

v1.6.0 is a feature release. It is supported for all usage.
Expand Down
11 changes: 8 additions & 3 deletions MIGRATION.md
Original file line number Diff line number Diff line change
Expand Up @@ -303,9 +303,14 @@ producerRun().then(consumerRun).catch(console.error);
- The `heartbeat()` no longer needs to be called by the user in the `eachMessage/eachBatch` callback.
Heartbeats are automatically managed by librdkafka.
- The `partitionsConsumedConcurrently` is supported by both `eachMessage` and `eachBatch`.
- An API compatible version of `eachBatch` is available, but the batch size calculation is not
as per configured parameters, rather, a constant maximum size is configured internally. This is subject
to change.
- An API compatible version of `eachBatch` is available, maximum batch size
can be configured through the `js.consumer.max.batch.size` configuration property
and defaults to 32. It is not dependent on the size of the produced batches
present on the broker, as these are constructed client-side. Similar to
the Java client configuration `max.poll.records`.
`js.consumer.max.cache.size.per.worker.ms` allows to
configure the cache size estimated based on consumption rate and defaults
to the cache being sized to 1.5s worth of messages.
The property `eachBatchAutoResolve` is supported.
Within the `eachBatch` callback, use of `uncommittedOffsets` is unsupported,
and within the returned batch, `offsetLag` and `offsetLagLow` are supported.
Expand Down
406 changes: 406 additions & 0 deletions ci/tests/run_perf_test.js

Large diffs are not rendered by default.

64 changes: 0 additions & 64 deletions ci/tests/run_perf_test.sh

This file was deleted.

2 changes: 1 addition & 1 deletion ci/update-version.js
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ function getPackageVersion(tag, branch) {

// publish with a -devel suffix for EA and RC releases.
if (tag.prerelease.length > 0) {
baseVersion += '-' + tag.prerelease.join('-');
baseVersion += '-' + tag.prerelease.join('.');
}

console.log(`Package version is "${baseVersion}"`);
Expand Down
Loading