Performance test improvements

[emasab]

measure eachBatch rate, time and lag, avg and max memory usage

[Copilot]

Pull Request Overview

This PR enhances performance testing capabilities by adding comprehensive metrics collection for consumer performance, including batch processing rates, end-to-end latency measurements, memory usage tracking, and offset lag monitoring. The changes also improve cache management in the consumer implementation and fix issues with message handling during rebalancing.

Key changes:

• Added configurable batch size (js.consumer.max.batch.size) and cache size (js.consumer.max.cache.size.per.worker.ms) parameters for consumers
• Implemented dynamic cache sizing based on consumption rate rather than static increments
• Enhanced performance test infrastructure with latency percentiles, memory tracking, and lag monitoring capabilities

Reviewed Changes

Copilot reviewed 9 out of 10 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
test/promisified/producer/flush.spec.js	Added producer creation and cleanup for timeout test
test/promisified/consumer/consumerCacheTests.spec.js	Adjusted message counts and timing for more reliable cache testing
test/promisified/consumer/consumeMessages.spec.js	Modified concurrency tests to use dynamic batch processing expectations
test/promisified/admin/fetch_offsets.spec.js	Fixed timing of message push to occur after commit check
package.json, schemaregistry/package.json, lib/util.js	Version bump to 1.6.1
lib/kafkajs/_producer.js	Refactored to avoid mutating input message objects
lib/kafkajs/_consumer_cache.js	Added support for returning messages to cache head
lib/kafkajs/_consumer.js	Replaced static cache sizing with dynamic rate-based sizing, improved message return handling during pending operations
examples/performance/*.js	Added comprehensive performance testing infrastructure with latency, memory, and lag metrics
ci/update-version.js	Fixed prerelease version formatting
ci/tests/run_perf_test.*	Migrated performance tests from bash to Node.js with enhanced metrics
MIGRATION.md, CHANGELOG.md	Updated documentation for new consumer configuration options
.semaphore/semaphore.yml	Updated CI to use new Node.js-based performance test runner

Comments suppressed due to low confidence (1)

ci/tests/run_perf_test.js:1

• Use let instead of var for block-scoped variable declaration, consistent with modern JavaScript practices and the rest of the codebase.

#!/usr/bin/env node

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Variable staticValueBytes is assigned without declaration. Add let or const before staticValueBytes.

Suggested change

	let staticValueLength = Math.floor(msgSize * (1 - randomness));
	let staticValueLength = Math.floor(msgSize * (1 - randomness));
	let staticValueBytes;

[milindl]

Minor change worth making.

[sonarqube-confluent]

Quality Gate failed

Failed conditions
1968 New issues
52 Security Hotspots
41.3% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

[milindl]

Thanks for the changes.