Merged
Conversation
Increase the default SDK retry count from 3 to 10 for Kinesis, DynamoDB and CloudWatch clients used by KCL.
When a consumer group rebalances, the Kafka consumer must commit offsets before partitions are revoked. Previously we were using "Eager" rebalance revoke mode from fs2-kafka, with the following consequences: 1. Partitions were revoked immediately without waiting for the application to commit offsets. This could lead to duplicates. 2. The commit occasionally failed with a RebalanceInProgressException, if the application tried to commit offsets after the rebalance had started. This change switches to "Graceful" rebalance revoke mode. It works as follows: 1. Source waits up to session.timeout.ms for the fs2 stream to finalize. This includes committing outstanding offsets. 2. Rebalancing only proceeds after the fs2 stream has finalized, or after session.timeout.ms. This reduces the possibility of duplicates due to re-processing. 3. We catch and ignore the RebalanceInProgressException in case the downstream application cannot finalize the fs2 stream within the session timeout. This is needed e.g. for Lake Loader which is slow to finalize a window.
Common streams is built around the concept of a stream of streams: `Stream[F, Stream[F, A]]`. The nesting of `Stream` is needed so the Source has a way to force the app to immediately checkpoint. This is needed e.g. at the end of a kinesis shard, or when kafka rebalances. We already allowed eager processing of inner streams under some circumstances, i.e. the app can start processing the next inner stream while the preceding inner stream is still finalizing. But we did not have eager processing under all circumstances. This PR allows the eager processing feature to work in more scenarios. It will have most impact on the kinesis source near a shard end, and in particular an app that works with timed windows, e.g. Lake Loader.
Oguzhan Unlu (oguzhanunlu)
force-pushed
the
release/0.19.0
branch
from
58e2638 to
41da12f
Compare
Ian Streeter (istreeter)
approved these changes
Jan 8, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.