You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
* Updated samples/sample.properties reflecting support for InitialPositionInStreamExtended
195
-
* Related: [#804](https://github.com/awslabs/amazon-kinesis-client/pull/804) Allowing user to specify an initial timestamp in which daemon will process records.
196
-
* Feature released with previous [release 2.0.4](https://github.com/awslabs/amazon-kinesis-client-python/releases/tag/v2.0.4)
* Updated to version 2.1.0 of the Amazon Kinesis Client for Java
253
-
* Version 2.1.0 now defaults to using [`RegisterStreamConsumer` Kinesis API](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_RegisterStreamConsumer.html), which provides dedicated throughput compared to `GetRecords`.
254
-
* Version 2.1.0 now defaults to using [`SubscribeToShard` Kinesis API](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_SubscribeToShard.html), which provides lower latencies than `GetRecords`.
255
-
*__WARNING: `RegisterStreamConsumer` and `SubscribeToShard` are new APIs, and may require updating any explicit IAM policies__
256
-
* For more information about Enhaced Fan-Out and Polling with the KCL check out the [announcement](https://aws.amazon.com/blogs/aws/kds-enhanced-fanout/) and [developer documentation](https://docs.aws.amazon.com/streams/latest/dev/introduction-to-enhanced-consumers.html).
257
-
* Introducing version 3 of the `RecordProcessorBase` which supports the new `ShardRecordProcessor` interface
258
-
* The `shutdown` method from version 2 has been removed and replaced by `leaseLost` and `shardEnded` methods.
259
-
* Introducing `leaseLost` method, which takes `LeaseLostInput` object and is invoked when a lease is lost.
260
-
* Introducing `shardEnded` method, which takes `ShardEndedInput` object and is invoked when all records from a split/merge have been processed.
261
-
* Updated AWS SDK version to 2.2.0
262
-
* MultiLangDaemon now uses logging using logback
263
-
* MultiLangDaemon supports custom logback.xml file via the `--log-configuration` option.
264
-
*`amazon_kclpy_helper` script supports `--log-configuration` option for command generation.
265
-
266
-
### Release 1.5.1 (January 2, 2019)
267
-
* Updated to version 1.9.3 of the Amazon Kinesis Client Library for Java.
* Updated to version 1.9.0 of the Amazon Kinesis Client Library for Java
276
-
* Version 1.9.0 now uses the [`ListShards` Kinesis API](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_ListShards.html), which provides a higher call rate than `DescribeStream`.
277
-
*__WARNING: `ListShards` is a new API, and may require updating any explicit IAM policies__
To use this feature the record processor must implement the `shutdown_requested` operation from the respective processor module.
287
-
See [v2/processor.py](https://github.com/awslabs/amazon-kinesis-client-python/blob/master/amazon_kclpy/v2/processor.py#L76) or [kcl.py](https://github.com/awslabs/amazon-kinesis-client-python/blob/master/amazon_kclpy/kcl.py#L223) for the required API.
288
-
289
-
### Release 1.4.4 (April 7, 2017)
290
-
*[PR #47](https://github.com/awslabs/amazon-kinesis-client-python/pull/47): Update to release 1.7.5 of the Amazon Kinesis Client.
291
-
* Additionally updated to version 1.11.115 of the AWS Java SDK.
* Added a new v2 record processor class that allows access to updated features.
307
-
* Record processor initialization
308
-
* The initialize method receives an InitializeInput object that provides shard id, and the starting sequence and sub sequence numbers.
309
-
* Process records calls
310
-
* The process_records calls now receives a ProcessRecordsInput object that, in addition to the records, now includes the millisBehindLatest for the batch of records
311
-
* Records are now represented as a Record object that adds new data, and includes some convenience methods
312
-
* Adds a `binary_data` method that handles the base 64 decode of the data.
313
-
* Includes the sub sequence number of the record.
314
-
* Includes the approximate arrival time stamp of the record.
315
-
* Record processor shutdown
316
-
* The method `shutdown` now receives a `ShutdownInput` object.
317
-
* Checkpoint methods now accept a sub sequence number in addition to the sequence number.
318
-
319
-
### Release 1.3.1
320
-
* Version number increase to stay inline with PyPI.
321
-
322
-
### Release 1.3.0
323
-
* Updated dependency to Amazon KCL version 1.6.4
324
-
325
-
### Release 1.2.0
326
-
* Updated dependency to Amazon KCL version 1.6.1
327
-
328
-
### Release 1.1.0 (January 27, 2015)
329
-
***Python 3 support** All Python files are compatible with Python 3
330
-
331
-
### Release 1.0.0 (October 21, 2014)
332
-
***amazon_kclpy** module exposes an interface to allow implementation of record processor executables that are compatible with the MultiLangDaemon
333
-
***samples** module provides a sample putter application using [boto][boto] and a sample processing app using `amazon_kclpy`
* KCL 3.x introduces a new lease assignment and load balancing algorithm. It assigns leases among workers based on worker utilization metrics and throughput on each lease, replacing the previous lease count-based lease assignment algorithm.
152
+
* When KCL detects higher variance in CPU utilization among workers, it proactively reassigns leases from over-utilized workers to under-utilized workers for even load balancing. This ensures even CPU utilization across workers and removes the need to over-provision the stream processing compute hosts.
153
+
* Optimized DynamoDB RCU usage
154
+
* KCL 3.x optimizes DynamoDB read capacity unit (RCU) usage on the lease table by implementing a global secondary index with leaseOwner as the partition key. This index mirrors the leaseKey attribute from the base lease table, allowing workers to efficiently discover their assigned leases by querying the index instead of scanning the entire table.
155
+
* This approach significantly reduces read operations compared to earlier KCL versions, where workers performed full table scans, resulting in higher RCU consumption.
156
+
* Graceful lease handoff
157
+
* KCL 3.x introduces a feature called "graceful lease handoff" to minimize data reprocessing during lease reassignments. Graceful lease handoff allows the current worker to complete checkpointing of processed records before transferring the lease to another worker. For graceful lease handoff, you should implement checkpointing logic within the existing `shutdownRequested()` method.
158
+
* This feature is enabled by default in KCL 3.x, but you can turn off this feature by adjusting the configuration property `isGracefulLeaseHandoffEnabled`.
159
+
* While this approach significantly reduces the probability of data reprocessing during lease transfers, it doesn't completely eliminate the possibility. To maintain data integrity and consistency, it's crucial to design your downstream consumer applications to be idempotent. This ensures that the application can handle potential duplicate record processing without adverse effects.
160
+
* New DynamoDB metadata management artifacts
161
+
* KCL 3.x introduces two new DynamoDB tables for improved lease management:
162
+
* Worker metrics table: Records CPU utilization metrics from each worker. KCL uses these metrics for optimal lease assignments, balancing resource utilization across workers. If CPU utilization metric is not available, KCL assigns leases to balance the total sum of shard throughput per worker instead.
163
+
* Coordinator state table: Stores internal state information for workers. Used to coordinate in-place migration from KCL 2.x to KCL 3.x and leader election among workers.
164
+
* Follow this [documentation](https://docs.aws.amazon.com/streams/latest/dev/kcl-migration-from-2-3.html#kcl-migration-from-2-3-IAM-permissions) to add required IAM permissions for your KCL application.
165
+
* Other improvements and changes
166
+
* Dependency on the AWS SDK for Java 1.x has been fully removed.
167
+
* The Glue Schema Registry integration functionality no longer depends on AWS SDK for Java 1.x. Previously, it required this as a transient dependency.
168
+
* Multilangdaemon has been upgraded to use AWS SDK for Java 2.x. It no longer depends on AWS SDK for Java 1.x.
169
+
*`idleTimeBetweenReadsInMillis` (PollingConfig) now has a minimum default value of 200.
170
+
* This polling configuration property determines the [publishers](https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/retrieval/polling/PrefetchRecordsPublisher.java) wait time between GetRecords calls in both success and failure cases. Previously, setting this value below 200 caused unnecessary throttling. This is because Amazon Kinesis Data Streams supports up to five read transactions per second per shard for shared-throughput consumers.
171
+
* Shard lifecycle management is improved to deal with edge cases around shard splits and merges to ensure records continue being processed as expected.
172
+
* Migration
173
+
* The programming interfaces of KCL 3.x remain identical with KCL 2.x for an easier migration. For detailed migration instructions, please refer to the [Migrate consumers from KCL 2.x to KCL 3.x](https://docs.aws.amazon.com/streams/latest/dev/kcl-migration-from-2-3.html) page in the Amazon Kinesis Data Streams developer guide.
174
+
* Configuration properties
175
+
* New configuration properties introduced in KCL 3.x are listed in this [doc](https://github.com/awslabs/amazon-kinesis-client/blob/master/docs/kcl-configurations.md#new-configurations-in-kcl-3x).
176
+
* Deprecated configuration properties in KCL 3.x are listed in this [doc](https://github.com/awslabs/amazon-kinesis-client/blob/master/docs/kcl-configurations.md#discontinued-configuration-properties-in-kcl-3x). You need to keep the deprecated configuration properties during the migration from any previous KCL version to KCL 3.x.
177
+
* Metrics
178
+
* New CloudWatch metrics introduced in KCL 3.x are explained in the [Monitor the Kinesis Client Library with Amazon CloudWatch](https://docs.aws.amazon.com/streams/latest/dev/monitoring-with-kcl.html) in the Amazon Kinesis Data Streams developer guide. The following operations are newly added in KCL 3.x:
0 commit comments