Is your feature request related to a problem? Please describe.
pw.io.kinesis.read and pw.io.kinesis.write currently have no way to configure server-side encryption (SSE) for the target stream. Amazon Kinesis Data Streams supports SSE via AWS KMS, which encrypts all data at rest using a specified KMS key. There is no way to enable, update, or verify this setting through the Pathway connector, making it impossible to enforce encryption requirements as part of the pipeline definition.
Describe the solution you'd like
Add encryption configuration support to both pw.io.kinesis.read and pw.io.kinesis.write. Concretely, this means accepting two new optional parameters (e.g. encryption_type and key_id) that map directly to the corresponding fields in the Kinesis StartStreamEncryption API:
encryption_type — the encryption type to use; currently the only valid value in Kinesis is "KMS".
key_id — the KMS key identifier: a globally unique key ID, a full key ARN, an alias ARN, or an alias name prefixed with "alias/". The AWS-managed default key can be specified as "alias/aws/kinesis".
When these parameters are provided, the connector should call StartStreamEncryption before beginning to read or write, ensuring the stream is encrypted before any data is produced or consumed. If the stream is already encrypted with the specified key, the call is a no-op.
Describe alternatives you've considered
- Enabling SSE manually via the AWS Management Console or AWS CLI (
aws kinesis start-stream-encryption) before running the Pathway pipeline — works, but requires out-of-band setup steps and cannot be enforced or automated at the pipeline level.
- Using an AWS Lambda or infrastructure-as-code tool (e.g. Terraform) to manage stream encryption separately — adds operational overhead and couples the pipeline to external provisioning tooling.
Additional context
Note that once SSE is enabled on a Kinesis stream, encryption and decryption are fully transparent to producers and consumers: PutRecord, PutRecords, and GetRecords calls work without any changes. Therefore, this feature request is specifically about configuring encryption on the stream, not about handling encrypted payloads inside the connector.
The KMS key specified via key_id must be accessible to the IAM role used by the connector. If a customer-managed key (CMK) is used instead of the default aws/kinesis key, producers and consumers must be explicitly granted kms:GenerateDataKey and kms:Decrypt permissions on that key, otherwise PutRecord/GetRecords calls will fail.
Is your feature request related to a problem? Please describe.
pw.io.kinesis.readandpw.io.kinesis.writecurrently have no way to configure server-side encryption (SSE) for the target stream. Amazon Kinesis Data Streams supports SSE via AWS KMS, which encrypts all data at rest using a specified KMS key. There is no way to enable, update, or verify this setting through the Pathway connector, making it impossible to enforce encryption requirements as part of the pipeline definition.Describe the solution you'd like
Add encryption configuration support to both
pw.io.kinesis.readandpw.io.kinesis.write. Concretely, this means accepting two new optional parameters (e.g.encryption_typeandkey_id) that map directly to the corresponding fields in the KinesisStartStreamEncryptionAPI:encryption_type— the encryption type to use; currently the only valid value in Kinesis is"KMS".key_id— the KMS key identifier: a globally unique key ID, a full key ARN, an alias ARN, or an alias name prefixed with"alias/". The AWS-managed default key can be specified as"alias/aws/kinesis".When these parameters are provided, the connector should call
StartStreamEncryptionbefore beginning to read or write, ensuring the stream is encrypted before any data is produced or consumed. If the stream is already encrypted with the specified key, the call is a no-op.Describe alternatives you've considered
aws kinesis start-stream-encryption) before running the Pathway pipeline — works, but requires out-of-band setup steps and cannot be enforced or automated at the pipeline level.Additional context
Note that once SSE is enabled on a Kinesis stream, encryption and decryption are fully transparent to producers and consumers:
PutRecord,PutRecords, andGetRecordscalls work without any changes. Therefore, this feature request is specifically about configuring encryption on the stream, not about handling encrypted payloads inside the connector.The KMS key specified via
key_idmust be accessible to the IAM role used by the connector. If a customer-managed key (CMK) is used instead of the defaultaws/kinesiskey, producers and consumers must be explicitly grantedkms:GenerateDataKeyandkms:Decryptpermissions on that key, otherwisePutRecord/GetRecordscalls will fail.