Skip to content

Support encryption in Kinesis connectors #223

@zxqfd555

Description

@zxqfd555

Is your feature request related to a problem? Please describe.

pw.io.kinesis.read and pw.io.kinesis.write currently have no way to configure server-side encryption (SSE) for the target stream. Amazon Kinesis Data Streams supports SSE via AWS KMS, which encrypts all data at rest using a specified KMS key. There is no way to enable, update, or verify this setting through the Pathway connector, making it impossible to enforce encryption requirements as part of the pipeline definition.

Describe the solution you'd like

Add encryption configuration support to both pw.io.kinesis.read and pw.io.kinesis.write. Concretely, this means accepting two new optional parameters (e.g. encryption_type and key_id) that map directly to the corresponding fields in the Kinesis StartStreamEncryption API:

  • encryption_type — the encryption type to use; currently the only valid value in Kinesis is "KMS".
  • key_id — the KMS key identifier: a globally unique key ID, a full key ARN, an alias ARN, or an alias name prefixed with "alias/". The AWS-managed default key can be specified as "alias/aws/kinesis".

When these parameters are provided, the connector should call StartStreamEncryption before beginning to read or write, ensuring the stream is encrypted before any data is produced or consumed. If the stream is already encrypted with the specified key, the call is a no-op.

Describe alternatives you've considered

  • Enabling SSE manually via the AWS Management Console or AWS CLI (aws kinesis start-stream-encryption) before running the Pathway pipeline — works, but requires out-of-band setup steps and cannot be enforced or automated at the pipeline level.
  • Using an AWS Lambda or infrastructure-as-code tool (e.g. Terraform) to manage stream encryption separately — adds operational overhead and couples the pipeline to external provisioning tooling.

Additional context

Note that once SSE is enabled on a Kinesis stream, encryption and decryption are fully transparent to producers and consumers: PutRecord, PutRecords, and GetRecords calls work without any changes. Therefore, this feature request is specifically about configuring encryption on the stream, not about handling encrypted payloads inside the connector.

The KMS key specified via key_id must be accessible to the IAM role used by the connector. If a customer-managed key (CMK) is used instead of the default aws/kinesis key, producers and consumers must be explicitly granted kms:GenerateDataKey and kms:Decrypt permissions on that key, otherwise PutRecord/GetRecords calls will fail.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions