Merge branch 'develop' into develop

Author: leandrodamascena

.github/workflows/publish_v2_layer.yml

@@ -124,7 +124,7 @@ jobs:
 
       - name: Set up Docker Buildx
         id: builder
-        uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
+        uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
         with:
           install: true
           driver: docker

.github/workflows/publish_v3_layer.yml

@@ -146,7 +146,7 @@ jobs:
 
       - name: Set up Docker Buildx
         id: builder
-        uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
+        uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
         with:
           install: true
           driver: docker

.github/workflows/quality_code_cdk_constructor.yml

@@ -57,7 +57,7 @@ jobs:
         # NOTE: we need QEMU to build Layer against a different architecture (e.g., ARM)
       - name: Set up Docker Buildx
         id: builder
-        uses: docker/setup-buildx-action@b5ca514318bd6ebac0fb2aedd5d36ec1b5c232a2 # v3.10.0
+        uses: docker/setup-buildx-action@e468171a9de216ec08956ac3ada2f0791b6bd435 # v3.11.1
         with:
           install: true
           driver: docker

CHANGELOG.md

@@ -4,22 +4,65 @@
 <a name="unreleased"></a>
 # Unreleased
 
+## Maintenance
+
+* **deps:** bump redis from 5.3.0 to 6.2.0 ([#6827](https://github.com/aws-powertools/powertools-lambda-python/issues/6827))
+
+
+<a name="v3.15.1"></a>
+## [v3.15.1] - 2025-06-20
+## Features
+
+* **kafka:** add logic to handle protobuf deserialization ([#6841](https://github.com/aws-powertools/powertools-lambda-python/issues/6841))
+
+## Maintenance
+
+* version bump
+* **ci:** new pre-release 3.15.1a0 ([#6839](https://github.com/aws-powertools/powertools-lambda-python/issues/6839))
+
+
+<a name="v3.15.0"></a>
+## [v3.15.0] - 2025-06-19
+## Bug Fixes
+
+* **bedrock_agent:** fix querystring field resolution ([#6777](https://github.com/aws-powertools/powertools-lambda-python/issues/6777))
+
 ## Documentation
 
+* **kafka:** add kafka documentation ([#6834](https://github.com/aws-powertools/powertools-lambda-python/issues/6834))
 * **public_reference:** add Instil as a public reference ([#6763](https://github.com/aws-powertools/powertools-lambda-python/issues/6763))
 
+## Features
+
+* **kafka:** add support for Confluence Producers ([#6833](https://github.com/aws-powertools/powertools-lambda-python/issues/6833))
+* **kafka:** New Kafka utility ([#6821](https://github.com/aws-powertools/powertools-lambda-python/issues/6821))
+
 ## Maintenance
 
+* version bump
+* **ci:** new pre-release 3.14.1a6 ([#6830](https://github.com/aws-powertools/powertools-lambda-python/issues/6830))
+* **ci:** new pre-release 3.14.1a5 ([#6820](https://github.com/aws-powertools/powertools-lambda-python/issues/6820))
 * **ci:** new pre-release 3.14.1a0 ([#6773](https://github.com/aws-powertools/powertools-lambda-python/issues/6773))
+* **ci:** new pre-release 3.14.1a4 ([#6812](https://github.com/aws-powertools/powertools-lambda-python/issues/6812))
+* **ci:** new pre-release 3.14.1a3 ([#6797](https://github.com/aws-powertools/powertools-lambda-python/issues/6797))
 * **ci:** new pre-release 3.14.1a1 ([#6778](https://github.com/aws-powertools/powertools-lambda-python/issues/6778))
-* **deps:** bump mkdocstrings-python from 1.16.11 to 1.16.12 ([#6765](https://github.com/aws-powertools/powertools-lambda-python/issues/6765))
+* **ci:** new pre-release 3.14.1a2 ([#6788](https://github.com/aws-powertools/powertools-lambda-python/issues/6788))
 * **deps:** bump mkdocstrings-python from 1.16.11 to 1.16.12 in /docs ([#6768](https://github.com/aws-powertools/powertools-lambda-python/issues/6768))
+* **deps:** bump mkdocstrings-python from 1.16.11 to 1.16.12 ([#6765](https://github.com/aws-powertools/powertools-lambda-python/issues/6765))
+* **deps:** bump protobuf from 6.31.0 to 6.31.1 ([#6815](https://github.com/aws-powertools/powertools-lambda-python/issues/6815))
 * **deps-dev:** bump boto3-stubs from 1.38.29 to 1.38.30 ([#6772](https://github.com/aws-powertools/powertools-lambda-python/issues/6772))
 * **deps-dev:** bump aws-cdk from 2.1017.1 to 2.1018.0 ([#6775](https://github.com/aws-powertools/powertools-lambda-python/issues/6775))
-* **deps-dev:** bump aws-cdk-aws-lambda-python-alpha from 2.200.0a0 to 2.200.1a0 ([#6766](https://github.com/aws-powertools/powertools-lambda-python/issues/6766))
-* **deps-dev:** bump aws-cdk-lib from 2.200.0 to 2.200.1 ([#6767](https://github.com/aws-powertools/powertools-lambda-python/issues/6767))
+* **deps-dev:** bump boto3-stubs from 1.38.33 to 1.38.35 ([#6796](https://github.com/aws-powertools/powertools-lambda-python/issues/6796))
+* **deps-dev:** bump aws-cdk from 2.1018.0 to 2.1018.1 ([#6803](https://github.com/aws-powertools/powertools-lambda-python/issues/6803))
 * **deps-dev:** bump boto3-stubs from 1.38.30 to 1.38.31 ([#6776](https://github.com/aws-powertools/powertools-lambda-python/issues/6776))
+* **deps-dev:** bump requests from 2.32.3 to 2.32.4 ([#6789](https://github.com/aws-powertools/powertools-lambda-python/issues/6789))
 * **deps-dev:** bump boto3-stubs from 1.38.28 to 1.38.29 ([#6764](https://github.com/aws-powertools/powertools-lambda-python/issues/6764))
+* **deps-dev:** bump ruff from 0.11.12 to 0.11.13 ([#6780](https://github.com/aws-powertools/powertools-lambda-python/issues/6780))
+* **deps-dev:** bump boto3-stubs from 1.38.31 to 1.38.33 ([#6786](https://github.com/aws-powertools/powertools-lambda-python/issues/6786))
+* **deps-dev:** bump aws-cdk-aws-lambda-python-alpha from 2.200.0a0 to 2.200.1a0 ([#6766](https://github.com/aws-powertools/powertools-lambda-python/issues/6766))
+* **deps-dev:** bump aws-cdk-lib from 2.200.0 to 2.200.1 ([#6767](https://github.com/aws-powertools/powertools-lambda-python/issues/6767))
+* **deps-dev:** bump pytest-cov from 6.1.1 to 6.2.1 ([#6800](https://github.com/aws-powertools/powertools-lambda-python/issues/6800))
+* **deps-dev:** bump requests from 2.32.3 to 2.32.4 ([#6787](https://github.com/aws-powertools/powertools-lambda-python/issues/6787))
 
 
 <a name="v3.14.0"></a>
@@ -6634,7 +6677,9 @@
 * Merge pull request [#5](https://github.com/aws-powertools/powertools-lambda-python/issues/5) from jfuss/feat/python38
 
 
-[Unreleased]: https://github.com/aws-powertools/powertools-lambda-python/compare/v3.14.0...HEAD
+[Unreleased]: https://github.com/aws-powertools/powertools-lambda-python/compare/v3.15.1...HEAD
+[v3.15.1]: https://github.com/aws-powertools/powertools-lambda-python/compare/v3.15.0...v3.15.1
+[v3.15.0]: https://github.com/aws-powertools/powertools-lambda-python/compare/v3.14.0...v3.15.0
 [v3.14.0]: https://github.com/aws-powertools/powertools-lambda-python/compare/v3.13.0...v3.14.0
 [v3.13.0]: https://github.com/aws-powertools/powertools-lambda-python/compare/v3.12.0...v3.13.0
 [v3.12.0]: https://github.com/aws-powertools/powertools-lambda-python/compare/v3.11.0...v3.12.0

aws_lambda_powertools/event_handler/api_gateway.py

@@ -407,7 +407,7 @@ def __init__(
 
         # OpenAPI spec only understands paths with { }. So we'll have to convert Powertools' < >.
         # https://swagger.io/specification/#path-templating
-        self.openapi_path = re.sub(r"<(.*?)>", lambda m: f"{{{''.join(m.group(1))}}}", self.path)
+        self.openapi_path = re.sub(r"<(.*?)>", lambda m: f"{{{''.join(m.group(1))}}}", self.path)  # type: ignore[arg-type]
 
         self.rule = rule
         self.func = func

aws_lambda_powertools/shared/version.py

@@ -1,3 +1,3 @@
 """Exposes version constant to avoid circular dependencies."""
 
-VERSION = "3.14.1a2"
+VERSION = "3.15.1"

aws_lambda_powertools/utilities/data_classes/kafka_event.py

@@ -10,7 +10,19 @@
     from collections.abc import Iterator
 
 
-class KafkaEventRecord(DictWrapper):
+class KafkaEventRecordSchemaMetadata(DictWrapper):
+    @property
+    def data_format(self) -> str | None:
+        """The data format of the Kafka record."""
+        return self.get("dataFormat", None)
+
+    @property
+    def schema_id(self) -> str | None:
+        """The schema id of the Kafka record."""
+        return self.get("schemaId", None)
+
+
+class KafkaEventRecordBase(DictWrapper):
     @property
     def topic(self) -> str:
         """The Kafka topic."""
@@ -36,6 +48,24 @@ def timestamp_type(self) -> str:
         """The Kafka record timestamp type."""
         return self["timestampType"]
 
+    @property
+    def key_schema_metadata(self) -> KafkaEventRecordSchemaMetadata | None:
+        """The metadata of the Key Kafka record."""
+        return (
+            None if self.get("keySchemaMetadata") is None else KafkaEventRecordSchemaMetadata(self["keySchemaMetadata"])
+        )
+
+    @property
+    def value_schema_metadata(self) -> KafkaEventRecordSchemaMetadata | None:
+        """The metadata of the Value Kafka record."""
+        return (
+            None
+            if self.get("valueSchemaMetadata") is None
+            else KafkaEventRecordSchemaMetadata(self["valueSchemaMetadata"])
+        )
+
+
+class KafkaEventRecord(KafkaEventRecordBase):
     @property
     def key(self) -> str | None:
         """
@@ -83,18 +113,7 @@ def decoded_headers(self) -> dict[str, bytes]:
         return CaseInsensitiveDict((k, bytes(v)) for chunk in self.headers for k, v in chunk.items())
 
 
-class KafkaEvent(DictWrapper):
-    """Self-managed or MSK Apache Kafka event trigger
-    Documentation:
-    --------------
-    - https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html
-    - https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html
-    """
-
-    def __init__(self, data: dict[str, Any]):
-        super().__init__(data)
-        self._records: Iterator[KafkaEventRecord] | None = None
-
+class KafkaEventBase(DictWrapper):
     @property
     def event_source(self) -> str:
         """The AWS service from which the Kafka event record originated."""
@@ -115,6 +134,19 @@ def decoded_bootstrap_servers(self) -> list[str]:
         """The decoded Kafka bootstrap URL."""
         return self.bootstrap_servers.split(",")
 
+
+class KafkaEvent(KafkaEventBase):
+    """Self-managed or MSK Apache Kafka event trigger
+    Documentation:
+    --------------
+    - https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html
+    - https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html
+    """
+
+    def __init__(self, data: dict[str, Any]):
+        super().__init__(data)
+        self._records: Iterator[KafkaEventRecord] | None = None
+
     @property
     def records(self) -> Iterator[KafkaEventRecord]:
         """The Kafka records."""

aws_lambda_powertools/utilities/kafka/__init__.py

@@ -0,0 +1,9 @@
+from aws_lambda_powertools.utilities.kafka.consumer_records import ConsumerRecords
+from aws_lambda_powertools.utilities.kafka.kafka_consumer import kafka_consumer
+from aws_lambda_powertools.utilities.kafka.schema_config import SchemaConfig
+
+__all__ = [
+    "kafka_consumer",
+    "ConsumerRecords",
+    "SchemaConfig",
+]

aws_lambda_powertools/utilities/kafka/consumer_records.py

@@ -0,0 +1,159 @@
+from __future__ import annotations
+
+import logging
+from functools import cached_property
+from typing import TYPE_CHECKING, Any
+
+from aws_lambda_powertools.utilities.data_classes.common import CaseInsensitiveDict
+from aws_lambda_powertools.utilities.data_classes.kafka_event import KafkaEventBase, KafkaEventRecordBase
+from aws_lambda_powertools.utilities.kafka.deserializer.deserializer import get_deserializer
+from aws_lambda_powertools.utilities.kafka.serialization.serialization import serialize_to_output_type
+
+if TYPE_CHECKING:
+    from collections.abc import Iterator
+
+    from aws_lambda_powertools.utilities.kafka.schema_config import SchemaConfig
+
+logger = logging.getLogger(__name__)
+
+
+class ConsumerRecordRecords(KafkaEventRecordBase):
+    """
+    A Kafka Consumer Record
+    """
+
+    def __init__(self, data: dict[str, Any], schema_config: SchemaConfig | None = None):
+        super().__init__(data)
+        self.schema_config = schema_config
+
+    @cached_property
+    def key(self) -> Any:
+        key = self.get("key")
+
+        # Return None if key doesn't exist
+        if not key:
+            return None
+
+        logger.debug("Deserializing key field")
+
+        # Determine schema type and schema string
+        schema_type = None
+        schema_value = None
+        output_serializer = None
+
+        if self.schema_config and self.schema_config.key_schema_type:
+            schema_type = self.schema_config.key_schema_type
+            schema_value = self.schema_config.key_schema
+            output_serializer = self.schema_config.key_output_serializer
+
+        # Always use get_deserializer if None it will default to DEFAULT
+        deserializer = get_deserializer(
+            schema_type=schema_type,
+            schema_value=schema_value,
+            field_metadata=self.key_schema_metadata,
+        )
+        deserialized_value = deserializer.deserialize(key)
+
+        # Apply output serializer if specified
+        if output_serializer:
+            return serialize_to_output_type(deserialized_value, output_serializer)
+
+        return deserialized_value
+
+    @cached_property
+    def value(self) -> Any:
+        value = self["value"]
+
+        # Determine schema type and schema string
+        schema_type = None
+        schema_value = None
+        output_serializer = None
+
+        logger.debug("Deserializing value field")
+
+        if self.schema_config and self.schema_config.value_schema_type:
+            schema_type = self.schema_config.value_schema_type
+            schema_value = self.schema_config.value_schema
+            output_serializer = self.schema_config.value_output_serializer
+
+        # Always use get_deserializer if None it will default to DEFAULT
+        deserializer = get_deserializer(
+            schema_type=schema_type,
+            schema_value=schema_value,
+            field_metadata=self.value_schema_metadata,
+        )
+        deserialized_value = deserializer.deserialize(value)
+
+        # Apply output serializer if specified
+        if output_serializer:
+            return serialize_to_output_type(deserialized_value, output_serializer)
+
+        return deserialized_value
+
+    @property
+    def original_value(self) -> str:
+        """The original (base64 encoded) Kafka record value."""
+        return self["value"]
+
+    @property
+    def original_key(self) -> str | None:
+        """
+        The original (base64 encoded) Kafka record key.
+
+        This key is optional; if not provided,
+        a round-robin algorithm will be used to determine
+        the partition for the message.
+        """
+
+        return self.get("key")
+
+    @property
+    def original_headers(self) -> list[dict[str, list[int]]]:
+        """The raw Kafka record headers."""
+        return self["headers"]
+
+    @cached_property
+    def headers(self) -> dict[str, bytes]:
+        """Decodes the headers as a single dictionary."""
+        return CaseInsensitiveDict((k, bytes(v)) for chunk in self.original_headers for k, v in chunk.items())
+
+
+class ConsumerRecords(KafkaEventBase):
+    """Self-managed or MSK Apache Kafka event trigger
+    Documentation:
+    --------------
+    - https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html
+    - https://docs.aws.amazon.com/lambda/latest/dg/with-msk.html
+    """
+
+    def __init__(self, data: dict[str, Any], schema_config: SchemaConfig | None = None):
+        super().__init__(data)
+        self._records: Iterator[ConsumerRecordRecords] | None = None
+        self.schema_config = schema_config
+
+    @property
+    def records(self) -> Iterator[ConsumerRecordRecords]:
+        """The Kafka records."""
+        for chunk in self["records"].values():
+            for record in chunk:
+                yield ConsumerRecordRecords(data=record, schema_config=self.schema_config)
+
+    @property
+    def record(self) -> ConsumerRecordRecords:
+        """
+        Returns the next Kafka record using an iterator.
+
+        Returns
+        -------
+        ConsumerRecordRecords
+            The next Kafka record.
+
+        Raises
+        ------
+        StopIteration
+            If there are no more records available.
+
+        """
+        if self._records is None:
+            self._records = self.records
+        return next(self._records)