|
1 | 1 | ---
|
2 | 2 | title: Kafka Consumer
|
3 | 3 | description: Utility
|
4 |
| -status: new |
5 | 4 | ---
|
6 | 5 |
|
7 | 6 | The Kafka Consumer utility transparently handles message deserialization, provides an intuitive developer experience, and integrates seamlessly with the rest of the Powertools for AWS Lambda ecosystem.
|
@@ -38,7 +37,7 @@ The Kafka Consumer utility transparently handles message deserialization, provid
|
38 | 37 | Lambda processes Kafka messages as discrete events rather than continuous streams, requiring a different approach to consumer development that Powertools for AWS helps standardize.
|
39 | 38 |
|
40 | 39 | | Aspect | Traditional Kafka Consumers | Lambda Kafka Consumer |
|
41 |
| -|-----------------------|-------------------------------------|----------------------------------------------------------------| |
| 40 | +| --------------------- | ----------------------------------- | -------------------------------------------------------------- | |
42 | 41 | | **Model** | Pull-based (you poll for messages) | Push-based (Lambda invoked with messages) |
|
43 | 42 | | **Scaling** | Manual scaling configuration | Automatic scaling to partition count |
|
44 | 43 | | **State** | Long-running application with state | Stateless, ephemeral executions |
|
@@ -201,15 +200,15 @@ The Kafka consumer utility supports multiple serialization formats to match your
|
201 | 200 | === "Supported Formats"
|
202 | 201 |
|
203 | 202 | | Format | Schema Type | Description | Required Parameters |
|
204 |
| - |----------------------|--------------|-----------------------------------|--------------------------------------| |
| 203 | + | -------------------- | ------------ | --------------------------------- | ------------------------------------ | |
205 | 204 | | **JSON** | `"JSON"` | Human-readable text format | None |
|
206 | 205 | | **Avro** | `"AVRO"` | Compact binary format with schema | `value.schema` (Avro schema string) |
|
207 | 206 | | **Protocol Buffers** | `"PROTOBUF"` | Efficient binary format | `value.schema` (Proto message class) |
|
208 | 207 |
|
209 | 208 | === "Format Comparison"
|
210 | 209 |
|
211 | 210 | | Feature | JSON | Avro | Protocol Buffers |
|
212 |
| - |-------------------------------|----------|----------------------|-------------------------| |
| 211 | + | ----------------------------- | -------- | -------------------- | ----------------------- | |
213 | 212 | | **Schema Definition** | Optional | Required JSON schema | Required Protobuf class |
|
214 | 213 | | **Schema Evolution** | None | Strong support | Strong support |
|
215 | 214 | | **Size Efficiency** | Low | High | Highest |
|
@@ -241,7 +240,7 @@ For debugging purposes, you can also access the original key, value, and headers
|
241 | 240 | #### Available metadata properties
|
242 | 241 |
|
243 | 242 | | Property | Description | Example Use Case |
|
244 |
| -|-----------------------|------------------------------------------------------------------|---------------------------------------------------------------------| |
| 243 | +| --------------------- | ---------------------------------------------------------------- | ------------------------------------------------------------------- | |
245 | 244 | | `topic` | Topic name the record was published to | Routing logic in multi-topic consumers |
|
246 | 245 | | `partition` | Kafka partition number | Tracking message distribution |
|
247 | 246 | | `offset` | Position in the partition | De-duplication, exactly-once processing |
|
@@ -304,7 +303,7 @@ Handle errors gracefully when processing Kafka messages to ensure your applicati
|
304 | 303 | #### Error types
|
305 | 304 |
|
306 | 305 | | Exception | Description | Common Causes |
|
307 |
| -|--------------------------------------|-----------------------------------------------|-----------------------------------------------------------------------------| |
| 306 | +| ------------------------------------ | --------------------------------------------- | --------------------------------------------------------------------------- | |
308 | 307 | | `KafkaConsumerError`. | Base class for all Kafka consumer errors | General unhandled errors |
|
309 | 308 | | `KafkaConsumerDeserializationError` | Thrown when message deserialization fails | Corrupted message data, schema mismatch, or wrong schema type configuration |
|
310 | 309 | | `KafkaConsumerMissingSchemaError` | Thrown when a required schema is not provided | Missing schema for AVRO or PROTOBUF formats (required parameter) |
|
|
0 commit comments