Skip to content

Commit 9575b4c

Browse files
author
Tom McCarthy
committed
docs: simplify documentation
more SQS specific focus Update for sqs_batch_processor interface
1 parent 11e385d commit 9575b4c

File tree

1 file changed

+51
-37
lines changed

1 file changed

+51
-37
lines changed

docs/content/utilities/batch.mdx

Lines changed: 51 additions & 37 deletions
Original file line numberDiff line numberDiff line change
@@ -1,65 +1,81 @@
11
---
2-
title: Batch
2+
title: SQS Batch Processing
33
description: Utility
44
---
55

66
import Note from "../../src/components/Note"
77

8-
One very attractive feature of Lambda functions is the ability to integrate them with a plethora of internal and external [event sources][1]. Some of these event providers allows a feature called "Batch processing" in which [predefined number][2] of events is sent to lambda function at once.
9-
10-
The proposed batch utility aims to provide an abstraction to handle a partial failure during a batch execution from a SQS event source, providing a base class (`BasePartialProcessor`) allowing you to create your **own** batch processor.
8+
The SQS batch processing utility provides a way to handle partial failures when processing batches of messages from SQS.
119

1210
**Key Features**
1311

14-
* Removal of successful messages for [AWS SQS](https://aws.amazon.com/sqs/) batch - in case of partial failure;
15-
* Build your own batch processor using the base classes.
12+
* Prevent succesfully processed messages being returned to SQS
13+
* Simple interface for individually processing messages from a batch
14+
* Build your own batch processor using the base classes
1615

17-
**IAM Permissions**
16+
**Background**
17+
18+
When using SQS as a Lambda event source mapping, functions can be triggered with a batch of messages from SQS. If the Lambda function fails when processing the batch,
19+
all messages in the batch will be returned to the queue. With this utility, messages within a batch are handled individually - only messages that were not successfully processed
20+
are returned to the queue. More details on how Lambda works with SQS can be found in the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html).
1821

19-
This utility requires additional permissions to work as expected. See the following table:
22+
<Note type="warning">
23+
While this utility lowers the chance of processing messages more than once, it is not guaranteed. We recommend implementing processing logic in an idempotent manner wherever possible.
24+
</Note><br/>
2025

21-
Processor | Function/Method | IAM Permission
22-
|---------|-----------------|---------------|
23-
PartialSQSProcessor | `_clean` | `sqs:DeleteMessageBatch`
2426

25-
### PartialSQSProcessor
27+
**IAM Permissions**
28+
29+
This utility requires additional permissions to work as expected. Lambda functions using this utility require the `sqs:DeleteMessageBatch` permission.
2630

27-
SQS integration with Lambda is one of the most well established ones and pretty useful when building asynchronous applications. One common approach to maximize the performance of this integration is to enable the batch processing feature, resulting in higher throughput with less invocations.
31+
## Processing messages from SQS
2832

29-
As any function call, you may face errors during execution, in one or more records belonging to a batch. SQS's native behavior is to redrive the **whole** batch to the queue again, reprocessing all of them again, including successful ones. This cycle can happen multiple times depending on your [configuration][3], until the whole batch succeeds or the maximum number of attempts is reached. Your application may face some problems with such behavior, especially if there's no idempotency.
33+
There are 2 ways to use this utility for processing SQS messages:
3034

31-
A *naive* approach to solving this problem is to delete successful records from the queue before redriving's phase. The `PartialSQSProcessor` class offers this solution both as context manager and middleware, removing all successful messages from the queue case one or more failures occurred during lambda's execution. Two examples are provided below, displaying the behavior of this class.
35+
**With a decorator:**
3236

33-
**Examples:**
37+
Using the `sqs_batch_processor` decorator with your lambda handler function, you provide a `record_handler` which is responsible for processing individual messages. It should raise an exception if
38+
it is unable to process the record - this will lead to the message returning to the queue. If the function does not return an exception, the message will be deleted from the queue. When using the decorator, you
39+
will not have accessed to the processed messages within the lambda handler - all processing should be handled from the `record_handler` function.
3440

35-
```python:title=context_manager.py
36-
from aws_lambda_powertools.utilities.batch import batch_processor, PartialSQSProcessor
41+
```python:title=app.py
42+
from aws_lambda_powertools.utilities.batch import sqs_batch_processor
3743

3844
def record_handler(record):
39-
return record["body"]
45+
# This will be called for each individual message from a batch
46+
# It should raise an exception if the message was not processed successfully
47+
return_value = do_something_with(record["body"])
48+
return return_value
4049

50+
@sqs_batch_processor(record_handler=record_handler)
4151
def lambda_handler(event, context):
42-
records = event["Records"]
52+
return {"statusCode": 200}
53+
```
4354

44-
# highlight-start
45-
with processor(records, record_handler):
46-
result = processor.process()
47-
# highlight-end
55+
**With a context manager:**
4856

49-
return result
50-
```
57+
If you require access to the result of processed messages, you can use the context manager. The result from calling `process()` on the context manager will be a list of
58+
all the return values from your `record_handler` function.
5159

52-
```python:title=middleware.py
53-
from aws_lambda_powertools.utilities.batch import batch_processor, PartialSQSProcessor
60+
```python:title=app.py
61+
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
5462

5563
def record_handler(record):
56-
return record["body"]
64+
# This will be called for each individual message from a batch
65+
# It should raise an exception if the message was not processed successfully
66+
return_value = do_something_with(record["body"])
67+
return return_value
68+
5769

58-
# highlight-start
59-
@batch_processor(record_handler=record_handler, processor=PartialSQSProcessor())
60-
# highlight-end
6170
def lambda_handler(event, context):
62-
return {"statusCode": 200}
71+
records = event["Records"]
72+
73+
processor = PartialSQSProcessor()
74+
75+
with processor(records, record_handler):
76+
result = processor.process() # Returns a list of all results from record_handler
77+
78+
return result
6379
```
6480

6581
## Create your own partial processor
@@ -68,6 +84,8 @@ You can create your own partial batch processor by inheriting the `BasePartialPr
6884

6985
All processing logic is handled by `_process_record()` whilst `_prepare()` and `clean()` take care of doing a setup/teardown of the processor, being called at start/end of processor's execution, respectively.
7086

87+
You can then use this class as a context manager, or pass it to `batch_processor` to use as a decorator on your Lambda handler function.
88+
7189
**Example:**
7290

7391
```python:title=custom_processor.py
@@ -131,7 +149,3 @@ class MyPartialProcessor(BasePartialProcessor):
131149
def lambda_handler(event, context):
132150
return {"statusCode": 200}
133151
```
134-
135-
[1]: https://aws.amazon.com/eventbridge/integrations/
136-
[2]: https://docs.aws.amazon.com/lambda/latest/dg/API_CreateEventSourceMapping.html
137-
[3]: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html

0 commit comments

Comments
 (0)