You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/content/utilities/batch.mdx
+51-37Lines changed: 51 additions & 37 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,65 +1,81 @@
1
1
---
2
-
title: Batch
2
+
title: SQS Batch Processing
3
3
description: Utility
4
4
---
5
5
6
6
importNotefrom"../../src/components/Note"
7
7
8
-
One very attractive feature of Lambda functions is the ability to integrate them with a plethora of internal and external [event sources][1]. Some of these event providers allows a feature called "Batch processing" in which [predefined number][2] of events is sent to lambda function at once.
9
-
10
-
The proposed batch utility aims to provide an abstraction to handle a partial failure during a batch execution from a SQS event source, providing a base class (`BasePartialProcessor`) allowing you to create your **own** batch processor.
8
+
The SQS batch processing utility provides a way to handle partial failures when processing batches of messages from SQS.
11
9
12
10
**Key Features**
13
11
14
-
* Removal of successful messages for [AWS SQS](https://aws.amazon.com/sqs/) batch - in case of partial failure;
15
-
* Build your own batch processor using the base classes.
12
+
* Prevent succesfully processed messages being returned to SQS
13
+
* Simple interface for individually processing messages from a batch
14
+
* Build your own batch processor using the base classes
16
15
17
-
**IAM Permissions**
16
+
**Background**
17
+
18
+
When using SQS as a Lambda event source mapping, functions can be triggered with a batch of messages from SQS. If the Lambda function fails when processing the batch,
19
+
all messages in the batch will be returned to the queue. With this utility, messages within a batch are handled individually - only messages that were not successfully processed
20
+
are returned to the queue. More details on how Lambda works with SQS can be found in the [AWS documentation](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html).
18
21
19
-
This utility requires additional permissions to work as expected. See the following table:
22
+
<Notetype="warning">
23
+
While this utility lowers the chance of processing messages more than once, it is not guaranteed. We recommend implementing processing logic in an idempotent manner wherever possible.
This utility requires additional permissions to work as expected. Lambda functions using this utility require the `sqs:DeleteMessageBatch` permission.
26
30
27
-
SQS integration with Lambda is one of the most well established ones and pretty useful when building asynchronous applications. One common approach to maximize the performance of this integration is to enable the batch processing feature, resulting in higher throughput with less invocations.
31
+
## Processing messages from SQS
28
32
29
-
As any function call, you may face errors during execution, in one or more records belonging to a batch. SQS's native behavior is to redrive the **whole** batch to the queue again, reprocessing all of them again, including successful ones. This cycle can happen multiple times depending on your [configuration][3], until the whole batch succeeds or the maximum number of attempts is reached. Your application may face some problems with such behavior, especially if there's no idempotency.
33
+
There are 2 ways to use this utility for processing SQS messages:
30
34
31
-
A *naive* approach to solving this problem is to delete successful records from the queue before redriving's phase. The `PartialSQSProcessor` class offers this solution both as context manager and middleware, removing all successful messages from the queue case one or more failures occurred during lambda's execution. Two examples are provided below, displaying the behavior of this class.
35
+
**With a decorator:**
32
36
33
-
**Examples:**
37
+
Using the `sqs_batch_processor` decorator with your lambda handler function, you provide a `record_handler` which is responsible for processing individual messages. It should raise an exception if
38
+
it is unable to process the record - this will lead to the message returning to the queue. If the function does not return an exception, the message will be deleted from the queue. When using the decorator, you
39
+
will not have accessed to the processed messages within the lambda handler - all processing should be handled from the `record_handler` function.
34
40
35
-
```python:title=context_manager.py
36
-
from aws_lambda_powertools.utilities.batch importbatch_processor, PartialSQSProcessor
41
+
```python:title=app.py
42
+
from aws_lambda_powertools.utilities.batch importsqs_batch_processor
37
43
38
44
defrecord_handler(record):
39
-
return record["body"]
45
+
# This will be called for each individual message from a batch
46
+
# It should raise an exception if the message was not processed successfully
If you require access to the result of processed messages, you can use the context manager. The result from calling `process()` on the context manager will be a list of
58
+
all the return values from your `record_handler` function.
51
59
52
-
```python:title=middleware.py
53
-
from aws_lambda_powertools.utilities.batch importbatch_processor, PartialSQSProcessor
60
+
```python:title=app.py
61
+
from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
54
62
55
63
defrecord_handler(record):
56
-
return record["body"]
64
+
# This will be called for each individual message from a batch
65
+
# It should raise an exception if the message was not processed successfully
result = processor.process() # Returns a list of all results from record_handler
77
+
78
+
return result
63
79
```
64
80
65
81
## Create your own partial processor
@@ -68,6 +84,8 @@ You can create your own partial batch processor by inheriting the `BasePartialPr
68
84
69
85
All processing logic is handled by `_process_record()` whilst `_prepare()` and `clean()` take care of doing a setup/teardown of the processor, being called at start/end of processor's execution, respectively.
70
86
87
+
You can then use this class as a context manager, or pass it to `batch_processor` to use as a decorator on your Lambda handler function.
88
+
71
89
**Example:**
72
90
73
91
```python:title=custom_processor.py
@@ -131,7 +149,3 @@ class MyPartialProcessor(BasePartialProcessor):
0 commit comments