Skip to content

Commit 8b4a7b3

Browse files
heitorlessaheitorlessa
committed
docs(idempotency): cleanup redis usage and link with setup/infra
1 parent c77869a commit 8b4a7b3

File tree

5 files changed

+133
-130
lines changed

5 files changed

+133
-130
lines changed

‎docs/utilities/idempotency.md

Lines changed: 107 additions & 112 deletions
Original file line numberDiff line numberDiff line change
@@ -73,12 +73,18 @@ We currently support Amazon DynamoDB and Redis as a storage layer. The following
7373
If you're not [changing the default configuration for the DynamoDB persistence layer](#dynamodbpersistencelayer), this is the expected default configuration:
7474

7575
| Configuration | Value | Notes |
76-
| ------------------ | ------------ |-------------------------------------------------------------------------------------|
76+
| ------------------ | ------------ | ----------------------------------------------------------------------------------- |
7777
| Partition key | `id` | |
7878
| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console |
7979

80-
???+ tip "Tip: You can share a single state table for all functions"
81-
You can reuse the same DynamoDB table to store idempotency state. We add `module_name` and [qualified name for classes and functions](https://peps.python.org/pep-3155/){target="_blank" rel="nofollow"} in addition to the idempotency key as a hash key.
80+
| Configuration | Value | Notes |
81+
| ------------------ | ------------ | -------------------------------------------------------------------------------------------------------- |
82+
| Partition key | `id` | Primary key looks like: <br> `{lambda_fn_name}.{module_name}.{fn_qualified_name}#{idempotency_key_hash}` |
83+
| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console |
84+
85+
Note that `fn_qualified_name` means the [qualified name for classes and functions](https://peps.python.org/pep-3155/){target="_blank" rel="nofollow"} defined in PEP-3155.
86+
87+
##### DynamoDB IaC examples
8288

8389
=== "AWS Serverless Application Model (SAM) example"
8490

@@ -109,7 +115,24 @@ If you're not [changing the default configuration for the DynamoDB persistence l
109115

110116
On subsequent invocations with the same payload, you can expect just 1 `PutItem` request to DynamoDB.
111117

112-
**Note:** While we try to minimize requests to DynamoDB to 1 per invocation, if your boto3 version is lower than `1.26.194`, you may experience 2 requests in every invocation. Ensure to check your boto3 version and review the [DynamoDB pricing documentation](https://aws.amazon.com/dynamodb/pricing/){target="_blank"} to estimate the cost.
118+
We recommend you start with a Redis compatible management services such as [Amazon ElastiCache for Redis](https://aws.amazon.com/elasticache/redis/){target="_blank"} or [Amazon MemoryDB for Redis](https://aws.amazon.com/memorydb/){target="_blank"}.
119+
120+
In both services and self-hosting Redis, you'll need to configure [VPC access](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html){target="_blank"} to your AWS Lambda.
121+
122+
!!! tip "First time setting it all up? Checkout the official tutorials for [Amazon ElastiCache for Redis](https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/LambdaRedis.html) or [Amazon MemoryDB for Redis](https://aws.amazon.com/blogs/database/access-amazon-memorydb-for-redis-from-aws-lambda/)"
123+
124+
##### Redis IaC examples
125+
126+
=== "AWS CloudFormation example"
127+
128+
```yaml hl_lines="5 21"
129+
--8<-- "examples/idempotency/templates/cfn_redis_serverless.yaml"
130+
```
131+
132+
1. Replace the Security Group ID and Subnet ID to match your VPC settings.
133+
2. Replace the Security Group ID and Subnet ID to match your VPC settings.
134+
135+
Once setup, you can find quick start and advanced examples for Redis in [the persistent layers section](RedisCachePersistenceLayer).
113136

114137
<!-- markdownlint-enable MD013 -->
115138
### Idempotent decorator
@@ -353,6 +376,8 @@ This persistence layer is built-in, allowing you to use an existing DynamoDB tab
353376
--8<-- "examples/idempotency/src/customize_persistence_layer.py"
354377
```
355378

379+
##### DynamoDB defaults
380+
356381
When using DynamoDB as the persistence layer, you can customize the attribute names by passing the following parameters during the initialization of the persistence layer:
357382

358383
| Parameter | Required | Default | Description |
@@ -369,23 +394,82 @@ When using DynamoDB as the persistence layer, you can customize the attribute na
369394

370395
#### RedisPersistenceLayer
371396

372-
This persistence layer is built-in, allowing you to use an existing Redis service. For optimal performance and compatibility, it is strongly recommended to use a Redis service version 7 or higher.
397+
!!! info "We recommend Redis version 7 or higher for optimal performance."
398+
399+
For a quick start, initialize `RedisCachePersistenceLayer` and pass your cluster host endpoint along with the port to connect to.
400+
401+
For security, we enforce SSL connections by default; to disable it, set `ssl=False`.
402+
403+
=== "Redis quick start"
404+
```python hl_lines="7-9 12 26"
405+
--8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_config.py"
406+
```
407+
408+
=== "Using an existing Redis client"
409+
```python hl_lines="4 9-11 14 22 36"
410+
--8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_client.py"
411+
```
412+
413+
=== "Sample event"
414+
415+
```json
416+
--8<-- "examples/idempotency/src/getting_started_with_idempotency_payload.json"
417+
```
418+
419+
##### Redis SSL connections
420+
421+
We recommend using AWS Secrets Manager to store and rotate certificates safely, and the [Parameters feature](./parameters.md){target="_blank"} to fetch and cache optimally.
422+
423+
For advanced configurations, we also recommend using an existing Redis client for optimal compatibility like SSL certificates and timeout.
424+
425+
=== "Advanced configuration using AWS Secrets"
426+
```python hl_lines="9-11 13 15 25"
427+
--8<-- "examples/idempotency/src/using_redis_client_with_aws_secrets.py"
428+
```
429+
430+
1. JSON stored:
431+
```json
432+
{
433+
"REDIS_ENDPOINT": "127.0.0.1",
434+
"REDIS_PORT": "6379",
435+
"REDIS_PASSWORD": "redis-secret"
436+
}
437+
```
438+
439+
=== "Advanced configuration with local certificates"
440+
```python hl_lines="14 25-27"
441+
--8<-- "examples/idempotency/src/using_redis_client_with_local_certs.py"
442+
```
443+
444+
1. JSON stored:
445+
```json
446+
{
447+
"REDIS_ENDPOINT": "127.0.0.1",
448+
"REDIS_PORT": "6379",
449+
"REDIS_PASSWORD": "redis-secret"
450+
}
451+
```
452+
2. redis_user.crt file stored in the "certs" directory of your Lambda function
453+
3. redis_user_private.key file stored in the "certs" directory of your Lambda function
454+
4. redis_ca.pem file stored in the "certs" directory of your Lambda function
455+
456+
##### Redis defaults
457+
458+
You can customize attribute names when instantiating `RedisCachePersistenceLayer` with the following parameters:
459+
460+
| Parameter | Required | Default | Description |
461+
| --------------------------- | -------- | ------------------------ | --------------------------------------------------------------------------------------------- |
462+
| **in_progress_expiry_attr** | | `in_progress_expiration` | Unix timestamp of when record expires while in progress (in case of the invocation times out) |
463+
| **status_attr** | | `status` | Stores status of the Lambda execution during and after invocation |
464+
| **data_attr** | | `data` | Stores results of successfully executed Lambda handlers |
465+
| **validation_key_attr** | | `validation` | Hashed representation of the parts of the event used for validation |
373466

374467
=== "Customizing RedisPersistenceLayer to suit your data structure"
375468

376469
```python hl_lines="9-16"
377470
--8<-- "examples/idempotency/src/customize_persistence_layer_redis.py"
378471
```
379472

380-
When using Redis as the persistence layer, you can customize the attribute names by providing the following parameters upon initialization of the persistence layer:
381-
382-
| Parameter | Required | Default | Description |
383-
| --------------------------- | ------------------ | ------------------------------------ | -------------------------------------------------------------------------------------------------------- |
384-
| **in_progress_expiry_attr** | | `in_progress_expiration` | Unix timestamp of when record expires while in progress (in case of the invocation times out) |
385-
| **status_attr** | | `status` | Stores status of the Lambda execution during and after invocation |
386-
| **data_attr** | | `data` | Stores results of successfully executed Lambda handlers |
387-
| **validation_key_attr** | | `validation` | Hashed representation of the parts of the event used for validation |
388-
389473
### Idempotency request flow
390474

391475
The following sequence diagrams explain how the Idempotency feature behaves under different scenarios.
@@ -638,110 +722,21 @@ graph TD;
638722
<i>Race condition with Redis</i>
639723
</center>
640724

641-
## Redis as persistent storage layer provider
642-
643-
### Redis resources
644-
645-
Before setting up Redis as the persistent storage layer provider, you must have an existing Redis service. We recommend you to use Redis compatible services such as [Amazon ElastiCache for Redis](https://aws.amazon.com/elasticache/redis/){target="_blank"} or [Amazon MemoryDB for Redis](https://aws.amazon.com/memorydb/){target="_blank"} as your persistent storage layer provider.
646-
647-
???+ tip "No existing Redis service?"
648-
If you don't have an existing Redis service, we recommend using [DynamoDB](#dynamodbpersistencelayer) as the persistent storage layer provider.
649-
650-
=== "AWS CloudFormation example"
651-
652-
```yaml hl_lines="5"
653-
--8<-- "examples/idempotency/templates/cfn_redis_serverless.yaml"
654-
```
655-
656-
1. Replace the Security Group ID and Subnet ID to match your VPC settings.
657-
658-
### VPC Access
659-
660-
Your Lambda Function must have network access to the Redis endpoint before using it as the idempotency persistent storage layer. In most cases, you will need to [configure VPC access](https://docs.aws.amazon.com/lambda/latest/dg/configuration-vpc.html){target="_blank"} for your Lambda Function.
661-
662-
???+ tip "Amazon ElastiCache/MemoryDB for Redis as persistent storage layer provider"
663-
If you plan to use Amazon ElastiCache for Redis as the idempotency persistent storage layer, you may find [this AWS tutorial](https://docs.aws.amazon.com/lambda/latest/dg/services-elasticache-tutorial.html){target="_blank"} helpful.
664-
For those using Amazon MemoryDB for Redis, refer to [this AWS tutorial](https://aws.amazon.com/blogs/database/access-amazon-memorydb-for-redis-from-aws-lambda/){target="_blank"} specifically for the VPC setup guidance.
665-
666-
After completing the VPC setup, you can use the templates provided below to set up Lambda functions with access to VPC internal subnets.
667-
668-
=== "AWS Serverless Application Model (SAM) example"
669-
670-
```yaml hl_lines="9"
671-
--8<-- "examples/idempotency/templates/sam_redis_vpc.yaml"
672-
```
673-
674-
1. Replace the Security Group ID and Subnet ID to match your VPC settings.
675-
676-
### Configuring Redis persistence layer
677-
678-
You can quickly get started by initializing the `RedisCachePersistenceLayer` class and applying the `idempotent` decorator to your Lambda handler. For a detailed example of using the `RedisCachePersistenceLayer`, refer to the [Persistence layers section](#redispersistencelayer).
679-
680-
???+ info
681-
We enforce security best practices by using SSL connections in the `RedisCachePersistenceLayer`; to disable it, set `ssl=False`
682-
683-
=== "Use Persistence Layer with Redis config variables"
684-
```python hl_lines="7-9 12 26"
685-
--8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_config.py"
686-
```
687-
688-
=== "Use established Redis Client"
689-
```python hl_lines="4 9-11 14 22 36"
690-
--8<-- "examples/idempotency/src/getting_started_with_idempotency_redis_client.py"
691-
```
692-
693-
=== "Sample event"
694-
695-
```json
696-
--8<-- "examples/idempotency/src/getting_started_with_idempotency_payload.json"
697-
```
698-
699-
### Custom advanced settings
700-
701-
For advanced configurations, such as setting up SSL certificates or customizing parameters like a custom timeout, you can utilize the Redis client to tailor these specific settings to your needs.
702-
703-
=== "Advanced configuration using AWS Secrets"
704-
```python hl_lines="7-9 11 13 23"
705-
--8<-- "examples/idempotency/src/using_redis_client_with_aws_secrets.py"
706-
```
707-
708-
1. JSON stored:
709-
{
710-
"REDIS_ENDPOINT": "127.0.0.1",
711-
"REDIS_PORT": "6379",
712-
"REDIS_PASSWORD": "redis-secret"
713-
}
714-
715-
=== "Advanced configuration with local certificates"
716-
```python hl_lines="12 23-25"
717-
--8<-- "examples/idempotency/src/using_redis_client_with_local_certs.py"
718-
```
719-
720-
1. JSON stored:
721-
{
722-
"REDIS_ENDPOINT": "127.0.0.1",
723-
"REDIS_PORT": "6379",
724-
"REDIS_PASSWORD": "redis-secret"
725-
}
726-
2. redis_user.crt file stored in the "certs" directory of your Lambda function
727-
3. redis_user_private.key file stored in the "certs" directory of your Lambda function
728-
4. redis_ca.pem file stored in the "certs" directory of your Lambda function
729-
730725
## Advanced
731726

732727
### Customizing the default behavior
733728

734729
Idempotent decorator can be further configured with **`IdempotencyConfig`** as seen in the previous example. These are the available options for further configuration
735730

736-
| Parameter | Default | Description |
737-
|---------------------------------|---------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
738-
| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](./jmespath_functions.md#built-in-jmespath-functions){target="_blank"} |
739-
| **payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload |
740-
| **raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request |
741-
| **expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired |
742-
| **use_local_cache** | `False` | Whether to locally cache idempotency results |
743-
| **local_cache_max_items** | 256 | Max number of items to store in local cache |
744-
| **hash_function** | `md5` | Function to use for calculating hashes, as provided by [hashlib](https://docs.python.org/3/library/hashlib.html){target="_blank" rel="nofollow"} in the standard library. |
731+
| Parameter | Default | Description |
732+
| ------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
733+
| **event_key_jmespath** | `""` | JMESPath expression to extract the idempotency key from the event record using [built-in functions](./jmespath_functions.md#built-in-jmespath-functions){target="_blank"} |
734+
| **payload_validation_jmespath** | `""` | JMESPath expression to validate whether certain parameters have changed in the event while the event payload |
735+
| **raise_on_no_idempotency_key** | `False` | Raise exception if no idempotency key was found in the request |
736+
| **expires_after_seconds** | 3600 | The number of seconds to wait before a record is expired |
737+
| **use_local_cache** | `False` | Whether to locally cache idempotency results |
738+
| **local_cache_max_items** | 256 | Max number of items to store in local cache |
739+
| **hash_function** | `md5` | Function to use for calculating hashes, as provided by [hashlib](https://docs.python.org/3/library/hashlib.html){target="_blank" rel="nofollow"} in the standard library. |
745740
| **response_hook** | `None` | Function to use for processing the stored Idempotent response. This function hook is called when an existing idempotent response is found. See [Manipulating The Idempotent Response](idempotency.md#manipulating-the-idempotent-response) |
746741

747742
### Handling concurrent executions with the same payload

‎examples/idempotency/src/using_redis_client_with_aws_secrets.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
from __future__ import annotations
2+
13
from typing import Any
24

35
from redis import Redis
@@ -8,7 +10,7 @@
810
RedisCachePersistenceLayer,
911
)
1012

11-
redis_values: Any = parameters.get_secret("redis_info", transform="json") # (1)!
13+
redis_values: dict[str, Any] = parameters.get_secret("redis_info", transform="json") # (1)!
1214

1315
redis_client = Redis(
1416
host=redis_values.get("REDIS_HOST"),

‎examples/idempotency/src/using_redis_client_with_local_certs.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
from __future__ import annotations
2+
13
from typing import Any
24

35
from redis import Redis
@@ -9,7 +11,7 @@
911
RedisCachePersistenceLayer,
1012
)
1113

12-
redis_values: Any = parameters.get_secret("redis_info", transform="json") # (1)!
14+
redis_values: dict[str, Any] = parameters.get_secret("redis_info", transform="json") # (1)!
1315

1416

1517
redis_client = Redis(

0 commit comments

Comments
 (0)