Skip to content

Commit 019f751

Browse files
srikanthjgkolchfa-aws
srikanthjg
and
kolchfa-aws
authored
Addressing aws lambda documentation changes (#11432)
* Addressing aws lambda documentation changes Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestion from @kolchfa-aws Co-authored-by: kolchfa-aws <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> * Apply suggestions from code review Signed-off-by: kolchfa-aws <[email protected]> --------- Signed-off-by: Srikanth Govindarajan <[email protected]> Signed-off-by: Srikanth Govindarajan <[email protected]> Signed-off-by: kolchfa-aws <[email protected]> Co-authored-by: kolchfa-aws <[email protected]>
1 parent d30a678 commit 019f751

File tree

1 file changed

+58
-23
lines changed

1 file changed

+58
-23
lines changed

‎_data-prepper/pipelines/configuration/processors/aws-lambda.md‎

Lines changed: 58 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -20,25 +20,30 @@ You can configure the processor using the following configuration options.
2020

2121
Field | Type | Required | Description
2222
-------------------- | ------- | -------- | ----------------------------------------------------------------------------
23-
`function_name` | String | Required | The name of the AWS Lambda function to invoke.
24-
`invocation_type` | String | Required | Specifies the invocation type, either `request-response` or `event`. Default is `request-response`.
23+
`function_name` | String | Required | The name of the AWS Lambda function to invoke. Must be 3--500 characters.
24+
`invocation_type` | String | Optional | Specifies the invocation type, either `request-response` or `event`. Default is `request-response`.
2525
`aws.region` | String | Required | The AWS Region in which the Lambda function is located.
26-
`aws.sts_role_arn` | String | Optional | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function.
27-
`max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`.
26+
`aws.sts_role_arn` | String | Optional | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function. Must be 20--2048 characters.
27+
`aws.sts_external_id` | String | Optional | An external ID for STS role assumption. Must be 2--1224 characters.
28+
`aws.sts_header_overrides` | Map | Optional | STS header overrides. Maximum of 5 headers supported.
29+
`client.max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`.
30+
`client.api_call_timeout` | Duration | Optional | The API call timeout. Default is `60s`.
31+
`client.connection_timeout` | Duration | Optional | The SDK connection timeout. Default is `60s`.
32+
`client.max_concurrency` | Integer | Optional | The maximum number of concurrent threads on the client. Default is `200`.
33+
`client.base_delay` | Duration | Optional | The base delay for the exponential backoff. Default is `100ms`.
34+
`client.max_backoff` | Duration | Optional | The maximum backoff time for the exponential backoff. Default is `20s`.
2835
`batch` | Object | Optional | The batch settings for the Lambda invocations. Default is `key_name = "events"`. Default threshold is `event_count=100`, `maximum_size="5mb"`, and `event_collect_timeout = 10s`.
2936
`lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor.
3037
`response_codec` | Object | Optional | A codec configuration for parsing Lambda responses. Default is `json`.
31-
`tags_on_match_failure` | List | Optional | A list of tags to add to events when Lambda matching fails or encounters an unexpected error.
32-
`sdk_timeout` | Duration| Optional | Configures the SDK's client connection timeout period. Default is `60s`.
38+
`tags_on_failure` | List | Optional | A list of tags to add to events when the Lambda function fails or encounters an exception.
3339
`response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`.
34-
`client` | Object | Optional | The client configuration.
35-
`api_call_timeout` | Duration | Optional | The amount of time that the SDK maintains the API call before timing out, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations).
36-
`base_delay` | Duration | Optional | The base delay for exponential backoff, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations).
37-
`connection_timeout` | Duration | Optional | The amount of time that the SDK maintains the connection to the client before timing out, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations).
38-
`max_backoff` | Duration | Optional | The maximum backoff time for exponential backoff, in [ISO 8601 format](https://en.wikipedia.org/wiki/ISO_8601#Durations).
39-
`max_concurrency` | Integer | Optional | The maximum concurrency defined on the client side.
40-
`max_retries` | Integer | Optional | The maximum number of retries before failing.
41-
40+
`response_mode` | String | Optional | The response handling mode, either `replace` or `merge`. Default is `replace`.
41+
`keys` | List | Optional | Keys to send to the Lambda function.
42+
`cache` | Object | Optional | The cache configuration. Only valid when `response_mode` is `merge` and `keys` are specified.
43+
`cache.ttl` | Long | Optional | The cache time-to-live.
44+
`cache.max_size` | Long | Optional | The maximum cache size. Must be between 1048576 and 10485760.
45+
`circuit_breaker_retries` | Integer | Optional | The maximum number of circuit breaker checks before proceeding. Default is `0`.
46+
`circuit_breaker_wait_interval` | Long | Optional | The amount of time, in milliseconds, between circuit breaker checks. Default is `1000ms`.
4247

4348
#### Example configuration
4449

@@ -48,20 +53,31 @@ processors:
4853
function_name: "my-lambda-function"
4954
invocation_type: "request-response"
5055
response_events_match: false
51-
client:
52-
connection_timeout: PT5M
53-
api_call_timeout: PT5M
56+
response_mode: "replace"
5457
aws:
5558
region: "us-east-1"
5659
sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-role"
57-
max_retries: 3
60+
client:
61+
max_retries: 3
62+
api_call_timeout: "PT60S"
63+
connection_timeout: "PT60S"
64+
max_concurrency: 200
65+
base_delay: "PT0.1S"
66+
max_backoff: "PT20S"
5867
batch:
5968
key_name: "events"
6069
threshold:
6170
event_count: 100
6271
maximum_size: "5mb"
63-
event_collect_timeout: PT10S
64-
lambda_when: "event['status'] == 'process'"
72+
event_collect_timeout: "PT10S"
73+
lambda_when: "/some_key == null"
74+
keys: ["key1", "key2"]
75+
cache:
76+
ttl: 3600
77+
max_size: 5242880
78+
circuit_breaker_retries: 0
79+
circuit_breaker_wait_interval: 1000
80+
tags_on_failure: ["lambda_failed"]
6581
```
6682
{% include copy.html %}
6783
@@ -71,9 +87,28 @@ The processor supports the following invocation types:
7187
7288
- `request-response`: The processor waits for Lambda function completion before proceeding.
7389
- `event`: The function is triggered asynchronously without waiting for a response.
74-
- `batch`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Batch thresholds control the event count, size limit, and timeout.
75-
- `codec`: JSON is used for both request and response codecs. Lambda must return JSON array outputs.
76-
- `tags_on_match_failure`: Custom tags can be applied to events when Lambda processing fails or encounters unexpected issues.
90+
91+
### Batch processing
92+
93+
When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Batch thresholds control the event count, size limit, and timeout.
94+
95+
### Response handling
96+
97+
The processor supports two response modes:
98+
- `replace`: Lambda response replaces the original event data (default)
99+
- `merge`: Lambda response is merged with the original event data
100+
101+
### Caching
102+
103+
When `response_mode` is set to `merge` and `keys` are specified, the processor can cache Lambda responses to improve performance for repeated requests.
104+
105+
### Circuit breaker
106+
107+
The processor includes circuit breaker functionality to handle memory pressure situations gracefully.
108+
109+
### Tags on failure
110+
111+
Custom tags can be applied to events when Lambda processing fails or encounters exceptions using the `tags_on_failure` configuration.
77112

78113
## Behavior
79114

0 commit comments

Comments
 (0)