From efc3245a1b58046a2a393c31632ebf0fc75ecf74 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 13 May 2025 15:02:05 -0500 Subject: [PATCH 1/7] Address feedback on Lamda page Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../configuration/processors/aws-lambda.md | 26 +++++++++++-------- 1 file changed, 15 insertions(+), 11 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index 0ef9dfd7d74..67706de2d28 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -10,7 +10,6 @@ nav_order: 10 The [AWS Lambda](https://aws.amazon.com/lambda/) integration allows developers to use serverless computing capabilities within their OpenSearch Data Prepper pipelines for flexible event processing and data routing. -## AWS Lambda processor configuration The `aws_lambda` processor enables invocation of an AWS Lambda function within your Data Prepper pipeline in order to process events. It supports both synchronous and asynchronous invocations based on your use case. @@ -18,8 +17,8 @@ The `aws_lambda` processor enables invocation of an AWS Lambda function within y You can configure the processor using the following configuration options. -Field | Type | Required | Description --------------------- | ------- | -------- | ---------------------------------------------------------------------------- +Field | Type | Required | Description +:--- | :------- | :--- | :--- `function_name` | String | Required | The name of the AWS Lambda function to invoke. `invocation_type` | String | Required | Specifies the invocation type, either `request-response` or `event`. Default is `request-response`. `aws.region` | String | Required | The AWS Region in which the Lambda function is located. @@ -29,18 +28,24 @@ Field | Type | Required | Description `lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor. `response_codec` | Object | Optional | A codec configuration for parsing Lambda responses. Default is `json`. `tags_on_match_failure` | List | Optional | A list of tags to add to events when Lambda matching fails or encounters an unexpected error. -`sdk_timeout` | Duration| Optional | Configures the SDK's client connection timeout period. Default is `60s`. +`sdk_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. -#### Example configuration +The invocation_type field is not supported in Amazon OpenSearch Ingestion pipelines using the aws_lambda processor. Including this field will result in a validation error. +{: .note} + + +## Usage + +This example configures the `aws_lambda` processor to invoke a Lambda function with advanced batching, conditional invocation, retry logic, and cross-region AWS credentials: ``` processors: - aws_lambda: - function_name: "my-lambda-function" - invocation_type: "request-response" - response_events_match: false - aws: + arn: "arn:aws:lambda:us-west-2:123456789012:function:my-processing-function" + timeout: 2000 + batch_size: 10 + aws: region: "us-east-1" sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-role" max_retries: 3 @@ -51,11 +56,10 @@ processors: maximum_size: "5mb" event_collect_timeout: PT10S lambda_when: "event['status'] == 'process'" - ``` {% include copy-curl.html %} -## Usage +## Invocation types The processor supports the following invocation types: From 2f0a15e41b4ec703ed6590da1dfc0880bf7b1252 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 22 May 2025 16:16:37 -0500 Subject: [PATCH 2/7] Apply suggestions from code review Co-authored-by: David Venable Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _data-prepper/pipelines/configuration/processors/aws-lambda.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index 67706de2d28..bda874d2351 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -31,7 +31,7 @@ Field | Type | Required | Description `sdk_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. -The invocation_type field is not supported in Amazon OpenSearch Ingestion pipelines using the aws_lambda processor. Including this field will result in a validation error. +The `invocation_type` field is not supported in Amazon OpenSearch Ingestion pipelines using the `aws_lambda` processor. Including this field will result in a validation error. {: .note} From 770e93db9bf35a86177960582547dd3f7b77e918 Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Thu, 12 Jun 2025 18:17:09 -0500 Subject: [PATCH 3/7] Apply suggestions from code review Co-authored-by: David Venable Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../pipelines/configuration/processors/aws-lambda.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index bda874d2351..c195bc82503 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -28,7 +28,7 @@ Field | Type | Required | Description `lambda_when` | String | Optional | A conditional expression that determines when to invoke the Lambda processor. `response_codec` | Object | Optional | A codec configuration for parsing Lambda responses. Default is `json`. `tags_on_match_failure` | List | Optional | A list of tags to add to events when Lambda matching fails or encounters an unexpected error. -`sdk_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. +`connection_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. The `invocation_type` field is not supported in Amazon OpenSearch Ingestion pipelines using the `aws_lambda` processor. Including this field will result in a validation error. @@ -42,9 +42,11 @@ This example configures the `aws_lambda` processor to invoke a Lambda function w ``` processors: - aws_lambda: - arn: "arn:aws:lambda:us-west-2:123456789012:function:my-processing-function" - timeout: 2000 - batch_size: 10 + function_name: "arn:aws:lambda:us-west-2:123456789012:function:my-processing-function" + batch: + threshold: + event_collect_timeout: 30s + event_count: 10 aws: region: "us-east-1" sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-role" From 13f92379aa18af8e54561b45439cf399be891d0d Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Wed, 25 Jun 2025 13:45:08 -0500 Subject: [PATCH 4/7] Update aws-lambda.md Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- .../configuration/processors/aws-lambda.md | 15 +++------------ 1 file changed, 3 insertions(+), 12 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index c195bc82503..c5b7330c042 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -20,7 +20,7 @@ You can configure the processor using the following configuration options. Field | Type | Required | Description :--- | :------- | :--- | :--- `function_name` | String | Required | The name of the AWS Lambda function to invoke. -`invocation_type` | String | Required | Specifies the invocation type, either `request-response` or `event`. Default is `request-response`. +`invocation_type` | String | Required | Specifies the [invocation type](#invocation-type), either `request-response` or `event`. Default is `request-response`. `aws.region` | String | Required | The AWS Region in which the Lambda function is located. `aws.sts_role_arn` | String | Optional | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function. `max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`. @@ -32,7 +32,7 @@ Field | Type | Required | Description `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. The `invocation_type` field is not supported in Amazon OpenSearch Ingestion pipelines using the `aws_lambda` processor. Including this field will result in a validation error. -{: .note} +{: .warning} ## Usage @@ -46,17 +46,11 @@ processors: batch: threshold: event_collect_timeout: 30s - event_count: 10 + maximum_size: "5mb" aws: region: "us-east-1" sts_role_arn: "arn:aws:iam::123456789012:role/my-lambda-role" max_retries: 3 - batch: - key_name: "events" - threshold: - event_count: 100 - maximum_size: "5mb" - event_collect_timeout: PT10S lambda_when: "event['status'] == 'process'" ``` {% include copy-curl.html %} @@ -67,9 +61,6 @@ The processor supports the following invocation types: - `request-response`: The processor waits for Lambda function completion before proceeding. - `event`: The function is triggered asynchronously without waiting for a response. -- `batch`: When enabled, events are aggregated and sent in bulk to optimize Lambda invocations. Batch thresholds control the event count, size limit, and timeout. -- `codec`: JSON is used for both request and response codecs. Lambda must return JSON array outputs. -- `tags_on_match_failure`: Custom tags can be applied to events when Lambda processing fails or encounters unexpected issues. ## Behavior From f297671dc627b4dbc3d2e55f94472af8a3d7e3bb Mon Sep 17 00:00:00 2001 From: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> Date: Tue, 8 Jul 2025 06:20:28 -0500 Subject: [PATCH 5/7] Apply suggestions from code review Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com> --- _data-prepper/pipelines/configuration/processors/aws-lambda.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index c5b7330c042..6b0dc246462 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -20,7 +20,7 @@ You can configure the processor using the following configuration options. Field | Type | Required | Description :--- | :------- | :--- | :--- `function_name` | String | Required | The name of the AWS Lambda function to invoke. -`invocation_type` | String | Required | Specifies the [invocation type](#invocation-type), either `request-response` or `event`. Default is `request-response`. +`invocation_type` | String | Required | Specifies the [invocation type](#invocation-types), either `request-response` or `event`. Default is `request-response`. `aws.region` | String | Required | The AWS Region in which the Lambda function is located. `aws.sts_role_arn` | String | Optional | The Amazon Resource Name (ARN) of the role to assume before invoking the Lambda function. `max_retries` | Integer | Optional | The maximum number of retries for failed invocations. Default is `3`. From e12218604ff6d9fd5d5561e15129aad5e39713fd Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Mon, 20 Oct 2025 15:10:15 +0100 Subject: [PATCH 6/7] taking over the PR and adding details Signed-off-by: Anton Rubin --- .../pipelines/configuration/processors/aws-lambda.md | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index 6b0dc246462..405c834c58e 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -31,7 +31,13 @@ Field | Type | Required | Description `connection_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. -The `invocation_type` field is not supported in Amazon OpenSearch Ingestion pipelines using the `aws_lambda` processor. Including this field will result in a validation error. +### Invocation types + +The `request-response` invocation type is used for synchronous processing, the pipeline waits for the Lambda result. This is the default for `aws_lambda` processor. + +The `event` invocation type sends the request to Lambda asynchronously and Data Prepper doesn't wait for a response. Default for `aws_lambda` sink. + +The `invocation_type` field is not supported in [Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html) pipelines using the `aws_lambda` processor. Including this field will result in a validation error. {: .warning} From 37187131b63124e5f509c0609b3f403e0ce41b2e Mon Sep 17 00:00:00 2001 From: Anton Rubin Date: Mon, 20 Oct 2025 15:13:58 +0100 Subject: [PATCH 7/7] removing duplicates Signed-off-by: Anton Rubin --- .../pipelines/configuration/processors/aws-lambda.md | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/_data-prepper/pipelines/configuration/processors/aws-lambda.md b/_data-prepper/pipelines/configuration/processors/aws-lambda.md index 405c834c58e..c9706ef04a2 100644 --- a/_data-prepper/pipelines/configuration/processors/aws-lambda.md +++ b/_data-prepper/pipelines/configuration/processors/aws-lambda.md @@ -31,12 +31,6 @@ Field | Type | Required | Description `connection_timeout` | Duration | Optional | Configures the SDK's client connection timeout period. Default is `60s`. `response_events_match` | Boolean | Optional | Specifies how Data Prepper interprets and processes Lambda function responses. Default is `false`. -### Invocation types - -The `request-response` invocation type is used for synchronous processing, the pipeline waits for the Lambda result. This is the default for `aws_lambda` processor. - -The `event` invocation type sends the request to Lambda asynchronously and Data Prepper doesn't wait for a response. Default for `aws_lambda` sink. - The `invocation_type` field is not supported in [Amazon OpenSearch Ingestion](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/ingestion.html) pipelines using the `aws_lambda` processor. Including this field will result in a validation error. {: .warning} @@ -66,7 +60,7 @@ processors: The processor supports the following invocation types: - `request-response`: The processor waits for Lambda function completion before proceeding. -- `event`: The function is triggered asynchronously without waiting for a response. +- `event`: The function is triggered asynchronously without waiting for a response. ## Behavior