New component: aws lambda receiver

[MichaelKatsoulis]

The purpose and use-cases of the new component

Summary

Add a new awslambdareceiver component that receives AWS Lambda invocations and decodes messages using provided encoding extensions. This receiver is designed to run as part of an OpenTelemetry Collector deployed as an AWS Lambda function.

Motivation

AWS Lambda is a popular serverless service used extensively for event-driven architectures. Many AWS services (S3, CloudWatch, SNS, SQS) can trigger Lambda functions, making it an ideal entry point for collecting data from AWS services.

The awslambdareceiver enables users to:

• Collect logs stored in AWS S3 buckets (VPC Flow Logs, ELB Access Logs, CloudTrail, WAF, S3 Access Logs, etc.)
• Ingest CloudWatch Logs via subscription filters
• Process custom JSON logs stored in S3
• Leverage OpenTelemetry's encoding extensions for message parsing

Primary Use Cases

1. S3-triggered Log Collection: Lambda is invoked by S3 event notifications when new log files are created
2. CloudWatch Logs: Lambda is invoked by CloudWatch Logs subscription filters

Design

The awslambdareceiver operates as follows:

• Accepts Lambda Invocations
• Identifies the event source (S3, CloudWatch, etc.)
• Uses configured encoding extensions to parse the data
• Creates OpenTelemetry log records
• Passes parsed logs to the collector pipeline

Supported Event Sources

• S3 Event Notifications (s3:ObjectCreated:*)
  • Fetches objects from S3
  • Decodes using specified encoding extension
• CloudWatch Logs Subscription Filters
  • Decodes CloudWatch Logs data
  • Extracts log events

Example configuration for the component

Example 1: VPC Flow Logs from S3

receivers:
  awslambda:
    s3_encoding: awslogs_encoding
    failure_bucket_arn: <Optional arn of an existing s3 bucket to store failed lambda invocation events>

extensions:
  awslogs_encoding:
    format: vpcflow
    vpcflow:
      file_format: plain-text

exporters:
  otlphttp:
    endpoint: "https://my-backend:443"

service:
  extensions:
    - awslogs_encoding
  pipelines:
    logs:
      receivers: [awslambda]
      exporters: [otlphttp]

In this example, awslambdareceiver receives a notification when a new VPC flow log file is stored in an S3 bucket. The receiver fetches the log file from S3 and parses it using the awslogs_encoding extension with vpcflow format. The parsed logs are then sent to an OTLP listener using the otlphttp exporter.

Example 2: CloudWatch Logs Subscription

receivers:
  awslambda:
    failure_bucket_arn: "my-bucket-arn"

exporters:
  otlphttp:
    endpoint: "https://my-backend:443"

service:
  pipelines:
    logs:
      receivers: [awslambda]
      exporters: [otlphttp]

In this example, awslambdareceiver is invoked by a CloudWatch Logs subscription filter. The receiver automatically parses the CloudWatch Logs data using the default awslogs_encoding extension with cloudwatch format. No explicit encoding configuration is needed. The parsed logs are then sent to an OTLP listener using the otlphttp exporter.

Telemetry data types supported

• Logs (Primary support)
• Metrics (Future consideration)

Code Owner(s)

@MichaelKatsoulis, @Kavindu-Dodan

Sponsor (optional)

@axw

Additional context

This component is currently implemented and used by Elastic's EDOT Cloud Forwarder for AWS. We propose open sourcing it to benefit the broader OpenTelemetry community.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[atoulme]

@constanca-m @mxiamxia please review

[constanca-m]

Did you mean to tag me @atoulme ? I work with the codeowners, so I see value in this component as well since we are using it internally already. I am pretty biased 😅

[axw]

For the record: I also work with these folks, so also biased, but I think this would be a good addition to the OpenTelemetry Collector project. Ideally we would have a sponsor from outside of Elastic.

[atoulme]

I make no assumptions and appreciate all your reviews, as I have worked with you previously. That said I also meant to tag @Aneurysm9 along with @mxiamxia. Anyone from the AWS/ADOT team with a vested interest in supporting the AWS ecosystem with an opinion on this approach?

[MovieStoreGuy]

👋🏽

I wanted to understand how this is potentially different from something like the https://github.com/open-telemetry/opentelemetry-lambda ?

I understand the above is an extension deployed to run along side with existing code, however, would this be better suited as part of the lambda project?

Also, am I correct in understanding that the intent here is to the run the collector in FaaS? (AWS Lambda directly)
If so, it would also be nice to include some documentation / best practices of what a collector deployment looks like in a FaaS

[axw]

@MovieStoreGuy good question! They serve different purposes. We could challenge that, but I think it's probably better to keep the two use cases separate since they are quite different.

The opentelemetry-lambda extension is for exporting telemetry from a Lambda, i.e. it's auxiliary to the Lambda (e.g. some custom business application).

The receiver proposed here is intended to be used where the collector is the Lambda, i.e. the receiver implements the Lambda handler, for integrating with S3 event notifications & CloudWatch subscription.

Also, am I correct in understanding that the intent here is to the run the collector in FaaS? (AWS Lambda directly)

Yes

If so, it would also be nice to include some documentation / best practices of what a collector deployment looks like in a FaaS

Anything specific you're looking for here?

[MovieStoreGuy]

What I'd look for in documentation is something like:

• Can FaaS deployments support agent mode or gateway modes?
• What is the recommended right sizing?
• What considerations is there for using a FaaS deployment?

[axw]

I'll let @MichaelKatsoulis or @Kavindu-Dodan respond to the other points, but I can address this one:

Can FaaS deployments support agent mode or gateway modes?

Neither really - the proposed name might be misleading.

The receiver has a fairly specific function: it handles Lambda events from S3 and CloudWatch. In the S3 case it will go and fetch the related objects from S3, parse them using an encoding extension, and then send them down the pipeline. The CloudWatch case is a little simpler, as the logs come in the event payload.

For context, we built this with the intention of creating a vendor-neutral replacement for Elastic Serverless Forwarder for AWS. There are similar proprietary products out there, e.g. Datadog Forwarder and Honeycomb Agentless Integrations for AWS.

[MovieStoreGuy]

Ack,

I am keen to hear more details just so we can avoid any confusion or potential project overlap before it starts.

[Kavindu-Dodan]

@MovieStoreGuy Good questions, I will try to answer the deployment & sizing aspects,

Can FaaS deployments support agent mode or gateway modes?

We have only seen the need for the Agent mode and haven't come across use cases for the gateway mode. But given the proposed component is a receiver which gets deployed as an OTel collector, there shouldn't be any blocker to deploy it in gateway mode if there's a valid use case.

What is the recommended right sizing?

I assume you are referring to Lambda configurations such as memory & reserved concurrency. If so, it will depend on the signal type and log generation frequency.

For example, when comparing VPC & CloudTrail logs, CloudTrail logs have more attributes. Therefore, if the same number of logs gets pushed into a single S3 entry (note - both VPC and CloudTrail logs are aggregated at 5 minute intervals), then the collector handling CloudTrail logs needs more memory.

Also, if the log creation frequency is high, for example, if logs get generated by multiple VPCs and if they all get forwarded to the same S3 bucket, which awslambdareceiver configured collector listens, then the reserved concurrency needs to be increased.

Altogether, sizing is a little complex. But in our tests, we have seen receiver -> encoder -> exporter working efficiently when deployed as a Lambda. For example, one of our tests showed 25K CloudTrail log S3 entry gets processed within 5 seconds. And that's with 512MB Lambda memory.