Add AWS Lambda logs integration with Rotel

Author: mheffner

docs/use-cases/observability/clickstack/integration-examples/aws-lambda.md

@@ -0,0 +1,329 @@
+---
+slug: /use-cases/observability/clickstack/integrations/aws-lambda
+title: 'Monitoring AWS Lambda Logs with ClickStack'
+sidebar_label: 'AWS Lambda Logs'
+pagination_prev: null
+pagination_next: null
+description: 'Monitoring AWS Lambda Logs with ClickStack'
+doc_type: 'guide'
+keywords: ['AWS', 'Lambda', 'OTEL', 'ClickStack', 'logs', 'CloudWatch']
+---
+
+import Image from '@theme/IdealImage';
+import useBaseUrl from '@docusaurus/useBaseUrl';
+import log_view from '@site/static/images/clickstack/lambda/lambda-log-view.png';
+import log from '@site/static/images/clickstack/lambda/lambda-log.png';
+import { TrackedLink } from '@site/src/components/GalaxyTrackedLink/GalaxyTrackedLink';
+
+# Monitoring AWS Lambda Logs with ClickStack {#lambda-clickstack}
+
+:::note[TL;DR]
+This guide shows you how to monitor AWS Lambda functions with ClickStack by using the Rotel Lambda Extension to collect and forward function logs, extension logs, and OpenTelemetry data directly to ClickHouse. You'll learn how to:
+
+- Deploy the Rotel Lambda Extension layer to your Lambda functions
+- Configure the extension to export logs and traces to ClickStack
+- Optionally disable CloudWatch Logs to reduce costs
+- Use pre-built dashboards to visualize Lambda metrics (invocations, errors, duration, cold starts)
+
+This approach can **significantly reduce your Lambda observability costs** by bypassing CloudWatch Logs entirely.
+
+Time Required: 5-10 minutes
+:::
+
+## Integration with existing Lambda functions {#existing-lambda}
+
+This section covers configuring your existing AWS Lambda functions to send logs and traces to ClickStack using the Rotel Lambda Extension.
+
+### Prerequisites {#prerequisites}
+- ClickStack instance running
+- AWS Lambda function(s) to monitor
+- AWS CLI configured with appropriate permissions
+- Lambda execution role with permissions to add layers
+
+<VerticalStepper headerLevel="h4">
+
+#### Choose the appropriate Rotel Lambda Extension layer {#choose-layer}
+
+The [Rotel Lambda Extension](https://github.com/streamfold/rotel-lambda-extension) is available as a pre-built AWS Lambda layer. Choose the layer ARN that matches your Lambda function's architecture:
+
+| Architecture | ARN Pattern | Latest Version |
+|--------------|-------------|----------------|
+| x86-64/amd64 | `arn:aws:lambda:{region}:418653438961:layer:rotel-extension-amd64-alpha:{version}` | ![Version](https://img.shields.io/github/v/release/streamfold/rotel-lambda-extension?filter=*alpha&label=version&labelColor=%2338BDF8&color=%23312E81&cacheSeconds=600) |
+| arm64        | `arn:aws:lambda:{region}:418653438961:layer:rotel-extension-arm64-alpha:{version}` | ![Version](https://img.shields.io/github/v/release/streamfold/rotel-lambda-extension?filter=*alpha&label=version&labelColor=%2338BDF8&color=%23312E81&cacheSeconds=600) |
+
+**Available regions:**
+- us-east-{1, 2}, us-west-{1, 2}
+- eu-central-1, eu-north-1, eu-west-{1, 2, 3}
+- ca-central-1
+- ap-southeast-{1, 2}, ap-northeast-{1, 2}
+- ap-south-1
+- sa-east-1
+
+#### Add the Rotel layer to your Lambda function {#add-layer}
+
+_In these examples replace `{arch}`, `{region}`, and `{version}` with the appropriate values above._
+
+##### Option 1: AWS Console {#console}
+
+1. Open the AWS Lambda console
+2. Navigate to your Lambda function
+3. Scroll to the **Layers** section and click **Add a layer**
+4. Select **Specify an ARN**
+5. Enter the Rotel layer ARN:
+   ```
+   arn:aws:lambda:{region}:418653438961:layer:rotel-extension-{arch}-alpha:{version}
+   ```
+6. Click **Add**
+
+##### Option 2: AWS CLI {#cli}
+
+```bash
+aws lambda update-function-configuration \
+  --function-name my-function \
+  --layers arn:aws:lambda:{region}:418653438961:layer:rotel-extension-{arch}-alpha:{version}
+```
+
+##### Option 3: AWS SAM {#sam}
+
+```yaml
+Resources:
+  MyFunction:
+    Type: AWS::Serverless::Function
+    Properties:
+      # ... other configuration ...
+      Layers:
+        - arn:aws:lambda:{version}:418653438961:layer:rotel-extension-{arch}-alpha:{version}
+```
+
+#### Configure the extension to export to ClickStack {#configure-extension}
+
+The Rotel Lambda Extension is configured using environment variables. You need to configure the OTLP exporter endpoint to point to your ClickStack instance. The examples assume your AWS Lambda function is able to reach the ClickStack instance.
+
+##### Basic Configuration (Environment Variables) {#basic-config}
+
+Add these environment variables to your Lambda function:
+
+```bash
+# Required: ClickStack OTLP endpoint
+ROTEL_OTLP_EXPORTER_ENDPOINT=https://clickstack.example.com:4317
+
+# Optional: Authentication headers
+ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=<YOUR_INGESTION_API_KEY>"
+
+# Optional: Service name (defaults to Lambda function name)
+ROTEL_OTEL_RESOURCE_ATTRIBUTES="service.name=my-lambda-api,service.version=1.0.0"
+```
+
+##### Advanced Configuration (Using .env file) {#advanced-config}
+
+For more complex configurations, create a `rotel.env` file in your Lambda function bundle:
+
+**rotel.env:**
+```shell
+ROTEL_OTLP_EXPORTER_ENDPOINT=https://clickstack.example.com:4317
+ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=<YOUR_INGESTION_API_KEY>"
+ROTEL_OTEL_RESOURCE_ATTRIBUTES="service.name=my-lambda-api,deployment.environment=production"
+```
+
+Then set the environment variable to point to this file:
+```bash
+ROTEL_ENV_FILE=/var/task/rotel.env
+```
+
+##### Using AWS Secrets Manager or Parameter Store {#secrets}
+
+For production deployments, store sensitive values like API keys in AWS Secrets Manager or Parameter Store:
+
+**AWS Secrets Manager Example:**
+```shell
+ROTEL_OTLP_EXPORTER_ENDPOINT=https://clickstack.example.com:4317
+ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=${arn:aws:secretsmanager:us-east-1:123456789012:secret:clickstack-api-key-abc123}"
+```
+
+**AWS Parameter Store Example:**
+```shell
+ROTEL_OTLP_EXPORTER_ENDPOINT=https://clickstack.example.com:4317
+ROTEL_OTLP_EXPORTER_CUSTOM_HEADERS="Authorization=${arn:aws:ssm:us-east-1:123456789012:parameter/clickstack-api-key}"
+```
+
+**Required IAM Permissions:**
+
+Add these permissions to your Lambda execution role:
+
+For Secrets Manager:
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "secretsmanager:GetSecretValue",
+        "secretsmanager:BatchGetSecretValue"
+      ],
+      "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:clickstack-api-key-*"
+    }
+  ]
+}
+```
+
+For Parameter Store:
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": [
+        "ssm:GetParameters"
+      ],
+      "Resource": "arn:aws:ssm:us-east-1:123456789012:parameter/clickstack-api-key"
+    }
+  ]
+}
+```
+
+:::note
+AWS API calls for secret retrieval add 100-150ms to cold start latency. Secrets are retrieved in batches (up to 10) and only on initialization, so subsequent invocations are not impacted.
+:::
+
+#### Test the integration {#test-integration}
+
+Invoke your Lambda function to verify logs are being sent to ClickStack:
+
+```bash
+aws lambda invoke \
+  --function-name my-function \
+  --payload '{"test": "data"}' \
+  response.json
+```
+
+Check the Lambda logs for any errors:
+```bash
+aws logs tail /aws/lambda/my-function --follow
+```
+
+#### Verify logs in HyperDX {#verify-logs}
+
+Once configured, log into HyperDX (ClickStack's UI) and verify that logs are flowing:
+
+<Image img={log_view} alt="Lambda Log View"/>
+
+<Image img={log} alt="Lambda Log Detail"/>
+
+Look for these key attributes in the logs:
+- `service.name`: Your Lambda function name
+- `faas.name`: AWS Lambda function name
+- `faas.invocation_id`: Unique invocation ID
+- `cloud.provider`: "aws"
+- `cloud.platform`: "aws_lambda"
+
+</VerticalStepper>
+
+## Disabling CloudWatch Logs (Cost Optimization) {#disable-cloudwatch}
+
+By default, AWS Lambda sends all logs to CloudWatch Logs, which can be expensive at scale. Once you've verified that logs are flowing to ClickStack, you can disable CloudWatch logging to reduce costs.
+
+<VerticalStepper headerLevel="h4">
+
+#### Remove CloudWatch permissions from the execution role {#remove-permissions}
+
+1. Open the AWS Console and navigate to **AWS Lambda**
+2. Navigate to your Lambda function
+3. Select **Configuration** → **Permissions**
+4. Click the execution role name to open the IAM console
+5. Edit the role and remove any `logs:*` actions:
+   - If using a custom policy, edit to remove `logs:CreateLogGroup`, `logs:CreateLogStream`, and `logs:PutLogEvents`
+   - If using the AWS managed policy `AWSLambdaBasicExecutionRole`, remove it from the role
+6. Save the role
+
+#### Verify CloudWatch logging is disabled {#verify-disabled}
+
+Invoke your function again and verify that:
+1. No new CloudWatch log streams are created
+2. Logs continue to appear in ClickStack/HyperDX
+
+```bash
+# This should show no new log streams after the policy change
+aws logs describe-log-streams \
+  --log-group-name /aws/lambda/my-function \
+  --order-by LastEventTime \
+  --descending \
+  --max-items 5
+```
+
+</VerticalStepper>
+
+## Adding OpenTelemetry auto-instrumentation {#auto-instrumentation}
+
+The Rotel Lambda Extension works seamlessly with OpenTelemetry auto-instrumentation layers to collect distributed traces and metrics in addition to logs.
+
+<VerticalStepper headerLevel="h4">
+
+#### Choose your language instrumentation layer {#choose-instrumentation}
+
+AWS provides OpenTelemetry auto-instrumentation layers for multiple languages:
+
+| Language | Layer ARN Pattern |
+|----------|-------------------|
+| Node.js  | `arn:aws:lambda:{region}:901920570463:layer:aws-otel-nodejs-{arch}-ver-{version}` |
+| Python   | `arn:aws:lambda:{region}:901920570463:layer:aws-otel-python-{arch}-ver-{version}` |
+| Java     | `arn:aws:lambda:{region}:901920570463:layer:aws-otel-java-agent-{arch}-ver-{version}` |
+
+Find the latest versions in the [AWS OpenTelemetry Lambda repository](https://github.com/aws-observability/aws-otel-lambda).
+
+#### Add both layers to your function {#add-both-layers}
+
+Add **both** the Rotel extension layer and the instrumentation layer:
+
+```bash
+aws lambda update-function-configuration \
+  --function-name my-function \
+  --layers \
+    arn:aws:lambda:{region}:418653438961:layer:rotel-extension-{arch}-alpha:{version} \
+    arn:aws:lambda:{region}:901920570463:layer:aws-otel-nodejs-{arch}-ver-1-30-2:1
+```
+
+#### Configure the auto-instrumentation {#configure-instrumentation}
+
+Set the `AWS_LAMBDA_EXEC_WRAPPER` environment variable to enable auto-instrumentation:
+
+**For Node.js:**
+```bash
+AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
+```
+
+**For Python:**
+```bash
+AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
+```
+
+**For Java:**
+```bash
+AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-handler
+```
+
+#### Verify traces in HyperDX {#verify-traces}
+
+After invoking your function:
+
+1. Navigate to the **Traces** view in HyperDX
+2. You should see traces with spans from your Lambda function
+3. Traces will be correlated with logs via `trace_id` and `span_id` attributes
+
+</VerticalStepper>
+
+## Example applications {#examples}
+
+Checkout the example Python app demonstrating the Rotel Lambda Extension:
+
+- **[Python + ClickHouse](https://github.com/streamfold/python-aws-lambda-clickhouse-example)**: Python application with manual OpenTelemetry instrumentation, sending traces and logs directly to ClickHouse
+
+## Join the Rotel community
+
+If you have questions about Rotel, please join the [Rotel Discord server](https://rotel.dev) and share your feedback or questions. Check out the [Rotel Lambda Extension](https://github.com/streamfold/rotel-lambda-extension) to contribute any improvements.
+
+## Additional resources {#resources}
+
+- **[Rotel Lambda Extension](https://github.com/streamfold/rotel-lambda-extension)**: Source code and detailed documentation
+- **[Rotel Core](https://github.com/streamfold/rotel)**: The lightweight OTEL data plane powering the extension

docs/use-cases/observability/clickstack/integration-examples/index.md

@@ -25,4 +25,5 @@ Several of these integration guides use ClickStack's built-in OpenTelemetry Coll
 | [PostgreSQL Logs](/use-cases/observability/clickstack/integrations/postgresql-logs) | Quick start guide for PostgreSQL Logs |
 | [PostgreSQL Metrics](/use-cases/observability/clickstack/integrations/postgresql-metrics) | Quick start guide for PostgreSQL Metrics |
 | [Redis Logs](/use-cases/observability/clickstack/integrations/redis) | Quick start guide for Redis Logs |
-| [Redis Metrics](/use-cases/observability/clickstack/integrations/redis-metrics) | Quick start guide for Redis Metrics |
+| [Redis Metrics](/use-cases/observability/clickstack/integrations/redis-metrics) | Quick start guide for Redis Metrics |
+| [AWS Lambda Logs](/use-cases/observability/clickstack/integrations/aws-lambda) | Quick start guide for AWS Lambda Logs |