|
1 | 1 | # Lambda Functions |
2 | 2 |
|
3 | | -This directory contains AWS Lambda functions for Kinesis Firehose data transformation. |
| 3 | +Firehose transformation Lambda functions that enrich CloudWatch data with AWS resource tags and properties before delivery to Better Stack. |
| 4 | + |
| 5 | +See the [main README](../README.md) for an overview or the [CloudFormation deployment guide](../cloudformation/full/README.md) for deployment instructions. |
4 | 6 |
|
5 | 7 | ## Functions |
6 | 8 |
|
7 | | -| Function | Description | |
8 | | -|----------|-------------| |
9 | | -| `firehose_logs_tag_enrichment.rb` | Enriches CloudWatch Logs with AWS resource tags before delivery | |
10 | | -| `firehose_metrics_tag_enrichment.rb` | Enriches CloudWatch Metrics with AWS resource tags before delivery | |
| 9 | +### firehose_metrics_tag_enrichment.rb |
11 | 10 |
|
12 | | -## Deployment |
| 11 | +Enriches CloudWatch Metric Stream data with resource tags and properties. |
| 12 | + |
| 13 | +**Input:** Base64-encoded NDJSON (newline-delimited JSON) from Firehose |
| 14 | +**Output:** Enriched NDJSON with `tags` and `properties` fields added |
| 15 | + |
| 16 | +**Supported Namespaces:** |
| 17 | +- `AWS/EC2` - Instance tags + properties (type, family, size, architecture, AZ, lifecycle) |
| 18 | +- `AWS/EBS` - Volume tags + properties (type, size, IOPS, throughput) |
| 19 | +- `AWS/RDS` - Database tags + properties (class, engine, version, storage type) |
| 20 | +- `AWS/Lambda` - Function tags + properties (runtime, memory, timeout, architecture) |
| 21 | +- `AWS/DynamoDB` - Table tags |
| 22 | +- `AWS/SQS` - Queue tags |
| 23 | +- `AWS/SNS` - Topic tags |
| 24 | +- `AWS/S3` - Bucket tags |
| 25 | +- `AWS/ELB`, `AWS/ApplicationELB`, `AWS/NetworkELB` - Load balancer tags |
| 26 | + |
| 27 | +### firehose_logs_tag_enrichment.rb |
| 28 | + |
| 29 | +Enriches CloudWatch Logs with resource tags extracted from log group/stream names. |
| 30 | + |
| 31 | +**Input:** Base64-encoded gzip-compressed CloudWatch Logs subscription data |
| 32 | +**Output:** Enriched logs with `tags`, `resource_name`, `environment`, and `team` fields |
13 | 33 |
|
14 | | -These Lambda functions are automatically provisioned by the CloudFormation stack when `EnableTagEnrichment` is set to `true`. The stack handles packaging, deployment, and IAM permissions required for the Resource Groups Tagging API integration. |
| 34 | +**Supported Log Patterns:** |
| 35 | +- `/aws/lambda/{function-name}` - Lambda function tags |
| 36 | +- `/aws/rds/instance/{db-instance}/{type}` - RDS instance tags |
| 37 | +- `RDSOSMetrics` - RDS Enhanced Monitoring (extracts `instanceID` from message body) |
| 38 | +- `/ecs/{cluster}/...` - ECS cluster tags |
| 39 | +- `/aws/api-gateway/{api-id}` - API Gateway tags |
| 40 | +- Log streams containing `i-xxxxxxxxx` - EC2 instance tags |
15 | 41 |
|
16 | | -Lambda packages are hosted in regional S3 buckets following the pattern: |
| 42 | +## Configuration |
| 43 | + |
| 44 | +Environment variables (set via CloudFormation): |
| 45 | + |
| 46 | +| Variable | Default | Description | |
| 47 | +|----------|---------|-------------| |
| 48 | +| `CACHE_TTL_MINUTES` | `10` | How long to cache tags/properties before refreshing | |
| 49 | +| `ACCOUNT_ID` | (required) | AWS account ID for ARN construction | |
| 50 | +| `AWS_REGION` | `us-east-1` | AWS region (auto-set by Lambda runtime) | |
| 51 | +| `DEBUG` | `false` | Enable verbose logging | |
| 52 | + |
| 53 | +## Architecture |
17 | 54 |
|
18 | 55 | ``` |
19 | | -s3://better-stack-lambda-${AWS::Region}/<function>.zip |
| 56 | +CloudWatch Metrics ─┐ |
| 57 | + ├─> Firehose ─> Lambda (enrichment) ─> Firehose -> Better Stack |
| 58 | +CloudWatch Logs ────┘ |
20 | 59 | ``` |
21 | 60 |
|
22 | | -For example, in `us-east-1`: |
23 | | -- `s3://better-stack-lambda-us-east-1/firehose_metrics_tag_enrichment.zip` |
24 | | -- `s3://better-stack-lambda-us-east-1/firehose_logs_tag_enrichment.zip` |
| 61 | +Both functions: |
| 62 | +1. Receive batched records from Kinesis Firehose |
| 63 | +2. Extract resource identifiers (ARNs) from the data |
| 64 | +3. Batch-fetch tags via the Resource Groups Tagging API (max 100 ARNs per call) |
| 65 | +4. Cache tags in-memory to minimize API calls |
| 66 | +5. Return enriched records to Firehose for delivery |
| 67 | + |
| 68 | +## Local Development |
| 69 | + |
| 70 | +### Prerequisites |
| 71 | + |
| 72 | +```bash |
| 73 | +bundle install |
| 74 | +``` |
| 75 | + |
| 76 | +### Running Tests |
| 77 | + |
| 78 | +```bash |
| 79 | +bundle exec rspec spec/lambda/ |
| 80 | +``` |
| 81 | + |
| 82 | +Or run specific tests: |
| 83 | + |
| 84 | +```bash |
| 85 | +bundle exec rspec spec/lambda/firehose_metrics_tag_enrichment_spec.rb |
| 86 | +bundle exec rspec spec/lambda/firehose_logs_tag_enrichment_spec.rb |
| 87 | +``` |
| 88 | + |
| 89 | +### Dependencies |
| 90 | + |
| 91 | +- `aws-sdk-resourcegroupstaggingapi` - Tag lookups |
| 92 | +- `aws-sdk-ec2` - EC2/EBS property lookups |
| 93 | +- `aws-sdk-rds` - RDS property lookups |
| 94 | +- `aws-sdk-lambda` - Lambda property lookups |
| 95 | + |
| 96 | +## Deployment |
| 97 | + |
| 98 | +Lambda code is deployed via CloudFormation from regional S3 buckets (`better-stack-lambda-{region}`), referenced in the CloudFormation stack. See the [CloudFormation README](../cloudformation/full/README.md) for deployment commands. |
0 commit comments