Skip to content

Commit 91fbd65

Browse files
edwardchchenedwardchchen
committed
feat(typescript/ddb-stream-lambda-sns) Add an example of dynamodb stream integration with lambda and sns
1 parent 7f2d778 commit 91fbd65

File tree

13 files changed

+510
-0
lines changed

13 files changed

+510
-0
lines changed

‎typescript/ddb-stream-lambda-sns/.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
*.js
2+
!jest.config.js
3+
*.d.ts
4+
node_modules
5+
6+
# CDK asset staging directory
7+
.cdk.staging
8+
cdk.out
9+
.DS_Store

‎typescript/ddb-stream-lambda-sns/.npmignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out

‎typescript/ddb-stream-lambda-sns/README.md‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
DynamoDB Stream Constructs for AWS CDK
2+
---
3+
4+
<!--BEGIN STABILITY BANNER-->
5+
---
6+
7+
![Stability: Developer Preview](https://img.shields.io/badge/stability-Developer--Preview-important.svg?style=for-the-badge)
8+
9+
> **This is an experimental example. It may not build out of the box**
10+
>
11+
> This example is built on Construct Libraries marked "Developer Preview" and may not be updated for latest breaking changes.
12+
>
13+
> It may additionally requires infrastructure prerequisites that must be created before successful build.
14+
>
15+
> If build is unsuccessful, please create an [issue](https://github.com/aws-samples/aws-cdk-examples/issues/new) so that we may debug the problem
16+
---
17+
<!--END STABILITY BANNER-->
18+
19+
## Overview
20+
21+
This repository provides both L2 and L3 constructs example usage for working with DynamoDB streams [AWS Cloud Development Kit (CDK)](https://aws.amazon.com/cdk/) with TypeScript. It showcases the integration of DynamoDB streams with AWS Lambda and Amazon SNS (Simple Notification Service), providing an example of real-time data processing and notification workflows.
22+
23+
This solution demonstrates a use case for real-time notifications: alerting users about low inventory of an item in the system.
24+
25+
## Architecture Diagram
26+
27+
![Architecture Diagram](images/architecture.jpg)
28+
29+
## Features
30+
31+
- L2 (low-level) construct for fine-grained control over DynamoDB streams
32+
- [L3 (high-level)](https://docs.aws.amazon.com/solutions/latest/constructs/aws-dynamodbstreams-lambda.html) construct for simplified, best-practice implementations of DynamoDB streams
33+
- Integration with Lambda functions for stream processing
34+
- Implements an SQS Dead Letter Queue (DLQ) for the Lambda function processing the DynamoDB stream, enhancing reliability and error handling
35+
- Shows how to use Amazon SNS to distribute stream processing results or notifications.
36+
37+
38+
## Build, Deploy and Testing
39+
40+
### Prerequisites
41+
42+
Before you begin, ensure you have met the following requirements:
43+
44+
* You have installed the latest version of [Node.js and npm](https://nodejs.org/en/download/)
45+
* You have installed the [AWS CLI](https://aws.amazon.com/cli/) and configured it with your credentials
46+
* You have installed the [AWS CDK Toolkit](https://docs.aws.amazon.com/cdk/latest/guide/cli.html) globally
47+
* You have an AWS account and have set up your [AWS credentials](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html)
48+
* You have [bootstrapped your AWS account](https://docs.aws.amazon.com/cdk/latest/guide/bootstrapping.html) for CDK
49+
50+
51+
52+
### Build
53+
To build this app, you need to be in this example's root folder. Then run the following:
54+
55+
```bash
56+
npm install
57+
npm run build
58+
```
59+
60+
This will install the dependencies for this example.
61+
62+
### Deploy
63+
64+
Run `cdk deploy`. This will deploy the Stack to your AWS Account.
65+
66+
Post deployment, you should see the table arn, lambda function arn and sns topic arn on the output of your CLI.
67+
68+
## Testing
69+
```bash
70+
npm run test
71+
```
72+
73+
## Usage
74+
75+
### Configuring SNS Notification Subscription
76+
77+
1. After deploying the stack, locate the SNS topic Amazon Resource Name (ARN) from the CLI output.
78+
79+
2. To subscribe an email address to the SNS topic:
80+
81+
```bash
82+
aws sns subscribe --topic-arn <your-sns-topic-arn> --protocol email --notification-endpoint [email protected]
83+
```
84+
Replace <your-sns-topic-arn> with the actual ARN of your SNS topic, and [email protected] with the email address you want to subscribe.
85+
86+
3. Check your email inbox for a confirmation message from AWS. Click the link in the email to confirm your subscription.
87+
88+
### Creating an Item in DynamoDB with id, itemName, and count
89+
To trigger the stream processing and email notification, you need to create an item in your DynamoDB table with the fields id, itemName, and count. You can do this using the AWS CLI or AWS Management Console.
90+
91+
Example item.json provided in this repo:
92+
```bash
93+
{
94+
"id": {
95+
"S": "1"
96+
},
97+
"count": {
98+
"N": "10"
99+
},
100+
"itemName": {
101+
"S": "Coffee Beans"
102+
}
103+
}
104+
```
105+
106+
1. Use the following command to put the item into your DynamoDB table:
107+
108+
```bash
109+
aws dynamodb put-item --table-name <your-table-name> --item file://item.json
110+
```
111+
112+
Replace <your-table-name> with the actual name of your DynamoDB table.
113+
114+
2. Whenever you update the count field of the item to 0, the DynamoDB stream will trigger the Lambda function, which will process the data and send a notification to the subscribed email address via SNS.
115+
116+
117+
118+
## Cleanup
119+
120+
To avoid incurring future charges, please destroy the resources when they are no longer needed:
121+
122+
```bash
123+
cdk destroy
124+
```

‎typescript/ddb-stream-lambda-sns/bin/ddb-stream.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
#!/usr/bin/env node
2+
import 'source-map-support/register';
3+
import * as cdk from 'aws-cdk-lib';
4+
import { DdbStreamStack } from '../lib/ddb-stream-stack';
5+
6+
const app = new cdk.App();
7+
new DdbStreamStack(app, 'DdbStreamStack', {});

‎typescript/ddb-stream-lambda-sns/cdk.json‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/ddb-stream.ts",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"**/*.d.ts",
11+
"**/*.js",
12+
"tsconfig.json",
13+
"package*.json",
14+
"yarn.lock",
15+
"node_modules",
16+
"test"
17+
]
18+
},
19+
"context": {
20+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
21+
"@aws-cdk/core:checkSecretUsage": true,
22+
"@aws-cdk/core:target-partitions": [
23+
"aws",
24+
"aws-cn"
25+
],
26+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
27+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
28+
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
29+
"@aws-cdk/aws-iam:minimizePolicies": true,
30+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
31+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
32+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
33+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
34+
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
35+
"@aws-cdk/core:enablePartitionLiterals": true,
36+
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
37+
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
38+
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
39+
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
40+
"@aws-cdk/aws-route53-patters:useCertificate": true,
41+
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
42+
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
43+
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
44+
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
45+
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
46+
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
47+
"@aws-cdk/aws-redshift:columnId": true,
48+
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
49+
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
50+
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
51+
"@aws-cdk/aws-kms:aliasNameRef": true,
52+
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
53+
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
54+
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
55+
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
56+
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
57+
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
58+
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
59+
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
60+
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
61+
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
62+
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
63+
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
64+
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
65+
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
66+
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
67+
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
68+
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
69+
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false,
70+
"@aws-cdk/aws-s3:keepNotificationInImportedBucket": false
71+
}
72+
}

‎typescript/ddb-stream-lambda-sns/images/architecture.jpg‎

70 KB
Loading

‎typescript/ddb-stream-lambda-sns/item.json‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
{
2+
"id": {
3+
"S": "1"
4+
},
5+
"count": {
6+
"N": "10"
7+
},
8+
"itemName": {
9+
"S": "Coffee Beans"
10+
}
11+
}

‎typescript/ddb-stream-lambda-sns/jest.config.js‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
module.exports = {
2+
testEnvironment: 'node',
3+
roots: ['<rootDir>/test'],
4+
testMatch: ['**/*.test.ts'],
5+
transform: {
6+
'^.+\\.tsx?$': 'ts-jest'
7+
}
8+
};

‎typescript/ddb-stream-lambda-sns/lib/ddb-stream-stack.ts‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
import * as cdk from 'aws-cdk-lib';
2+
import { Construct } from 'constructs';
3+
import * as dynamodb from 'aws-cdk-lib/aws-dynamodb';
4+
import * as lambda from 'aws-cdk-lib/aws-lambda';
5+
import * as lambdaEventSources from 'aws-cdk-lib/aws-lambda-event-sources';
6+
import { DynamoDBStreamsToLambda } from '@aws-solutions-constructs/aws-dynamodbstreams-lambda';
7+
import * as sns from 'aws-cdk-lib/aws-sns';
8+
import * as kms from 'aws-cdk-lib/aws-kms';
9+
import * as sqs from 'aws-cdk-lib/aws-sqs';
10+
11+
export class DdbStreamStack extends cdk.Stack {
12+
constructor(scope: Construct, id: string, props?: cdk.StackProps) {
13+
super(scope, id, props);
14+
const aws_sns_kms_key = kms.Alias.fromAliasName(
15+
this,
16+
"aws-managed-sns-kms-key",
17+
"alias/aws/sns",
18+
)
19+
20+
const snsTopic = new sns.Topic(this, 'ddb-stream-topic', {
21+
topicName: 'ddb-stream-topic',
22+
displayName: 'SNS Topic for DDB streams',
23+
enforceSSL: true,
24+
masterKey: aws_sns_kms_key,
25+
});
26+
27+
//L2 CDK Construct
28+
const deadLetterQueueL2 = new sqs.Queue(this, 'ddb-stream-l2-dlq', {
29+
queueName: 'ddb-stream-l2-dlq',
30+
encryption: sqs.QueueEncryption.KMS_MANAGED,
31+
retentionPeriod: cdk.Duration.days(4), // Adjust retention period as needed
32+
});
33+
34+
const itemL2Table = new dynamodb.Table(this, 'itemL2Table', {
35+
partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
36+
stream: dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
37+
billingMode: dynamodb.BillingMode.PAY_PER_REQUEST,
38+
encryption: dynamodb.TableEncryption.AWS_MANAGED,
39+
//If you wish to retain the table after running cdk destroy, comment out the line below
40+
removalPolicy: cdk.RemovalPolicy.DESTROY
41+
});
42+
43+
const itemL2TableLambdaFunction = new lambda.Function(this, 'itemL2TableLambdaFunction', {
44+
runtime: lambda.Runtime.NODEJS_20_X,
45+
handler: 'index.handler',
46+
tracing: lambda.Tracing.ACTIVE,
47+
code: lambda.Code.fromAsset('resources/lambda'),
48+
environment: {
49+
SNS_TOPIC_ARN: snsTopic.topicArn,
50+
AWS_NODEJS_CONNECTION_REUSE_ENABLED: '1'
51+
},
52+
});
53+
itemL2TableLambdaFunction.addEventSource(new lambdaEventSources.DynamoEventSource(itemL2Table, {
54+
startingPosition: lambda.StartingPosition.TRIM_HORIZON,
55+
onFailure: new lambdaEventSources.SqsDlq(deadLetterQueueL2),
56+
bisectBatchOnError: true,
57+
maxRecordAge: cdk.Duration.hours(24),
58+
retryAttempts: 500,
59+
}));
60+
61+
deadLetterQueueL2.grantSendMessages(itemL2TableLambdaFunction);
62+
63+
itemL2Table.grantStreamRead(itemL2TableLambdaFunction);
64+
65+
//L3 CDK Construct
66+
const itemL3Table = new DynamoDBStreamsToLambda(this, 'itemL3Table', {
67+
dynamoTableProps: {
68+
partitionKey: { name: 'id', type: dynamodb.AttributeType.STRING },
69+
stream: dynamodb.StreamViewType.NEW_AND_OLD_IMAGES,
70+
//If you wish to retain the table after running cdk destroy, comment out the line below
71+
removalPolicy: cdk.RemovalPolicy.DESTROY
72+
},
73+
lambdaFunctionProps: {
74+
code: lambda.Code.fromAsset('resources/lambda'),
75+
runtime: lambda.Runtime.NODEJS_20_X,
76+
handler: 'index.handler',
77+
environment: {
78+
SNS_TOPIC_ARN: snsTopic.topicArn,
79+
},
80+
},
81+
});
82+
83+
snsTopic.grantPublish(itemL2TableLambdaFunction);
84+
snsTopic.grantPublish(itemL3Table.lambdaFunction);
85+
86+
new cdk.CfnOutput(this, 'itemL2TableLambdaFunctionArn', { value: itemL2TableLambdaFunction.functionArn });
87+
new cdk.CfnOutput(this, 'itemL3TableLambdaFunctionArn', { value: itemL3Table.lambdaFunction.functionArn });
88+
new cdk.CfnOutput(this, 'l3TableArn', { value: itemL3Table.dynamoTableInterface.tableArn });
89+
new cdk.CfnOutput(this, 'l2TableArn', { value: itemL2Table.tableArn });
90+
new cdk.CfnOutput(this, 'topicArn', { value: snsTopic.topicArn });
91+
new cdk.CfnOutput(this, 'l2DLQArn', { value: deadLetterQueueL2.queueArn })
92+
}
93+
}

‎typescript/ddb-stream-lambda-sns/package.json‎

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
{
2+
"name": "ddb-stream-lambda-sns",
3+
"version": "0.1.0",
4+
"bin": {
5+
"ddb-stream": "bin/ddb-stream.js"
6+
},
7+
"scripts": {
8+
"build": "tsc",
9+
"watch": "tsc -w",
10+
"test": "jest",
11+
"cdk": "cdk"
12+
},
13+
"devDependencies": {
14+
"@types/jest": "^29.5.14",
15+
"@types/node": "22.9.0",
16+
"aws-cdk": "2.166.0",
17+
"jest": "^29.7.0",
18+
"ts-jest": "^29.2.5",
19+
"ts-node": "^10.9.2",
20+
"typescript": "~5.6.3"
21+
},
22+
"dependencies": {
23+
"@aws-solutions-constructs/aws-dynamodbstreams-lambda": "^2.74.0",
24+
"aws-cdk-lib": "2.167.0",
25+
"constructs": "^10.4.2",
26+
"source-map-support": "^0.5.21"
27+
}
28+
}

0 commit comments

Comments
 (0)