Skip to content

Commit 9749099

Browse files
committed
feat(typescript/amazon-mq-rabbitmq-lambda) Add an example of Amazon MQ RabbitMQ Integration with Lambda, Secrets Manager, and CloudWatch Logs
1 parent 39f4dc3 commit 9749099

File tree

12 files changed

+400
-0
lines changed

12 files changed

+400
-0
lines changed
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
*.js
2+
!jest.config.js
3+
*.d.ts
4+
node_modules
5+
6+
# CDK asset staging directory
7+
.cdk.staging
8+
cdk.out
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out
Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
# Amazon MQ RabbitMQ Integration with Lambda, Secrets Manager, and CloudWatch Logs
2+
<!--BEGIN STABILITY BANNER-->
3+
---
4+
![Stability: Experimental](https://img.shields.io/badge/stability-Experimental-important.svg?style=for-the-badge)
5+
6+
> **This is an experimental example. It may not build out of the box**
7+
>
8+
> This examples is built on Construct Libraries marked "Experimental" and may not be updated for latest breaking changes.
9+
>
10+
> It additionally requires infrastructure prerequisites that must be created before successful build.
11+
>
12+
> If build is unsuccessful, please create an [issue](https://github.com/aws-samples/aws-cdk-examples/issues/new) so that we may debug the problem
13+
14+
---
15+
<!--END STABILITY BANNER-->
16+
This example demonstrates how to set up an Amazon MQ RabbitMQ cluster, integrate it with AWS Lambda for message processing,
17+
use AWS Secrets Manager for storing sensitive information, and configure CloudWatch Logs for monitoring.
18+
19+
## Deploy
20+
21+
To deploy this app, you need to be in this example's root folder.
22+
23+
Run `cdk deploy`. This will deploy / redeploy your Stack to your AWS Account.
24+
25+
After the deployment you will see the RabbitMQ Broker's endpoints, which will be similar to the following:
26+
27+
```
28+
AmazonMqRabbitmqLambdaStack.AmqpEndpointPort = 5671
29+
AmazonMqRabbitmqLambdaStack.AmqpEndpointUrl = amqps://<broker-id>.mq.us-west-2.amazonaws.com:5671
30+
AmazonMqRabbitmqLambdaStack.WebConsolePort = 443
31+
AmazonMqRabbitmqLambdaStack.WebConsoleUrl = https://<broker-id>.mq.us-west-2.amazonaws.com
32+
```
33+
34+
This will install the necessary CDK, then this example's dependencies, and then build your TypeScript files and your CloudFormation template.
35+
36+
## Testing with [producer.py](producer.py)
37+
> **Important:**
38+
> Update the `username`, `password`, and `broker_endpoint` fields in the [producer.py](producer.py) script as indicated by the comments in the code.
39+
>
40+
> - The `username` and `password` can be retrieved from the secret stored in **AWS Secrets Manager**. You can find the secret created during deployment by searching for the secret name in the AWS Management Console.
41+
> - The `broker_endpoint` should be set to the broker's endpoint shown in the terminal after the cdk deploy command finishes. Specifically, use the URL format: <broker-id>.mq.us-west-2.amazonaws.com
42+
43+
You can test the example by running the [producer.py](producer.py) script provided in the repository.
44+
This script connects to the RabbitMQ broker using the provided credentials and publishes three test messages to the `testQueue`.
45+
46+
After running the producer.py script, you can check the CloudWatch Log Group for your Lambda function to see the output from [consumer.js](lambda/consumer.js).
47+
The [consumer.js](lambda/consumer.js) file includes an example output that shows the details of the received messages, including their Base64-encoded data.
48+
49+
## Stack Components
50+
51+
![Component Diagram](/images/amazon-mq-rabbitmq-lambda-diagram.png)
52+
53+
This stack includes:
54+
55+
- An Amazon MQ RabbitMQ single instance broker, provided by the [AWS::AmazonMQ L2 Construct Library](https://constructs.dev/packages/@cdklabs/cdk-amazonmq/v/0.0.1?lang=go#rabbitmq-brokers).
56+
- An AWS Lambda function for message processing, located in [consumer.js](lambda/consumer.js).
57+
- AWS Secrets Manager for storing RabbitMQ credentials.
58+
- A CloudWatch Log group for Lambda function logs.
59+
60+
## Testing
61+
```bash
62+
npm run test
63+
```
64+
65+
## Cleanup
66+
67+
To avoid incurring future charges, remember to destroy the resources:
68+
69+
```bash
70+
cdk destroy
71+
```
72+
73+
## Further Reading
74+
75+
- [Amazon MQ Documentation](https://docs.aws.amazon.com/amazon-mq/)
76+
- [AWS Lambda Developer Guide](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html)
77+
- [AWS Secrets Manager User Guide](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html)
78+
- [Amazon CloudWatch Logs User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html)
79+
80+
## Reference
81+
82+
- [AWS::AmazonMQ L2 Construct Library](https://constructs.dev/packages/@cdklabs/cdk-amazonmq/v/0.0.1?lang=go#rabbitmq-brokers)
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/usr/bin/env node
2+
import * as cdk from 'aws-cdk-lib';
3+
import { AmazonMqRabbitmqLambdaStack } from '../lib/amazon-mq-rabbitmq-lambda-stack';
4+
5+
const app = new cdk.App();
6+
new AmazonMqRabbitmqLambdaStack(app, 'AmazonMqRabbitmqLambdaStack');
Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/amazon-mq-rabbitmq-lambda.ts",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"**/*.d.ts",
11+
"**/*.js",
12+
"tsconfig.json",
13+
"package*.json",
14+
"yarn.lock",
15+
"node_modules",
16+
"test"
17+
]
18+
},
19+
"context": {
20+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
21+
"@aws-cdk/core:checkSecretUsage": true,
22+
"@aws-cdk/core:target-partitions": [
23+
"aws",
24+
"aws-cn"
25+
],
26+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
27+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
28+
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
29+
"@aws-cdk/aws-iam:minimizePolicies": true,
30+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
31+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
32+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
33+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
34+
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
35+
"@aws-cdk/core:enablePartitionLiterals": true,
36+
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
37+
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
38+
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
39+
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
40+
"@aws-cdk/aws-route53-patters:useCertificate": true,
41+
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
42+
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
43+
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
44+
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
45+
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
46+
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
47+
"@aws-cdk/aws-redshift:columnId": true,
48+
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
49+
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
50+
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
51+
"@aws-cdk/aws-kms:aliasNameRef": true,
52+
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
53+
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
54+
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
55+
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
56+
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
57+
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
58+
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
59+
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
60+
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
61+
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
62+
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
63+
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
64+
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
65+
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
66+
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
67+
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
68+
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
69+
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false,
70+
"@aws-cdk/aws-s3:keepNotificationInImportedBucket": false,
71+
"@aws-cdk/aws-ecs:reduceEc2FargateCloudWatchPermissions": true,
72+
"@aws-cdk/aws-ec2:ec2SumTImeoutEnabled": true
73+
}
74+
}
96.1 KB
Loading
Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
module.exports = {
2+
testEnvironment: 'node',
3+
roots: ['<rootDir>/test'],
4+
testMatch: ['**/*.test.ts'],
5+
transform: {
6+
'^.+\\.tsx?$': 'ts-jest'
7+
}
8+
};
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
import { Duration, Stack, StackProps, CfnOutput, RemovalPolicy } from 'aws-cdk-lib';
2+
import { Construct } from 'constructs';
3+
import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';
4+
import * as logs from 'aws-cdk-lib/aws-logs';
5+
import { Function, Runtime, Code } from 'aws-cdk-lib/aws-lambda';
6+
import { InstanceClass, InstanceSize, InstanceType } from 'aws-cdk-lib/aws-ec2';
7+
import { RabbitMqBrokerEngineVersion, RabbitMqBrokerInstance, RabbitMqEventSource } from '@cdklabs/cdk-amazonmq';
8+
9+
export class AmazonMqRabbitmqLambdaStack extends Stack {
10+
constructor(scope: Construct, id: string, props?: StackProps) {
11+
super(scope, id, props);
12+
13+
// Define the admin secret in AWS Secrets Manager
14+
const adminSecret = new secretsmanager.Secret(this, 'AdminSecret', {
15+
secretName: 'AdminCredentials',
16+
generateSecretString: {
17+
secretStringTemplate: JSON.stringify({
18+
username: 'admin' // Set a default username for RabbitMQ broker
19+
}),
20+
generateStringKey: 'password', // Auto-generate password
21+
excludePunctuation: true, // Avoid punctuation in password
22+
passwordLength: 12, // Set password length to 12 characters
23+
},
24+
removalPolicy: RemovalPolicy.DESTROY, // Ensure the secret is deleted on stack destroy
25+
});
26+
27+
// Create a RabbitMQ broker instance with specified version and instance type
28+
const broker = new RabbitMqBrokerInstance(this, 'RabbitMqBroker', {
29+
publiclyAccessible: true, // Publicly accessible RabbitMQ broker
30+
version: RabbitMqBrokerEngineVersion.V3_13, // Use RabbitMQ version 3.13
31+
instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.MICRO), // Instance type T3.micro
32+
admin: {
33+
username: adminSecret.secretValueFromJson('username').unsafeUnwrap(), // Use username from Secrets Manager
34+
password: adminSecret.secretValueFromJson('password'), // Use password from Secrets Manager
35+
},
36+
autoMinorVersionUpgrade: true, // Enable auto minor version upgrades
37+
});
38+
39+
// Output the AMQP and Web Console endpoints and ports for the RabbitMQ broker
40+
new CfnOutput(this, 'AmqpEndpointUrl', { value: broker.endpoints.amqp.url });
41+
new CfnOutput(this, 'AmqpEndpointPort', { value: broker.endpoints.amqp.port.toString() });
42+
new CfnOutput(this, 'WebConsoleUrl', { value: broker.endpoints.console.url });
43+
new CfnOutput(this, 'WebConsolePort', { value: broker.endpoints.console.port.toString() });
44+
45+
// Create a custom CloudWatch Log Group for the consumer Lambda function
46+
const consumerLambdaLogGroup = new logs.LogGroup(this, 'ConsumerLambdaLogGroup', {
47+
logGroupName: 'customLogGroup', // Custom log group name
48+
removalPolicy: RemovalPolicy.DESTROY, // Ensure it's deleted on stack destroy
49+
});
50+
51+
// Define the consumer Lambda function which will handle messages from RabbitMQ
52+
const consumer_lambda = new Function(this, 'consumer_lambdaFunction', {
53+
runtime: Runtime.NODEJS_20_X, // Use Node.js 20.x runtime for the Lambda function
54+
code: Code.fromAsset('lambda'), // Path to Lambda function code directory
55+
handler: 'consumer.handler', // The entry point (handler) for the Lambda function
56+
memorySize: 128, // Set memory size to 128 MB
57+
timeout: Duration.seconds(30), // Set timeout duration to 30 seconds
58+
logGroup: consumerLambdaLogGroup, // Attach the custom CloudWatch log group
59+
});
60+
61+
// Prevent the default log group creation
62+
consumer_lambda.logGroup.applyRemovalPolicy(RemovalPolicy.DESTROY);
63+
64+
// Add RabbitMQ as an event source for the Lambda function to consume messages from 'testQueue'
65+
consumer_lambda.addEventSource(new RabbitMqEventSource({
66+
broker, // Reference to the RabbitMQ broker instance
67+
credentials: adminSecret, // Use admin credentials from Secrets Manager
68+
queueName: 'testQueue', // Queue name in RabbitMQ from which Lambda will consume messages
69+
}));
70+
}
71+
}
Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
{
2+
"name": "amazon-mq-rabbitmq-lambda",
3+
"version": "0.1.0",
4+
"bin": {
5+
"amazon-mq-rabbitmq-lambda": "bin/amazon-mq-rabbitmq-lambda.js"
6+
},
7+
"scripts": {
8+
"build": "tsc",
9+
"watch": "tsc -w",
10+
"test": "jest",
11+
"cdk": "cdk"
12+
},
13+
"devDependencies": {
14+
"@types/jest": "^29.5.12",
15+
"@types/node": "22.5.4",
16+
"aws-cdk": "2.160.0",
17+
"jest": "^29.7.0",
18+
"ts-jest": "^29.2.5",
19+
"ts-node": "^10.9.2",
20+
"typescript": "~5.6.2"
21+
},
22+
"dependencies": {
23+
"@cdklabs/cdk-amazonmq": "^0.0.1",
24+
"aws-cdk-lib": "2.160.0",
25+
"constructs": "^10.0.0"
26+
}
27+
}
Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
import pika
2+
import ssl
3+
4+
def publish_messages():
5+
# Replace with your RabbitMQ connection details
6+
username = 'your_rabbitmq_username' # Retrieve your RabbitMQ username from AWS Secrets Manager
7+
password = 'your_rabbitmq_password' # Retrieve your RabbitMQ password from AWS Secrets Manager
8+
broker_endpoint = 'your_broker_id.mq.us-west-2.amazonaws.com' # Replace with your RabbitMQ broker endpoint (no protocol)
9+
port = 5671 # AMQPS port
10+
queue_name = 'testQueue'
11+
12+
# Create credentials
13+
credentials = pika.PlainCredentials(username, password)
14+
15+
# Create SSL context
16+
ssl_context = ssl.create_default_context()
17+
ssl_context.check_hostname = False # Change to True if you want hostname verification
18+
ssl_context.verify_mode = ssl.CERT_NONE # Change to ssl.CERT_REQUIRED if you have proper certificates
19+
20+
# Create connection parameters with SSL
21+
parameters = pika.ConnectionParameters(
22+
host=broker_endpoint,
23+
port=port,
24+
credentials=credentials,
25+
ssl_options=pika.SSLOptions(context=ssl_context) # Use ssl_options
26+
)
27+
28+
# Establish the connection
29+
connection = pika.BlockingConnection(parameters)
30+
channel = connection.channel()
31+
32+
# Declare the queue (it will create if it doesn't exist)
33+
channel.queue_declare(queue=queue_name, durable=True)
34+
35+
# Publish 3 test messages
36+
for i in range(1, 4):
37+
message = f'Hello World {i}'
38+
channel.basic_publish(
39+
exchange='',
40+
routing_key=queue_name,
41+
body=message,
42+
properties=pika.BasicProperties(
43+
delivery_mode=2, # Make message persistent
44+
)
45+
)
46+
print(f'Sent: {message}')
47+
48+
# Clean up
49+
connection.close()
50+
51+
if __name__ == '__main__':
52+
publish_messages()

0 commit comments

Comments
 (0)