Skip to content
Merged
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions typescript/amazon-mq-rabbitmq-lambda/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
*.js
!jest.config.js
*.d.ts
node_modules

# CDK asset staging directory
.cdk.staging
cdk.out
6 changes: 6 additions & 0 deletions typescript/amazon-mq-rabbitmq-lambda/.npmignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
*.ts
!*.d.ts

# CDK asset staging directory
.cdk.staging
cdk.out
90 changes: 90 additions & 0 deletions typescript/amazon-mq-rabbitmq-lambda/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
# Amazon MQ RabbitMQ Integration with Lambda, Secrets Manager, and CloudWatch Logs
<!--BEGIN STABILITY BANNER-->
---

![Stability: Developer Preview](https://img.shields.io/badge/stability-Developer--Preview-important.svg?style=for-the-badge)

> **This is an experimental example. It may not build out of the box**
>
> This example is built on Construct Libraries marked "Developer Preview" and may not be updated for latest breaking changes.
>
> It may additionally requires infrastructure prerequisites that must be created before successful build.
>
> If build is unsuccessful, please create an [issue](https://github.com/aws-samples/aws-cdk-examples/issues/new) so that we may debug the problem
---
<!--END STABILITY BANNER-->
This example demonstrates how to set up an Amazon MQ RabbitMQ cluster, integrate it with AWS Lambda for message processing,
use AWS Secrets Manager for storing sensitive information, and configure CloudWatch Logs for monitoring.

## Deploy

To deploy this app, you need to be in this example's root folder.

Run `cdk deploy`. This will deploy / redeploy your Stack to your AWS Account.

After the deployment you will see the RabbitMQ Broker's endpoints, which will be similar to the following:

```
AmazonMqRabbitmqLambdaStack.AmqpEndpointPort = 5671
AmazonMqRabbitmqLambdaStack.AmqpEndpointUrl = amqps://<broker-id>.mq.us-west-2.amazonaws.com:5671
AmazonMqRabbitmqLambdaStack.WebConsolePort = 443
AmazonMqRabbitmqLambdaStack.WebConsoleUrl = https://<broker-id>.mq.us-west-2.amazonaws.com
```

This will install the necessary CDK, then this example's dependencies, and then build your TypeScript files and your CloudFormation template.

## Testing with [producer.py](producer.py)
> **Important:**
> Update the `username`, `password`, and `broker_endpoint` fields in the [producer.py](producer.py) script as indicated by the comments in the code.
>
> - The `username` and `password` can be retrieved from the secret stored in **AWS Secrets Manager**. You can find the secret created during deployment by searching for the secret name in the AWS Management Console.
> - The `broker_endpoint` should be set to the broker's endpoint shown in the terminal after the cdk deploy command finishes. Specifically, use the URL format: <broker-id>.mq.us-west-2.amazonaws.com

You can test the example by running the [producer.py](producer.py) script provided in the repository.
This script connects to the RabbitMQ broker using the provided credentials and publishes three test messages to the `testQueue`.

After running the producer.py script, you can check the CloudWatch Log Group for your Lambda function to see the output from [consumer.js](lambda/consumer.js).
The [consumer.js](lambda/consumer.js) file includes an example output that shows the details of the received messages, including their Base64-encoded data.

Additionally, you can check the RabbitMQ management console to see the three messages that were published by producer.py
and consumed by the Lambda function, similar to the picture below.

![Queued Messages](images/queued-messages.png)

## Stack Components

![Component Diagram](images/amazon-mq-rabbitmq-lambda-diagram.png)

This stack includes:

- An Amazon MQ RabbitMQ single instance broker, provided by the [AWS::AmazonMQ L2 Construct Library](https://constructs.dev/packages/@cdklabs/cdk-amazonmq/v/0.0.1?lang=go#rabbitmq-brokers).
- An AWS Lambda function for message processing, located in [consumer.js](lambda/consumer.js).
- AWS Secrets Manager for storing RabbitMQ credentials.
- A CloudWatch Log group for Lambda function logs.

## Testing
```bash
npm run test
```

## Cleanup

To avoid incurring future charges, remember to destroy the resources:

```bash
cdk destroy
```

## Future Enhancements and Exploration Opportunities
![Future Enhancements and Exploration Opportunities](images/Future-Enhancements-and-Exploration-Opportunities.png)

## Further Reading

- [Amazon MQ Documentation](https://docs.aws.amazon.com/amazon-mq/)
- [AWS Lambda Developer Guide](https://docs.aws.amazon.com/lambda/latest/dg/welcome.html)
- [AWS Secrets Manager User Guide](https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html)
- [Amazon CloudWatch Logs User Guide](https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html)

## Reference

- [AWS::AmazonMQ L2 Construct Library](https://constructs.dev/packages/@cdklabs/cdk-amazonmq/v/0.0.1?lang=go#rabbitmq-brokers)
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
#!/usr/bin/env node
import * as cdk from 'aws-cdk-lib';
import { AmazonMqRabbitmqLambdaStack } from '../lib/amazon-mq-rabbitmq-lambda-stack';

const app = new cdk.App();
new AmazonMqRabbitmqLambdaStack(app, 'AmazonMqRabbitmqLambdaStack');
74 changes: 74 additions & 0 deletions typescript/amazon-mq-rabbitmq-lambda/cdk.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
{
"app": "npx ts-node --prefer-ts-exts bin/amazon-mq-rabbitmq-lambda.ts",
"watch": {
"include": [
"**"
],
"exclude": [
"README.md",
"cdk*.json",
"**/*.d.ts",
"**/*.js",
"tsconfig.json",
"package*.json",
"yarn.lock",
"node_modules",
"test"
]
},
"context": {
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
"@aws-cdk/core:checkSecretUsage": true,
"@aws-cdk/core:target-partitions": [
"aws",
"aws-cn"
],
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
"@aws-cdk/aws-iam:minimizePolicies": true,
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
"@aws-cdk/core:enablePartitionLiterals": true,
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
"@aws-cdk/aws-route53-patters:useCertificate": true,
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
"@aws-cdk/aws-redshift:columnId": true,
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
"@aws-cdk/aws-kms:aliasNameRef": true,
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false,
"@aws-cdk/aws-s3:keepNotificationInImportedBucket": false,
"@aws-cdk/aws-ecs:reduceEc2FargateCloudWatchPermissions": true,
"@aws-cdk/aws-ec2:ec2SumTImeoutEnabled": true
}
}
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
8 changes: 8 additions & 0 deletions typescript/amazon-mq-rabbitmq-lambda/jest.config.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
module.exports = {
testEnvironment: 'node',
roots: ['<rootDir>/test'],
testMatch: ['**/*.test.ts'],
transform: {
'^.+\\.tsx?$': 'ts-jest'
}
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
import { Duration, Stack, StackProps, CfnOutput, RemovalPolicy } from 'aws-cdk-lib';
import { Construct } from 'constructs';
import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';
import * as logs from 'aws-cdk-lib/aws-logs';
import { Function, Runtime, Code } from 'aws-cdk-lib/aws-lambda';
import { InstanceClass, InstanceSize, InstanceType } from 'aws-cdk-lib/aws-ec2';
import { RabbitMqBrokerEngineVersion, RabbitMqBrokerInstance, RabbitMqEventSource } from '@cdklabs/cdk-amazonmq';

export class AmazonMqRabbitmqLambdaStack extends Stack {
constructor(scope: Construct, id: string, props?: StackProps) {
super(scope, id, props);

// Define the admin secret in AWS Secrets Manager
const adminSecret = new secretsmanager.Secret(this, 'AdminSecret', {
secretName: 'AdminCredentials',
generateSecretString: {
secretStringTemplate: JSON.stringify({
username: 'admin' // Set a default username for RabbitMQ broker
}),
generateStringKey: 'password', // Auto-generate password
excludePunctuation: true, // Avoid punctuation in password
passwordLength: 12, // Set password length to 12 characters
},
removalPolicy: RemovalPolicy.DESTROY, // Ensure the secret is deleted on stack destroy
});

// Create a RabbitMQ broker instance with specified version and instance type
const broker = new RabbitMqBrokerInstance(this, 'RabbitMqBroker', {
publiclyAccessible: true, // Publicly accessible RabbitMQ broker
version: RabbitMqBrokerEngineVersion.V3_13, // Use RabbitMQ version 3.13
instanceType: InstanceType.of(InstanceClass.T3, InstanceSize.MICRO), // Instance type T3.micro
admin: {
username: adminSecret.secretValueFromJson('username').unsafeUnwrap(), // Use username from Secrets Manager
password: adminSecret.secretValueFromJson('password'), // Use password from Secrets Manager
},
autoMinorVersionUpgrade: true, // Enable auto minor version upgrades
});

// Output the AMQP and Web Console endpoints and ports for the RabbitMQ broker
new CfnOutput(this, 'AmqpEndpointUrl', { value: broker.endpoints.amqp.url });
new CfnOutput(this, 'AmqpEndpointPort', { value: broker.endpoints.amqp.port.toString() });
new CfnOutput(this, 'WebConsoleUrl', { value: broker.endpoints.console.url });
new CfnOutput(this, 'WebConsolePort', { value: broker.endpoints.console.port.toString() });

// Create a custom CloudWatch Log Group for the consumer Lambda function
const consumerLambdaLogGroup = new logs.LogGroup(this, 'ConsumerLambdaLogGroup', {
logGroupName: 'customLogGroup', // Custom log group name
removalPolicy: RemovalPolicy.DESTROY, // Ensure it's deleted on stack destroy
});

// Define the consumer Lambda function which will handle messages from RabbitMQ
const consumer_lambda = new Function(this, 'consumer_lambdaFunction', {
runtime: Runtime.NODEJS_20_X, // Use Node.js 20.x runtime for the Lambda function
code: Code.fromAsset('lambda'), // Path to Lambda function code directory
handler: 'consumer.handler', // The entry point (handler) for the Lambda function
memorySize: 128, // Set memory size to 128 MB
timeout: Duration.seconds(30), // Set timeout duration to 30 seconds
logGroup: consumerLambdaLogGroup, // Attach the custom CloudWatch log group
});

consumer_lambda.logGroup.applyRemovalPolicy(RemovalPolicy.DESTROY);

// Add RabbitMQ as an event source for the Lambda function to consume messages from 'testQueue'
consumer_lambda.addEventSource(new RabbitMqEventSource({
broker, // Reference to the RabbitMQ broker instance
credentials: adminSecret, // Use admin credentials from Secrets Manager
queueName: 'testQueue', // Queue name in RabbitMQ from which Lambda will consume messages
}));
}
}
27 changes: 27 additions & 0 deletions typescript/amazon-mq-rabbitmq-lambda/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
{
"name": "amazon-mq-rabbitmq-lambda",
"version": "0.1.0",
"bin": {
"amazon-mq-rabbitmq-lambda": "bin/amazon-mq-rabbitmq-lambda.js"
},
"scripts": {
"build": "tsc",
"watch": "tsc -w",
"test": "jest",
"cdk": "cdk"
},
"devDependencies": {
"@types/jest": "^29.5.12",
"@types/node": "22.5.4",
"aws-cdk": "2.160.0",
"jest": "^29.7.0",
"ts-jest": "^29.2.5",
"ts-node": "^10.9.2",
"typescript": "~5.6.2"
},
"dependencies": {
"@cdklabs/cdk-amazonmq": "^0.0.1",
"aws-cdk-lib": "2.160.0",
"constructs": "^10.0.0"
}
}
52 changes: 52 additions & 0 deletions typescript/amazon-mq-rabbitmq-lambda/producer.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
import pika
import ssl

def publish_messages():
# Replace with your RabbitMQ connection details
username = 'your_rabbitmq_username' # Retrieve your RabbitMQ username from AWS Secrets Manager
password = 'your_rabbitmq_password' # Retrieve your RabbitMQ password from AWS Secrets Manager
broker_endpoint = 'your_broker_id.mq.us-west-2.amazonaws.com' # Replace with your RabbitMQ broker endpoint (no protocol)
port = 5671 # AMQPS port
queue_name = 'testQueue'

# Create credentials
credentials = pika.PlainCredentials(username, password)

# Create SSL context
ssl_context = ssl.create_default_context()
ssl_context.check_hostname = False # Change to True if you want hostname verification
ssl_context.verify_mode = ssl.CERT_NONE # Change to ssl.CERT_REQUIRED if you have proper certificates

# Create connection parameters with SSL
parameters = pika.ConnectionParameters(
host=broker_endpoint,
port=port,
credentials=credentials,
ssl_options=pika.SSLOptions(context=ssl_context) # Use ssl_options
)

# Establish the connection
connection = pika.BlockingConnection(parameters)
channel = connection.channel()

# Declare the queue (it will create if it doesn't exist)
channel.queue_declare(queue=queue_name, durable=True)

# Publish 3 test messages
for i in range(1, 4):
message = f'Hello World {i}'
channel.basic_publish(
exchange='',
routing_key=queue_name,
body=message,
properties=pika.BasicProperties(
delivery_mode=2, # Make message persistent
)
)
print(f'Sent: {message}')

# Clean up
connection.close()

if __name__ == '__main__':
publish_messages()
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
import * as cdk from 'aws-cdk-lib';
import { Template } from 'aws-cdk-lib/assertions';
import * as AmazonMqRabbitmqLambda from '../lib/amazon-mq-rabbitmq-lambda-stack';

test('RabbitMQ Broker and Lambda Function Created', () => {
const app = new cdk.App();
// WHEN
const stack = new AmazonMqRabbitmqLambda.AmazonMqRabbitmqLambdaStack(app, 'MyTestStack');
// THEN
const template = Template.fromStack(stack);

// Check that RabbitMQ Broker instance is created with the correct properties
template.hasResourceProperties('AWS::AmazonMQ::Broker', {
EngineType: 'RABBITMQ', // Corrected to match actual output (uppercase)
PubliclyAccessible: true,
AutoMinorVersionUpgrade: true,
});

// Check that a Lambda function is created
template.hasResourceProperties('AWS::Lambda::Function', {
Runtime: 'nodejs20.x',
MemorySize: 128,
Timeout: 30,
});

// Check that the Secret for admin credentials is created
template.hasResourceProperties('AWS::SecretsManager::Secret', {
Name: 'AdminCredentials'
});

// Verify that the custom CloudWatch Log Group for the Lambda function is created
template.hasResourceProperties('AWS::Logs::LogGroup', {
LogGroupName: 'customLogGroup'
});
});
Loading