Skip to content

Commit e055d3d

Browse files
jfan9Jacky FanKihoon KwonEC2 Default Usersonal-aws
authored
Issue#820 - add new CDK example demonstrating AWS CodePipeline based CICD setup (#1054)
* initial commit * remove cd command * added github project path * rename stacks * rename stage app stacks * added vpc stack and cross stack reference the vpc in ecs stack * added cdk.context.json to .gitignore * added synthCodeBuildDefaults to fix build stage permission issue * added architecture diagram * added architecture diagram to README>md * add lambdaApiStack * added import PolicyStatement from cdk lib * Dockerfile and App code * lesser requirement * added datastore stack * Update pipeline-stack.ts * Update pipeline-stack.ts * updated import resource format * added waves to eu-west-1 & us-west-2 * fix container missing waitress module error * Update README.md * Added sample stack for event sources mapped asynchronous lambda functions It create a SNS topic, a SQS queue, and a EventBridge rule that trigger asynchronous lambda functions. The source of event is the Amazon ECS events that are published into the default EventBridge event bus. We use this event to push messages to the SNS topic and to the SQS queue. * added @types/aws-lambda: ^8.10.140 * updated a new version of diagram * Update README.md * relocate async lambda function code to assets * explicit add account id in the pipeline * moved variables to app.ts * upgrade cdk app version to v2.162.1 * Update README.md Update diagram image * Delete typescript/aws-codepipeline-ecs-lambda/test/aws-codepipeline-ecs-lambda.test.ts No test implemented * Update package.json No test implemented * Delete typescript/aws-codepipeline-ecs-lambda/jest.config.js No test implemented * Update Dockerfile update python 3.12 * fix: add DNA for cross account and lookup --------- Co-authored-by: Jacky Fan <[email protected]> Co-authored-by: Kihoon Kwon <[email protected]> Co-authored-by: EC2 Default User <[email protected]> Co-authored-by: Sonal <[email protected]> Co-authored-by: Jacky Fan <[email protected]> Co-authored-by: ruchirshetye-aws <[email protected]> Co-authored-by: Junseong Jang <[email protected]> Co-authored-by: Michael Kaiser <[email protected]> Co-authored-by: Michael Kaiser <[email protected]> Co-authored-by: Michael Kaiser <[email protected]>
1 parent a25bb9e commit e055d3d

26 files changed

+689
-0
lines changed

‎typescript/aws-codepipeline-ecs-lambda/.gitignore‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
*.js
2+
!jest.config.js
3+
*.d.ts
4+
node_modules
5+
6+
# CDK asset staging directory
7+
.cdk.staging
8+
cdk.out
9+
cdk.context.json

‎typescript/aws-codepipeline-ecs-lambda/.npmignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out

‎typescript/aws-codepipeline-ecs-lambda/DO_NOT_AUTOTEST‎

Whitespace-only changes.

‎typescript/aws-codepipeline-ecs-lambda/README.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# AWS Codepipeline CI/CD Solution for ECS Fargate and Lambda
2+
3+
## Architect Design:
4+
![](./static_images/Architecture_diagram.png)
5+
6+
## Overview
7+
8+
This CDK package provides a production-grade template for setting up AWS resources to enable smooth migration from monolithic EC2-based architectures to cloud-native solutions on AWS. It's designed for startups looking to scale their infrastructure efficiently.
9+
10+
Key features:
11+
- CICD pipeline using AWS CodePipeline
12+
- Lambda functions (async triggered and REST endpoints behind API Gateway)
13+
- ECS Fargate based service with automatic deployment
14+
- Integration with private GitHub repositories
15+
- Reference to CDK created VPCs
16+
- Creates RDS Instances with credentials managed by AWS Secrets Manager
17+
- Event-driven architecture using SQS, SNS, and EventBridge
18+
19+
## Getting Started
20+
21+
### Prerequisites
22+
23+
- Create a Private Github Repository with source code inside 'aws-codepipeline-ecs-lambda' directory.
24+
- Create a connection to GitHub or GitHub Enterprise Cloud, see [Create a connection to GitHub](https://docs.aws.amazon.com/dtconsole/latest/userguide/connections-create-github.html).
25+
- Modify the `connectionArn` in `pipeline-stack.ts` file.
26+
- Modify `githubOrg`, `githubRepo`, `githubBranch` with your private repository details.
27+
28+
## Project Structure
29+
30+
The project is organized into six main stacks:
31+
32+
1. `VpcStack`: Network infrastructure resources
33+
2. `DataStoresStack`: Database resources
34+
3. `PubSubStack`: Event-based infrastructure (SQS, SNS, EventBridge)
35+
4. `AsyncLambdasStack`: Asynchronously triggered Lambda functions
36+
5. `LambdaApisStack`: Lambda functions as REST endpoints behind API Gateway
37+
6. `EcsFargateStack`: ECS Fargate Service, Cluster, Tasks, and Containers
38+
39+
You can easily customize the infrastructure by modifying or removing specific stacks in the `lib/pipeline-stage.ts` file.
40+
41+
## Solution overview
42+
43+
The CDK application is structured as follows:
44+
45+
`lib/pipeline-stack.ts` contains the definition of the CI/CD pipeline. The main component here is the CodePipeline construct that creates the pipeline for us
46+
47+
`lib/stage-app.ts` contains definitions of all the six stacks which the pipeline will deploy.
48+
49+
`lib/stage-app-vpc-stack.ts` creates new vpc resource along with subnets, nat gateways and remaining networking infrastructure.
50+
51+
`lib/stage-app-datastore-stack.ts` creates a Aurora Serverless V2 Cluster along with KMS key to encrypt the database.
52+
53+
`lib/stage-app-ecs-fargate-stack.ts` builds the `Dockerfile` and creates ecs fargate service along with loadbalancer.
54+
55+
`lib/stage-app-lambda-api-stack.ts` creates lambda functions as REST endpoints behind API Gateway resource.
56+
57+
`lib/PubSubStack.ts` creates sns, eventbridge and lambda function.
58+
59+
---

‎typescript/aws-codepipeline-ecs-lambda/assets/lambda-functions/events_handler.ts‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
import { EventBridgeEvent, Context } from 'aws-lambda';
2+
3+
export const handler = async (event: EventBridgeEvent<string, any>) => {
4+
console.log('LogEvent');
5+
console.log('Received event:', JSON.stringify(event, null, 2));
6+
return 'Finished';
7+
};

‎typescript/aws-codepipeline-ecs-lambda/assets/lambda-functions/queue_message_handler.ts‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
import { SQSEvent, SQSBatchResponse, SQSBatchItemFailure } from 'aws-lambda';
2+
3+
export const handler = (event: SQSEvent): SQSBatchResponse | void => {
4+
if (event) {
5+
const batchItemFailures: SQSBatchItemFailure[] = [];
6+
event?.Records.forEach(record => {
7+
try {
8+
// process record
9+
} catch (err) {
10+
batchItemFailures.push({
11+
itemIdentifier: record.messageId
12+
});
13+
}
14+
});
15+
16+
return { batchItemFailures };
17+
}
18+
};

‎typescript/aws-codepipeline-ecs-lambda/assets/lambda-functions/topic_message_handler.ts‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
import { SNSEvent, SNSEventRecord } from 'aws-lambda';
2+
3+
export const handler = async (event: SNSEvent) => {
4+
for (const record of event.Records) {
5+
await processMessageAsync(record);
6+
}
7+
console.info("done");
8+
};
9+
10+
async function processMessageAsync(record: SNSEventRecord) {
11+
try {
12+
const message = JSON.stringify(record.Sns.Message);
13+
console.log(`Processed message ${message}`);
14+
await Promise.resolve(1); //Placeholder for actual async work
15+
} catch (err) {
16+
console.error("An error occurred");
17+
throw err;
18+
}
19+
}

‎typescript/aws-codepipeline-ecs-lambda/cdk.json‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts lib/app.ts",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"**/*.d.ts",
11+
"**/*.js",
12+
"tsconfig.json",
13+
"package*.json",
14+
"yarn.lock",
15+
"node_modules",
16+
"test"
17+
]
18+
},
19+
"context": {
20+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
21+
"@aws-cdk/core:checkSecretUsage": true,
22+
"@aws-cdk/core:target-partitions": [
23+
"aws",
24+
"aws-cn"
25+
],
26+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
27+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
28+
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
29+
"@aws-cdk/aws-iam:minimizePolicies": true,
30+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
31+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
32+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
33+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
34+
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
35+
"@aws-cdk/core:enablePartitionLiterals": true,
36+
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
37+
"@aws-cdk/aws-iam:standardizedServicePrincipals": true,
38+
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
39+
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
40+
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
41+
"@aws-cdk/aws-route53-patters:useCertificate": true,
42+
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
43+
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
44+
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
45+
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
46+
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
47+
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
48+
"@aws-cdk/aws-redshift:columnId": true,
49+
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
50+
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
51+
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
52+
"@aws-cdk/aws-kms:aliasNameRef": true,
53+
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
54+
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
55+
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
56+
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
57+
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
58+
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
59+
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
60+
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
61+
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
62+
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
63+
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
64+
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
65+
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
66+
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
67+
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
68+
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
69+
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
70+
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false
71+
}
72+
}

‎typescript/aws-codepipeline-ecs-lambda/lib/app.ts‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#!/usr/bin/env node
2+
import 'source-map-support/register';
3+
import * as cdk from 'aws-cdk-lib';
4+
import { pipelineStack } from './pipeline-stack';
5+
6+
const app = new cdk.App();
7+
const env = { account: process.env.CDK_DEFAULT_ACCOUNT, region: process.env.CDK_DEFAULT_REGION }
8+
const pipelineAccountId = process.env.PIPELINE_ACCOUNT_ID || "111111111111"; // replace with your pipeline account id
9+
const pipelineRegion = process.env.PIPELINE_REGION || "us-east-1"; // replace with your pipeline region
10+
const githubOrg = process.env.GITHUB_ORG || "aws-6w8hnx"; // replace with your GitHub Org
11+
const githubRepo = process.env.GITHUB_REPO || "aws-codepipeline-ecs-lambda"; // replace with your GitHub Repo
12+
const githubBranch = process.env.GITHUB_BRANCH || "main"; // replace with your GitHub repo branch
13+
const devEnv = process.env.DEV_ENV || "dev"; // replace with your environment
14+
const devAccountId = process.env.DEV_ACCOUNT_ID || "222222222222"; // replace with your dev account id
15+
const primaryRegion = process.env.PRIMARY_REGION || "us-west-2"; // replace with your primary region
16+
const secondaryRegion = process.env.SECONDARY_REGION || "eu-west-1"; // replace with your secondary region
17+
18+
19+
const pipeline_stack = new pipelineStack(app, 'aws-codepipeline-stack', {
20+
env,
21+
pipelineAccountId,
22+
pipelineRegion,
23+
githubOrg,
24+
githubRepo,
25+
githubBranch,
26+
devEnv,
27+
devAccountId,
28+
primaryRegion,
29+
secondaryRegion,
30+
});
31+
cdk.Tags.of(pipeline_stack).add('managedBy', 'cdk');
32+
cdk.Tags.of(pipeline_stack).add('environment', 'dev');
33+
34+
app.synth();

‎typescript/aws-codepipeline-ecs-lambda/lib/pipeline-stack.ts‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
import * as cdk from 'aws-cdk-lib';
2+
import { Construct } from 'constructs';
3+
import { CodePipeline, CodePipelineSource, ManualApprovalStep, ShellStep, Wave } from 'aws-cdk-lib/pipelines';
4+
import { pipelineAppStage } from './stage-app';
5+
import { PolicyStatement } from 'aws-cdk-lib/aws-iam';
6+
7+
export interface pipelineProps extends cdk.StackProps {
8+
readonly pipelineAccountId: string;
9+
readonly pipelineRegion: string;
10+
readonly githubOrg: string;
11+
readonly githubRepo: string;
12+
readonly githubBranch: string;
13+
readonly devEnv: string;
14+
readonly devAccountId: string;
15+
readonly primaryRegion: string;
16+
readonly secondaryRegion: string;
17+
}
18+
19+
export class pipelineStack extends cdk.Stack {
20+
constructor(scope: Construct, id: string, props: pipelineProps) {
21+
super(scope, id, props);
22+
23+
const pipeline = new CodePipeline(this, 'pipeline', {
24+
selfMutation: true,
25+
crossAccountKeys: true,
26+
reuseCrossRegionSupportStacks: true,
27+
synth: new ShellStep('Synth', {
28+
input: CodePipelineSource.connection(`${props.githubOrg}/${props.githubRepo}`, `${props.githubBranch}`,{
29+
// You need to replace the below code connection arn:
30+
connectionArn: `arn:aws:codestar-connections:${props.pipelineRegion}:${props.pipelineAccountId}:connection/0ce75950-a29b-4ee4-a9d3-b0bad3b2c0a6`
31+
}),
32+
commands: [
33+
'npm ci',
34+
'npm run build',
35+
'npx cdk synth'
36+
]
37+
}),
38+
synthCodeBuildDefaults: {
39+
rolePolicy: [
40+
new PolicyStatement({
41+
resources: [ '*' ],
42+
actions: [ 'ec2:DescribeAvailabilityZones' ],
43+
}),
44+
]}
45+
});
46+
47+
const devStage = pipeline.addStage(new pipelineAppStage(this, `${props.devEnv}`, {
48+
env: { account: `${props.pipelineAccountId}`, region: `${props.pipelineRegion}`}
49+
}));
50+
devStage.addPost(new ManualApprovalStep('approval'));
51+
52+
// add waves:
53+
const devWave = pipeline.addWave(`${props.devEnv}Wave`);
54+
55+
devWave.addStage(new pipelineAppStage(this, `${props.devEnv}-${props.primaryRegion}`, {
56+
env: { account: `${props.devAccountId}`, region: `${props.primaryRegion}`}
57+
}));
58+
59+
devWave.addStage(new pipelineAppStage(this, `${props.devEnv}-${props.secondaryRegion}`, {
60+
env: { account: `${props.devAccountId}`, region: `${props.secondaryRegion}`}
61+
}));
62+
}
63+
}

0 commit comments

Comments
 (0)