Skip to content

Commit f097848

Browse files
architecarchitec
committed
feat:(typescript/airflow-lambda-dynamodb-approval)
1 parent f968615 commit f097848

File tree

12 files changed

+806
-0
lines changed

12 files changed

+806
-0
lines changed

‎typescript/airflow-lambda-dynamodb-approval/.gitignore‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
*.js
2+
!jest.config.js
3+
*.d.ts
4+
node_modules
5+
6+
# CDK asset staging directory
7+
.cdk.staging
8+
cdk.out
9+
10+
# Python
11+
__pycache__/
12+
*.py[cod]
13+
*$py.class
14+
*.so
15+
.Python
16+
build/
17+
develop-eggs/
18+
dist/
19+
downloads/
20+
eggs/
21+
.eggs/
22+
lib/
23+
lib64/
24+
parts/
25+
sdist/
26+
var/
27+
wheels/
28+
*.egg-info/
29+
.installed.cfg
30+
*.egg
31+
MANIFEST
32+
33+
# PyInstaller
34+
*.manifest
35+
*.spec
36+
37+
# Unit test / coverage reports
38+
htmlcov/
39+
.tox/
40+
.coverage
41+
.coverage.*
42+
.cache
43+
nosetests.xml
44+
coverage.xml
45+
*.cover
46+
.hypothesis/
47+
.pytest_cache/
48+
49+
# Virtual environments
50+
.env
51+
.venv
52+
env/
53+
venv/
54+
ENV/
55+
env.bak/
56+
venv.bak/
57+
58+
# IDE
59+
.vscode/
60+
.idea/
61+
*.swp
62+
*.swo
63+
*~
64+
65+
# OS
66+
.DS_Store
67+
Thumbs.db

‎typescript/airflow-lambda-dynamodb-approval/.npmignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out

‎typescript/airflow-lambda-dynamodb-approval/README.md‎

Lines changed: 118 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,118 @@
1+
<!--BEGIN STABILITY BANNER-->
2+
---
3+
4+
![Stability: Stable](https://img.shields.io/badge/stability-Stable-success.svg?style=for-the-badge)
5+
6+
> **This is a stable example.** It should successfully build out of the box
7+
8+
---
9+
<!--END STABILITY BANNER-->
10+
11+
# AWS MWAA (Managed Workflows for Apache Airflow) with CDK
12+
13+
This sample demonstrates how to deploy AWS Managed Workflows for Apache Airflow (MWAA) using AWS CDK with example DAGs for basic workflows, Lambda integration, and human approval processes.
14+
15+
## Overview
16+
17+
This CDK project creates:
18+
- **VPC** with public/private subnets and NAT Gateway
19+
- **S3 bucket** for DAG storage with automatic deployment
20+
- **MWAA environment** with proper IAM roles and security groups
21+
- **DynamoDB table** for human approval workflows
22+
- **Lambda function** for integration testing
23+
- **Three example DAGs** demonstrating different patterns
24+
25+
### DAG Workflows
26+
27+
```mermaid
28+
graph TD
29+
subgraph "example_dag"
30+
A1[Start] --> A2[Print Hello]
31+
A2 --> A3[Print Goodbye]
32+
A3 --> A4[End]
33+
end
34+
35+
subgraph "lambda_invoke_dag"
36+
B1[List Lambda Functions] --> B2[Invoke Demo Lambda]
37+
B2 --> B3[Process Response]
38+
end
39+
40+
subgraph "ddb_approval_dag"
41+
C1[Create Approval Request] --> C5[(DynamoDB Table)]
42+
C1 --> C2[Wait for Human Approval]
43+
C5 --> C2
44+
C2 --> C3[Process Transaction]
45+
C3 --> C4[Complete]
46+
47+
C6[Manual Approval via AWS Console] --> C5
48+
end
49+
```
50+
51+
**Example DAGs included:**
52+
1. **`example_dag.py`** - Basic Airflow workflow with simple tasks
53+
2. **`lambda_invoke_dag.py`** - Demonstrates Lambda function invocation from Airflow
54+
3. **`ddb_approval_dag.py`** - Human approval workflow using DynamoDB sensors
55+
56+
## Build
57+
58+
```bash
59+
npm install
60+
npm run build
61+
```
62+
63+
## Deploy
64+
65+
```bash
66+
cdk deploy
67+
```
68+
69+
**Note the outputs** after deployment:
70+
- **MwaaWebServerUrl** - Access the Airflow web interface
71+
- **S3BucketName** - Where your DAGs are stored
72+
- **ApprovalTableName** - DynamoDB table for approval workflows
73+
74+
## Usage
75+
76+
### Access Airflow Web UI
77+
1. **Login to AWS Console first** - Ensure you're logged into the AWS Console in your browser
78+
2. **Use the deployment output URL** - Copy the `MwaaWebServerUrl` from the deployment outputs
79+
3. **Access Airflow** - Open the URL in the same browser where you're logged into AWS Console
80+
81+
**Note:** MWAA requires AWS authentication even with `PUBLIC_ONLY` access mode. You must be logged into the AWS Console to access the Airflow web interface.
82+
83+
### Test Human Approval Workflow
84+
1. Trigger the `dynamodb_human_approval_pipeline` DAG
85+
2. Go to AWS Console → DynamoDB → Tables → `mwaa-approval-table-{region}`
86+
3. Find your process record and change `approval_status` from `PENDING` to `APPROVED`
87+
4. Watch the workflow complete automatically
88+
89+
### Test Lambda Integration
90+
1. Trigger the `lambda_invoke_example` DAG
91+
2. View logs to see Lambda function listing and invocation results
92+
93+
## Clean up
94+
95+
```bash
96+
cdk destroy
97+
```
98+
99+
All resources are configured with `RemovalPolicy.DESTROY` for easy cleanup.
100+
101+
## Architecture
102+
103+
- **Environment Class**: `mw1.small` (1-2 workers)
104+
- **Airflow Version**: 2.7.2
105+
- **Web Access**: Public (configure private access for production)
106+
- **Logging**: All log types enabled at INFO level
107+
108+
## Security Notes
109+
110+
This sample uses demo-friendly settings. For production:
111+
112+
- **Web Access**: Change from `PUBLIC_ONLY` to `PRIVATE_ONLY` for MWAA environment
113+
- **IAM Permissions**: Replace wildcard (`'*'`) permissions with specific resource ARNs
114+
- **S3 Encryption**: Enable server-side encryption for the DAGs bucket
115+
- **VPC Endpoints**: Add endpoints for S3, DynamoDB, and Lambda to avoid internet traffic
116+
- **Resource Policies**: Use `RETAIN` instead of `DESTROY` for production resources
117+
- **DynamoDB**: Enable point-in-time recovery (currently disabled)
118+
- **Monitoring**: Enable CloudTrail and CloudWatch alarms for security monitoring

‎typescript/airflow-lambda-dynamodb-approval/bin/aws-mwaa-cdk.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/usr/bin/env node
2+
import * as cdk from 'aws-cdk-lib';
3+
import { AwsMwaaCdkStack } from '../lib/aws-mwaa-cdk-stack';
4+
5+
const app = new cdk.App();
6+
new AwsMwaaCdkStack(app, 'AwsMwaaCdkStack');

‎typescript/airflow-lambda-dynamodb-approval/cdk.json‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts bin/aws-mwaa-cdk.ts",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"**/*.d.ts",
11+
"**/*.js",
12+
"tsconfig.json",
13+
"package*.json",
14+
"yarn.lock",
15+
"node_modules",
16+
"test"
17+
]
18+
},
19+
"context": {
20+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
21+
"@aws-cdk/core:checkSecretUsage": true,
22+
"@aws-cdk/core:target-partitions": [
23+
"aws",
24+
"aws-cn"
25+
],
26+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
27+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
28+
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
29+
"@aws-cdk/aws-iam:minimizePolicies": true,
30+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
31+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
32+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
33+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
34+
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
35+
"@aws-cdk/core:enablePartitionLiterals": true,
36+
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
37+
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
38+
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
39+
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
40+
"@aws-cdk/aws-route53-patters:useCertificate": true,
41+
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
42+
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
43+
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
44+
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
45+
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
46+
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
47+
"@aws-cdk/aws-redshift:columnId": true,
48+
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
49+
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
50+
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
51+
"@aws-cdk/aws-kms:aliasNameRef": true,
52+
"@aws-cdk/aws-kms:applyImportedAliasPermissionsToPrincipal": true,
53+
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
54+
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
55+
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
56+
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
57+
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
58+
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
59+
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
60+
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
61+
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
62+
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
63+
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true,
64+
"@aws-cdk/aws-codepipeline:crossAccountKeysDefaultValueToFalse": true,
65+
"@aws-cdk/aws-codepipeline:defaultPipelineTypeToV2": true,
66+
"@aws-cdk/aws-kms:reduceCrossAccountRegionPolicyScope": true,
67+
"@aws-cdk/aws-eks:nodegroupNameAttribute": true,
68+
"@aws-cdk/aws-ec2:ebsDefaultGp3Volume": true,
69+
"@aws-cdk/aws-ecs:removeDefaultDeploymentAlarm": true,
70+
"@aws-cdk/custom-resources:logApiResponseDataPropertyTrueDefault": false,
71+
"@aws-cdk/aws-s3:keepNotificationInImportedBucket": false,
72+
"@aws-cdk/core:explicitStackTags": true,
73+
"@aws-cdk/aws-ecs:enableImdsBlockingDeprecatedFeature": false,
74+
"@aws-cdk/aws-ecs:disableEcsImdsBlocking": true,
75+
"@aws-cdk/aws-ecs:reduceEc2FargateCloudWatchPermissions": true,
76+
"@aws-cdk/aws-dynamodb:resourcePolicyPerReplica": true,
77+
"@aws-cdk/aws-ec2:ec2SumTImeoutEnabled": true,
78+
"@aws-cdk/aws-appsync:appSyncGraphQLAPIScopeLambdaPermission": true,
79+
"@aws-cdk/aws-rds:setCorrectValueForDatabaseInstanceReadReplicaInstanceResourceId": true,
80+
"@aws-cdk/core:cfnIncludeRejectComplexResourceUpdateCreatePolicyIntrinsics": true,
81+
"@aws-cdk/aws-lambda-nodejs:sdkV3ExcludeSmithyPackages": true,
82+
"@aws-cdk/aws-stepfunctions-tasks:fixRunEcsTaskPolicy": true,
83+
"@aws-cdk/aws-ec2:bastionHostUseAmazonLinux2023ByDefault": true,
84+
"@aws-cdk/aws-route53-targets:userPoolDomainNameMethodWithoutCustomResource": true,
85+
"@aws-cdk/aws-elasticloadbalancingV2:albDualstackWithoutPublicIpv4SecurityGroupRulesDefault": true,
86+
"@aws-cdk/aws-iam:oidcRejectUnauthorizedConnections": true,
87+
"@aws-cdk/core:enableAdditionalMetadataCollection": true,
88+
"@aws-cdk/aws-lambda:createNewPoliciesWithAddToRolePolicy": false,
89+
"@aws-cdk/aws-s3:setUniqueReplicationRoleName": true,
90+
"@aws-cdk/aws-events:requireEventBusPolicySid": true,
91+
"@aws-cdk/core:aspectPrioritiesMutating": true,
92+
"@aws-cdk/aws-dynamodb:retainTableReplica": true,
93+
"@aws-cdk/aws-stepfunctions:useDistributedMapResultWriterV2": true,
94+
"@aws-cdk/s3-notifications:addS3TrustKeyPolicyForSnsSubscriptions": true,
95+
"@aws-cdk/aws-ec2:requirePrivateSubnetsForEgressOnlyInternetGateway": true,
96+
"@aws-cdk/aws-s3:publicAccessBlockedByDefault": true,
97+
"@aws-cdk/aws-lambda:useCdkManagedLogGroup": true
98+
}
99+
}

0 commit comments

Comments
 (0)