Skip to content

Commit c135380

Browse files
julianwoodjulianwood
authored
Merge pull request #2379 from jeromevdl/jeromevdl-feature-s3-trigger-fargate-task
New serverless pattern: s3 trigger fargate task
2 parents 952ce38 + 98d1b66 commit c135380

File tree

16 files changed

+592
-0
lines changed

16 files changed

+592
-0
lines changed

‎s3-eventbridge-fargate-cdk/.gitignore‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
*.js
2+
!jest.config.js
3+
*.d.ts
4+
node_modules
5+
6+
# CDK asset staging directory
7+
.cdk.staging
8+
cdk.out
9+
10+
.idea

‎s3-eventbridge-fargate-cdk/.npmignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.ts
2+
!*.d.ts
3+
4+
# CDK asset staging directory
5+
.cdk.staging
6+
cdk.out

‎s3-eventbridge-fargate-cdk/README.md‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
# Amazon S3 to AWS Fargate
2+
3+
This pattern demonstrates how to invoke an AWS Fargate task when an object is uploaded to Amazon S3.
4+
This pattern is commonly implemented with an AWS Lambda function, but this is not always possible:
5+
- Processing > 15 min
6+
- Docker image > 10G
7+
- GPU required
8+
9+
When you need more (more time, more memory, more power), but still want to use serverless service, you can look at Fargate.
10+
11+
Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example.
12+
13+
## Requirements
14+
15+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
16+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
17+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
18+
* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) (AWS CDK) installed
19+
20+
## Deployment Instructions
21+
22+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
23+
```
24+
git clone https://github.com/aws-samples/serverless-patterns
25+
```
26+
2. Change directory to the pattern directory:
27+
```
28+
cd serverless-patterns/s3-eventbridge-fargate-cdk
29+
```
30+
3. From the command line, use AWS SAM to deploy the AWS resources for the pattern as specified in the template.yml file:
31+
```
32+
npm install && npx cdk deploy
33+
```
34+
4. During the prompts:
35+
* Allow CDK to create resources and IAM roles with the required permissions.
36+
37+
5. Note the outputs from the CDK deployment process. These contain the resource names and/or ARNs which are used for testing.
38+
39+
## How it works
40+
41+
![Architecture](doc/archi.png)
42+
43+
1. When a file is uploaded to S3, in the specified bucket (eventually with a specific prefix), an event is triggered
44+
2. EventBridge catch this event and triggers the ECS Fargate task
45+
3. Fargate bootstrap a container to run the task
46+
47+
## Testing
48+
49+
1. Once CDK / CloudFormation deployed the stack, upload a file to S3 using the following command.
50+
(replace `your-bucket-name` with the CDK output `S3TriggerFargateTaskStack.IngestionBucket`) :
51+
52+
```shell
53+
aws s3api put-object --bucket your-bucket-name --key file-name --body file-name
54+
```
55+
56+
2. The S3 file upload will trigger the EventBridge Rule which will run the ECS task. The ECS task will execute and print the content of the document in CloudWatch.
57+
58+
3. Check the CloudWatch log group to see the ECS task execution details. Logs can be found in `/ecs/doc-ingestion` log stream.
59+
Use the following commands to get the logs:
60+
61+
```shell
62+
# get log streams
63+
aws logs describe-log-streams --log-group-name /ecs/doc-ingestion
64+
# use the latest log stream name in the next command to get logs
65+
aws logs get-log-events --log-group-name /ecs/doc-ingestion --log-stream-name doc-ingestion-logs/DocIngestion/...
66+
```
67+
68+
4. The file contents are displayed in the log.
69+
70+
## Cleanup
71+
72+
1. Delete the stack
73+
```bash
74+
npx cdk destroy
75+
```
76+
2. Confirm the stack has been deleted
77+
```bash
78+
aws cloudformation list-stacks --query "StackSummaries[?contains(StackName,'S3TriggerFargateTaskStack')].StackStatus"
79+
```
80+
----
81+
Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
82+
83+
SPDX-License-Identifier: MIT-0

‎s3-eventbridge-fargate-cdk/cdk.json‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
{
2+
"app": "npx ts-node --prefer-ts-exts src/s3-trigger-fargate-task.ts",
3+
"watch": {
4+
"include": [
5+
"**"
6+
],
7+
"exclude": [
8+
"README.md",
9+
"cdk*.json",
10+
"**/*.d.ts",
11+
"**/*.js",
12+
"tsconfig.json",
13+
"package*.json",
14+
"yarn.lock",
15+
"node_modules",
16+
"test"
17+
]
18+
},
19+
"context": {
20+
"@aws-cdk/aws-lambda:recognizeLayerVersion": true,
21+
"@aws-cdk/core:checkSecretUsage": true,
22+
"@aws-cdk/core:target-partitions": [
23+
"aws",
24+
"aws-cn"
25+
],
26+
"@aws-cdk-containers/ecs-service-extensions:enableDefaultLogDriver": true,
27+
"@aws-cdk/aws-ec2:uniqueImdsv2TemplateName": true,
28+
"@aws-cdk/aws-ecs:arnFormatIncludesClusterName": true,
29+
"@aws-cdk/aws-iam:minimizePolicies": true,
30+
"@aws-cdk/core:validateSnapshotRemovalPolicy": true,
31+
"@aws-cdk/aws-codepipeline:crossAccountKeyAliasStackSafeResourceName": true,
32+
"@aws-cdk/aws-s3:createDefaultLoggingPolicy": true,
33+
"@aws-cdk/aws-sns-subscriptions:restrictSqsDescryption": true,
34+
"@aws-cdk/aws-apigateway:disableCloudWatchRole": true,
35+
"@aws-cdk/core:enablePartitionLiterals": true,
36+
"@aws-cdk/aws-events:eventsTargetQueueSameAccount": true,
37+
"@aws-cdk/aws-iam:standardizedServicePrincipals": true,
38+
"@aws-cdk/aws-ecs:disableExplicitDeploymentControllerForCircuitBreaker": true,
39+
"@aws-cdk/aws-iam:importedRoleStackSafeDefaultPolicyName": true,
40+
"@aws-cdk/aws-s3:serverAccessLogsUseBucketPolicy": true,
41+
"@aws-cdk/aws-route53-patters:useCertificate": true,
42+
"@aws-cdk/customresources:installLatestAwsSdkDefault": false,
43+
"@aws-cdk/aws-rds:databaseProxyUniqueResourceName": true,
44+
"@aws-cdk/aws-codedeploy:removeAlarmsFromDeploymentGroup": true,
45+
"@aws-cdk/aws-apigateway:authorizerChangeDeploymentLogicalId": true,
46+
"@aws-cdk/aws-ec2:launchTemplateDefaultUserData": true,
47+
"@aws-cdk/aws-secretsmanager:useAttachedSecretResourcePolicyForSecretTargetAttachments": true,
48+
"@aws-cdk/aws-redshift:columnId": true,
49+
"@aws-cdk/aws-stepfunctions-tasks:enableEmrServicePolicyV2": true,
50+
"@aws-cdk/aws-ec2:restrictDefaultSecurityGroup": true,
51+
"@aws-cdk/aws-apigateway:requestValidatorUniqueId": true,
52+
"@aws-cdk/aws-kms:aliasNameRef": true,
53+
"@aws-cdk/aws-autoscaling:generateLaunchTemplateInsteadOfLaunchConfig": true,
54+
"@aws-cdk/core:includePrefixInUniqueNameGeneration": true,
55+
"@aws-cdk/aws-efs:denyAnonymousAccess": true,
56+
"@aws-cdk/aws-opensearchservice:enableOpensearchMultiAzWithStandby": true,
57+
"@aws-cdk/aws-lambda-nodejs:useLatestRuntimeVersion": true,
58+
"@aws-cdk/aws-efs:mountTargetOrderInsensitiveLogicalId": true,
59+
"@aws-cdk/aws-rds:auroraClusterChangeScopeOfInstanceParameterGroupWithEachParameters": true,
60+
"@aws-cdk/aws-appsync:useArnForSourceApiAssociationIdentifier": true,
61+
"@aws-cdk/aws-rds:preventRenderingDeprecatedCredentials": true,
62+
"@aws-cdk/aws-codepipeline-actions:useNewDefaultBranchForCodeCommitSource": true,
63+
"@aws-cdk/aws-cloudwatch-actions:changeLambdaPermissionLogicalIdForLambdaAction": true
64+
}
65+
}

‎s3-eventbridge-fargate-cdk/doc/archi.drawio‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
<mxfile host="Electron" modified="2024-07-12T07:38:35.598Z" agent="5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) draw.io/15.2.7 Chrome/93.0.4577.63 Electron/14.0.1 Safari/537.36" etag="tnOOLYDqQkU0KmawHNMB" version="15.2.7" type="device"><diagram id="0d4nrQkaPeM4ShJWhUgr" name="Page-1">7Vptd5owFP41fpwHCAh+LKjd69nO3DnbPu2kEiErEBei1f763UBAMNTVM/qyWauWPHnhJs997g1pByhIt5ccr+IPLCTJwDLC7QBNBpY1tkfwLYFdCYxcqwQiTsMSMvfAnN4SBRoKXdOQ5K2GgrFE0FUbXLAsIwvRwjDn7KbdbMmS9l1XOCIaMF/gREe/0lDEJepZ7h5/TWgUV3c2R+OyJsVVYzWTPMYhu2lAaDpAAWdMlFfpNiCJXLtqXcp+sztqa8M4ycR9OsxFaH3z32a/bj8a32eX7y5fkY+vbGWb2FUTJiHMXxUZFzGLWIaT6R71OVtnIZGjGlDat3nP2ApAE8CfRIidIhOvBQMoFmmiasFgvvum+heF77IwdKriZNusnOyapU+E05QIwhVYTkBafee6KChna74gRxaj8i/MIyKOtEM1e+D1hIE1fAf9OEmwoJu2HVj5X1S321MEF4qlExhT425wslZ30ijMr4lYxGp9VoxmojDC8eENPhKUHweaBhIZWk4H2IW5OmjqzeCX2XWHQ7ALc3XQ1JvJUmV1G+zCXEe3+LC32dHbPOgNb3D3tUhoRoI63Mg1XrJMBCxhvFh/BD8zyagfcRxS0qobGRcIuY26CeUwEGUZ1GdSTHI8miTN8VwX4grgueDsmjRqlsULakKcx7UkN4QLCiHsPb4iySeWUzX8FROCpY0GFwmNZIWQwvWxKi3AKimvpmTlDJWeTasqK4+Tt8T5qlyOJd1KO3yIditZmW4jmReG+Ca3h5yUGnyzkPb4UCyv2q1yVMtaGkq2x4WtC7HqMFKRd3dQvtkHctcrobgRwyusd+lamnQvUnwLC2AZc6SpGCYu2iRoBB3ymNIwLEM0yektvqrpaQcB6ccyKueKUM2zMpaRAzdUUB+0OOMWLRbyNFo8Q6elyqO90zJ6ihzYY9pC90xbzlOmLfSStv7ltDWb2TPPPy1t+YGJnPNJW2QDBlzBQ0zUU6BEzy1/OS8a/qc17HpTwz5NwxPDCUz3bDS8hByKRU/6dZ6bft0T9ftXDtexhyzyyHhsGHdtOJ+dKxURjPCpDO15tVXuci+yyH8InF/34zkj58+eg4yhh3TnsR9ql+zd/fBSLI9fZr5zeYpByP7jU4yJHvMxpjoxfVp5q23ifyZvvk7Ij56OJWz3QNyO7jm2M/Q6MsODids89VTxxXfunxo2hWP04Tr1ucgx1+mIOQ/nN0eOtKbBHL5nakd1LnnBOcwLtp66Hzkv6EcvhdOeDSWW67RV00GJ86iM2BojnyHBnA0hWgbsIKQrjD0cIfrJxhe5mT8XQkaw37Ato36ZbXo6trY90QPF/R/Ai7rGfxGg6W8=</diagram></mxfile>

‎s3-eventbridge-fargate-cdk/doc/archi.png‎

24.4 KB
Loading

‎s3-eventbridge-fargate-cdk/docker/doc-ingestion/Dockerfile‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
FROM public.ecr.aws/docker/library/python:3.10-slim
2+
3+
RUN apt-get update
4+
5+
RUN python -m pip install --upgrade setuptools pip
6+
7+
COPY requirements.txt ./
8+
RUN pip install --default-timeout=100 -r requirements.txt
9+
CMD ["pip", "freeze"]
10+
11+
COPY app.py ./
12+
CMD [ "python", "app.py" ]

‎s3-eventbridge-fargate-cdk/docker/doc-ingestion/app.py‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
import os
2+
import boto3
3+
4+
s3_bucket = os.environ['S3_BUCKET']
5+
s3_object = os.environ['S3_OBJECT_KEY']
6+
7+
s3_client = boto3.client('s3')
8+
try:
9+
s3_response = s3_client.get_object(Bucket=s3_bucket, Key=s3_object)
10+
data = s3_response['Body'].read().decode('utf-8').splitlines(True)
11+
print(data)
12+
13+
# implement business logic to process file
14+
15+
except Exception as e:
16+
print(f"Error: {e}")
17+
raise e
18+

‎s3-eventbridge-fargate-cdk/docker/doc-ingestion/requirements.txt‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
boto3
2+
botocore

‎s3-eventbridge-fargate-cdk/jest.config.js‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
module.exports = {
2+
testEnvironment: 'node',
3+
roots: ['<rootDir>/test'],
4+
testMatch: ['**/*.test.ts'],
5+
transform: {
6+
'^.+\\.tsx?$': 'ts-jest'
7+
}
8+
};

0 commit comments

Comments
 (0)