feat: aws lambda cdn handler #7292
+359
−58
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| --- | ||
| 'hive': minor | ||
| --- | ||
|
|
||
| Add AWS Lambda CDN Artifact Handler. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,74 @@ | ||
| import { readFileSync } from 'node:fs'; | ||
| import { resolve } from 'node:path'; | ||
| import * as aws from '@pulumi/aws'; | ||
| import * as pulumi from '@pulumi/pulumi'; | ||
| import { Environment } from './environment'; | ||
| import { S3 } from './s3'; | ||
|
|
||
| export function deployAWSArtifactsLambdaFunction(args: { | ||
| environment: Environment; | ||
| /** Note: We run this mirror only on the AWS S3 Bucket on purpose. */ | ||
| s3Mirror: S3; | ||
| }) { | ||
| const lambdaRole = new aws.iam.Role('awsLambdaArtifactsHandlerRole', { | ||
| assumeRolePolicy: { | ||
| Version: '2012-10-17', | ||
| Statement: [ | ||
| { | ||
| Effect: 'Allow', | ||
| Principal: { Service: 'lambda.amazonaws.com' }, | ||
| Action: 'sts:AssumeRole', | ||
| }, | ||
| ], | ||
| }, | ||
| }); | ||
|
|
||
| new aws.iam.RolePolicyAttachment('lambdaBasicExecution', { | ||
| role: lambdaRole.name, | ||
| policyArn: aws.iam.ManagedPolicy.AWSLambdaBasicExecutionRole, | ||
| }); | ||
|
|
||
| const awsLambdaArtifactsHandler = new aws.lambda.Function('awsLambdaArtifactsHandler', { | ||
| name: `hive-artifacts-handler-${args.environment.envName}`, | ||
| runtime: aws.lambda.Runtime.NodeJS22dX, | ||
| handler: 'index.handler', | ||
| packageType: 'Zip', | ||
| architectures: ['arm64'], | ||
| code: new pulumi.asset.AssetArchive({ | ||
| 'index.mjs': new pulumi.asset.StringAsset( | ||
| readFileSync( | ||
| process.env.AWS_LAMBDA_ARTIFACT_PATH || | ||
| resolve(__dirname, '../../packages/services/cdn-worker/dist/index.lambda.mjs'), | ||
| 'utf-8', | ||
| ), | ||
| ), | ||
| }), | ||
| role: lambdaRole.arn, | ||
| region: 'us-east-2', | ||
| environment: { | ||
| variables: { | ||
| // This could be done better with secrets manager etc. | ||
| // But it adds a lot of complexity and overhead and runtime logic | ||
| AWS_S3_ENDPOINT: args.s3Mirror.secret.raw.endpoint, | ||
| AWS_S3_BUCKET_NAME: args.s3Mirror.secret.raw.bucket, | ||
| AWS_S3_ACCESS_KEY_ID: args.s3Mirror.secret.raw.accessKeyId, | ||
| AWS_S3_ACCESSS_KEY_SECRET: args.s3Mirror.secret.raw.secretAccessKey, | ||
| }, | ||
| }, | ||
| // 448mb | ||
| memorySize: 448, | ||
|
Collaborator
There was a problem hiding this comment. lambda memory impacts CPU also.
Contributor
Author
There was a problem hiding this comment. I did some different tests, and 448mb had the best trade-off in terms of cost and runtime. See https://www.notion.so/theguildoss/AWS-Lambda-CDN-Handler-Fallback-2b0b6b71848a80968ce8ff8d756adf89?source=copy_link#2b0b6b71848a8001be26f7cf15136032 for more context |
||
| // 10 seconds | ||
| timeout: 10, | ||
| }); | ||
|
|
||
| const example = new aws.lambda.FunctionUrl('awsLambdaArtifactsHandlerUrl', { | ||
| functionName: awsLambdaArtifactsHandler.arn, | ||
| authorizationType: 'NONE', | ||
| invokeMode: 'BUFFERED', | ||
| region: 'us-east-2', | ||
| }); | ||
|
|
||
| return { | ||
| functionUrl: example.functionUrl, | ||
| }; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| import type { APIGatewayProxyEventV2, APIGatewayProxyResult, Context } from 'aws-lambda'; | ||
| import { z } from 'zod'; | ||
| import { createServerAdapter } from '@whatwg-node/server'; | ||
| import { createArtifactRequestHandler } from './artifact-handler'; | ||
| import { ArtifactStorageReader } from './artifact-storage-reader'; | ||
| import { AwsClient } from './aws'; | ||
| import { createIsAppDeploymentActive } from './is-app-deployment-active'; | ||
| import { createIsKeyValid } from './key-validation'; | ||
|
|
||
| const env = z | ||
| .object({ | ||
| AWS_S3_ACCESS_KEY_ID: z.string(), | ||
| AWS_S3_ACCESSS_KEY_SECRET: z.string(), | ||
| AWS_S3_ENDPOINT: z.string(), | ||
| AWS_S3_BUCKET_NAME: z.string(), | ||
| }) | ||
| .parse((globalThis as any).process.env); | ||
|
|
||
| const s3 = { | ||
| client: new AwsClient({ | ||
| accessKeyId: env.AWS_S3_ACCESS_KEY_ID, | ||
| secretAccessKey: env.AWS_S3_ACCESSS_KEY_SECRET, | ||
| service: 's3', | ||
| }), | ||
| endpoint: env.AWS_S3_ENDPOINT, | ||
| bucketName: env.AWS_S3_BUCKET_NAME, | ||
| }; | ||
|
|
||
| const s3Mirror = null; | ||
|
|
||
| const artifactStorageReader = new ArtifactStorageReader(s3, s3Mirror, null, null); | ||
|
|
||
| const artifactHandler = createArtifactRequestHandler({ | ||
| isKeyValid: createIsKeyValid({ | ||
| artifactStorageReader, | ||
| analytics: null, | ||
| breadcrumb(message: string) { | ||
| console.log(message); | ||
|
Contributor
Author
There was a problem hiding this comment. it gives some context in case there is an exception being thrown in the cloudwatch logs (e.g. what step are we in). breadcrumb is basically a |
||
| }, | ||
| getCache: null, | ||
| waitUntil: null, | ||
| captureException() {}, | ||
| }), | ||
| artifactStorageReader, | ||
| isAppDeploymentActive: createIsAppDeploymentActive({ | ||
| artifactStorageReader, | ||
| getCache: null, | ||
| waitUntil: null, | ||
| }), | ||
| }); | ||
|
|
||
| const artifactRouteHandler = createServerAdapter(artifactHandler as any); | ||
|
|
||
| export async function handler( | ||
| event: APIGatewayProxyEventV2, | ||
| lambdaContext: Context, | ||
| ): Promise<APIGatewayProxyResult> { | ||
| console.log(event.requestContext.http.method, event.rawPath); | ||
| const url = new URL(event.rawPath, 'http://localhost'); | ||
| if (event.queryStringParameters != null) { | ||
| for (const name in event.queryStringParameters) { | ||
| const value = event.queryStringParameters[name]; | ||
| if (value != null) { | ||
| url.searchParams.set(name, value); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| const response = await artifactRouteHandler.fetch( | ||
| url, | ||
| { | ||
| method: event.requestContext.http.method, | ||
| headers: event.headers as HeadersInit, | ||
| body: undefined, | ||
| }, | ||
| { | ||
| event, | ||
| lambdaContext, | ||
| }, | ||
| ); | ||
|
|
||
| if (!response) { | ||
| return { | ||
| statusCode: 404, | ||
| body: '', | ||
| isBase64Encoded: false, | ||
| }; | ||
| } | ||
|
|
||
| const responseHeaders: Record<string, string> = {}; | ||
|
|
||
| response.headers.forEach((value, name) => { | ||
| responseHeaders[name] = value; | ||
| }); | ||
|
|
||
| return { | ||
| statusCode: response.status, | ||
| headers: responseHeaders, | ||
| body: await response.text(), | ||
| isBase64Encoded: false, | ||
| }; | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If anyone insists I can do the extra step of storing these in the AWS Secret Manager and reading it within the lambda during function startup from there instead.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this should be completely fine. And secret manager is expensive