Deploying a CDK StackSet using ServiceCatalog ProductStack causes a Race Condition

[billyjbryant]

Background

I am attempting to deploy a Lambda Singleton Function in a StackSet using CDK. I am using TypeScript CDK to build my stack. I am not able to use the cdk-labs/cdk-stacksets package because I was running into other errors with it due to our deployment strategy. I have built a template that builds a S3 Bucket, A ServiceCatalog CloudFormation Template from a ProductStack and a CfnStackSet. My issue is that the CfnStackSet is attempting to deploy before the S3 Deployment of the Service Catalog Template assets is finished, thus creating a race condition where the StackSet attempts to deploy (wherein the stack references a Lambda object in a bucket and the object does not yet exist).

Does anyone have a good way of doing this?

Here is my code so far:

Code for Stack.ts

import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as lambda from 'aws-cdk-lib/aws-lambda';
import * as s3 from 'aws-cdk-lib/aws-s3';
import * as servicecatalog from 'aws-cdk-lib/aws-servicecatalog';

export class stackTemplate extends servicecatalog.ProductStack {
    constructor(scope: Construct, id: string, props?: productStackProps) {
        super(scope, id, props);

       [...]

        const svcAccountLambda = new lambda.SingletonFunction(
            this,
            'svcAccountLambda',
            {
                uuid: uuid(),
                runtime: lambda.Runtime.PYTHON_3_12,
                handler: 'index.getServiceAccount',
                code: lambda.Code.fromAsset('./lambda'),
                timeout: cdk.Duration.seconds(60),
                role: p0LambdaRole,
                functionName: `${this.stackName}-svcAccountLambda`
            }
        );

       [...]
}
export class stackSetStack extends cdk.Stack {
    constructor(scope: Construct, id: string, props?: stackSetStackProps) {
        super(scope, id, props);

        const tags = props?.tags || {};
        const target = props?.target || undefined;

        if (!target) {
            throw new Error('Target is required for stackSetStack');
        }

        const deployBucket = new s3.Bucket(this, 'deployBucket', {
            bucketName: `${this.stackName}-assets`,
            versioned: false,
            removalPolicy: cdk.RemovalPolicy.DESTROY,
            lifecycleRules: [
                {
                    expiration: cdk.Duration.days(7)
                }
            ],
        });

        const assetBucketPolicy = new s3.BucketPolicy(
            this,
            'deployBucketPolicy',
            {
                bucket: deployBucket,
                removalPolicy: cdk.RemovalPolicy.DESTROY,
            }
        );

        assetBucketPolicy.node.addDependency(deployBucket);

        deployBucket.policy = assetBucketPolicy;
        // Enforce that bucket uses SSL for all requests
        assetBucketPolicy.document.addStatements(
            new iam.PolicyStatement({
                actions: ['s3:*'],
                resources: [
                    deployBucket.bucketArn,
                    `${deployBucket.bucketArn}/*`
                ],
                effect: iam.Effect.DENY,
                principals: [new iam.StarPrincipal()],
                conditions: {
                    Bool: {
                        'aws:SecureTransport': 'false'
                    }
                }
            })
        );
        // Permit Get* List* from all principals in the organization
        assetBucketPolicy.document.addStatements(
            new iam.PolicyStatement({
                actions: [
                    's3:Get*',
                    's3:List*'
                ],
                resources: [
                    deployBucket.bucketArn,
                    `${deployBucket.bucketArn}/*`
                ],
                effect: iam.Effect.ALLOW,
                principals: [new iam.OrganizationPrincipal(orgId)]
            })
        );


        for (const [key, value] of Object.entries(tags)) {
            cdk.Tags.of(deployBucket).add(key, value);
        }
        const stackSetBucket = s3.Bucket.fromBucketName(this, 'stackSetBucket', deployBucket.bucketName);

        const stackSetTemplate = new stackTemplate(
            this,
            'P0JITStackTemplate',
            {
                env: props?.env || {},
                tags: props?.tags || {},
                assetBucket: stackSetBucket
            }
        );

        const stackSetServiceCatalog =
            servicecatalog.CloudFormationTemplate.fromProductStack(
                stackSetTemplate
            );

        const stackSet = new cdk.CfnStackSet(this, 'stackSet', {
            stackSetName: this.stackName,
            templateUrl: stackSetServiceCatalog.bind(this).httpUrl,
            callAs: 'DELEGATED_ADMIN',
            permissionModel: 'SERVICE_MANAGED',
            autoDeployment: {
                enabled: true,
                retainStacksOnAccountRemoval: false
            },
            capabilities: ['CAPABILITY_NAMED_IAM', 'CAPABILITY_AUTO_EXPAND'],
            stackInstancesGroup: [
                {
                    deploymentTargets: {
                        organizationalUnitIds: target?.organizationalUnits
                    },
                    regions: target?.regions || ['us-east-1']
                }
            ]
        } as cdk.CfnStackSetProps);
    }
}

Code for app.ts

import * as cdk from 'aws-cdk-lib';
import * as s3 from 'aws-cdk-lib/aws-s3';
import { CustomSynthesizer } from 'custom-synthesizer-package';
import {
    stackSetStack
} from '../lib/stack';

const app = new cdk.App({});     
     
new stackSetStack(app, 'stackSetStack', {
    env,
    stackName: `${stackName}-stackset`,
    tags,
    synthesizer: await new CustomSynthesizer({
        account: env.account,
        region: env.region,
        stackName: `${stackName}-stackset`
    }).synthesize(),
    target,
});

I basically get this error message:

Resource handler returned message: "Resource of type 'Stack set operation [5955205d-d408-4f4f-8dc1-a4f9f2d55ec2] was unexpectedly stopped or failed. status reason(s): [ResourceLogicalId:SingletonLambda487523757a0d4fa18148caa92abefc0f41D847E0, ResourceType:AWS::Lambda::Function, ResourceStatusReason:Resource handler returned message: "Error occurred while GetObject. S3 Error Code: NoSuchKey. S3 Error Message: The specified key does not exist. (Service: Lambda, Status Code: 400, Request ID: e4f39eb7-8fec-424b-a6d9-1e4b53e2b365)"

[pemattr]

So i think i found a solution to this. The problem is that the template is deployed by a custom resource Custom::CDKBucketDeployment and if it happens to run after the stackset (most commonly when the Productstack needs bundling), then the url cannot resolve. To use the above code, you just need to apply a dependency between the asset bucket and the stackset, which will include all the nested resources under the asset bucket:

    stackSet.node.addDependency(stackSetBucket);