CloudFormation stack deployment takes very long before failing

[malikalimoekhamedov]

Hello friends,

I'm trying to initialise my Redshift Serverless Workgroup/Namespace either from a snapshot or with SQL statements. Both approaches leave the CloudFormation hanging for a while before throwing the following error:

CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [blah-blah]. If you are using the Python cfn-response module, you may need to update your Lambda function code so that CloudFormation can attach the updated version.

Here's the code of my AwsCustomResource:

new AwsCustomResource(this, 'Restore', {
  resourceType: 'Custom::DataWarehouseButlerRestore',
  functionName: 'data-warehouse-butler-restore',
  logRetention: RetentionDays.ONE_DAY,
  vpc: props.vpc,
  vpcSubnets: { subnets: props.vpc.isolatedSubnets },
  policy: AwsCustomResourcePolicy.fromSdkCalls({
    resources: AwsCustomResourcePolicy.ANY_RESOURCE,
  }),
  onCreate: {
    service: 'RedshiftServerless',
    action: 'restoreFromSnapshot',
    physicalResourceId: PhysicalResourceId.of(props.workgroupName),
    parameters: {
      workgroupName: props.workgroupName,
      namespaceName: props.namespaceName,
      snapshotName: props.finalSnapshotName,
    },
  },
});

I feel like I'm following the documentation here as I should.
The fact that it takes very long before throwing the error makes me thing it could be a role/permissions issue, but I explicitly allow access to ANY_RESOURCE to the Lambda policy as seen in the code above.

What am I doing wrong?

[peterwoodworth]

There will be logs for the custom resource provider (Lambda Function) in CloudWatch - Could you view these logs to see if it gives any insight?

If you think it might be a permissions error, try using fromStatements() and granting admin permissions to see if that helps

[malikalimoekhamedov]

Thanks @peterwoodworth,

I did try doing that, indeed. Here's the modified (more opinionated) version of the construct:

new AwsCustomResource(this, 'Restore', {
  role: new Role(this, 'RedshiftServerlessDataRestoration', {
    assumedBy: new ServicePrincipal('lambda.amazonaws.com'),
    description: 'IAM role Lambda can use to restore the database from a snapshot.',
    managedPolicies: [
      ManagedPolicy.fromAwsManagedPolicyName(
        'service-role/AWSLambdaVPCAccessExecutionRole'
      ),
      ManagedPolicy.fromAwsManagedPolicyName('AmazonRedshiftFullAccess'),
    ],
  }),
  resourceType: 'Custom::DataWarehouseButlerRestore',
  functionName: 'data-warehouse-butler-restore',
  logRetention: RetentionDays.ONE_DAY,
  vpc: props.vpc,
  vpcSubnets: { subnets: props.vpc.isolatedSubnets },
  policy: AwsCustomResourcePolicy.fromStatements([
    new PolicyStatement({ actions: ['redshift-serverless:*'], resources: ['*'] }),
  ]),
  onCreate: {
    service: '@aws-sdk/client-redshift-serverless',
    action: 'RestoreFromSnapshotCommand',
    physicalResourceId: PhysicalResourceId.of('Restore'),
    parameters: {
      namespaceName: props.namespaceName,
      workgroupName: props.workgroupName,
      //snapshotName: props.finalSnapshotName,
      snapshotArn:
        'arn:aws:redshift-serverless:MY_REGION:MY_ACCOUNT_NUMBER:snapshot/BLAH_BLAH',
    },
  },
});

Unfortunately, this didn't move the needle, and I'm experiencing the same issue.
The Lambda function logs indicate that it attempts to perform the operation 3 times before calling it quits. I've read on other forums that this is the default amount of attempts before capitulation.

Here's what it prints out 3 times:

{
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:MY_REGION:MY_ACCOUNT:function:data-warehouse-butler-restore",
    "ResponseURL": "...",
    "StackId": "arn:aws:cloudformation:MY_REGION:MY_ACCOUNT:stack/DataWarehouse/BLAH_BLAH",
    "RequestId": "BLAH_BLAH",
    "LogicalResourceId": "DataWarehouseButlerRestoreCAD1C5C6",
    "ResourceType": "Custom::DataWarehouseButlerRestore",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:MY_REGION:MY_ACCOUNT:function:data-warehouse-butler-restore",
        "InstallLatestAwsSdk": "false",
        "Create": {
            "service": "@aws-sdk/client-redshift-serverless",
            "action": "RestoreFromSnapshotCommand",
            "physicalResourceId": {
                "id": "Restore"
            },
            "parameters": {
                "namespaceName": "MY_NAMESPACE",
                "workgroupName": "MY_WORKGROUP",
                "snapshotArn": "arn:aws:redshift-serverless:MY_REGION:MY_ACCOUNT:snapshot/BLAH_BLAH"
            }
        }
    }
}

I'm out of ideas.

[peterwoodworth]

There should be a different handler - the logs you're looking at here are the logs responsible for sending the request to the handler. You should be able to view the actual handler's logs in CloudWatch. Check for the lambda function that is actually running the RestoreFromSnapshotCommand Api call in code

[malikalimoekhamedov]

Thanks, @peterwoodworth.

However, I don't know if there's anything else I have logs for. The only other lambda output related to this deployment is the logs retention lambda one. I'm providing it here just in case, but I'm certain that's not what you were referring to.

{
    "RequestType": "Create",
    "ServiceToken": "arn:aws:lambda:MY_REGION:MY_ACCOUNT:function:DataWarehouse-LogRetentionUUID",
    "ResponseURL": "...",
    "StackId": "arn:aws:cloudformation:MY_REGION:MY_ACCOUNT:stack/DataWarehouse/UUID",
    "RequestId": "UUID",
    "LogicalResourceId": "AWSUUIDLogRetentionCE72797A",
    "ResourceType": "Custom::LogRetention",
    "ResourceProperties": {
        "ServiceToken": "arn:aws:lambda:MY_REGION:MY_ACCOUNT:function:DataWarehouse-LogRetentionUUID",
        "RetentionInDays": "1",
        "LogGroupName": "/aws/lambda/data-warehouse-butler-restore"
    }
}

{
  clientName: 'CloudWatchLogsClient',
  commandName: 'PutRetentionPolicyCommand',
  input: {
    logGroupName: '/aws/lambda/data-warehouse-butler-restore',
    retentionInDays: 1
  },
  output: {},
  metadata: {
    httpStatusCode: 200,
    requestId: 'UUID',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

{
  clientName: 'CloudWatchLogsClient',
  commandName: 'CreateLogGroupCommand',
  input: {
    logGroupName: '/aws/lambda/DataWarehouse-LogRetentionUUID'
  },
  output: {},
  metadata: {
    httpStatusCode: 200,
    requestId: 'UUID',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

{
  clientName: 'CloudWatchLogsClient',
  commandName: 'PutRetentionPolicyCommand',
  input: {
    logGroupName: '/aws/lambda/DataWarehouse-LogRetentionUUID',
    retentionInDays: 1
  },
  output: {},
  metadata: {
    httpStatusCode: 200,
    requestId: 'UUID',
    extendedRequestId: undefined,
    cfId: undefined,
    attempts: 1,
    totalRetryDelay: 0
  }
}

{
    "Status": "SUCCESS",
    "Reason": "OK",
    "PhysicalResourceId": "/aws/lambda/data-warehouse-butler-restore",
    "StackId": "arn:aws:cloudformation:MY_REGION:MY_ACCOUNT:stack/DataWarehouse/UUID",
    "RequestId": "UUID",
    "LogicalResourceId": "AWSUUIDLogRetentionCE72797A",
    "Data": {
        "LogGroupName": "/aws/lambda/data-warehouse-butler-restore"
    }
}

By the way, it looks like these "LogRetentionUUID" log groups never get deleted. Is there anything I can do about it? Looking at the removalPolicy parameter, I see the default is RemovalPolicy.Destroy.

Any clues?

[malikalimoekhamedov]

Actually, after a while, I can see another log stream in the /aws/lambda/data-warehouse-butler-restore lambda logs:

{
    "Status": "SUCCESS",
    "Reason": "OK",
    "PhysicalResourceId": "DataWarehouse-DataWarehouseButlerRestoreUUID",
    "StackId": "arn:aws:cloudformation:MY_REGION:MY_ACCOUNT:stack/DataWarehouse/UUID",
    "RequestId": "UUID",
    "LogicalResourceId": "DataWarehouseButlerRestoreCAD1C5C6",
    "NoEcho": false,
    "Data": {}
}

This log also appears 3 times.

Yet, the entire deployment fails and rolls back just as slowly as it takes to attempt to deploy.

I hope this information completes the picture.

[malikalimoekhamedov]

Some food for thought...

On this page, it says whenever one creates a custom resource in a VPC that has an endpoint (which is my case), the IAM policy of this endpoint should allow for communication with a global AWS S3 bucket dedicated to responses from custom resources.

When using the VPC endpoint feature, grant access to CloudFormation-specific S3 buckets for resources in a VPC that must respond to a custom resource request or a wait condition.

If you use CloudFormation to create resources in a VPC with a VPC endpoint, you might need to modify your IAM endpoint policy so that it permits access to specific S3 buckets.

CloudFormation has S3 buckets in each Region to monitor responses to a custom resource request or a wait condition. If a template includes custom resources or wait conditions in a VPC, the VPC endpoint policy must allow users to send responses to the following buckets:

For custom resources, permit traffic to the cloudformation-custom-resource-response-region bucket. When using custom resources, region names don't contain dashes. For example, uswest2.

Here's how I create my VPC:

new Vpc(this, 'VPC', {
  vpcName: 'Main',
  maxAzs: 3,
  subnetConfiguration: [
    {
      subnetType: SubnetType.PUBLIC,
      name: 'Public',
    },
    {
      subnetType: SubnetType.PRIVATE_WITH_EGRESS,
      name: 'PrivateWithEgress',
    },
    {
      subnetType: SubnetType.PRIVATE_ISOLATED,
      name: 'PrivateIsolated',
    },
  ],
});

CDK creates one VPC Interface Endpoint associated with one default security group. The VPC endpoint and security group are attached to the Redshift Serverless workgroup/namespace.

This security group allows for all inbound traffic from within itself and outbound traffic to anything (0.0.0.0/0).

I manually added an inbound rule allowing all inbound traffic (0.0.0.0/0) for the sake of the test but to no avail.

I also notice that the Redshift Serverless dashboard allows you to create a Redshift-managed VPC endpoint. I'm not sure this could help.

My custom resource lambda function has its own security group. How can I (using CDK) make sure it can "talk" to the security group of Redshift Serverless)? As you can see above, I do have the following policy statements ManagedPolicy.fromAwsManagedPolicyName('AmazonRedshiftFullAccess') and new PolicyStatement({ actions: ['redshift-serverless:*'], resources: ['*'] }). Yet, apparently, this is not enough to make CDK create a communication channel.

All ideas are welcome. Thanks in advance.

[malikalimoekhamedov]

Manually added permissive ingress/egress rules (0.0.0.0/0) to both security groups (Redshift Serverless Workgroup and custom resource lambda). Nothing. Nada.