Automate the resolution of dependency deadlocks

[stowns]

Hi all. Like most of us, I have a multi-stack deployment with cross-stack dependencies. I often find myself encountering dependency deadlock. This is especially frequent during early development of infrastructure. This occurs due to the following scenario

1. StackA exposes myProperty
2. StackB references stackA.myProperty which results in StackA including a CfnExport in the template
3. StackB later removes the reference to stackA.myProperty.
4. cdk recognizes that StackA no longer needs the export and removes the CfnExport for myProperty
5. deployment of StackA fails due to CloudFormation error stating that the CfnExport 'cannot be deleted as it is in use by StackB'.

While it may seem obvious to just deploy StackB, typically StackA is something like a NetworkStack while StackB is Persistence or Compute. StackA likely contains changes which need to be deployed prior to StackB. The guidance in the CDK documentation is to temporarily add a CfnOutput until the dependency. But, is there a reason why CDK can't do this for us?

ie)
Mark Outputs that will be removed as CDK_TEMP_MIGRATE

1. Compare the list of the CfnOutput's for the currently deployed stack to the one's in the ChangeSet to be deployed
2. If CfnOutput's exist in the deployed stack but do not in the ChangeSet (are being removed), add them to the ChangeSet with a description of CDK_TEMP_MIGRATE
3. Deploy

Remove CDK_TEMP_MIGRATE Outputs on subsequent deployment

1. Compare the list of the CfnOutput's for the currently deployed stack to the one's in the ChangeSet to be deployed
   • If the status of the deployed stack is UPDATE_COMPLETE|CREATE_COMPLETE (previous deployment success)
     • ignore deployed CfnOutputs which have a description of CDK_TEMP_MIGRATE as those were part
       of a previous Output migration and it's assumed that dependent stacks have already been deployed and the dependency deadlock resolved (they no longer need to be deployed)
2. If there are remaining CfnOutput's which exist in the deployed stack but not in the proposed ChangeSet, add them to the ChangeSet with a description of CDK_TEMP_MIGRATE
3. Deploy

This would not solve for scenarios where a stack is deployed, Outputs are marked as CDK_TEMP_MIGRATE, and then the stack redeployed prior to its dependent stacks. You would need to manually add CfnOutput's as is suggested today to resolve the deadlock. This would however solve dependency issues for those of us which have CICD that updates all stacks on each deployment.