az stack sub delete times out consistently.

[preslavgerchev]

Describe the bug

Bug Report

I am trying to deploy a simple ARM template via a deployment stack. It gets deployed successfully but then deleting it times out consistently, takes exactly one hour and then fails. This is now happening multiple times across multiple subscriptions that we've tested.

To reproduce:
This is the first bicep file, main.bicep

targetScope = 'subscription'

@description('Name of the resource group to create')
param resourceGroupName string 

@description('Location for the resource group')
@minLength(1)
param resourceGroupLocation string

module resourceGrp 'resourceGroup.bicep' = {
  name: 'test_deploy_rg_123'
  params: {
    resourceGroupName: resourceGroupName
    resourceGroupLocation: resourceGroupLocation
  }
}

And the referenced resourceGroup.bicep:

targetScope = 'subscription'

@description('Name of the resource group to create.')
param resourceGroupName string

@description('Location of the resource group to create.')
param resourceGroupLocation string

resource rg 'Microsoft.Resources/resourceGroups@2023-07-01' = {
  name: resourceGroupName
  location: resourceGroupLocation
}

Also vars.bicepparam file for completeness:

using './main.bicep'

param resourceGroupLocation = 'uksouth'
param resourceGroupName = 'some-rg-name'

Then run

az stack sub create --name <name>
    --action-on-unmanage deleteAll \
    --deny-settings-mode None \
    --template-file main.bicep \
    --location uksouth \
    --parameters vars.bicepparam \
    --subscription <sub>

The stack gets deployed successfully. Then try to run this:

az stack sub delete --name <name> --subscription <sub> --action-on-unmanage deleteAll --yes

This takes an hour and then times out. Upon inspection, the RGs are deleted but for some reason the deployment stack still has them linked and thinks they are not deleted.

The resource group under the deployment stack has this error:

Deletion of this resource was bypassed because of a failure in another resource deletion.

 (Code: ResourceDeletionBypassed, Target: /subscriptions/id/rg-name)

Deleting with --action-on-unmanage detach works after.

Related command

az stack sub delete --action-on-unmanage deleteAll

Errors

az stack sub delete --name <name> --subscription <sub> --action-on-unmanage deleteAll --yes
(DeploymentStackDeleteResourcesFailed) One or more resources could not be deleted. Correlation id: '73439f74-2996-4a7d-be35-1eb70523cc38'.
Code: DeploymentStackDeleteResourcesFailed
Message: One or more resources could not be deleted. Correlation id: '73439f74-2996-4a7d-be35-1eb70523cc38'.
Exception Details:      (DeploymentStackDeleteResourcesFailed) An error occurred while deleting resources. These resources are still present in the stack but can be deleted manually. Please see the FailedResources property for specific error information. Deletion failures that are known limitations are documented here: https://aka.ms/DeploymentStacksKnownLimitations
        Code: DeploymentStackDeleteResourcesFailed
        Message: An error occurred while deleting resources. These resources are still present in the stack but can be deleted manually. Please see the FailedResources property for specific error information. Deletion failures that are known limitations are documented here: https://aka.ms/DeploymentStacksKnownLimitations

Issue script & Debug output

Did not gather this. I suspect this is not a CLI issue, but I can rerun it again if debug logs are required

Expected behavior

The stack gets deleted

Environment Summary

az version                                             
{
  "azure-cli": "2.74.0",
  "azure-cli-core": "2.74.0",
  "azure-cli-telemetry": "1.1.0",
  "extensions": {
    "application-insights": "1.2.2",
    "quota": "1.0.0",
    "resource-graph": "2.1.0",
    "ssh": "2.0.5"
  }
}

Additional context

No response

[yonzhan]

Thank you for opening this issue, we will look into it.

[preslavgerchev]

Is there any update on this? We're still seeing this bug when deleting deployment stacks that involve resource groups.

[niinasii]

I also get this error via Azure Devops pipeline with this command:

az stack sub delete \
--subscription subID  \
--name previously-created-stack \
--action-on-unmanage deleteAll \
--yes

My bicep deploys resource group and keyvault. Deletion resource group fails everytime, causing retry loop and stack provisioning state is stuck in 'deletion' until timeout. However first attempt to delete resource group actually succeeds de facto because next retries indicates that resource group is not found and activity logs actually shows it. Also recreating new resource group within this deletion timeframe will also delete this new resource group even if it was created manually in Azure portal.

Error json from activity logs:
"statusMessage": "{\"error\":{\"code\":\"ResourceGroupNotFound\",\"message\":\"Resource group 'previously-created-stack' could not be found.\"}}",