Deployment error upgrading to release 0.24.0 - A resource with the ID `pip-fw-management` already exists

[jonnyry]

Currently running a TRE on v0.23.0. Attempted an upgrade to v0.25.0 which resulted in the deployment error below during the Deploy TRE Core job.

I suspect it is related to this PR #4342 since it contains a sizable firewall change.

I am running an Azure Firewall with a Basic SKU.

---

module.resource_processor_vmss_porter[0].terraform_data.vm_linux_reimage: Still creating... [50s elapsed]
module.resource_processor_vmss_porter[0].terraform_data.vm_linux_reimage: Still creating... [1m0s elapsed]
module.resource_processor_vmss_porter[0].terraform_data.vm_linux_reimage: Still creating... [1m10s elapsed]
module.resource_processor_vmss_porter[0].terraform_data.vm_linux_reimage: Still creating... [1m20s elapsed]
module.resource_processor_vmss_porter[0].terraform_data.vm_linux_reimage: Still creating... [1m30s elapsed]
module.resource_processor_vmss_porter[0].terraform_data.vm_linux_reimage: Creation complete after 1m32s [id=XXX]
╷
│ Error: A resource with the ID "/subscriptions/XXX/resourceGroups/rg-***/providers/Microsoft.Network/publicIPAddresses/pip-fw-management-***" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_public_ip" for more information.
│ 
│   with module.firewall.azurerm_public_ip.fwmanagement[0],
│   on firewall/firewall.tf line 18, in resource "azurerm_public_ip" "fwmanagement":
│   18: resource "azurerm_public_ip" "fwmanagement" ***
│ 
╵
Script done.
Terraform Error

Setting key vault kv-*** back to private access...
 Key vault set back to private access


Disabling public access on storage account

Storage account name: ***

Resource group name: ***
 Public access has been disabled successfully

make: *** [Makefile:136: deploy-core] Error 1

[marrobi]

Did you manage to work around it?

This line in the PR looks like it should remove it from state:

'remove_if_present azurerm_public_ip.fwmanagement[0]'

[marrobi]

And this import it:

  # Firewall IPs 
   if [[ "${FIREWALL_SKU:-}" == "Basic" ]]; then 
     import_if_exists module.firewall.azurerm_public_ip.fwmanagement[0] "/subscriptions/${ARM_SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP_NAME}/providers/Microsoft.Network/publicIPAddresses/pip-fw-management-${TRE_ID}" 
   fi

[jonnyry]

Hmmm, I can see the removal happening:

Checking azurerm_public_ip.fwmanagement[0] ... removing
Removed azurerm_public_ip.fwmanagement[0]
Successfully removed 1 resource instance(s).

But the import is missing the management IP:

IMPORTING STATE FOR FIREWALL...
Resource group rg-*** exists, proceeding with import.
Checking if module.firewall.azurerm_firewall.fw exists in Terraform state...
Checking if module.firewall.azurerm_firewall.fw exists in Azure...
IMPORTING module.firewall.azurerm_firewall.fw /subscriptions/XXX/resourceGroups/rg-***/providers/Microsoft.Network/azureFirewalls/fw-***

[...]

Checking if module.firewall.azurerm_public_ip.fwtransit[0] exists in Terraform state...
Checking if module.firewall.azurerm_public_ip.fwtransit[0] exists in Azure...
IMPORTING module.firewall.azurerm_public_ip.fwtransit[0] /subscriptions/XXX/resourceGroups/rg-***/providers/Microsoft.Network/publicIPAddresses/pip-fw-***

[...]

Checking if module.firewall.azurerm_firewall_policy.root exists in Terraform state...
Checking if module.firewall.azurerm_firewall_policy.root exists in Azure...
IMPORTING module.firewall.azurerm_firewall_policy.root /subscriptions/XXX/resourceGroups/rg-***/providers/Microsoft.Network/firewallPolicies/fw-policy-***

[...]

 Checking if module.firewall.azurerm_firewall_policy_rule_collection_group.core exists in Terraform state...
Checking if module.firewall.azurerm_firewall_policy_rule_collection_group.core exists in Azure...
IMPORTING module.firewall.azurerm_firewall_policy_rule_collection_group.core /subscriptions/XXX/resourceGroups/rg-***/providers/Microsoft.Network/firewallPolicies/fw-policy-***/ruleCollectionGroups/rcg-core


[...]

Checking if module.firewall.azurerm_monitor_diagnostic_setting.firewall exists in Terraform state...
Checking if module.firewall.azurerm_monitor_diagnostic_setting.firewall exists in Azure...
IMPORTING module.firewall.azurerm_monitor_diagnostic_setting.firewall /subscriptions/XXX/resourceGroups/rg-***/providers/Microsoft.Network/azureFirewalls/fw-***|diagnostics-fw-***

[...]

Checking if azurerm_route_table.rt exists in Terraform state...
Checking if azurerm_route_table.rt exists in Azure...
IMPORTING azurerm_route_table.rt /subscriptions/XXX/resourceGroups/rg-***/providers/Microsoft.Network/routeTables/rt-***

[...]

Here's the firewall SKU:

[marrobi]

Sure I checked this flow.

if [[ "${FIREWALL_SKU:-}" == "Basic" ]]; then must be evaluating to false?

Can you echo out FIREWALL_SKU?

[jonnyry]

Ah the var is passed through as TF_VAR_firewall_sku here but not (also) FIREWALL_SKU:

AzureTRE/.github/actions/devcontainer_run_command/action.yml

Line 297 in 34186de

-e TF_VAR_firewall_sku=${{ inputs.FIREWALL_SKU != '' && inputs.FIREWALL_SKU || 'Standard' }} \

[marrobi]

Ah, ok, maybe this is specific to the workflows?

If change FIREWALL_SKU to TF_VAR_firewall_sku think that will work for both local and CI scenarios?

[jonnyry]

I think so... though the pattern in devcontainer_run_command/action.yml seems to be sending the parameter through twice:

e.g.

          -e PRIVATE_AGENT_SUBNET_ID=${{ inputs.PRIVATE_AGENT_SUBNET_ID }} \
          -e TF_VAR_private_agent_subnet_id=${{ inputs.PRIVATE_AGENT_SUBNET_ID }} \

will give this a try

[marrobi]

Yeah, feels like some consolidation needed. Expect ether will work.

[jonnyry]

OK that looks like its worked:

Checking if module.firewall.azurerm_public_ip.fwmanagement[0] exists in Terraform state...
Checking if module.firewall.azurerm_public_ip.fwmanagement[0] exists in Azure...
IMPORTING module.firewall.azurerm_public_ip.fwmanagement[0] /subscriptions/XXX/resourceGroups/rg-***/providers/Microsoft.Network/publicIPAddresses/pip-fw-management-***
Acquiring state lock. This may take a few moments...

[...]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

just waiting for the full deploy to complete

[marrobi]

Which did you do?

[jonnyry]

I added this line to devcontainer_run_command/action.yml:

-e FIREWALL_SKU=${{ inputs.FIREWALL_SKU != '' && inputs.FIREWALL_SKU || 'Standard' }} \

---

Here's the final output from the firewall shared service (which looks ok to me):

Using TEST_ACCOUNT_CLIENT_ID to sign in to tre CLI
Login details saved
Deploying shared service tre-shared-service-firewall of version 1.5.0
An older or failed version is already deloyed. Trying to update...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
Operation state: updating (action=upgrade) - refreshing...
id                                    status    action    resourcePath                                           message
------------------------------------  --------  --------  -----------------------------------------------------  ----------------------------------------------------------------------------
66f31cb2-b6ac-448a-8eb7-828bef49ead9  updated   upgrade   /shared-services/b7ff633b-762c-437d-9cdb-85fd58a6d1cc  b7ff633b-762c-437d-9cdb-85fd58a6d1cc: upgrade action completed successfully.
Deployed shared service tre-shared-service-firewall

[jonnyry]

I can PR this over?

79d8216

[marrobi]

yes please.