Weird diff behaviour for aci_application_epg.relation_to_domains - objects not lining up when importing? (DCNE-637)

[pv2b]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v1.13.3
on windows_amd64

• provider registry.terraform.io/ciscodevnet/aci v2.18.0
• provider registry.terraform.io/hashicorp/http v3.5.0

APIC version and APIC Platform

• v6.0(7e) on-prem

Affected Resource(s)

• aci_application_epg

Terraform Configuration Files

resource "aci_application_epg" "test" {
  parent_dn = "uni/tn-CAGCommon/ap-AppProf-CAGCommon"
  name        = "EPG-JuniperExt"
  relation_to_bridge_domain = {
    bridge_domain_name = "BD-CAGCommon-JuniperExt"
  }
  relation_to_domains = [
    {
        target_dn = "uni/phys-PhysDom-CustPort"
    },
    {
        target_dn = "uni/phys-PhysDom-PaloAltoFW-Cust"
    },
    {
        target_dn = "uni/vmmp-VMware/dom-ACI-dvSwitch"
        resolution_immediacy = "pre-provision"
        deployment_immediacy = "immediate"
    }
  ]
}

import {
  to = aci_application_epg.test
  id = "uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt"
}

Debug Output

Panic Output

Expected Behavior

Because uni/tn-CAGCommon/ap-AppProf-CAGCommon/EPG-JuniperExt is already present in the APIC, and is already related to the specified bridge domains, I would expect the relation_to_domains object to not materially change.

Actual Behavior

In fact, when running terraform plan, this is what we get:

aci_application_epg.test: Preparing import... [id=uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt]
aci_application_epg.test: Refreshing state... [id=uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # aci_application_epg.test will be updated in-place
  # (imported from "uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt")
  ~ resource "aci_application_epg" "test" {
        admin_state                            = "no"
      ~ annotation                             = "" -> "orchestrator:terraform"
        annotations                            = []
      ~ application_profile_dn                 = "uni/tn-CAGCommon/ap-AppProf-CAGCommon" -> (known after apply)
        contract_exception_tag                 = ""
        description                            = "Juniper_Priv_Net"
        epg_useg_block_statement               = {}
      ~ exception_tag                          = null -> (known after apply)
        flood_in_encapsulation                 = "disabled"
      ~ flood_on_encap                         = "disabled" -> (known after apply)
        forwarding_control                     = "none"
      ~ fwd_ctrl                               = "none" -> (known after apply)
      ~ has_mcast_source                       = "no" -> (known after apply)
        has_multicast_source                   = "no"
        id                                     = "uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt"
        intra_epg_isolation                    = "unenforced"
      ~ is_attr_based_epg                      = "no" -> (known after apply)
        match_criteria                         = "AtleastOne"
      ~ match_t                                = "AtleastOne" -> (known after apply)
        name                                   = "EPG-JuniperExt"
        name_alias                             = ""
        parent_dn                              = "uni/tn-CAGCommon/ap-AppProf-CAGCommon"
      ~ pc_enf_pref                            = "unenforced" -> (known after apply)
      ~ pc_tag                                 = "16395" -> (known after apply)
      ~ pref_gr_memb                           = "exclude" -> (known after apply)
        preferred_group_member                 = "exclude"
      ~ prio                                   = "unspecified" -> (known after apply)
        priority                               = "unspecified"
      + relation_fv_rs_aepg_mon_pol            = (known after apply)
      ~ relation_fv_rs_bd                      = "uni/tn-CAGCommon/BD-BD-CAGCommon-JuniperExt" -> (known after apply)
      ~ relation_fv_rs_cons                    = [] -> (known after apply)
      ~ relation_fv_rs_cons_if                 = [] -> (known after apply)
      ~ relation_fv_rs_cust_qos_pol            = "uni/tn-common/qoscustom-default" -> (known after apply)
      + relation_fv_rs_dpp_pol                 = (known after apply)
      ~ relation_fv_rs_fc_path_att             = [] -> (known after apply)
      ~ relation_fv_rs_intra_epg               = [] -> (known after apply)
        relation_fv_rs_path_att                = []
      ~ relation_fv_rs_prot_by                 = [] -> (known after apply)
      ~ relation_fv_rs_prov                    = [] -> (known after apply)
        relation_fv_rs_prov_def                = []
      ~ relation_fv_rs_sec_inherited           = [] -> (known after apply)
      + relation_fv_rs_trust_ctrl              = (known after apply)
        relation_to_bridge_domain              = {
            annotation         = ""
            annotations        = []
            bridge_domain_name = "BD-CAGCommon-JuniperExt"
            tags               = []
        }
        relation_to_consumed_contracts         = []
        relation_to_contract_masters           = []
        relation_to_custom_qos_policy          = {
            annotation             = ""
            annotations            = []
            custom_qos_policy_name = ""
            tags                   = []
        }
        relation_to_data_plane_policing_policy = {}
      ~ relation_to_domains                    = [
            {
                annotation                    = null
                annotations                   = []
                binding_type                  = "none"
                class_preference              = "encap"
                custom_epg_name               = null
                delimiter                     = null
                deployment_immediacy          = "immediate"
                enable_netflow                = "disabled"
                encapsulation                 = "unknown"
                encapsulation_mode            = "auto"
                epg_cos                       = "Cos0"
                epg_cos_pref                  = "disabled"
                ipam_dhcp_override            = "0.0.0.0"
                ipam_enabled                  = "no"
                ipam_gateway                  = "0.0.0.0"
                lag_policy_name               = null
                netflow_direction             = "both"
                number_of_ports               = "0"
                port_allocation               = "none"
                primary_encapsulation         = "unknown"
                primary_encapsulation_inner   = "unknown"
                resolution_immediacy          = "pre-provision"
                secondary_encapsulation_inner = "unknown"
                switching_mode                = "native"
                tags                          = []
                target_dn                     = "uni/vmmp-VMware/dom-ACI-dvSwitch"
                untagged                      = "no"
            },
          - {
                annotation                    = null
              - annotations                   = [] -> null
              - binding_type                  = "none" -> null
              - class_preference              = "encap" -> null
                custom_epg_name               = null
                delimiter                     = null
              - deployment_immediacy          = "lazy" -> null
              - enable_netflow                = "disabled" -> null
              - encapsulation                 = "unknown" -> null
              - encapsulation_mode            = "auto" -> null
              - epg_cos                       = "Cos0" -> null
              - epg_cos_pref                  = "disabled" -> null
              - ipam_dhcp_override            = "0.0.0.0" -> null
              - ipam_enabled                  = "no" -> null
              - ipam_gateway                  = "0.0.0.0" -> null
                lag_policy_name               = null
              - netflow_direction             = "both" -> null
              - number_of_ports               = "0" -> null
              - port_allocation               = "none" -> null
              - primary_encapsulation         = "unknown" -> null
              - primary_encapsulation_inner   = "unknown" -> null
              - resolution_immediacy          = "immediate" -> null
              - secondary_encapsulation_inner = "unknown" -> null
              - switching_mode                = "native" -> null
              - tags                          = [] -> null
              - target_dn                     = "uni/phys-PhysDom-PaloAltoFW-Cust" -> null
              - untagged                      = "no" -> null
            },
          - {
              - annotation                    = "orchestrator:terraform" -> null
              - annotations                   = [] -> null
              - binding_type                  = "none" -> null
              - class_preference              = "encap" -> null
                custom_epg_name               = null
                delimiter                     = null
              - deployment_immediacy          = "lazy" -> null
              - enable_netflow                = "disabled" -> null
              - encapsulation                 = "unknown" -> null
              - encapsulation_mode            = "auto" -> null
              - epg_cos                       = "Cos0" -> null
              - epg_cos_pref                  = "disabled" -> null
              - ipam_dhcp_override            = "0.0.0.0" -> null
              - ipam_enabled                  = "no" -> null
              - ipam_gateway                  = "0.0.0.0" -> null
                lag_policy_name               = null
              - netflow_direction             = "both" -> null
              - number_of_ports               = "0" -> null
              - port_allocation               = "none" -> null
              - primary_encapsulation         = "unknown" -> null
              - primary_encapsulation_inner   = "unknown" -> null
              - resolution_immediacy          = "lazy" -> null
              - secondary_encapsulation_inner = "unknown" -> null
              - switching_mode                = "native" -> null
              - tags                          = [] -> null
              - target_dn                     = "uni/phys-PhysDom-CustPort" -> null
              - untagged                      = "no" -> null
            },
          + {
                annotation                    = null
              + annotations                   = []
              + binding_type                  = "none"
              + class_preference              = "encap"
                custom_epg_name               = null
                delimiter                     = null
              + deployment_immediacy          = "lazy"
              + enable_netflow                = "disabled"
              + encapsulation                 = "unknown"
              + encapsulation_mode            = "auto"
              + epg_cos                       = "Cos0"
              + epg_cos_pref                  = "disabled"
              + ipam_dhcp_override            = "0.0.0.0"
              + ipam_enabled                  = "no"
              + ipam_gateway                  = "0.0.0.0"
                lag_policy_name               = null
              + netflow_direction             = "both"
              + number_of_ports               = "0"
              + port_allocation               = "none"
              + primary_encapsulation         = "unknown"
              + primary_encapsulation_inner   = "unknown"
              + resolution_immediacy          = "immediate"
              + secondary_encapsulation_inner = "unknown"
              + switching_mode                = "native"
              + tags                          = []
              + target_dn                     = "uni/phys-PhysDom-CustPort"
              + untagged                      = "no"
            },
          + {
              + annotation                    = "orchestrator:terraform"
              + annotations                   = []
              + binding_type                  = "none"
              + class_preference              = "encap"
                custom_epg_name               = null
                delimiter                     = null
              + deployment_immediacy          = "lazy"
              + enable_netflow                = "disabled"
              + encapsulation                 = "unknown"
              + encapsulation_mode            = "auto"
              + epg_cos                       = "Cos0"
              + epg_cos_pref                  = "disabled"
              + ipam_dhcp_override            = "0.0.0.0"
              + ipam_enabled                  = "no"
              + ipam_gateway                  = "0.0.0.0"
                lag_policy_name               = null
              + netflow_direction             = "both"
              + number_of_ports               = "0"
              + port_allocation               = "none"
              + primary_encapsulation         = "unknown"
              + primary_encapsulation_inner   = "unknown"
              + resolution_immediacy          = "lazy"
              + secondary_encapsulation_inner = "unknown"
              + switching_mode                = "native"
              + tags                          = []
              + target_dn                     = "uni/phys-PhysDom-PaloAltoFW-Cust"
              + untagged                      = "no"
            },
        ]
        relation_to_fibre_channel_paths        = []
        relation_to_imported_contracts         = []
        relation_to_intra_epg_contracts        = []
        relation_to_monitoring_policy          = {}
        relation_to_provided_contracts         = []
        relation_to_static_leafs               = []
        relation_to_static_paths               = []
        relation_to_taboo_contracts            = []
        relation_to_trust_control_policy       = {}
      ~ scope                                  = "2818051" -> (known after apply)
      ~ shutdown                               = "no" -> (known after apply)
        tags                                   = []
        useg_epg                               = "no"
    }

Plan: 1 to import, 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

As we can see, the relation to uni/vmmp-VMware/dom-ACI-dvSwitch remains unchanged. However, it then seems to delete the relations to uni/phys-PhysDom-PaloAltoFW-Cust and uni/phys-PhysDom-CustPort - only to re-create them again. However, when re-creating them it seems to be inconsistent about deployment_immediacy. For uni/phys-PhysDom-PaloAltoFW-Cust, resolution_immediacy is changed from immediate to lazy, and for uni/phys-PhysDom-CustPort it's changing from lazy to immediate.

Changing the order of the relationships does not seem to affect anything.

Steps to Reproduce

1. Somehow have an ACI environment with a similar EPG and relation to physdoms. Not sure what fields exactly need to be the same.
2. Run a terraform plan and observe the output.

Important Factoids

I can't actually find any reference to the meaning of deployment_immediacy or resolution_immediacy in the context of relations to physical domains, it's not settable or viewable through the APIC GUI, only for VMM domains. Looking at live objects deployed, it seems these attributes is set inconsistently with no clear pattern or mechanism by which those would have been set, but discussion of that is probably not in scope of this bug - the main concern here is that when importing a state it doesn't appear like the "sub-objects" are being changed, instead being deleted and re-added. Also it's not clear where the "desired value" for these objects is even coming from.

References

[shrsr]

@pv2b Thank you for opening the issue. I briefly went through it and am wondering what you'll see if you tried the following import config instead?

resource "aci_application_epg" "test" {
  parent_dn = "uni/tn-CAGCommon/ap-AppProf-CAGCommon"
  name        = "EPG-JuniperExt"
  annotation = ""
}

import {
  to = aci_application_epg.test
  id = "uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt"
}

I have added this to our to-do and someone from the team will get back to you after looking at it thoroughly.

[pv2b]

@pv2b Thank you for opening the issue. I briefly went through it and am wondering what you'll see if you tried the following import config instead?

resource "aci_application_epg" "test" {
  parent_dn = "uni/tn-CAGCommon/ap-AppProf-CAGCommon"
  name        = "EPG-JuniperExt"
  annotation = ""
}

import {
  to = aci_application_epg.test
  id = "uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt"
}

Thank you for your response. :-)

When running the exact config you mentioned above, this is what I get:

aci_application_epg.test: Preparing import... [id=uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt]
aci_application_epg.test: Refreshing state... [id=uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt]

Terraform will perform the following actions:

  # aci_application_epg.test will be imported
    resource "aci_application_epg" "test" {
        admin_state                            = "no"
        annotation                             = ""
        annotations                            = []
        application_profile_dn                 = "uni/tn-CAGCommon/ap-AppProf-CAGCommon"
        contract_exception_tag                 = ""
        description                            = "Juniper_Priv_Net"
        epg_useg_block_statement               = {}
        exception_tag                          = ""
        flood_in_encapsulation                 = "disabled"
        flood_on_encap                         = "disabled"
        forwarding_control                     = "none"
        fwd_ctrl                               = "none"
        has_mcast_source                       = "no"
        has_multicast_source                   = "no"
        id                                     = "uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt"
        intra_epg_isolation                    = "unenforced"
        is_attr_based_epg                      = "no"
        match_criteria                         = "AtleastOne"
        match_t                                = "AtleastOne"
        name                                   = "EPG-JuniperExt"
        name_alias                             = ""
        parent_dn                              = "uni/tn-CAGCommon/ap-AppProf-CAGCommon"
        pc_enf_pref                            = "unenforced"
        pc_tag                                 = "16395"
        pref_gr_memb                           = "exclude"
        preferred_group_member                 = "exclude"
        prio                                   = "unspecified"
        priority                               = "unspecified"
        relation_fv_rs_bd                      = "uni/tn-CAGCommon/BD-BD-CAGCommon-JuniperExt"
        relation_fv_rs_cons                    = []
        relation_fv_rs_cons_if                 = []
        relation_fv_rs_cust_qos_pol            = "uni/tn-common/qoscustom-default"
        relation_fv_rs_fc_path_att             = []
        relation_fv_rs_intra_epg               = []
        relation_fv_rs_path_att                = []
        relation_fv_rs_prot_by                 = []
        relation_fv_rs_prov                    = []
        relation_fv_rs_prov_def                = []
        relation_fv_rs_sec_inherited           = []
        relation_to_bridge_domain              = {
            annotation         = ""
            annotations        = []
            bridge_domain_name = "BD-CAGCommon-JuniperExt"
            tags               = []
        }
        relation_to_consumed_contracts         = []
        relation_to_contract_masters           = []
        relation_to_custom_qos_policy          = {
            annotation             = ""
            annotations            = []
            custom_qos_policy_name = ""
            tags                   = []
        }
        relation_to_data_plane_policing_policy = {}
        relation_to_domains                    = [
            {
                annotation                    = null
                annotations                   = []
                binding_type                  = "none"
                class_preference              = "encap"
                custom_epg_name               = null
                delimiter                     = null
                deployment_immediacy          = "immediate"
                enable_netflow                = "disabled"
                encapsulation                 = "unknown"
                encapsulation_mode            = "auto"
                epg_cos                       = "Cos0"
                epg_cos_pref                  = "disabled"
                ipam_dhcp_override            = "0.0.0.0"
                ipam_enabled                  = "no"
                ipam_gateway                  = "0.0.0.0"
                lag_policy_name               = null
                netflow_direction             = "both"
                number_of_ports               = "0"
                port_allocation               = "none"
                primary_encapsulation         = "unknown"
                primary_encapsulation_inner   = "unknown"
                resolution_immediacy          = "pre-provision"
                secondary_encapsulation_inner = "unknown"
                switching_mode                = "native"
                tags                          = []
                target_dn                     = "uni/vmmp-VMware/dom-ACI-dvSwitch"
                untagged                      = "no"
            },
            {
                annotation                    = null
                annotations                   = []
                binding_type                  = "none"
                class_preference              = "encap"
                custom_epg_name               = null
                delimiter                     = null
                deployment_immediacy          = "lazy"
                enable_netflow                = "disabled"
                encapsulation                 = "unknown"
                encapsulation_mode            = "auto"
                epg_cos                       = "Cos0"
                epg_cos_pref                  = "disabled"
                ipam_dhcp_override            = "0.0.0.0"
                ipam_enabled                  = "no"
                ipam_gateway                  = "0.0.0.0"
                lag_policy_name               = null
                netflow_direction             = "both"
                number_of_ports               = "0"
                port_allocation               = "none"
                primary_encapsulation         = "unknown"
                primary_encapsulation_inner   = "unknown"
                resolution_immediacy          = "immediate"
                secondary_encapsulation_inner = "unknown"
                switching_mode                = "native"
                tags                          = []
                target_dn                     = "uni/phys-PhysDom-PaloAltoFW-Cust"
                untagged                      = "no"
            },
            {
                annotation                    = "orchestrator:terraform"
                annotations                   = []
                binding_type                  = "none"
                class_preference              = "encap"
                custom_epg_name               = null
                delimiter                     = null
                deployment_immediacy          = "lazy"
                enable_netflow                = "disabled"
                encapsulation                 = "unknown"
                encapsulation_mode            = "auto"
                epg_cos                       = "Cos0"
                epg_cos_pref                  = "disabled"
                ipam_dhcp_override            = "0.0.0.0"
                ipam_enabled                  = "no"
                ipam_gateway                  = "0.0.0.0"
                lag_policy_name               = null
                netflow_direction             = "both"
                number_of_ports               = "0"
                port_allocation               = "none"
                primary_encapsulation         = "unknown"
                primary_encapsulation_inner   = "unknown"
                resolution_immediacy          = "lazy"
                secondary_encapsulation_inner = "unknown"
                switching_mode                = "native"
                tags                          = []
                target_dn                     = "uni/phys-PhysDom-CustPort"
                untagged                      = "no"
            },
        ]
        relation_to_fibre_channel_paths        = []
        relation_to_imported_contracts         = []
        relation_to_intra_epg_contracts        = []
        relation_to_monitoring_policy          = {}
        relation_to_provided_contracts         = []
        relation_to_static_leafs               = []
        relation_to_static_paths               = []
        relation_to_taboo_contracts            = []
        relation_to_trust_control_policy       = {}
        scope                                  = "2818051"
        shutdown                               = "no"
        tags                                   = []
        useg_epg                               = "no"
    }

Plan: 1 to import, 0 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

[pv2b]

Also, to anticipate what might be your next question - applied the plan which resulted in just an import with no changes into the test tfstate.

After this, I tried generating a plan for this configuration:

resource "aci_application_epg" "test" {
  parent_dn = "uni/tn-CAGCommon/ap-AppProf-CAGCommon"
  name        = "EPG-JuniperExt"
  annotation = ""
  /*relation_to_bridge_domain = {
    bridge_domain_name = "BD-CAGCommon-JuniperExt"
  }*/
  relation_to_domains = [
    {
        target_dn = "uni/phys-PhysDom-CustPort"
    },
    {
        target_dn = "uni/phys-PhysDom-PaloAltoFW-Cust"
    },
    {
        target_dn = "uni/vmmp-VMware/dom-ACI-dvSwitch"
        resolution_immediacy = "pre-provision"
        deployment_immediacy = "immediate"
    }
  ]
}

Which resulted in:

Terraform will perform the following actions:

  # aci_application_epg.test will be updated in-place
  ~ resource "aci_application_epg" "test" {
      ~ application_profile_dn                 = "uni/tn-CAGCommon/ap-AppProf-CAGCommon" -> (known after apply)
      ~ exception_tag                          = null -> (known after apply)
      ~ flood_on_encap                         = "disabled" -> (known after apply)
      ~ fwd_ctrl                               = "none" -> (known after apply)
      ~ has_mcast_source                       = "no" -> (known after apply)
        id                                     = "uni/tn-CAGCommon/ap-AppProf-CAGCommon/epg-EPG-JuniperExt"
      ~ is_attr_based_epg                      = "no" -> (known after apply)
      ~ match_t                                = "AtleastOne" -> (known after apply)
        name                                   = "EPG-JuniperExt"
      ~ pc_enf_pref                            = "unenforced" -> (known after apply)
      ~ pc_tag                                 = "16395" -> (known after apply)
      ~ pref_gr_memb                           = "exclude" -> (known after apply)
      ~ prio                                   = "unspecified" -> (known after apply)
      + relation_fv_rs_aepg_mon_pol            = (known after apply)
      ~ relation_fv_rs_bd                      = "uni/tn-CAGCommon/BD-BD-CAGCommon-JuniperExt" -> (known after apply)
      ~ relation_fv_rs_cons                    = [] -> (known after apply)
      ~ relation_fv_rs_cons_if                 = [] -> (known after apply)
      ~ relation_fv_rs_cust_qos_pol            = "uni/tn-common/qoscustom-default" -> (known after apply)
      + relation_fv_rs_dpp_pol                 = (known after apply)
      ~ relation_fv_rs_fc_path_att             = [] -> (known after apply)
      ~ relation_fv_rs_intra_epg               = [] -> (known after apply)
      ~ relation_fv_rs_prot_by                 = [] -> (known after apply)
      ~ relation_fv_rs_prov                    = [] -> (known after apply)
      ~ relation_fv_rs_sec_inherited           = [] -> (known after apply)
      + relation_fv_rs_trust_ctrl              = (known after apply)
      ~ relation_to_domains                    = [
          - {
              - annotations                   = [] -> null
              - binding_type                  = "none" -> null
              - class_preference              = "encap" -> null
              - deployment_immediacy          = "lazy" -> null
              - enable_netflow                = "disabled" -> null
              - encapsulation                 = "unknown" -> null
              - encapsulation_mode            = "auto" -> null
              - epg_cos                       = "Cos0" -> null
              - epg_cos_pref                  = "disabled" -> null
              - ipam_dhcp_override            = "0.0.0.0" -> null
              - ipam_enabled                  = "no" -> null
              - ipam_gateway                  = "0.0.0.0" -> null
              - netflow_direction             = "both" -> null
              - number_of_ports               = "0" -> null
              - port_allocation               = "none" -> null
              - primary_encapsulation         = "unknown" -> null
              - primary_encapsulation_inner   = "unknown" -> null
              - resolution_immediacy          = "immediate" -> null
              - secondary_encapsulation_inner = "unknown" -> null
              - switching_mode                = "native" -> null
              - tags                          = [] -> null
              - target_dn                     = "uni/phys-PhysDom-PaloAltoFW-Cust" -> null
              - untagged                      = "no" -> null
                # (4 unchanged attributes hidden)
            },
          - {
              - annotation                    = "orchestrator:terraform" -> null
              - annotations                   = [] -> null
              - binding_type                  = "none" -> null
              - class_preference              = "encap" -> null
              - deployment_immediacy          = "lazy" -> null
              - enable_netflow                = "disabled" -> null
              - encapsulation                 = "unknown" -> null
              - encapsulation_mode            = "auto" -> null
              - epg_cos                       = "Cos0" -> null
              - epg_cos_pref                  = "disabled" -> null
              - ipam_dhcp_override            = "0.0.0.0" -> null
              - ipam_enabled                  = "no" -> null
              - ipam_gateway                  = "0.0.0.0" -> null
              - netflow_direction             = "both" -> null
              - number_of_ports               = "0" -> null
              - port_allocation               = "none" -> null
              - primary_encapsulation         = "unknown" -> null
              - primary_encapsulation_inner   = "unknown" -> null
              - resolution_immediacy          = "lazy" -> null
              - secondary_encapsulation_inner = "unknown" -> null
              - switching_mode                = "native" -> null
              - tags                          = [] -> null
              - target_dn                     = "uni/phys-PhysDom-CustPort" -> null
              - untagged                      = "no" -> null
                # (3 unchanged attributes hidden)
            },
          + {
              + annotations                   = []
              + binding_type                  = "none"
              + class_preference              = "encap"
              + deployment_immediacy          = "lazy"
              + enable_netflow                = "disabled"
              + encapsulation                 = "unknown"
              + encapsulation_mode            = "auto"
              + epg_cos                       = "Cos0"
              + epg_cos_pref                  = "disabled"
              + ipam_dhcp_override            = "0.0.0.0"
              + ipam_enabled                  = "no"
              + ipam_gateway                  = "0.0.0.0"
              + netflow_direction             = "both"
              + number_of_ports               = "0"
              + port_allocation               = "none"
              + primary_encapsulation         = "unknown"
              + primary_encapsulation_inner   = "unknown"
              + resolution_immediacy          = "immediate"
              + secondary_encapsulation_inner = "unknown"
              + switching_mode                = "native"
              + tags                          = []
              + target_dn                     = "uni/phys-PhysDom-CustPort"
              + untagged                      = "no"
                # (4 unchanged attributes hidden)
            },
          + {
              + annotation                    = "orchestrator:terraform"
              + annotations                   = []
              + binding_type                  = "none"
              + class_preference              = "encap"
              + deployment_immediacy          = "lazy"
              + enable_netflow                = "disabled"
              + encapsulation                 = "unknown"
              + encapsulation_mode            = "auto"
              + epg_cos                       = "Cos0"
              + epg_cos_pref                  = "disabled"
              + ipam_dhcp_override            = "0.0.0.0"
              + ipam_enabled                  = "no"
              + ipam_gateway                  = "0.0.0.0"
              + netflow_direction             = "both"
              + number_of_ports               = "0"
              + port_allocation               = "none"
              + primary_encapsulation         = "unknown"
              + primary_encapsulation_inner   = "unknown"
              + resolution_immediacy          = "lazy"
              + secondary_encapsulation_inner = "unknown"
              + switching_mode                = "native"
              + tags                          = []
              + target_dn                     = "uni/phys-PhysDom-PaloAltoFW-Cust"
              + untagged                      = "no"
                # (3 unchanged attributes hidden)
            },
            # (1 unchanged element hidden)
        ]
      ~ scope                                  = "2818051" -> (known after apply)
      ~ shutdown                               = "no" -> (known after apply)
        tags                                   = []
        # (32 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

It would therefore seem that this behaviour isn't neccessarilly tied to an import as such, it can happen any time you try to bring relation_to_domains under management by Terraform.

Also, after sleeping on it, I think a possible workaround in my case would be to try to use aci_epg_to_domain instead. That way, there's less opportunity for terraform to get confused about which relation in the current ACI configuration matches the desired configuration under management for terraform. That doesn't solve this bug obviously, but it would at least make it possible for me to move forward with my project. (I'll just have to figure out how to migrate to this method without nuking existing managed relations...)

[akinross]

Hi @pv2b,

This is expected behaviour due to the introduction of default annotation attribute and global setting of this attribute. The provide enforces the default of "orchestrator:terraform" to annotation attributes of any class. In case of importing when this attribute is not set in your terraform configuration it will always attempt to set is based on precedence order. See more explanation: https://registry.terraform.io/providers/CiscoDevNet/aci/latest/docs/guides/migration#changed-annotation-behavior. I would recommend reading this entire guide carefully, since it explain a lot behaviour of new style terraform plugin framework resources inside the ACI provider.

[pv2b]

Hi @pv2b,

This is expected behaviour due to the introduction of default annotation attribute and global setting of this attribute. The provide enforces the default of "orchestrator:terraform" to annotation attributes of any class. In case of importing when this attribute is not set in your terraform configuration it will always attempt to set is based on precedence order. See more explanation: https://registry.terraform.io/providers/CiscoDevNet/aci/latest/docs/guides/migration#changed-annotation-behavior. I would recommend reading this entire guide carefully, since it explain a lot behaviour of new style terraform plugin framework resources inside the ACI provider.

I think there's more to it than this. Why would a mismatch of annotations cause Terraform to propose a plan that ends up deleting and re-creating domain relations, and also ends up changing the resolution_immediacy around between the relationships? Carefully examine the changes it wants to make to the relations.

[shrsr]

@pv2b This is a cosmetic quirk of Terraform when using type set, because:

1. Sets are unordered collections
2. Terraform compares elements by their complete attribute values
3. Even minor differences (like a missing or added annotation field) cause Terraform to display elements as being removed (-) and added (+) in the plan output
4. In reality from the provider's perspective, no actual deletion or recreation occurs, Terraform is simply showing the difference between the old and new representation of the same elements

[pv2b]

@pv2b This is a cosmetic quirk of Terraform when using type set, because:

1. Sets are unordered collections

2. Terraform compares elements by their complete attribute values

3. Even minor differences (like a missing or added annotation field) cause Terraform to display elements as being removed (-) and added (+) in the plan output

4. In reality from the provider's perspective, no actual deletion or recreation occurs, Terraform is simply showing the difference between the old and new representation of the same elements

Again, I don't think you've clearly looked at the changes it's proposing.

Perhaps it'll be more obvious if I filter to show only the "target_dn" and "resolution_immediacy" fields, see below:

# NOTE: THIS OUTPUT HAS BEEN EDITED FOR CLARITY
  ~ resource "aci_application_epg" "test" {
      ~ relation_to_domains                    = [
          - {
              - resolution_immediacy          = "immediate" -> null
              - target_dn                     = "uni/phys-PhysDom-PaloAltoFW-Cust" -> null
            },
          - {
              - resolution_immediacy          = "lazy" -> null
              - target_dn                     = "uni/phys-PhysDom-CustPort" -> null
            },
          + {
              + resolution_immediacy          = "immediate"
              + target_dn                     = "uni/phys-PhysDom-CustPort"
            },
          + {
              + resolution_immediacy          = "lazy"
              + target_dn                     = "uni/phys-PhysDom-PaloAltoFW-Cust"
            },
        ]
    }

See how it's proposing deleting a relation to uni/phys-PhysDom-PaloAltoFW-Cust with resolution_immediacy set to immediate, and then proposing re-creating it again with resolution_immediacy set to lazy. Also, uni/phys-PhysDom-CustPort is changed in a similar way with resolution_immediacy being changed from lazy to immediate.

Now, perhaps you are right in that it wouldn't actually delete and re-create any relations, but it would seem that the "desired state" given by the provider is showing attributes being changed without them being commanded!

[akinross]

Hi @pv2b,
This is expected behaviour due to the introduction of default annotation attribute and global setting of this attribute. The provide enforces the default of "orchestrator:terraform" to annotation attributes of any class. In case of importing when this attribute is not set in your terraform configuration it will always attempt to set is based on precedence order. See more explanation: https://registry.terraform.io/providers/CiscoDevNet/aci/latest/docs/guides/migration#changed-annotation-behavior. I would recommend reading this entire guide carefully, since it explain a lot behaviour of new style terraform plugin framework resources inside the ACI provider.

I think there's more to it than this. Why would a mismatch of annotations cause Terraform to propose a plan that ends up deleting and re-creating domain relations, and also ends up changing the resolution_immediacy around between the relationships? Carefully examine the changes it wants to make to the relations.

Hi @pv2b,

I am not sure what happend on GitHub but when I responded initially your second comment: "Also, to anticipate what might be your next question - applied the plan which resulted in just an import with no changes into the test tfstate." was not present.

What you describe here should indeed not happen, and to me this looks like a bug in the provider during plan modification. If this is the case, then I think this issue is not an import specific issue, but looking at the code appears to be an issue with any update operation. I will need to re-create this issue on Monday to see what is actually happening, and how these values are being switch.

As an alternative workaround, could you try providing the resolution_immediacy as config in the HCL code?

[akinross]

Hi @pv2b, I am able to reproduce your issue but am not able to determine the root cause yet. It appears that for some unknown reason the planned values are getting switched by terraform-plugin-framework logic prior to us doing any plan modifications. I will keep you updated on my findings.