Error waiting for Creating OrganizationSecurityPolicy

[BA-PoC]

TL;DR

While trying to deploy the terraform foundation, we are facing an error on Step 3-networks-dual-svpc.
We followed all the steps including exporting all the variables and granted all necessary permissions to the serviceAccount:

• ${ORGANIZATION_ID}
• ${ACCESS_CONTEXT_MANAGER_ID}
• ${CICD_PROJECT_ID}
• ${GOOGLE_IMPERSONATE_SERVICE_ACCOUNT}

On step 20.Run apply shared we are not able to create the google_compute_organization_security_policy. I mean the policy is created but seems not able to associate all the folders to the policy, failing after 21 retries everytime.

Expected behavior

Able to run the module ../../modules/hierarchical_firewall_policy properly without issue.

Observed behavior

module.hierarchical_firewall_policy.random_string.suffix: Creating...
module.hierarchical_firewall_policy.random_string.suffix: Creation complete after 0s [id=vo49]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Creating...
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [10s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [20s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [30s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [40s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [50s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [1m0s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [1m10s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [1m20s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [1m30s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [1m40s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [1m50s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [2m0s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [2m10s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [2m20s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [2m30s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [2m40s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [2m50s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [3m0s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [3m10s elapsed]
module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy: Still creating... [3m20s elapsed]

Error: Error waiting to create OrganizationSecurityPolicy: Error waiting for Creating OrganizationSecurityPolicy: couldn't find resource (21 retries)

  with module.hierarchical_firewall_policy.google_compute_organization_security_policy.policy,
  on ../../modules/hierarchical_firewall_policy/main.tf line 27, in resource "google_compute_organization_security_policy" "policy":
  27: resource "google_compute_organization_security_policy" "policy" {

There's more interesting information after adding DEBUG mode:

2025-01-10T16:07:18.552Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5: 2025/01/10 16:07:18 [DEBUG] Dismissed an error as retryable. Retry 404s for GET operation - googleapi: Error 404: The resource 'projects/null' was not found, notFound

2025-01-10T16:07:18.552Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5: 2025/01/10 16:07:18 [DEBUG] Dismissed retryable error on GET operation "org-[ORG_ID_REDACTED]-[UID_REDACTED]": googleapi: Error 404: The resource 'projects/null' was not found, notFound

2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5: {
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:   "error": {
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:     "code": 404,
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:     "message": "The resource 'projects/null' was not found",
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:     "errors": [
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:       {
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:         "message": "The resource 'projects/null' was not found",
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:         "domain": "global",
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:         "reason": "notFound"
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:       }
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:     ],
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:     "status": "NOT_FOUND"
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5:   }
2025-01-10T16:07:49.018Z [DEBUG] provider.terraform-provider-google-beta_v6.15.0_x5: }

As we can see above, the error message fails at looking for a project, however in the hierarchical_firewall.tf file the module is looking for folders, not projects:

module "hierarchical_firewall_policy" {
  source = "../../modules/hierarchical_firewall_policy/"

  parent = local.common_folder_name
  name   = "common-firewall-rules"
  associations = [
    local.common_folder_name,
    local.network_folder_name,
    local.bootstrap_folder_name,
    local.development_folder_name,
    local.production_folder_name,
    local.nonproduction_folder_name,
  ]

Terraform Configuration

We are using this fork https://github.com/Cloud-Gouvernance-Foundation/terraform-example-foundation configured with our environment.

Terraform Version

<>/$ terraform version
Terraform v1.5.7
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v6.15.0
+ provider registry.terraform.io/hashicorp/google-beta v6.15.0
+ provider registry.terraform.io/hashicorp/random v3.6.3

Your version of Terraform is out of date! The latest version
is 1.10.4. You can update by downloading from https://www.terraform.io/downloads.html

### Additional information

UTC -4 timezone

[BA-PoC]

We found something interesting in DEBUG:

{
   "kind": "compute#operation",
   "id": "[REDACTED]",
   "name": "org-[ORG_ID_REDACTED]-[UID_REDACTED]",
   "operationType": "createFirewallPolicy",
   "status": "RUNNING",
   "user": "[SA_REDACTED]",
   "progress": 0,
   "insertTime": "2025-01-10T08:43:12.968-08:00",
   "startTime": "2025-01-10T08:43:12.972-08:00",
   "warnings": [
     {
       "code": "DEPRECATED_RESOURCE_USED",
       "message": "compute.organizationSecurityPolicies.create is deprecated. Please use compute.firewallPolicies.create instead."
     }
   ],
   "selfLink": "https://www.googleapis.com/compute/beta/locations/global/operations/org-[ORD_ID_REDACTED]-[UID_REDACTED]"
 }

With this deprecated error in mind, we took a look at the running version of the module/hierarchical_firewall_policy/versions.tf :

terraform {
  required_version = ">= 0.13"
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = ">= 3.50"
    }
    google-beta = {
      source  = "hashicorp/google-beta"
      version = ">= 3.50"
    }
    random = {
      source = "hashicorp/random"
    }
  }

Despite the effort to change the module version to 3.82, that doesnt fixed our issue.

[daniel-cit]

See #1354

[BA-PoC]

Thanks @daniel-cit , we applied the version and api method change.
But seems the error persist:

module.hierarchical_firewall_policy.google_compute_firewall_policy.policy: Creating...

Error: Error creating FirewallPolicy: googleapi: Error 404: The resource 'projects/null' was not found, notFound

  with module.hierarchical_firewall_policy.google_compute_firewall_policy.policy,
  on ../../modules/hierarchical_firewall_policy/main.tf line 27, in resource "google_compute_firewall_policy" "policy":
  27: resource "google_compute_firewall_policy" "policy" {

From the official doc both, the org or folder are supported. However, for some reason we dont understand why the debug logs point to a project not found issue.

module "hierarchical_firewall_policy" {
  source = "../../modules/hierarchical_firewall_policy/"

  parent = local.common_folder_name
  name   = "common-firewall-rules"
  associations = [
    local.common_folder_name,
    local.network_folder_name,
    local.bootstrap_folder_name,
    local.development_folder_name,
    local.production_folder_name,
    local.nonproduction_folder_name,
  ]

debug level logs:

2025-01-17T14:24:52.720Z [DEBUG] State storage *remote.State declined to persist a state snapshot
2025-01-17T14:24:52.721Z [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2025-01-17T14:24:52.722Z [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/random/3.6.3/linux_amd64/terraform-provider-random_v3.6.3_x5 pid=3066
2025-01-17T14:24:52.722Z [DEBUG] provider: plugin exited
2025-01-17T14:24:53.064Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [INFO] Authenticating using DefaultClient...
2025-01-17T14:24:53.064Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [INFO]   -- Scopes: [https://www.googleapis.com/auth/cloud-platform https://www.googleapis.com/auth/userinfo.email]
2025-01-17T14:24:53.070Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG] Waiting for state to become: [success]
2025-01-17T14:24:53.131Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [INFO] Terraform is configured with service account impersonation, original identity: <USER_ACCOUNT_REDACTED>, impersonated identity: <SERVICE_ACCOUNT_REDACTED>
2025-01-17T14:24:53.141Z [INFO]  Starting apply for module.hierarchical_firewall_policy.google_compute_firewall_policy.policy
2025-01-17T14:24:53.142Z [DEBUG] module.hierarchical_firewall_policy.google_compute_firewall_policy.policy: applying the planned Create change
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [RequestID:b0wvvzfn] Beginning ApplyFirewallPolicy...
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG][DCL INFO] [RequestID:b0wvvzfn]  User specified desired state: {
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Name: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Id: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  CreationTimestamp: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Description: "",
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Fingerprint: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  SelfLink: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  SelfLinkWithId: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  RuleTupleCount: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  ShortName: "common-firewall-rules-<REDACTED>",
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Parent: "folders/<REDACTED>",
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: }
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [RequestID:b0wvvzfn] Fetching initial state...
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG][DCL INFO] [RequestID:b0wvvzfn]  Created plan: []compute.firewallPolicyApiOperation{(*compute.createFirewallPolicyOperation)(0xc00011fea8)}
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG][DCL INFO] [RequestID:b0wvvzfn]  Performing operation *compute.createFirewallPolicyOperation &{response:map[]}
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG][DCL INFO] Attempting to create {
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Name: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Id: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  CreationTimestamp: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Description: "",
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Fingerprint: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  SelfLink: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  SelfLinkWithId: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  RuleTupleCount: nil,
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  ShortName: "common-firewall-rules-<REDACTED>",
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  Parent: "folders/<REDACTED>",
2025-01-17T14:24:53.144Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: }
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG][DCL INFO] [RequestID:b0wvvzfn]  Google API Request: (id vbzdg)
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: -----------[REQUEST]----------
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: POST /compute/v1/locations/global/firewallPolicies?alt=json&parentId=folders%<REDACTED>HTTP/1.1
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Host: compute.googleapis.com
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: User-Agent: Terraform/1.5.7 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/6.16.0 blueprints/terraform/terraform-example-foundation:hierarchical_firewall_policy/v4.1.0 DeclarativeClientLib/0.0.1
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Content-Length: 74
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Content-Type: application/json
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Accept-Encoding: gzip
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: {"parent":"folders/<REDACTED>","shortName":"common-firewall-rules-<REDACTED>"}
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: -------[END REQUEST]--------
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG] Retry Transport: starting RoundTrip retry loop
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG] Retry Transport: request attempt 0
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG] Google API Request Details:
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: ---[ REQUEST ]---------------------------------------
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: POST /compute/v1/locations/global/firewallPolicies?alt=json&parentId=folders%2F<REDACTED> HTTP/1.1
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Host: compute.googleapis.com
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: User-Agent: Terraform/1.5.7 (+https://www.terraform.io) Terraform-Plugin-SDK/2.33.0 terraform-provider-google/6.16.0 blueprints/terraform/terraform-example-foundation:hierarchical_firewall_policy/v4.1.0 DeclarativeClientLib/0.0.1
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Content-Length: 74
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Content-Type: application/json
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Accept-Encoding: gzip
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: {
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  "parent": "folders/<REDACTED>",
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:  "shortName": "common-firewall-rules-<REDACTED>"
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: }
2025-01-17T14:24:53.145Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: -----------------------------------------------------
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 2025/01/17 14:24:53 [DEBUG] Google API Response Details:
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: ---[ RESPONSE ]--------------------------------------
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: HTTP/2.0 404 Not Found
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Content-Type: application/json; charset=UTF-8
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Date: Fri, 17 Jan 2025 14:24:53 GMT
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Server: ESF
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Vary: Origin
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Vary: X-Origin
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: Vary: Referer
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: X-Content-Type-Options: nosniff
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: X-Frame-Options: SAMEORIGIN
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: X-Xss-Protection: 0
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: {
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:   "error": {
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:     "code": 404,
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:     "message": "The resource 'projects/null' was not found",
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:     "errors": [
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:       {
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:         "message": "The resource 'projects/null' was not found",
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:         "domain": "global",
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:         "reason": "notFound"
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:       }
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:     ],
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:     "status": "NOT_FOUND"
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5:   }
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: }
2025-01-17T14:24:53.420Z [DEBUG] provider.terraform-provider-google_v6.16.0_x5: 

[BA-PoC]

Long story short;

terraform state list | grep firewallPolicies
terraform state rm (the results from the grep above)
terraform plan
terraform apply

Unfortunately, I'm still not sure why this happened, but the issue has been resolved by deleting the impacted states.