Issue importing resources when upgraded to v1.14.0 from v1.13.5.

[tomelliot16]

Terraform Version

v1.14.0

Terraform Configuration Files

...terraform config...

Debug Output

does not work on v1.14.0

data.external.git_version: Reading...
data.external.git_version: Read complete after 0s [id=-]
aws_vpc.default: Importing from ID "vpc-HIDDEN"...
data.aws_api_gateway_rest_api.probe: Reading...
data.aws_acm_certificate.issued: Reading...
module.target_account.data.aws_region.current: Reading...
module.target_account.data.aws_caller_identity.current: Reading...
aws_vpc.default: Import prepared!
  Prepared aws_vpc for import
aws_vpc.default: Refreshing state... [id=vpc-HIDDEN]
module.target_account.data.aws_region.current: Read complete after 0s [id=us-east-1]
module.target_account.data.aws_caller_identity.current: Read complete after 0s [id=HIDDEN]
data.aws_api_gateway_rest_api.probe: Read complete after 0s [id=HIDDEN]
data.aws_acm_certificate.issued: Read complete after 0s [id=arn:aws:acm:us-east-1:HIDDEN:certificate/HIDDEN]
module.terraform_state_account.data.aws_caller_identity.current: Reading...
module.terraform_state_account.data.aws_region.current: Reading...
module.terraform_state_account.data.aws_region.current: Read complete after 0s [id=us-east-1]
module.terraform_state_account.data.aws_caller_identity.current: Read complete after 0s [id=HIDDEN]
╷
│ Error: Invalid for_each argument
│
│   on ../modules/peer/main.tf line 23, in resource "aws_route" "ipv4_private_routes":
│   23:   for_each = var.destination_cidr_block != "" ? var.private_route_table_ids : {}
│     ├────────────────
│     │ var.destination_cidr_block is "10.1.0.0/16"
│     │ var.private_route_table_ids is a map of string, known only after apply
│
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.
╵

╷
│ Error: Invalid for_each argument
│
│   on ../modules/private_dns_zone/main.tf line 12, in resource "aws_route53_record" "private":
│   12:   for_each = {
│   13:     for record in var.records :
│   14:     record.name => record
│   15:   }
│     ├────────────────
│     │ var.records is a list of object, known only after apply
│
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.
╵

works fine on on v1.13.5

terraform import -var-file input.json aws_vpc.default vpc-HIDDEN
data.external.git_version: Reading...
data.external.git_version: Read complete after 0s [id=-]
module.terraform_state_account.data.aws_caller_identity.current: Reading...
module.terraform_state_account.data.aws_region.current: Reading...
module.terraform_state_account.data.aws_region.current: Read complete after 0s [id=us-east-1]
module.terraform_state_account.data.aws_caller_identity.current: Read complete after 0s [id=HIDDEN]
aws_vpc.default: Importing from ID "HIDDEN"...
data.aws_api_gateway_rest_api.probe: Reading...
data.aws_acm_certificate.issued: Reading...
aws_vpc.default: Import prepared!
  Prepared aws_vpc for import
module.target_account.data.aws_caller_identity.current: Reading...
aws_vpc.default: Refreshing state... [id=vpc-HIDDEN]
module.target_account.data.aws_region.current: Reading...
module.target_account.data.aws_region.current: Read complete after 0s [id=us-east-1]
module.target_account.data.aws_caller_identity.current: Read complete after 0s [id=HIDDEN]
data.aws_api_gateway_rest_api.probe: Read complete after 0s [id=HIDDEN]
data.aws_acm_certificate.issued: Read complete after 1s [id=arn:aws:acm:us-east-1:HIDDEN:certificate/HIDDEN]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

Expected Behavior

As seem in the above v1.13.5 the resource should have successfully been imported

Actual Behavior

It fails given an error which is not helpful as we need these resources to be dynamic.

Steps to Reproduce

terraform import aws_vpc.default vpc-123

Additional Context

terraform apply works fine in both versions. I believe the issue is when you are importing and tfstate is not fully flushed out.

References

No response

Generative AI / LLM assisted development?

No response

[jbardin]

Hi @tomelliot16,

Thanks for filing the issue. Can you supply a configuration to reproduce the issue? Without a minimal reproducible example it's going to be difficult to pinpoint what might have changed to block your import command from working.

Thanks!

[tomelliot16]

Okay I was able to strip it down

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.99.1"
    }
  }
}

variable "azs" {
  type = map(string)
  default = {
    "us-east-1a" = "us-east-1a"
    "us-east-1b" = "us-east-1b"
  }
}

resource "aws_vpc" "subject" {
    cidr_block                       = "10.0.0.0/16"
    enable_dns_hostnames             = true
    enable_dns_support               = true
}

resource "aws_route_table" "private" {
  for_each = var.azs
  vpc_id = aws_vpc.subject.id
}

resource "aws_internet_gateway" "gw" {
    vpc_id = aws_vpc.subject.id
}

resource "aws_route" "route" {
  for_each = aws_route_table.private
  route_table_id = each.value.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id = aws_internet_gateway.gw.id
}

This will require a vpc to be created I just ran it and then removed the statefile

> terraform init
> terraform version
Terraform v0.14.0
+ provider registry.terraform.io/hashicorp/aws v5.99.1

> terraform import aws_vpc.subject vpc-hidden
aws_vpc.subject: Importing from ID "vpc-hidden"...
aws_vpc.subject: Import prepared!
  Prepared aws_vpc for import
aws_vpc.subject: Refreshing state... [id=vpc-hidden]
╷
│ Error: Invalid for_each argument
│ 
│   on main.tf line 34, in resource "aws_route" "route":
│   34:   for_each = aws_route_table.private
│     ├────────────────
│     │ aws_route_table.private will be known only after apply
│ 
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.
│ 
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│ 
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.
╵

Then remove the statefile (important) and try with the older version

> terraform init
> terraform version
Terraform v0.13.7
+ provider registry.terraform.io/hashicorp/aws v5.99.1

> terraform import aws_vpc.subject vpc-hidden
aws_vpc.subject: Importing from ID "vpc-hidden"...
aws_vpc.subject: Import prepared!
  Prepared aws_vpc for import
aws_vpc.subject: Refreshing state... [id=vpc-hidden]

Import successful!                                                                                                                                                                                                                                                              
                                                                                                                                                                                                                                                                                
The resources that were imported are shown above. These resources are now in                                                                                                                                                                                                    
your Terraform state and will henceforth be managed by Terraform.

[jbardin]

Hi @tomelliot16,

Thanks for the example. The issue was created referencing v1.13.5, but your repro here uses v0.13 and v0.14. Those are very old versions of Terraform, and don't represent what is supported today. Are you certain this worked with v1.13?

[tomelliot16]

Interesting yeah that was a total typo. And yes I run with v1.13.5 I cannot import successful with the above. What's crazy I know that using a large terraform project that I'm trying to simplify from we did succeed for months with v1.13/v1.12 and are able to run it as seen in the ticket description. I will keep digging into it. Funny enough I may have found a workaround for v1.14.0 but I still don't understand why it seems to work in v1.13.5.

[stefano-franco]

I can also successfully reproduce this bug without creating any resources (requires an aws account though).
this is a simplified code snippet which replicates the bug:

provider "aws" {
  region = "eu-central-1"
}

# ------------------------------------------------------------------------------------------------------
# ¦ LOCALS
# ------------------------------------------------------------------------------------------------------
locals {
  increase_aws_service_quotas = {
    # NOTE: if we set the value to 0 the import will fail with terraform 1.14.0 due to a bug
    # with < 1.14.0 the exact same code will successfully import resources
    # if we set the value to > 0 the import works fine also with 1.14.0
    "codebuild_concurrent_runs_linux_small" = 0
  }
  service_quota_mapping = {
    "codebuild_concurrent_runs_linux_small" = { 
      service_code = "codebuild", 
      quota_name = "Concurrently running builds for Linux/Small environment" 
    }
  }
}

# ------------------------------------------------------------------------------------------------------
# ¦ BUG REPRODUCTION
# ------------------------------------------------------------------------------------------------------
data "aws_servicequotas_service_quota" "servicequotas" {
  for_each = {
    for quota_key, quota_value in local.increase_aws_service_quotas : quota_key => quota_value
    # only fetch quotas for which an increase is requested
    # NOTE: this condition is causing the import bug with terraform 1.14.0 when the condition is false
    if quota_value > 0
  }

  quota_name   = local.service_quota_mapping[each.key].quota_name
  service_code = local.service_quota_mapping[each.key].service_code
}

resource "aws_servicequotas_service_quota" "servicequotas" {
  for_each = {
    for quota_key, quota_value in local.increase_aws_service_quotas : quota_key => quota_value
    # only request increases for quotas where the requested value is higher than the current value
    if quota_value > try(data.aws_servicequotas_service_quota.servicequotas[quota_key].value, 999999)
  }

  quota_code   = data.aws_servicequotas_service_quota.servicequotas[each.key].quota_code
  service_code = data.aws_servicequotas_service_quota.servicequotas[each.key].service_code
  value        = each.value
}

# ------------------------------------------------------------------------------------------------------
# ¦ DUMMY RESOURCE TO IMPORT
# ------------------------------------------------------------------------------------------------------
resource "aws_ebs_encryption_by_default" "test" {
  enabled = true
}

# To reproduce the bug:
# terraform import aws_ebs_encryption_by_default.test default

# To clean up the state after reproducing the bug:
# terraform state rm aws_ebs_encryption_by_default.test

setting the local increase_aws_service_quotas.codebuild_concurrent_runs_linux_small to 0 will fail:

❯ terraform version
Terraform v1.14.0
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v6.23.0

❯ terraform import aws_ebs_encryption_by_default.test default
aws_ebs_encryption_by_default.test: Importing from ID "default"...
aws_ebs_encryption_by_default.test: Import prepared!
  Prepared aws_ebs_encryption_by_default for import
aws_ebs_encryption_by_default.test: Refreshing state... [id=default]
╷
│ Error: Invalid for_each argument
│
│   on /debug.tf line 39, in resource "aws_servicequotas_service_quota" "servicequotas":
│   39:   for_each = {
│   40:     for quota_key, quota_value in local.increase_aws_service_quotas : quota_key => quota_value
│   41:     # only request increases for quotas where the requested value is higher than the current value
│   42:     if quota_value > try(data.aws_servicequotas_service_quota.servicequotas[quota_key].value, 999999)
│   43:   }
│     ├────────────────
│     │ data.aws_servicequotas_service_quota.servicequotas will be known only after apply
│     │ local.increase_aws_service_quotas is object with 1 attribute "codebuild_concurrent_runs_linux_small"
│
│ The "for_each" map includes keys derived from resource attributes that cannot be determined until apply, and so Terraform cannot determine the full set of keys that will identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to define the map keys statically in your configuration and place apply-time results only in the map values.
│
│ Alternatively, you could use the -target planning option to first apply only the resources that the for_each value depends on, and then apply a second time to fully converge.
╵

setting the local increase_aws_service_quotas.codebuild_concurrent_runs_linux_small to 1 will succeed:

❯ terraform version
Terraform v1.14.0
on darwin_arm64
+ provider registry.terraform.io/hashicorp/aws v6.23.0

❯ terraform import aws_ebs_encryption_by_default.test default
aws_ebs_encryption_by_default.test: Importing from ID "default"...
data.aws_servicequotas_service_quota.servicequotas["codebuild_concurrent_runs_linux_small"]: Reading...
aws_ebs_encryption_by_default.test: Import prepared!
  Prepared aws_ebs_encryption_by_default for import
aws_ebs_encryption_by_default.test: Refreshing state... [id=default]
data.aws_servicequotas_service_quota.servicequotas["codebuild_concurrent_runs_linux_small"]: Read complete after 0s [id=arn:aws:servicequotas:eu-central-1::codebuild/L-9D07B6EF]

Import successful!

The resources that were imported are shown above. These resources are now in
your Terraform state and will henceforth be managed by Terraform.

❯ terraform state rm aws_ebs_encryption_by_default.test
Removed aws_ebs_encryption_by_default.test
Successfully removed 1 resource instance(s).

[jbardin]

Note that it's not surprising that this doesn't work, I wouldn't expect it to. What I'm more interested in is if there was some form of this which recently stopped working, because there shouldn't have been any changes affecting import between 1.13 and 1.14.

The fact that there was a change in behavior between 0.13 and 0.14 is expected, because that's about when we started working in planed imports, so that resources could be imported into more complex configurations which could not work with the import command.

Using the current method of planning imports with import blocks in the configuration will avoid the problems shown here. Any time you need to import multiple resources at once, or resources that participate in expansion (count, for_each) will need to be imported in this way. The old import command was maintained best we could for backwards compatibility, but it cannot work with all configurations.

[stefano-franco]

the error message is a bit strange:
│ data.aws_servicequotas_service_quota.servicequotas will be known only after apply
this statement shouldn't be true. the data source is known before apply and a terraform plan also works without any issues.

I can confirm that using the import block statement works with v1.14.0 so this is a specific issue only for the terraform import command.

works with terraform v1.14.0:

import {
  to = aws_ebs_encryption_by_default.test
  id = "default"
}

fails with terraform v1.14.0:

terraform import aws_ebs_encryption_by_default.test default

maybe this is related to #37515?

[jbardin]

│ data.aws_servicequotas_service_quota.servicequotas will be known only after apply
this statement shouldn't be true. the data source is known before apply and a terraform plan also works without any issues.

The terraform import command can't do a complete plan, which is why it was replaced with the more declarative import block system. We try to maintain the existing import command as closely as we can, but most improvements require working within a full plan.

#37515 shouldn't change anything, as that is an optional call which isn't yet implemented by providers. But that's why I'm keeping this open for a reproduction though, because we don't want to inadvertently break anything ;) If there is an identical state+config which works on v1.13 and not on v1.14, I'd at least like to confirm what the change was as it might have impacts in other areas too.