Terraform should automatically re-download providers on init when TF_PLUGIN_CACHE_DIR is removed after initial init

[dee-kryvenko]

Terraform Version

Was able to reproduce it on a variety of versions starting from 0.13.5 and up to the latest.

Terraform Configuration Files

# main.tf
resource "null_resource" "cluster" {}
resource "random_string" "random" {
    length = 5
}

Expected Behavior

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

Actual Behavior

Error: Failed to install provider from shared cache

Error while importing hashicorp/null v3.1.0 from the shared cache directory:
after linking registry.terraform.io/hashicorp/null from provider cache at
/Users/kryvenkd/repos/tmp/tf-plugin-cache it is still not detected in the
target directory; this is a bug in Terraform.

Steps to Reproduce

1. export TF_PLUGIN_CACHE_DIR=~/repos/tmp/tf-plugin-cache
2. tf init
3. rm -rf ${TF_PLUGIN_CACHE_DIR}/registry.terraform.io/hashicorp/random
4. tf init

This is tricky one to reproduce - there must be at least two providers at play, and also deleting null provider doesn't reproduce this - but other providers does.

Additional Context

The goal here is basically to do as explained here https://learn.hashicorp.com/tutorials/terraform/automate-terraform#plan-and-apply-on-different-machines.
In my CI I archive everything from the workspace during the plan and then restore for apply. The trick is TF_PLUGIN_CACHE_DIR volume might be different than have used during the plan, and it might not have providers that were previously installed during the plan. That resulting in .terraform/providers having broken symlinks. I think TF should not freak out if the symlink leads to nowhere but simply fallback to re-downloading the provider.

[mildwonkey]

Hi @dee-kryvenko ,
If I'm reading this correctly, your request here is to change terraform's behavior when the local .terraform directory has broken symlinks to the plugin cache dir. You would rather it automatically re-download providers instead of emit an error.

This is an intentional design choice, so I am going to re-label this issue as an enhancement and change the title a bit to reflect that.

Thanks!

EDIT: I should add that the usual workflow after doing something like removing the entire plugin cache dir is to also remove the .terraform directory. It's an error because we tend to assume that, in this scenario, the user would want to fix the problem with the local directory system themselves instead of having Terraform arbitrarily remove links (we can't know, in every situation, why the links are broken and what the correct way of fixing it is).

[dee-kryvenko]

This is an intentional design choice

I'm sorry but this is contradicting to

this is a bug in Terraform.

EDIT: I should add that the usual workflow after doing something like removing the entire plugin cache dir is to also remove the .terraform directory. It's an error because we tend to assume that, in this scenario, the user would want to fix the problem with the local directory system themselves instead of having Terraform arbitrarily remove links (we can't know, in every situation, why the links are broken and what the correct way of fixing it is).

Honestly I'd say you are assuming too much about the typical user. These symlinks in the .terraform/providers are implementation details and should be treated as a black box by a typical user. A typical user expects that if they point TF to the certain directory in the TF_PLUGIN_CACHE_DIR - TF will reuse what's in there and download what's missing. A typical user wouldn't know these symlinks exists. As simple as that.

I want also note there was no removing or manual alterations to the cache dir in my scenario. I run CI/CD in K8s. For TF it basically creates a pod for tf plan, destroys a pod, presenting user with the diff in the plan and asking for approval, then upon approval creates a new pod to run tf apply. I did as it says here https://learn.hashicorp.com/tutorials/terraform/automate-terraform#plan-and-apply-on-different-machines and turns out this document is misleading because it doesn't mention anything specific about .terraform folder. If you want to use TF_PLUGIN_CACHE_DIR in such a k8s setup like above - you use a pool of reclaimable PVs. And there is no guarantee whatsoever that the same PV will happen to be mounted by both plan and apply pods. It might so happen that the one claimed by the apply pod just never had some of the plugins in it. I'd say this is a pretty typical scenario.

To sum up - misleading error message suggesting I ran into a bug in TF, misleading documentation that directs me to archive entire .terraform folder and a typical user who doesn't know internal implementation details and contents of this .terraform folder - I view this as a bug, not as an enhancement request.

[mildwonkey]

👍 that's fair, I'll change it back!

[Nowaker]

Fresh TF install on a new system:

% mkdir -p $TF_PLUGIN_CACHE_DIR

% ti

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v3.47.0...
╷
│ Error: Failed to install provider
│
│ Error while installing hashicorp/aws v3.47.0: the current package for registry.terraform.io/hashicorp/aws 3.47.0 doesn't match any of the checksums previously recorded in the
│ dependency lock file
╵

% terraform providers lock -platform=darwin_amd64 -platform=darwin_arm64 -platform=linux_amd64

- Fetching hashicorp/aws 3.47.0 for darwin_amd64...
╷
│ Error: Could not retrieve providers for locking
│
│ Terraform failed to fetch the requested providers for darwin_amd64 in order to calculate their checksums: some providers could not be installed:
│ - registry.terraform.io/hashicorp/aws: the current package for registry.terraform.io/hashicorp/aws 3.47.0 doesn't match any of the checksums previously recorded in the
│ dependency lock file.

% unset $TF_PLUGIN_CACHE_DIR

% terraform providers lock -platform=darwin_amd64 -platform=darwin_arm64 -platform=linux_amd64

- Fetching hashicorp/aws 3.47.0 for darwin_amd64...
╷
│ Error: Could not retrieve providers for locking
│
│ Terraform failed to fetch the requested providers for darwin_amd64 in order to calculate their checksums: some providers could not be installed:
│ - registry.terraform.io/hashicorp/aws: the current package for registry.terraform.io/hashicorp/aws 3.47.0 doesn't match any of the checksums previously recorded in the
│ dependency lock file.
╵

Totally unusable, regardless of whether $TF_PLUGIN_CACHE_DIR is set or not.

Missing the old Terraform, 0.13 and below.

[adk-swisstopo]

Five years later, I am hitting this as well.

For anyone who is confused, the symlinks are deeper than you might expect below the version level. E.g. in .terraform/providers/registry.terraform.io/hashicorp/aws/1.2.3/linux_amd64 -> ~/.terraform.d/plugin-cache/registry.terraform.io/hashicorp/aws/1.2.3/linux_amd64.

[apparentlymart]

FWIW, the advice in Plan and Apply on different machines was originally written long before Terraform supported the global plugin cache directory, so the text there is not considering that the .terraform/providers directory might contain symlinks rather than direct package directories.

Separately from whatever changes in Terraform in response to this issue, it might be worth updating the docs to also note that if a plugin cache directory is enabled during planning then it should also be enabled and referring to the same directory during apply, or else the cache won't be effective.