new ephemeral resource: local_file

[bschaatsbergen]

Background

In Terraform 1.10, we introduced a new provider language construct: ephemeral, enabling provider plugin developers to create resources that do not persist any value in the Terraform state. Ephemeral resources expect all their dependencies to exist and are executed in both the plan and apply cycles. Additionally, 1.10 introduced the ability to reference ephemeral resources and values (variables) within provider and provisioner blocks. Building on this, in Terraform 1.11, we are prototyping write-only attributes, which would allow users to reference an ephemeral attribute directly in a resource’s input argument (as shown in the use cases below).

The purpose of this issue is to serve as a public record to exchange ideas. A possible outcome could be deciding not to implement this.

Use Cases or Problem Statement

The addition of an ephemeral local_file resource would provide a secure way to read sensitive files during Terraform plan and apply cycles, enabling users to reference file content directly in other language constructs (e.g. provider or resource blocks) while minimizing the risk of exposing sensitive data in the Terraform state. This ephemeral resource would simplify user workflows by eliminating the need for external tools or manual cleanup for temporary sensitive files.

Why this ephemeral resource?

While many resources require file content as input, there are valid reasons to avoid embedding large file content in your Terraform state.

Added, from a security best practices standpoint, it's strongly recommended that secrets be read and written through secret management solutions like HashiCorp Vault, rather than from the local filesystem. While educating users and guiding them away from reading secrets from their local filesystem is the ideal approach, I recognize that this transition won’t happen overnight. Introducing an ephemeral resource as a replacement for their current data source can help mitigate the immediate risk of sensitive data being exposed in the Terraform state, while also moving users one step closer to adopting a more secure approach.

Use cases

Writing a secret to HashiCorp Vault:

ephemeral "local_file" "example" {
  filename = "${path.module}/secret.json"
}

resource "vault_generic_secret" "example" {
  path          = "secret/example"
  data_json = ephemeral.local_file.example.content
}

Writing a secret to AWS Secret Manager:

ephemeral "local_file" "example" {
  filename = "${path.module}/secret.json"
}

resource "aws_secretsmanager_secret_version" "example" {
  secret_id     = aws_secretsmanager_secret.example.id
  secret_string = ephemeral.local_file.example.content
}

Writing a secret through other provider resources: kubernetes_secret_v1, google_secret_manager_secret_version, etc.

Proposal

Introduce an ephemeral resource: local_file with a similar implementation to the existing local_file data source.

How much impact is this issue causing?

Medium

Additional Information

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[ned1313]

What would the advantage be over using the file or templatefile functions to read data into a write-only attribute?

[novekm]

@ned1313 @bschaatsbergen This would be helpful for me as well. Happy to try to take a stab at adding this functionality to the provider, though I'm pretty freshly exposed to the new ephemeral functionality (just started looking into it a few days ago).

In my case, I'm trying to allow users to pass values for input variables into a module, and the module would do the following:

1. Either accept a completed local file (using file()) and upload to S3
2. Or, dynamically create the file (a somewhat lengthy, but predictable php file that includes key/value pairs used for configuration), injecting the variables into the file where necessary as those values. Upload this file to S3.
3. This file (config.php) would be consumed by ECS (pulled from S3 with a sidecar container that does an aws s3 cp command) to configure a service running on it (Perforce Helix Swarm). With Helix Swarm, it runs based on the configuration defined in the config.php file, so when the sidecar copies it to the helix-swarm container, it would use the new values and continuously update/reconfigure whenever the local file is changed (and new object is uploaded to S3).

I haven't extensively used templatefile however local_file has bit a bit more useful for things I've done in the past. Also, in this scenario local_file helps keep the code a lot more readable/understandable for the project (open source, active contributions) than templatefile.

[ArtsiomAntropau]

I have one more use case.

We have some providers which are expecting file to be, but file may contain some sensitive info which we don't want to save forever.

For an example config_path

provider "kubernetes" {
  config_path    = "~/.kube/config"
}

https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#example-usage

so this kubeconfig may be received as an ephemeral data source from some external provider, but must be written to disk so having ephemeral local_sensitive_file here will be a great feature.

[jcmcken]

Another use case: Terraform Cloud or Terraform Enterprise

In these cases, the thing executing the plans is remote and ephemeral itself. Therefore writing local files and maintaining state about them doesn't make a lot of sense, since every single plan would need to recreate the file and update the state unnecessarily.

In my case, we have a complex, multi-account AWS environment. I need to temporarily write a ~/.aws/config to set the right IAM role to use in a particular profile used by a particular shell resource. The configuration for this IAM role cannot live in the environment, it's only supported in the config file.

I'm sure there are many use cases like this -- take some info that Terraform has and write it temporarily to some file for the duration of the plan/apply sequence.

[Vokunne]

Hi folks,
It'd be a really useful feature 'cause sometimes you ought to operate with such sensitive items like private keys and certificates. My case is importing certs to Fortinet devices. I can't wait for it.