New `local_exec` data source

[mwarkentin]

There are several issues (hashicorp/terraform-provider-external#13, hashicorp/terraform#12256, hashicorp/terraform#13991) over the past few years about issues dealing with the external data source's support for complex JSON or other types of program output. @apparentlymart had a potential design in a comment on terraform-provider-external which recommended opening an issue on the local provider, so I'm doing so here. His comments are below:

This provider was originally written before Terraform had a jsondecode function. I agree that with that function now present it would be better to have a new data source that can just run a program and capture its output as a string.

I would suggest doing that as a new data source in the local provider though, not as an extension of the external data source, because the external data source was designed for use with programs tailored to work with Terraform (as a middle-ground to allow writing small glue programs rather than entire new Terraform providers), not for executing arbitrary existing software with no modifications.

As an example, the new data source might look like this:

data "local_exec" "example" {
  program = ["example", "program", "generating", "csv"]
}

output "example" {
  value = csvdecode(data.local_exec.example.stdout)
}

As with external, it would be important to use this only for programs that don't have externally-visible side-effects, because the program would be run during the planning phase rather than the apply phase. But unlike external it would impose no constraints on the output except that it be UTF-8 encoded (because Terraform language strings are Unicode, not raw bytes) and leave the user to decide how and whether to parse it.

I don't work directly on either the external or the local providers, so I'm suggesting the above just as a design starting point, and I can't promise it would be accepted exactly like that. If you're interested in working on such a thing I'd suggest starting by opening an issue in the local provider repository to describe what you're intending to do and get feedback from the team that maintains that provider before doing work on it, in case there are design constraints for that provider that I'm not considering.

Originally posted by @apparentlymart in hashicorp/terraform-provider-external#13 (comment)