Consul module fails to read the aws provider credentials

[SahanaPrabhakar]

My configuration file looks like below

provider "aws" {
  access_key = "${var.access_key}"
  secret_key = "${var.secret_key}"
  region     = "eu-west-2"
  version = "~> 1.10"
}

module "consul" {
  source = "hashicorp/consul/aws"
  version = "0.1.2"
  aws_region  = "eu-west-2"
  num_servers = "3"
}

When i run #terraform.exe plan I get the following error on Windows. Note i entered the aws_key and secret_key as part of command line input.

Error: Error refreshing state: 1 error(s) occurred:

• module.consul.provider.aws: No valid credential sources found for AWS Provider.
  Please see https://terraform.io/docs/providers/aws/index.html for more information on
  providing credentials for the AWS Provider

I dont know how to fix this. So far i have tried #terraform.exe init -upgrade to ensure the aws provider and consul module provider are current.

[brikis98]

Hm, I believe this is due to some confusion between "modules" and "examples" in the Terraform Registry. The registry requires that each repo has Terraform code in the root, but for complicated modules such as Consul, the actual implementation consists of several submodules in the modules folder. The code in the root is actually example code meant to be cloned and deployed directly, rather than used with the module keyword, which is why it has its own provider block and isn't using yours.

@KFishner What are your thoughts on what to do about this? If we remove the provider block from the root example, it won't work as a standalone example. If we keep the provider block, it doesn't work with the module keyword. All of this weirdness goes away if we just allow complicated modules to not have Terraform code in the root and instead maintain a clear separation between examples in the examples folder, which should be deployed directly and have their own provider blocks, and modules in the modules folder, which are meant to be deployed using the module keyword.

[SahanaPrabhakar]

Is there a workaround till this is fixed ?

[brikis98]

1. Copy the code in the root example: https://github.com/hashicorp/terraform-aws-vault/blob/master/main.tf.
2. Follow the instructions in the comments of the code.
3. Run terraform init
4. Run terraform apply

[SahanaPrabhakar]

Thank you.

[mitchellh]

Hey @brikis98, thanks for being quick to respond and for your feedback. I think this is more of an issue caused by the changes to provider configuration in 0.11 (which as a reminder were to resolve mostly inconsistent relationships between provider configuration and modules). I imagine you know most of the information below but being verbose for readers.

Prior to 0.11, we would've merged those configurations, so both cases you describe would've worked: if you ran it directly or if you configured a provider in a parent consuming this as a module.

In 0.11, a major change we introduced was a consistent approach to provider configuration for modules. A backwards breaking change that is part of this is that provider configurations are never merged anymore. So by the root Terraform configs in this module setting provider, it is overriding the default "aws" provider in the root. For full details, see: https://www.terraform.io/upgrade-guides/0-11.html#interactions-between-providers-and-modules

I believe an easy fix for this would be to up the major version of the module and remove the provider block in the root and remove the aws_region variable completely and switch to using the aws_region data source for resources that may need the region.

The behavior would then be:

1.) Running via the registry instructions (as the poster of this issue did): Any provider configuration, as was done here, would work correctly. They can set the region and the module would load it properly.

2.) Running directly via init -from-module or cloning this repo: Terraform would by default prompt for configuration (via CLI input). Users could set env vars, or use any other form of AWS auth to authenticate.

Both cases would then work, if I'm not mistaken.

EDIT: I should note that pre-0.11, there were really confusing scenarios even in this basic example, and I'm going to note that because it highlights why we made the provider configuration changes in 0.11. Provider merging in TF <= 0.10.x was inverted: parents overrode children. So setting aws_region actually would've had no effect if there was an outer config (and would've worked if there was not). For experienced TF users this was expected (most of the time), but for newcomers we found this was almost universally confusing. Module writers had no way to force the region, either. This ambiguity or confusion is gone with 0.11 and forward.

[brikis98]

@mitchellh Thanks for the explanation with the provider changes. It seems a bummer that you now must write code, or at least set env vars, just to try out the example code. Before, a terraform init and apply would've been enough.

We've discussed it before, but I still find the root folder sometimes containing a "standalone example" is confusing many users, who expect it to be a "reusable module." This provider issue is but one minor example of that. The main issue is that, while elegant, the suggested terraform init <repo>/<provider>/<module> or source = <repo>/<provider>/<module> usage sets the expectation that you're using a production-ready module, when for non-trivial modules, it's typically just an example that's optimized for learning and testing.

For example, the Vault, Consul, and Nomad root "modules", to make learning and testing easy, allow connections from all ports, deploy into the default VPC, don't encrypt certs, etc. We've seen that users are surprised by this time and again and often end up with the impression that the code in the Registry is just for learning and couldn't possibly be used in a real production setting.

It just seems more sane to tell users to run terraform init <repo>/<provider>/<module>/examples/<example-name> (e.g., hashicorp/aws/vault/examples/private-cluster) when they want to try out an example and to use source = <repo>/<provider>/<module>/modules/<module-name> (e.g., source = hashicorp/aws/vault/modules/vault-elb) when they want to use a production-ready, reusable module in their code. It avoids all the confusion about examples vs modules, provider configurations, and so on.

[mitchellh]

You don't need to write code to try it since Terraform should prompt you. I just tried it locally (with the recommendations I noted in my prior comment) and was able to start spinning up clusters without any code at all, I just had to answer the region which was prompted on terraform apply. If someone is pulling in via module, they sort of already have code and I think configuring a provider is not unexpected (though again, not even necessary since Terraform will prompt you).

Even in the prior unmodified version with TF 0.10.x, I don't see how I could've avoided code or env vars since the auth had to come from somewhere. In the new usage, I suppose you do have to at least tell Terraform the region though (but again, that can come from input via the CLI).

I understand your opinion on the root folder, but I still believe optimizing for repo/provider/module as a default UX is strictly less mental overhead than teaching a new user subfolders (either examples or submodules). And in a few months of private module registry usage (via Terraform Enterprise), simpler reusable modules where submodules are not necessary is the norm, and big complex modules are rarer. That sort of makes sense in a customer environment too where the modules are highly opinionated as designed (versus the public registry where they're instead trying to be overly generic).

Your last paragraph though is exactly right and the current standard module model doesn't invalidate that. That's completely usable today and a reasonable thing to do for a complex module such as Vault, Nomad, Consul. The root folder in that case is just there to do something since users would expect it from simpler modules. And I think as we discussed before the most reasonable thing in my mind is just a highly opinionated example for a quick out-of-box experience, but very likely not production ready.

[brikis98]

You don't need to write code to try it since Terraform should prompt you.

Sure, if you don't mind typing in the same thing over and over again :)

And in a few months of private module registry usage (via Terraform Enterprise), simpler reusable modules where submodules are not necessary is the norm, and big complex modules are rarer.

Are you optimizing the registry for private usage or public usage? Because with public usage, I'm guessing what most users are looking for are solutions for the complicated modules they can't do themselves: Vault, VPCs, Kafka, K8, etc.

[mitchellh]

Sure, if you don't mind typing in the same thing over and over again :)

Of course, I'm just thinking in an "example" or non-production use case. You can get up and running with no code, then write code after if you want to keep it around or terraform destroy (enter the same value again) and destroy it.

For production usage, you'll have to write code but I assume you'll be writing code no matter what so I don't think that should be a usability detractor.

Are you optimizing the registry for private usage or public usage?

Both, we dedicated a few months to building out the private registry but for the past couple months have worked almost exclusively on the public one (and have shipped big improvements like exposing both examples and submodules, more on the way!).

We don't want a fragmented use case. The standard module structure recommended supports both cases of simple and complex module. The complex module can easily satisfy the root requirement by technically having a blank TF file or a TF file that has an output that explains to look elsewhere, but I think a better UX is an opinionated example. It provides a uniform experience: the root will always do something. We never say that the root is always the production-ready module, we were careful when crafting docs and marketing to never say that. It can be, but its best to treat it as a starting off point before digging deeper (whether it has submodules or not).

If we instead said "there is no root, you must use submodules", then the complexity for simple modules is raised significantly for the benefit of complex modules. I think that cost is higher than the alternative, which is a relatively low ask (just call your submodules in an opinionated fashion, OR just put a blank TF file if you really don't want to).

It was a simple exercise to come to this conclusion last year: I wrote down the steps to publish and consume both a simple and complex module (matrix of 4). The spec we chose has the simplest workflows if I recall in every case (or all except 1, but only off by one step on one...).

We've also been monitoring feedback from trainings and customers and we've never heard a complaint about it. We've heard multiple times how "cool" it is that they can spin up complicated clusters by just copying and pasting the module usage text from the public registry. They probably won't use those complicated clusters in production. 😄 But that's the point!

And I want to note that your point about submodules has been extremely well received here. Prior to shipping the registry we made sure the structure documented submodules and we've always parsed them. And just a couple weeks ago, we deployed a new change to the public registry to expose them in the UI, too! They aren't second class citizens by any means.

Hope that helps. If you have any recommendations for improvements I'm happy to hear them, but the one thing we probably won't do at this point is get rid of the root config requirement.

[mitchellh]

Adding to the above, if you haven't seen, check out the new submodule UI: https://registry.terraform.io/modules/hashicorp/vault/aws/0.3.0/submodules/vault-elb

We have improvements planned including improving the usage instructions for submodules and examples to be even easier. But we wanted to at least expose the data we already had in a user-consumable way. We also include "internal" submodules but they're hidden behind a dropdown, see the submodules dropdown here: https://registry.terraform.io/modules/terraform-aws-modules/security-group/aws/1.15.0

[brikis98]

It was a simple exercise to come to this conclusion last year: I wrote down the steps to publish and consume both a simple and complex module (matrix of 4). The spec we chose has the simplest workflows if I recall in every case (or all except 1, but only off by one step on one...).

Number of steps != simple workflow. Clear, consistent, untangled mental model = simple workflow. (Rich Hickey's Simple Made Easy does a great job of distinguishing between these).

we've never heard a complaint about it

We've run into issues with this fairly regularly. E.g., here's one in the Vault repo. I also get questions about this frequently at meetups, talks, and when chatting with customers: e.g., "I tried the Vault module, but it spun up Consul, and I already have a Consul cluster, so I can't really use it..." or "I can't use the Nomad module because it deploys in the default VPC and at my company, everything has to go in our custom VPC..."

We've also been monitoring feedback from trainings

That may be the issue right there. I'm guessing most people who use the registries don't have official training. They just find it from Google, try it out, see it doesn't work immediately, and move on.

And I want to note that your point about submodules has been extremely well received here. Prior to shipping the registry we made sure the structure documented submodules and we've always parsed them. And just a couple weeks ago, we deployed a new change to the public registry to expose them in the UI, too! They aren't second class citizens by any means.

I'm very grateful for that. Your team responds very well to feedback, which is why I'm chancing some more here :)

Adding to the above, if you haven't seen, check out the new submodule UI: https://registry.terraform.io/modules/hashicorp/vault/aws/0.3.0/submodules/vault-elb

👍

Huge improvement. Same with examples. It's made it easier to explain away the confusion, but a simpler mental model might've avoided the confusion in the first place :)

If you have any recommendations for improvements I'm happy to hear them, but the one thing we probably won't do at this point is get rid of the root config requirement.

I share what I can and appreciate the parts you've taken into account.

[mitchellh]

I suppose we have to agree to disagree on the mental model. I think its simple: root module will always do something, more flexibility comes in submodules if they exist. But that 🚢 is currently sailing. If this ends up being wrong, we can reassess but I don't agree for now.

I hope that the confusion is cleared up by the additional UI elements. They can now be linked from READMEs and such to to better explain to look in certain places. If you have suggestions on how to improve the UI further let us know and we'll take that into account for sure!

[brikis98]

Our mental model is that modules go under /modules, examples under /examples, and tests under /tests. I'm not sure it can get any simpler or clearer than that. The structure is consistent (no corner cases or exceptions), extensible (e.g., what happens if a "simple" module in the root becomes complex?), encourages good habits for module creators (you should have examples and tests!), and the "overhead" (if you can call it that?) is negligible.

That said, I understand your perspective. We will continue contributing the to the registry and share more feedback when we can :)

[kevinsapp]

Hi. I'm experiencing the same problem that @SahanaPrabhakar reported. I'm simply following along with the getting started guide here. Maybe the problem is with the documentation. My config looks like this:

provider "aws" {
  access_key  = "${ var.access_key }"
  secret_key  = "${ var.secret_key }"
  region      = "us-east-1"
}

module "consul" {
  source = "hashicorp/consul/aws"

  aws_region  = "us-east-1" # should match provider region
  num_servers = "3"
}