Phantom .tfvars.json file when running nebari deploy

[mwengren]

I've been editing the Nebari TF code somewhat to try to adapt Nebari to work within my existing VPC and subnets on AWS. Xref to discussions I've created about that:

• https://github.com/orgs/nebari-dev/discussions/2559
• https://github.com/orgs/nebari-dev/discussions/3085

Part of my testing was to modify the TF variables Nebari accepts. At different times when deploying Nebari from an AWS instance where I have the nebari command/conda configured, I get the following warning message in my TF output. It looks like Nebari has created a .tfvars.json file somewhere, and Nebari is still reading variables from this file, rather than my nebari-config.yaml file that I pass to nebari deploy -c.

The following variables aren't an issue (because they've both been removed from my working version of the Nebari TF code), but I'm fairly certain it's getting values for other variables from the same temp file, rather than nebari-config.yaml, which is causing issues for my deployment, because I want those values disregarded (and I've commented them out in nebari-config.yaml).

[tofu]: │ Warning: Value for undeclared variable
[tofu]: │
[tofu]: │ The root module does not declare a variable named "existing_subnet_ids" but
[tofu]: │ a value was found in file "/tmp/tmp59j6kmj4.tfvars.json". If you meant to
[tofu]: │ use this value, add a "variable" block to the configuration.
[tofu]: │
[tofu]: │ To silence these warnings, use TF_VAR_... environment variables to provide
[tofu]: │ certain "global" settings to all configurations in your organization. To
[tofu]: │ reduce the verbosity of these warnings, use the -compact-warnings option.

[tofu]: │ Warning: Value for undeclared variable
[tofu]: │
[tofu]: │ The root module does not declare a variable named "vpc_id" but a value was
[tofu]: │ found in file "/tmp/tmp59j6kmj4.tfvars.json". If you meant to use this
[tofu]: │ value, add a "variable" block to the configuration.
[tofu]: │
[tofu]: │ To silence these warnings, use TF_VAR_... environment variables to provide
[tofu]: │ certain "global" settings to all configurations in your organization. To
[tofu]: │ reduce the verbosity of these warnings, use the -compact-warnings option.
[tofu]: ╵

The strange part is I can't find this file on the system where my nebari command and conda environment is set up. Any advice? How do I flush this file from my environment?

Is this the relevant part of the code: https://github.com/nebari-dev/nebari/blob/main/src/_nebari/provider/opentofu.py#L55-L57?

[viniciusdc]

Hey @mwengren, the contents for each .json file are defined during run time of the deploy command, and stored in a temporary file that is cleaned when the context ends, exactly where you pointed out

nebari/src/_nebari/provider/opentofu.py

Lines 55 to 59 in d680ca8

	with tempfile.NamedTemporaryFile(
	mode="w", encoding="utf-8", suffix=".tfvars.json"
	) as f:
	json.dump(input_vars, f.file)
	f.file.flush()

If you need this to persist for debugging, you can replace it with tempfile.NamedTemporaryFile and add delete=False to the arguments; this should prevent the file from being removed (keep in mind that this should only be done for debugging purposes).

Now, regarding your question per se, those files should only contain the variable inputs that exist within each stage as part of their input_vars method:

nebari/src/_nebari/stages/kubernetes_services/__init__.py

Line 607 in d680ca8

def input_vars(self, stage_outputs: Dict[str, Dict[str, Any]]):

Keep in mind that sometimes, outputs from previous stages are also sent out as inputs to other stages. Depending on what you are changing, it would receive precedence, since .json is sometimes considered a patch to .tf or .tofu files (check for overrides). Overall, the gist is that in the tofu code, .json is read first. Then they get merged with the remaining configuration. In Nebari's case, we primarily do this for provider settings, as they need to be dynamically configurable.

nebari/src/_nebari/stages/base.py

Lines 281 to 300 in d680ca8

	@contextlib.contextmanager
	def deploy(
	self,
	stage_outputs: Dict[str, Dict[str, Any]],
	disable_prompt: bool = False,
	tofu_init: bool = True,
	):
	deploy_config = dict(
	directory=str(self.output_directory / self.stage_prefix),
	input_vars=self.input_vars(stage_outputs),
	tofu_init=tofu_init,
	)
	state_imports = self.state_imports()
	if state_imports:
	deploy_config["tofu_import"] = True
	deploy_config["state_imports"] = state_imports
	self.set_outputs(stage_outputs, opentofu.deploy(**deploy_config))
	self.post_deploy(stage_outputs, disable_prompt)
	yield

Usually, nebari will only consider data from the nebari-config.yaml file if there is an associated pydantic model created under the target service, e.g for infra we have the main "exposed" config schema here:

nebari/src/_nebari/stages/infrastructure/__init__.py

Lines 237 to 255 in d680ca8

	class AWSInputVars(schema.Base):
	name: str
	environment: str
	existing_security_group_id: Optional[str] = None
	existing_subnet_ids: Optional[List[str]] = None
	region: str
	kubernetes_version: str
	eks_endpoint_access: Optional[
	Literal["private", "public", "public_and_private"]
	] = "public"
	eks_kms_arn: Optional[str] = None
	eks_public_access_cidrs: Optional[List[str]] = ["0.0.0.0/0"]
	node_groups: List[AWSNodeGroupInputVars]
	availability_zones: List[str]
	vpc_cidr_block: str
	permissions_boundary: Optional[str] = None
	kubeconfig_filename: str = get_kubeconfig_filename()
	tags: Dict[str, str] = {}
	efs_enabled: bool

while the data forwarded to terraform is consumed here:

nebari/src/_nebari/stages/infrastructure/__init__.py

Lines 937 to 973 in d680ca8

	elif self.config.provider == schema.ProviderEnum.aws:
	return AWSInputVars(
	name=self.config.escaped_project_name,
	environment=self.config.namespace,
	eks_endpoint_access=self.config.amazon_web_services.eks_endpoint_access,
	eks_public_access_cidrs=self.config.amazon_web_services.eks_public_access_cidrs,
	eks_kms_arn=self.config.amazon_web_services.eks_kms_arn,
	existing_subnet_ids=self.config.amazon_web_services.existing_subnet_ids,
	existing_security_group_id=self.config.amazon_web_services.existing_security_group_id,
	region=self.config.amazon_web_services.region,
	kubernetes_version=self.config.amazon_web_services.kubernetes_version,
	node_groups=[
	AWSNodeGroupInputVars(
	name=name,
	instance_type=node_group.instance,
	gpu=node_group.gpu,
	min_size=node_group.min_nodes,
	desired_size=node_group.min_nodes,
	max_size=node_group.max_nodes,
	single_subnet=node_group.single_subnet,
	permissions_boundary=node_group.permissions_boundary,
	launch_template=None,
	node_taints=node_group.taints,
	ami_type=construct_aws_ami_type(
	gpu_enabled=node_group.gpu,
	launch_template=None,
	),
	spot=node_group.spot,
	)
	for name, node_group in self.config.amazon_web_services.node_groups.items()
	],
	availability_zones=self.config.amazon_web_services.availability_zones,
	vpc_cidr_block=self.config.amazon_web_services.vpc_cidr_block,
	permissions_boundary=self.config.amazon_web_services.permissions_boundary,
	tags=self.config.amazon_web_services.tags,
	efs_enabled=self.config.storage.type == SharedFsEnum.efs,
	).model_dump()

This means that, if the associated template received a value that was not expecting, and was not catched adequately by the validators, it will then forward that as a variable to terraform, which can lead to issues, if the variable/field/value structure was not correct (sometime an open bracket, or identation) it could lead terraform to improperly assume variables that don't exist.

If this helps, I suggest you modify or expand the schema code of each stage from which you want to override values.

Additionally, and this is important, redeploying multiple times with different versions of schemas and resources can lead to a scrambled "stages" folder, in the sense that it may refer to variables, keys, and resources that no longer exist in the main code. However, locally they are still mapped -- just check your .tfstate files for "ghost" variables.

[mwengren]

@viniciusdc Thanks this is helpful! I'd found previously the AWSInputVars class you mentioned, and modified it to accept a few new parameters I'd been testing with.

I think I understand your explanation of how things work with the tfvars.json temp files and the Nebari stages, generally.

It is a bit confusing to develop against, or possibly for a general user to understand unexpected behavior, because Nebari includes default values for some variables and not others, and those that do seem to include them within the Nebari Pydantic classes such as AmazonWebServicesProvider, rather than within the Terraform code.

And I'm not sure those are documented anywhere in the user docs, which would be a nice addition. IMO there should be a page in the docs that lists all Nebari TF/config variables and default values, if any.

In my case I was running into the following issue with the vpc_cidr_block variable when doing some dev/testing:

Previously, I'd been testing deploying Nebari with a SG I'd manually created (comment) and passing the existing_security_group variable, but instead I wanted to deploy without the existing SG to eliminate that as a potential source of error and instead have Nebari create its own by removing the variable from nebari-config.yaml.

I modified the Nebari TF code to have it obtain the VPC subnet CIDRs rather than reference the vpc_cidr_block variable if none was present, so I was surprised when the SG was being created with a default of 10.10.0.0/16, which isn't relevant to our VPC network and I hadn't specified in nebari-config.yaml.

Then I found:

https://github.com/nebari-dev/nebari/blob/main/src/_nebari/stages/infrastructure/__init__.py#L573.

I found a better way that gets around the issue in the end, but at first in the commented-out line I was confused why 10.10.0.0/16 was being used.

Why are the variable defaults set in Pydantic vs Terraform? Thx!