feat: add example for GKE confidential nodes with GPU by arthurlapertosa · Pull Request #2426 · terraform-google-modules/terraform-google-kubernetes-engine

arthurlapertosa · 2025-08-28T03:55:34Z

No description provided.

arthurlapertosa · 2025-08-28T04:01:30Z

@apeabody could you please run the build for this PR?

apeabody · 2025-08-28T14:33:33Z

/gcbrun

arthurlapertosa · 2025-08-28T15:59:58Z

@apeabody I don't have access to the GCP cloud build project. Could you please send me the error?

apeabody · 2025-08-28T17:42:28Z

@apeabody I don't have access to the GCP cloud build project. Could you please send me the error?

Error: Reference to undeclared input variable

  on ../../modules/beta-autopilot-private-cluster/cluster.tf line 72, in resource "google_container_cluster" "primary":
  72:       confidential_instance_type = lookup(var.node_pools[0], "confidential_instance_type", null)

An input variable with the name "node_pools" has not been declared. This
variable can be declared with a variable "node_pools" {} block.}

arthurlapertosa · 2025-08-29T19:27:07Z

@apeabody could you please re-run the build?

apeabody · 2025-08-29T20:08:19Z

/gcbrun

arthurlapertosa · 2025-09-02T16:52:45Z

@apeabody I think the build wasn't properly triggered, could you please take a look?

apeabody · 2025-09-02T17:29:12Z

/gcbrun

apeabody · 2025-09-02T17:29:55Z

@apeabody I think the build wasn't properly triggered, could you please take a look?

Might have been too quick after the merge, it's running now.

apeabody

Thanks @arthurlapertosa!

autogen/main/cluster.tf.tmpl

apeabody · 2025-10-30T23:07:37Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new example for creating a GKE cluster with confidential nodes and GPUs. This is a valuable addition. The changes include modifications to several Terraform modules to support confidential_instance_type and guest_accelerator configurations, along with the new example files and corresponding integration tests. The implementation is mostly correct, but I've found a few issues related to version constraints, external dependencies, and a bug in the for_each logic that need to be addressed.

examples/confidential_gpu_public/network.tf

autogen/main/cluster.tf.tmpl

examples/confidential_gpu_public/main.tf

arthurlapertosa · 2025-11-04T18:30:43Z

@apeabody I don't have access to the build error, could you please send it to me?

arthurlapertosa · 2025-11-04T20:48:40Z

@apeabody could you please rerun the build and the checks?

apeabody · 2025-11-04T20:57:28Z

/gcbrun

apeabody · 2025-11-04T20:57:53Z

@apeabody I don't have access to the build error, could you please send it to me?

It's expired, triggered a fresh test

apeabody · 2025-11-04T22:46:56Z

@apeabody I don't have access to the build error, could you please send it to me?

It's expired, triggered a fresh test

Thanks - This change looks pretty good, but we still need to enable the new test. Can you please add to the cloudbuild similar to https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/pull/2458/files#diff-35f3e17c28a8a8de710f4be35fb5448d0e33c2d9b89fb2be7499b9830f890d12

arthurlapertosa · 2025-11-26T22:42:04Z

@apeabody sorry for the delay. Can you run the build, please?

apeabody · 2025-11-26T23:01:02Z

/gcbrun

apeabody · 2025-11-26T23:01:13Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new example for creating a GKE cluster with confidential nodes and GPUs. The implementation is solid, including updates to various modules to support confidential_instance_type and GPU configurations, and the addition of a comprehensive integration test. My review focuses on the new example files, where I've identified opportunities to simplify the Terraform code, remove unused resources, and improve documentation and test clarity. Overall, these are great additions that enhance the module's capabilities.

gemini-code-assist · 2025-11-26T23:03:35Z

examples/confidential_gpu_public/main.tf

+  cluster_type           = "confidential-gpu"
+  network_name           = "confidential-gpu-network-${random_string.suffix.result}"
+  subnet_name            = "confidential-gpu-subnet"
+  master_auth_subnetwork = "confidential-gpu-master-subnet"


The local variable master_auth_subnetwork is defined but never used in this public cluster example. This should be removed to avoid confusion. The corresponding subnet resource in network.tf should also be removed.

gemini-code-assist · 2025-11-26T23:03:35Z

examples/confidential_gpu_public/main.tf

+  region                            = var.region
+  zones                             = var.zones
+  network                           = module.gcp-network.network_name
+  subnetwork                        = local.subnet_names[index(module.gcp-network.subnets_names, local.subnet_name)]


This expression to look up the subnetwork name is unnecessarily complex. It can be simplified by using the local.subnet_name variable directly.

subnetwork = local.subnet_name

gemini-code-assist · 2025-11-26T23:03:36Z

examples/confidential_gpu_public/network.tf

+    {
+      subnet_name           = local.master_auth_subnetwork
+      subnet_ip             = "10.60.0.0/17"
+      subnet_region         = var.region
+      subnet_private_access = true
+    },


This subnet for master_auth_subnetwork is created but not used in this public cluster example. It should be removed to avoid provisioning unnecessary resources.

gemini-code-assist · 2025-11-26T23:03:36Z

examples/confidential_gpu_public/outputs.tf

+output "location" {
+  value = module.gke.location
+}


The location output is missing a description. Please add one for clarity and consistency with other outputs.

output "location" { description = "The location (region or zone) in which the cluster resides." value = module.gke.location }

gemini-code-assist · 2025-11-26T23:03:36Z

test/integration/confidential_gpu_public/confidential_gpu_public_test.go

+			"nodeConfig.diskType",
+			"nodeConfig.enableConfidentialStorage",
+			"nodeConfig.machineType",
+			"nodeConfig.diskType",


The JSON path nodeConfig.diskType is duplicated in the validateJSONPaths slice (it also appears on line 72). Please remove this duplicate entry to improve code clarity.

apeabody · 2025-11-27T00:27:20Z

Step #99 - "apply test-confidential-gpu-public":         	Error:      	Received unexpected error:
Step #99 - "apply test-confidential-gpu-public":         	            	FatalError{Underlying: error while running command: exit status 1; 
Step #99 - "apply test-confidential-gpu-public":         	            	Error: Error waiting for creating GKE cluster: Insufficient quota to satisfy the request: Not all instances running in IGM after 29.827554958s. Expected 1, running 0, transitioning 1. Current errors: [GCE_QUOTA_EXCEEDED]: Instance 'gke-confidential-gpu-clu-default-pool-ad60a6ba-xnqt' creation failed: Quota 'GPUS_PER_GPU_FAMILY' exceeded.  Limit: 0.0 in region us-central1.
Step #99 - "apply test-confidential-gpu-public":         	            	
Step #99 - "apply test-confidential-gpu-public":         	            	  with module.gke.google_container_cluster.primary,
Step #99 - "apply test-confidential-gpu-public":         	            	  on ../../modules/beta-public-cluster/cluster.tf line 22, in resource "google_container_cluster" "primary":
Step #99 - "apply test-confidential-gpu-public":         	            	  22: resource "google_container_cluster" "primary" {
Step #99 - "apply test-confidential-gpu-public":         	            	}
Step #99 - "apply test-confidential-gpu-public":         	Test:       	TestConfidentialGpuPublic

arthurlapertosa · 2025-11-27T04:55:33Z

@apeabody
Looks like the project used for testing has 0 GPUS_PER_GPU_FAMILY quota. This quota is necessary for this example to work, once it uses a container with a GPU.
Do you think we could get/request this quota?

apeabody · 2025-12-01T18:20:32Z

@apeabody Looks like the project used for testing has 0 GPUS_PER_GPU_FAMILY quota. This quota is necessary for this example to work, once it uses a container with a GPU. Do you think we could get/request this quota?

Hi @arthurlapertosa - They are ephemeral projects, so unfortunately that isn't really feasible. Could you switch to something more like NVIDIA T4 which we should have quota?

arthurlapertosa · 2026-01-20T21:45:01Z

I made some tests, and apparently only the nvidia H100 is supported with confidential computing. Also the docs only say it's only possible with H100: https://docs.cloud.google.com/confidential-computing/confidential-vm/docs/create-a-confidential-vm-instance-with-gpu.
Also, in the NVIDIA official website, it claims that the first GPU to support Confidential computing is the H100: https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/ ("starting with the NVIDIA Hopper™ architecture, which introduced confidential computing on the GPU").
Maybe we can have a preemptible quota, which is much easier to obtain? (https://docs.cloud.google.com/confidential-computing/confidential-vm/docs/create-a-confidential-vm-instance-with-gpu#request-preemptible-quota)

apeabody · 2026-01-21T00:12:58Z

I made some tests, and apparently only the nvidia H100 is supported with confidential computing. Also the docs only say it's only possible with H100: https://docs.cloud.google.com/confidential-computing/confidential-vm/docs/create-a-confidential-vm-instance-with-gpu. Also, in the NVIDIA official website, it claims that the first GPU to support Confidential computing is the H100: https://www.nvidia.com/en-us/data-center/solutions/confidential-computing/ ("starting with the NVIDIA Hopper™ architecture, which introduced confidential computing on the GPU"). Maybe we can have a preemptible quota, which is much easier to obtain? (https://docs.cloud.google.com/confidential-computing/confidential-vm/docs/create-a-confidential-vm-instance-with-gpu#request-preemptible-quota)

Unfortunately the test projects are ephemeral.

feat: add example for GKE confidential nodes with GPU

881ccb2

arthurlapertosa marked this pull request as ready for review August 28, 2025 03:59

arthurlapertosa requested review from a team, apeabody and ericyz as code owners August 28, 2025 03:59

fix: do not set confidential_instance_type for autopilot clusters

2664ab3

Merge branch 'main' into confidential-nodes-gpu

83b8aab

apeabody requested changes Sep 2, 2025

View reviewed changes

autogen/main/cluster.tf.tmpl Show resolved Hide resolved

Merge branch 'main' into confidential-nodes-gpu

8058b78

gemini-code-assist bot reviewed Oct 30, 2025

View reviewed changes

examples/confidential_gpu_public/network.tf Outdated Show resolved Hide resolved

autogen/main/cluster.tf.tmpl Show resolved Hide resolved

examples/confidential_gpu_public/main.tf Outdated Show resolved Hide resolved

examples/confidential_gpu_public/main.tf Show resolved Hide resolved

code review changes

cf2d7e8

arthurlapertosa and others added 2 commits November 26, 2025 19:32

Merge branch 'terraform-google-modules:main' into confidential-nodes-gpu

bbea9b8

enable tests

f5c41c8

gemini-code-assist bot reviewed Nov 26, 2025

View reviewed changes

Conversation

arthurlapertosa commented Aug 28, 2025

Uh oh!

arthurlapertosa commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

apeabody commented Aug 28, 2025

Uh oh!

arthurlapertosa commented Aug 28, 2025

Uh oh!

apeabody commented Aug 28, 2025

Uh oh!

arthurlapertosa commented Aug 29, 2025

Uh oh!

apeabody commented Aug 29, 2025

Uh oh!

arthurlapertosa commented Sep 2, 2025

Uh oh!

apeabody commented Sep 2, 2025

Uh oh!

apeabody commented Sep 2, 2025

Uh oh!

apeabody left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

apeabody commented Oct 30, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arthurlapertosa commented Nov 4, 2025

Uh oh!

arthurlapertosa commented Nov 4, 2025

Uh oh!

apeabody commented Nov 4, 2025

Uh oh!

apeabody commented Nov 4, 2025

Uh oh!

apeabody commented Nov 4, 2025

Uh oh!

arthurlapertosa commented Nov 26, 2025

Uh oh!

apeabody commented Nov 26, 2025

Uh oh!

apeabody commented Nov 26, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist bot Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist bot Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

apeabody commented Nov 27, 2025

Uh oh!

arthurlapertosa commented Nov 27, 2025

Uh oh!

apeabody commented Dec 1, 2025

Uh oh!

arthurlapertosa commented Jan 20, 2026

Uh oh!

apeabody commented Jan 21, 2026

arthurlapertosa commented Aug 28, 2025 •

edited

Loading