terraform-provider-kubernetes_v2.37.1_x5 plugin crashed

[medoni]

Terraform Version, Provider Version and Kubernetes Version

Terraform v1.12.2
on linux_amd64
+ provider registry.terraform.io/hashicorp/google v5.45.2
+ provider registry.terraform.io/hashicorp/helm v2.17.0
+ provider registry.terraform.io/hashicorp/kubernetes v2.37.1
+ provider registry.terraform.io/hashicorp/null v3.2.4

Affected Resource(s)

• kubernetes_manifest

Terraform Configuration Files

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

Debug Output

2025-07-05T06:33:23.280+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:   |   domain: {{branch-normalized}}.demo-app3.gke-auto.squirreldev.de
2025-07-05T06:33:23.280+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:   | image:
2025-07-05T06:33:23.280+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:   |   repository: ghcr.io/medoni/2024-helm-microservice-deployment/
2025-07-05T06:33:23.280+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:   | ">, "valuesObject":tftypes.DynamicPseudoType<null>, "version":tftypes.String<null>>, "kustomize":tftypes.Object["apiVersions":tftypes.List[tftypes.String], ...
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: goroutine 817 [running]:
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: github.com/hashicorp/terraform-plugin-go/tftypes.NewValue(...)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         github.com/hashicorp/terraform-plugin-go@v0.27.0/tftypes/value.go:278
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.DeepUnknown({0x26d0ad8, 0xc007fc4e10}, {{0x26d0ad8?, 0xc002094ea0?}, {0x1eadec0?, 0xc00194c9c0?}}, 0xc00194cd20)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:86 +0x1b11
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.DeepUnknown({0x26d0420, 0xc001b47d40}, {{0x26d0420?, 0xc0026486c0?}, {0x205b1a0?, 0xc001eedb60?}}, 0xc00194c1e0)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:33 +0x1e65
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.DeepUnknown({0x26d0420, 0xc001ee3200}, {{0x26d0420?, 0xc0027e4a80?}, {0x205b1a0?, 0xc001eed950?}}, 0xc00194c018)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:33 +0x1e65
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: github.com/hashicorp/terraform-provider-kubernetes/manifest/provider.(*RawProviderServer).PlanResourceChange(0xc000237280, {0x26c7208, 0xc00d3b70b0}, 0xc001772a20)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         github.com/hashicorp/terraform-provider-kubernetes/manifest/provider/plan.go:395 +0x36c5
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: github.com/hashicorp/terraform-plugin-mux/tf5muxserver.(*muxServer).PlanResourceChange(0xc000237200, {0x26c7208?, 0xc00d3b6cf0?}, 0xc001772a20)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         github.com/hashicorp/terraform-plugin-mux@v0.19.0/tf5muxserver/mux_server_PlanResourceChange.go:73 +0x2a2
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).PlanResourceChange(0xc0004ee460, {0x26c7208?, 0xc00d3b6030?}, 0xc000237500)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         github.com/hashicorp/terraform-plugin-go@v0.27.0/tfprotov5/tf5server/server.go:949 +0x3d9
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler({0x22b2080, 0xc0004ee460}, {0x26c7208, 0xc00d3b6030}, 0xc000237480, 0x0)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         github.com/hashicorp/terraform-plugin-go@v0.27.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:669 +0x1a6
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: google.golang.org/grpc.(*Server).processUnaryRPC(0xc000dfe000, {0x26c7208, 0xc00b52c030}, 0xc00130c2a0, 0xc000df71a0, 0x3898150, 0x0)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         google.golang.org/grpc@v1.72.1/server.go:1405 +0x1036
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: google.golang.org/grpc.(*Server).handleStream(0xc000dfe000, {0x26c78f0, 0xc000185a00}, 0xc00130c2a0)
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         google.golang.org/grpc@v1.72.1/server.go:1815 +0xb88
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: google.golang.org/grpc.(*Server).serveStreams.func2.1()
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         google.golang.org/grpc@v1.72.1/server.go:1035 +0x7f
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5: created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 53
2025-07-05T06:33:23.289+0200 [DEBUG] provider.terraform-provider-kubernetes_v2.37.1_x5:         google.golang.org/grpc@v1.72.1/server.go:1046 +0x11d
2025-07-05T06:33:23.294+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2025-07-05T06:33:23.294+0200 [ERROR] plugin.(*GRPCProvider).PlanResourceChange: error="rpc error: code = Unavailable desc = error reading from server: EOF"
2025-07-05T06:33:23.294+0200 [ERROR] provider: plugin process exited: plugin=.terraform/providers/registry.terraform.io/hashicorp/kubernetes/2.37.1/linux_amd64/terraform-provider-kubernetes_v2.37.1_x5 id=416212 error="exit status 2"
2025-07-05T06:33:23.294+0200 [ERROR] vertex "kubernetes_manifest.argocd_app_manifests[\"2025-demo-app3/app.yaml\"]" error: Plugin did not respond
2025-07-05T06:33:23.295+0200 [ERROR] vertex "kubernetes_manifest.argocd_app_manifests (expand)" error: Plugin did not respond
2025-07-05T06:33:23.305+0200 [WARN]  Planning encountered errors, so plan is not applyable

Panic Output

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Plugin did not respond
│
│   with kubernetes_manifest.argocd_app_manifests["2025-demo-app3/app.yaml"],
│   on argocd.projects.tf line 79, in resource "kubernetes_manifest" "argocd_app_manifests":
│   79: resource "kubernetes_manifest" "argocd_app_manifests" {
│
│ The plugin encountered an error, and failed to respond to the plugin.(*GRPCProvider).PlanResourceChange call. The plugin logs may contain more details.
╵
╷
│ Error: Request cancelled
│
│   with data.kubernetes_ingress_v1.argocd_server,
│   on dns.tf line 17, in data "kubernetes_ingress_v1" "argocd_server":
│   17: data "kubernetes_ingress_v1" "argocd_server" {
│
│ The plugin.(*GRPCProvider).ValidateDataResourceConfig request was cancelled.
╵

Stack trace from the terraform-provider-kubernetes_v2.37.1_x5 plugin:
panic: ElementKeyInt(0): can't use tftypes.Object["clusterDecisionResource":tftypes.Object["configMapRef":tftypes.String ...

goroutine 510 [running]:
github.com/hashicorp/terraform-plugin-go/tftypes.NewValue(...)
        github.com/hashicorp/terraform-plugin-go@v0.27.0/tftypes/value.go:278
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.DeepUnknown({0x26d0ad8, 0xc002d505d0}, {{0x26d0ad8?, 0xc002362270?}, {0x1eadec0?, 0xc0133b9788?}}, 0xc002ad2168)
        github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:86 +0x1b11
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.DeepUnknown({0x26d0420, 0xc001beb2f0}, {{0x26d0420?, 0xc002ad9830?}, {0x205b1a0?, 0xc001e0abd0?}}, 0xc0133b9f38)
        github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:33 +0x1e65
github.com/hashicorp/terraform-provider-kubernetes/manifest/morph.DeepUnknown({0x26d0420, 0xc001e0a1b0}, {{0x26d0420?, 0xc002dfaae0?}, {0x205b1a0?, 0xc001e0a420?}}, 0xc0133b9b60)
        github.com/hashicorp/terraform-provider-kubernetes/manifest/morph/scaffold.go:33 +0x1e65
github.com/hashicorp/terraform-provider-kubernetes/manifest/provider.(*RawProviderServer).PlanResourceChange(0xc00047e200, {0x26c7208, 0xc011c6a000}, 0xc002ece4e0)
        github.com/hashicorp/terraform-provider-kubernetes/manifest/provider/plan.go:395 +0x36c5
github.com/hashicorp/terraform-plugin-mux/tf5muxserver.(*muxServer).PlanResourceChange(0xc00047e100, {0x26c7208?, 0xc011ef98c0?}, 0xc002ece4e0)
        github.com/hashicorp/terraform-plugin-mux@v0.19.0/tf5muxserver/mux_server_PlanResourceChange.go:73 +0x2a2
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).PlanResourceChange(0xc000614500, {0x26c7208?, 0xc011ef8300?}, 0xc000e9e100)
        github.com/hashicorp/terraform-plugin-go@v0.27.0/tfprotov5/tf5server/server.go:949 +0x3d9
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_PlanResourceChange_Handler({0x22b2080, 0xc000614500}, {0x26c7208, 0xc011ef8300}, 0xc000e9e080, 0x0)
        github.com/hashicorp/terraform-plugin-go@v0.27.0/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:669 +0x1a6
google.golang.org/grpc.(*Server).processUnaryRPC(0xc0005ea000, {0x26c7208, 0xc011762030}, 0xc002ece180, 0xc001023a10, 0x3898150, 0x0)
        google.golang.org/grpc@v1.72.1/server.go:1405 +0x1036
google.golang.org/grpc.(*Server).handleStream(0xc0005ea000, {0x26c78f0, 0xc000d0e000}, 0xc002ece180)
        google.golang.org/grpc@v1.72.1/server.go:1815 +0xb88
google.golang.org/grpc.(*Server).serveStreams.func2.1()
        google.golang.org/grpc@v1.72.1/server.go:1035 +0x7f
created by google.golang.org/grpc.(*Server).serveStreams.func2 in goroutine 40
        google.golang.org/grpc@v1.72.1/server.go:1046 +0x11d

Error: The terraform-provider-kubernetes_v2.37.1_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

Steps to Reproduce

Expected Behavior

What should have happened?

Actual Behavior

What actually happened?

Important Factoids

References

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[medoni]

It seems this happens to the following ressource:

resource "kubernetes_manifest" "argocd_app_manifests" {
  for_each = toset(local.argocd_app_manifests)

  manifest = yamldecode(file("${local.argocd_projects_path}/${each.key}"))
}

The content of that manifest:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: 2025-demo-app3-stories
  namespace: argocd
spec:
  # generators:
  # - git:
  #     repoURL: https://github.com/medoni/2024-helm-microservice-deployment.git
  #     revision: HEAD
  #     branches:
  #     - "stories/*"
  #     requeueAfterSeconds: 30
  generators:
  - list:
      elements:
      - branch: stories/gcloud-k8s
        branchNormalized: stories-gcloud-k8s
      - branch: stories/frontend-svelte
        branchNormalized: stories-frontend-svelte
  template:
    metadata:
      name: '2025-demo-app3-{{branch-normalized}}'
    spec:
      project: 2025-demo-app3
      source:
        repoURL: https://github.com/medoni/2024-helm-microservice-deployment.git
        targetRevision: '{{branch}}'
        path: src/infrastructure/Kubernetes/helm/dev-local/PizzaService
        helm:
          values: |
            global:
              domain: {{branch-normalized}}.demo-app3.gke-auto.squirreldev.de
            image:
              repository: ghcr.io/medoni/2024-helm-microservice-deployment/
      destination:
        server: https://kubernetes.default.svc
        namespace: '2025-demo-app3-{{branch-normalized}}'
      syncPolicy:
        automated: {}

[uspiotrowska]

same issue but with different resource apiVersion: cluster.x-k8s.io/v1beta1, kind: Cluster

[willemvv]

I ran into the same issue with Kyverno permissions where the provider panics because the tuple type had a different amount of elements from the input. I think this testcase in scaffold_test.go TestDeepUnknown reproduces the issue:

"nested-tuple-object": {
	In: deepUnknownTestSampleInput{
		T: tftypes.Tuple{ElementTypes: []tftypes.Type{
			tftypes.Object{AttributeTypes: map[string]tftypes.Type{
				"first": tftypes.Tuple{ElementTypes: []tftypes.Type{
					tftypes.Object{AttributeTypes: map[string]tftypes.Type{
						"second": tftypes.Number,
					}},
				}},
			}},
		}},
		V: tftypes.NewValue(
			tftypes.Tuple{ElementTypes: []tftypes.Type{
				tftypes.Object{AttributeTypes: map[string]tftypes.Type{
					"first": tftypes.Tuple{ElementTypes: []tftypes.Type{
						tftypes.Object{AttributeTypes: map[string]tftypes.Type{
							"second": tftypes.Number,
						}},
						tftypes.Object{AttributeTypes: map[string]tftypes.Type{
							"second": tftypes.Number,
						}},
					}},
				}},
			}},
			[]tftypes.Value{
				tftypes.NewValue(
					tftypes.Object{AttributeTypes: map[string]tftypes.Type{
						"first": tftypes.Tuple{ElementTypes: []tftypes.Type{
							tftypes.Object{AttributeTypes: map[string]tftypes.Type{
								"second": tftypes.Number,
							}},
							tftypes.Object{AttributeTypes: map[string]tftypes.Type{
								"second": tftypes.Number,
							}},
						}},
					}},
					map[string]tftypes.Value{
						"first": tftypes.NewValue(
							tftypes.Tuple{ElementTypes: []tftypes.Type{
								tftypes.Object{AttributeTypes: map[string]tftypes.Type{
									"second": tftypes.Number,
								}},
								tftypes.Object{AttributeTypes: map[string]tftypes.Type{
									"second": tftypes.Number,
								}},
							}},
							[]tftypes.Value{
								tftypes.NewValue(
									tftypes.Object{AttributeTypes: map[string]tftypes.Type{
										"second": tftypes.Number,
									}},
									map[string]tftypes.Value{
										"second": tftypes.NewValue(tftypes.Number, 10),
									},
								),
								tftypes.NewValue(
									tftypes.Object{AttributeTypes: map[string]tftypes.Type{
										"second": tftypes.Number,
									}},
									map[string]tftypes.Value{
										"second": tftypes.NewValue(tftypes.Number, 10),
									},
								),
							},
						),
					},
				),
			},
		),
	},
	Out: tftypes.NewValue(
		tftypes.Tuple{ElementTypes: []tftypes.Type{
			tftypes.Object{AttributeTypes: map[string]tftypes.Type{
				"first": tftypes.Tuple{ElementTypes: []tftypes.Type{
					tftypes.Object{AttributeTypes: map[string]tftypes.Type{
						"second": tftypes.Number,
					}},
					tftypes.Object{AttributeTypes: map[string]tftypes.Type{
						"second": tftypes.Number,
					}},
				}},
			}},
		}},
		[]tftypes.Value{
			tftypes.NewValue(
				tftypes.Object{AttributeTypes: map[string]tftypes.Type{
					"first": tftypes.Tuple{ElementTypes: []tftypes.Type{
						tftypes.Object{AttributeTypes: map[string]tftypes.Type{
							"second": tftypes.Number,
						}},
						tftypes.Object{AttributeTypes: map[string]tftypes.Type{
							"second": tftypes.Number,
						}},
					}},
				}},
				map[string]tftypes.Value{
					"first": tftypes.NewValue(
						tftypes.Tuple{ElementTypes: []tftypes.Type{
							tftypes.Object{AttributeTypes: map[string]tftypes.Type{
								"second": tftypes.Number,
							}},
							tftypes.Object{AttributeTypes: map[string]tftypes.Type{
								"second": tftypes.Number,
							}},
						}},
						[]tftypes.Value{
							tftypes.NewValue(
								tftypes.Object{AttributeTypes: map[string]tftypes.Type{
									"second": tftypes.Number,
								}},
								map[string]tftypes.Value{
									"second": tftypes.NewValue(tftypes.Number, 10),
								},
							),
							tftypes.NewValue(
								tftypes.Object{AttributeTypes: map[string]tftypes.Type{
									"second": tftypes.Number,
								}},
								map[string]tftypes.Value{
									"second": tftypes.NewValue(tftypes.Number, 10),
								},
							),
						},
					),
				},
			),
		},
	),
},

which gives the following exception:

-- FAIL: TestDeepUnknown (0.00s)
    --- FAIL: TestDeepUnknown/nested-tuple-object (0.00s)
panic: ElementKeyInt(0): can't use tftypes.Object["first":tftypes.Tuple[tftypes.Object["second":tftypes.Number], tftypes.Object["second":tftypes.Number]]] as tftypes.Object["first":tftypes.Tuple[tftypes.Object["second":tftypes.Number]]] [recovered]
	panic: ElementKeyInt(0): can't use tftypes.Object["first":tftypes.Tuple[tftypes.Object["second":tftypes.Number], tftypes.Object["second":tftypes.Number]]] as tftypes.Object["first":tftypes.Tuple[tftypes.Object["second":tftypes.Number]]]

[fatmcgav]

#2800 should fix this one 🤞