capa-eks-e2e: "EKS cluster tests should create a cluster and add nodes" consistently failing

[nrb]

/kind bug
/area provider/eks
/area testing

What steps did you take and what happened:
[A clear and concise description of what the bug is.]

The EKS e2e tests are consistently failing on this test case:

capa-eks-e2e: [It] [managed] [general] [ipv6] EKS cluster tests should create a cluster and add nodes

What did you expect to happen:

e2e should continue passing.

Anything else you would like to add:

Last passing test: https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/kubernetes-sigs_cluster-api-provider-aws/5498/pull-cluster-api-provider-aws-e2e-eks/1927445735872139264

History: https://prow.k8s.io/job-history/gs/kubernetes-ci-logs/pr-logs/directory/pull-cluster-api-provider-aws-e2e-eks

Environment:

• Cluster-api-provider-aws version: main

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[richardcase]

/assign
/lifecycle active

[richardcase]

Do we have some examples of this failing?

[richardcase]

Seem to be looping on this:

I0609 11:12:30.068764       1 addons.go:75] "Executing addon procedure" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControlPlane" AWSManagedControlPlan │
│ I0609 11:12:30.068961       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:31.306109       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:33.379102       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:36.584944       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:42.420094       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:12:52.471935       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:13:08.014267       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:13:34.169904       1 endpoints.go:136] "Custom endpoint not found, using default endpoint" controller="awsmanagedcontrolplane" controllerGroup="controlplane.cluster.x-k8s.io" controllerKind="AWSManagedControl │
│ I0609 11:13:57.171748       1 reflector.go:871] pkg/mod/k8s.io/client-go@v0.31.3/tools/cache/reflector.go:243: Watch close - *v1beta1.Machine total 0 items received                                                      │
│ I0609 11:14:21.34

This might be a red-herring. Still looking.

[richardcase]

Also saw this error:

E0609 11:17:50.737080       1 controller.go:316] "Reconciler error" err="failed to reconcile machine pool for AWSManagedMachinePool eks-nodes-blnyr0/eks-nodes-0dja0i-pool-0: failed to wait for nodegroup to be active:  │
│ failed to wait for EKS nodegroup \"eks-nodes-blnyr0_eks-nodes-0dja0i-pool-0\": maximum wait time for waiter must be greater than zero" controller="awsmanagedmachinepool" controllerGroup="infrastructure.cluster.x-k8s.i │
│ o" controllerKind="AWSManagedMachinePool" AWSManagedMachinePool="eks-nodes-blnyr0/eks-nodes-0dja0i-pool-0" namespace="eks-nodes-blnyr0" name="eks-nodes-0dja0i-pool-0" reconcileID="b01f7221-9b82-4f14-a191-90a