Fails to delete CRD testclusters.test.cluster.x-k8s.io in TestReconcile/skip_all

[arshadd-b]

Which jobs are flaking?

capi-test-main
https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-cluster-api-test-main/1965630494284451840

Which tests are flaking?

TestReconcile/skip_all
sigs.k8s.io/cluster-api/controllers/crdmigrator: TestReconcile/skip_all

Since when has it been flaking?

8/28/2025 I can see firstoccurrence
https://storage.googleapis.com/k8s-triage/index.html?date=2025-09-11&text=testclusters.test.cluster.x-k8s.io%20is%20not%20being%20deleted%20from%20the%20testenv%20client%20cache&job=.*cluster-api.*(test%7Ce2e)-(mink8s-)*main&xjob=.*-provider-.*

Testgrid link

https://testgrid.k8s.io/cluster-api-core-main#capi-test-main

Reason for failure (if possible)

while cleaning up at the end of the test, it fails to delete this crd testclusters.test.cluster.x-k8s.io

    Expected success, but got an error:
        <errors.aggregate | len:1, cap:1>: 
        key /, Kind=, /testclusters.test.cluster.x-k8s.io is not being deleted from the testenv client cache: timed out waiting for the condition
        [
            <*errors.withStack | 0xc002552648>{
                error: <*errors.withMessage | 0xc002a4c420>{
                    cause: <wait.errInterrupted>{
                        cause: <*errors.errorString | 0xc0003383e0>{
                            s: "timed out waiting for the condition",
                        },
                    },
                    msg: "key /, Kind=, /testclusters.test.cluster.x-k8s.io is not being deleted from the testenv client cache",
                },
                stack: [0x3162376, 0x317410b, 0x3172d6a, 0x623c46, 0x4be5c1],
            },
        ]

Anything else we need to know?

No response

Label(s) to be applied

/kind flake
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[Shubhamag12]

I would like to take this up
/assign

[s3rj1k]

        message: "pping and waiting for non leader election runnables\"\nI0910 04:21:51.375114
          \  27914 internal.go:550] \"Stopping and waiting for warmup runnables\"\nI0910
          04:21:51.375190   27914 internal.go:559] \"Stopping and waiting for leader
          election runnables\"\nI0910 04:21:51.375257   27914 internal.go:567] \"Stopping
          and waiting for caches\"\nI0910 04:21:51.375348   27914 internal.go:571]
          \"Stopping and waiting for webhooks\"\nI0910 04:21:51.375546   27914 internal.go:574]
          \"Stopping and waiting for HTTP servers\"\nI0910 04:21:51.375639   27914
          internal.go:578] \"Wait completed, proceeding to shutdown the manager\"\n
          \   crd_migrator_test.go:166: \n        Expected success, but got an error:\n
          \           <errors.aggregate | len:1, cap:1>: \n            key /, Kind=,
          /testclusters.test.cluster.x-k8s.io is not being deleted from the testenv
          client cache: timed out waiting for the condition\n            [\n                <*errors.withStack
          | 0xc0020e2570>{\n                    error: <*errors.withMessage | 0xc0029571e0>{\n
          \                       cause: <wait.errInterrupted>{\n                            cause:
          <*errors.errorString | 0xc0003aa7c0>{\n                                s:
          \"timed out waiting for the condition\",\n                            },\n
          \                       },\n                        msg: \"key /, Kind=,
          /testclusters.test.cluster.x-k8s.io is not being deleted from the testenv
          client cache\",\n                    },\n                    stack: [0x3265836,
          0x32779eb, 0x327664a, 0x624446, 0x4be5c1],\n                },\n            ]\n\n===
          FAIL: controllers/crdmigrator TestReconcile (25.91s)\n\nDONE 6593 tests,
          12 skipped, 2 failures in 335.953s\nexit $(cat /logs/artifacts/junit.exitcode)\nmake:
          *** [Makefile:930: test-junit] Error 1\n+ EXIT_VALUE=2\n+ set +o xtrace\n{\"component\":\"entrypoint\",\"error\":\"wrapped
          process failed: exit status 2\",\"file\":\"sigs.k8s.io/prow/pkg/entrypoint/run.go:84\",\"func\":\"sigs.k8s.io/prow/pkg/entrypoint.Options.internalRun\",\"level\":\"error\",\"msg\":\"Error
          executing test process\",\"severity\":\"error\",\"time\":\"2025-09-10T04:25:39Z\"}\n"

happens somewhere after https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/manager/internal.go#L578

[fabriziopandini]

/triage accepted
/priority important-soon

[Shubhamag12]

I was trying reproduce this issue and am not able to do so locally.

In CleanupAndWait, we are checking cache for a certain period of time if the resource is deleted or not, and therefore I suspect that the cleanup is taking more time than expected (in some cases)

I added some debug logs and re-ran the TC.

I0918 14:13:18.045722  51696 environment.go:510] "[DEBUG]: object still exists in cache" gvk="/, Kind=" key="/testclusters.test.cluster.x-k8s.io" rv="343"
I0918 14:13:18.152894  51696 environment.go:510] "[DEBUG]: object still exists in cache" gvk="/, Kind=" key="/testclusters.test.cluster.x-k8s.io" rv="347"
I0918 14:13:18.310359  51696 environment.go:502] "[DEBUG]: object not found" gvk="/, Kind=" key="/testclusters.test.cluster.x-k8s.io" rv="347"

I feel like current timeout is already sufficient and we shouldn't increase this backoff period.
Open for suggestions.

[arshadd-b]

I see again this flake again today https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/periodic-cluster-api-test-main/1972524204335042560
@Shubhamag12 What i suggest we should check the logs and see why clean up is taking so much time.