Investigate Konnect timeout issue in tests #3633
Description
Problem statement
Recently, we've experienced intermittent client timeouts in tests running against Konnect in GHA.
For example: https://github.com/Kong/kong-operator/actions/runs/23238301487/job/67547894552
2026-03-18T09:44:41.0128575Z 2026-03-18T09:44:41Z ERROR KongService operation in Konnect API failed {"konnect_id": "8c4b4475-574e-43c7-8f8f-7cf30e6d2b3f", "op": "update", "duration": "10.000861398s", "error": "failed to update KongService f60f78f1-95db-47c0-b936-3acf27be5d3e/kongservice-784f5cb3: error sending request: Put \"https://us.api.konghq.tech/v2/control-planes/ec0672b4-fdd6-49ff-a473-6c9a7da744e0/core-entities/services/8c4b4475-574e-43c7-8f8f-7cf30e6d2b3f\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2026-03-18T09:44:41.0136542Z 2026-03-18T09:44:41Z ERROR KongRoute operation in Konnect API failed {"konnect_id": "aba3b4f0-0358-4a2c-b977-02b8b25eefb7", "op": "update", "duration": "10.000984258s", "error": "failed to update KongRoute f60f78f1-95db-47c0-b936-3acf27be5d3e/kongroute-fdf41eea: error sending request: Put \"https://us.api.konghq.tech/v2/control-planes/ec0672b4-fdd6-49ff-a473-6c9a7da744e0/core-entities/routes/aba3b4f0-0358-4a2c-b977-02b8b25eefb7\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2026-03-18T09:44:41.0237219Z 2026-03-18T09:44:41Z ERROR KongService failed to update {"konnect_id": "8c4b4475-574e-43c7-8f8f-7cf30e6d2b3f", "error": "failed to update KongService f60f78f1-95db-47c0-b936-3acf27be5d3e/kongservice-784f5cb3: error sending request: Put \"https://us.api.konghq.tech/v2/control-planes/ec0672b4-fdd6-49ff-a473-6c9a7da744e0/core-entities/services/8c4b4475-574e-43c7-8f8f-7cf30e6d2b3f\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2026-03-18T09:44:41.0240877Z github.com/kong/kong-operator/v2/controller/konnect.(*KonnectEntityReconciler[...]).Reconcile
2026-03-18T09:44:41.0242273Z /home/runner/work/kong-operator/kong-operator/controller/konnect/reconciler_generic.go:644
2026-03-18T09:44:41.0243697Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-03-18T09:44:41.0245116Z /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222
2026-03-18T09:44:41.0246564Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-03-18T09:44:41.0248051Z /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479
2026-03-18T09:44:41.0249347Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-03-18T09:44:41.0250729Z /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438
2026-03-18T09:44:41.0251778Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
2026-03-18T09:44:41.0254062Z /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313
2026-03-18T09:44:41.0258074Z 2026-03-18T09:44:41Z ERROR KongRoute failed to update {"konnect_id": "aba3b4f0-0358-4a2c-b977-02b8b25eefb7", "error": "failed to update KongRoute f60f78f1-95db-47c0-b936-3acf27be5d3e/kongroute-fdf41eea: error sending request: Put \"https://us.api.konghq.tech/v2/control-planes/ec0672b4-fdd6-49ff-a473-6c9a7da744e0/core-entities/routes/aba3b4f0-0358-4a2c-b977-02b8b25eefb7\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}
2026-03-18T09:44:41.0261656Z github.com/kong/kong-operator/v2/controller/konnect.(*KonnectEntityReconciler[...]).Reconcile
2026-03-18T09:44:41.0263139Z /home/runner/work/kong-operator/kong-operator/controller/konnect/reconciler_generic.go:644
2026-03-18T09:44:41.0264315Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Reconcile
2026-03-18T09:44:41.0265591Z /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:222
2026-03-18T09:44:41.0266875Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).reconcileHandler
2026-03-18T09:44:41.0268258Z /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:479
2026-03-18T09:44:41.0269621Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).processNextWorkItem
2026-03-18T09:44:41.0270993Z /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:438
2026-03-18T09:44:41.0272305Z sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller[...]).Start.func1.1
2026-03-18T09:44:41.0273897Z /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.23.3/pkg/internal/controller/controller.go:313
2026-03-18T09:44:41.0322719Z 2026-03-18T09:44:41Z DEBUG KongRoute reconciling {"konnect_id": "aba3b4f0-0358-4a2c-b977-02b8b25eefb7"}
2026-03-18T09:44:41.0341843Z 2026-03-18T09:44:41Z DEBUG KongService reconciling {"konnect_id": "8c4b4475-574e-43c7-8f8f-7cf30e6d2b3f"}
2026-03-18T09:44:41.0417572Z 2026-03-18T09:44:41Z DEBUG KongRoute Resource modified {"konnect_id": "aba3b4f0-0358-4a2c-b977-02b8b25eefb7"}
2026-03-18T09:44:41.0634913Z delete_test.go:22:
2026-03-18T09:44:41.0637232Z Error Trace: /home/runner/work/kong-operator/kong-operator/test/helpers/eventually/doesnotexist.go:40
2026-03-18T09:44:41.0639366Z /opt/hostedtoolcache/go/1.26.1/x64/src/runtime/asm_amd64.s:1771
2026-03-18T09:44:41.0640227Z Error: Should be true
2026-03-18T09:44:41.0641261Z delete_test.go:22:
2026-03-18T09:44:41.0643679Z Error Trace: /home/runner/work/kong-operator/kong-operator/test/helpers/eventually/doesnotexist.go:37
2026-03-18T09:44:41.0646442Z /home/runner/work/kong-operator/kong-operator/test/integration/delete_test.go:22
2026-03-18T09:44:41.0648488Z /opt/hostedtoolcache/go/1.26.1/x64/src/testing/testing.go:1317
2026-03-18T09:44:41.0650401Z /opt/hostedtoolcache/go/1.26.1/x64/src/testing/testing.go:1667
2026-03-18T09:44:41.0652847Z /opt/hostedtoolcache/go/1.26.1/x64/src/testing/testing.go:2030
2026-03-18T09:44:41.0653943Z Error: Condition never satisfied
2026-03-18T09:44:41.0656795Z Test: TestKonnectExtension/Konnect_hybrid_ControlPlane/Mirror_ControlPlane/KonnectExtension_with_KonnectNamespacedRef_control_plane_ref/automatic_secret_provisioning
2026-03-18T09:44:41.0658518Z Messages: *v1alpha2.KonnectExtension f60f78f1-95db-47c0-b936-3acf27be5d3e/konnect-extension-r72wp still exists
2026-03-18T09:44:41.0659989Z --- FAIL: TestKonnectExtension/Konnect_hybrid_ControlPlane/Mirror_ControlPlane/KonnectExtension_with_KonnectNamespacedRef_control_plane_ref/automatic_secret_provisioning (50.46s)
2026-03-18T09:44:41.0661848Z --- FAIL: TestKonnectExtension/Konnect_hybrid_ControlPlane/Mirror_ControlPlane/KonnectExtension_with_KonnectNamespacedRef_control_plane_ref (69.51s)
2026-03-18T09:44:41.0663995Z delete_test.go:17: Deleting f60f78f1-95db-47c0-b936-3acf27be5d3e/kongroute-fdf41eea and waiting for it to be gone
2026-03-18T09:44:41.0690868Z delete_test.go:22: Waiting for *v1alpha1.KongRoute f60f78f1-95db-47c0-b936-3acf27be5d3e/kongroute-fdf41eea to disappear
2026-03-18T09:44:41.5639240Z 2026-03-18T09:44:41Z ERROR KongDataPlaneClientCertificate operation in Konnect API failed {"konnect_id": "07afd8fb-1b17-4f70-afde-66910f7df6a3", "op": "delete", "duration": "10.001151984s", "error": "failed to delete KongDataPlaneClientCertificate on Konnect: error sending request: Delete \"https://us.api.konghq.tech/v2/control-planes/ec0672b4-fdd6-49ff-a473-6c9a7da744e0/dp-client-certificates/07afd8fb-1b17-4f70-afde-66910f7df6a3\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"}