flaky e2e failure in openshift-ci

[jiridanek]

https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/opendatahub-io_kubeflow/562/pull-ci-opendatahub-io-kubeflow-main-odh-notebook-controller-e2e/1901200109006229504

=== RUN   TestE2ENotebookController/validate_controllers/Validate_Kubeflow_notebook_controller
2025/03/16 09:40:01 Error in notebook-controller-deployment deployment
2025/03/16 09:40:11 Error in notebook-controller-deployment deployment
2025/03/16 09:40:21 Error in notebook-controller-deployment deployment
2025/03/16 09:40:31 Error in notebook-controller-deployment deployment
2025/03/16 09:40:41 Error in notebook-controller-deployment deployment
[202](https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/opendatahub-io_kubeflow/562/pull-ci-opendatahub-io-kubeflow-main-odh-notebook-controller-e2e/1901200109006229504#1:build-log.txt%3A202)5/03/16 09:40:51 Failed to get notebook-controller-deployment controller deployment
    notebook_controller_test.go:36: 
        	Error Trace:	/go/src/github.com/opendatahub-io/kubeflow/components/odh-notebook-controller/e2e/notebook_controller_test.go:36
        	Error:      	Received unexpected error:
        	            	client rate limiter Wait returned an error: context deadline exceeded
        	Test:       	TestE2ENotebookController/validate_controllers/Validate_Kubeflow_notebook_controller
        	Messages:   	error in validating notebook controller
=== RUN   TestE2ENotebookController/validate_controllers/Validate_ODH_notebook_controller
--- FAIL: TestE2ENotebookController (70.05s)
    --- FAIL: TestE2ENotebookController/validate_controllers (70.05s)
        --- FAIL: TestE2ENotebookController/validate_controllers/Validate_Kubeflow_notebook_controller (60.04s)
        --- PASS: TestE2ENotebookController/validate_controllers/Validate_ODH_notebook_controller (10.01s)
FAIL
FAIL	github.com/opendatahub-io/kubeflow/components/odh-notebook-controller/e2e	70.063s
FAIL
make[1]: *** [e2e-test-oauth] Error 1

[jiridanek]

would've been helpful to run must-gather before teardown

something like odh-e2e used to do, dumping stuff in all test namespaces

[jiridanek]

been playing with this

• wip dump info when waiting #565

and did not really finish it; printing more info while waiting would not be bad

at the time I filled the issue, this fixed itself; and when I was experimenting, prolonging the timeout from 1 minute also helped

openshift-ci was just slow at the moment

[jiridanek]

Again here https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/opendatahub-io_kubeflow/574/pull-ci-opendatahub-io-kubeflow-main-odh-notebook-controller-e2e/1914957017806540800

From PR

• NO-JIRA: chore(docs): Add guide for upgrading Go version and dependencies #574 (comment)

2025/04/23 08:59:37 Failed to get thoth-minimal-oauth-notebook statefulset
    notebook_creation_test.go:50: 
        	Error Trace:	/go/src/github.com/opendatahub-io/kubeflow/components/odh-notebook-controller/e2e/notebook_creation_test.go:50
        	Error:      	Received unexpected error:
        	            	error validating notebook StatefulSet: client rate limiter Wait returned an error: rate: Wait(n=1) would exceed context deadline
        	Test:       	TestE2ENotebookController/create/thoth-minimal-oauth-notebook/Notebook_Statefulset_Validation
        	Messages:   	error testing StatefulSet for Notebook 
=== RUN   TestE2ENotebookController/create/thoth-minimal-oauth-notebook/Notebook_OAuth_sidecar_Validation
    notebook_creation_test.go:58: 
        	Error Trace:	/go/src/github.com/opendatahub-io/kubeflow/components/odh-notebook-controller/e2e/notebook_creation_test.go:58
        	Error:      	Received unexpected error:
        	            	notebook pod thoth-minimal-oauth-notebook-0 is not in Running phase
        	Test:       	TestE2ENotebookController/create/thoth-minimal-oauth-notebook/Notebook_OAuth_sidecar_Validation
        	Messages:   	error testing sidecar for Notebook 
=== RUN   TestE2ENotebookController/create/thoth-minimal-oauth-notebook/Verify_Notebook_Traffic
    notebook_creation_test.go:63: 
        	Error Trace:	/go/src/github.com/opendatahub-io/kubeflow/components/odh-notebook-controller/e2e/notebook_creation_test.go:63
        	Error:      	Received unexpected error:
        	            	unexpected response from Notebook Endpoint:
        	            	HTTP/1.0 503 Service Unavailable
        	            	Connection: close
        	            	Cache-Control: private, max-age=0, no-cache, no-store
        	            	Content-Type: text/html
        	            	Pragma: no-cache
        	            	
        	Test:       	TestE2ENotebookController/create/thoth-minimal-oauth-notebook/Verify_Notebook_Traffic
        	Messages:   	error testing Notebook traffic