katib-integration test failed #201
Description
Bug Description
While testing Charmed Kubeflow bundle 1.10/candidate, SQA found one failing run where the katib-integration test failed.
To Reproduce
Juju deploy bundle Charmed Kubeflow 1.10/candidate (using FCE).
Environment
Jammy, Juju 3.6.8, Charmed Kubernetes 1.33/stable.
Please see the run's page for more information and logs.
Relevant Log Output
------------------------------ Captured log call -------------------------------
INFO test_notebooks:test_notebooks.py:48 Note that only CPU tests will be run. In order to run tests that use an NVIDIA GPU,use the `--include-gpu-tests` flag e.g. `tox -e kubeflow-local -- --include-gpu-tests`. To learn more, use `--help` or refer to the repository's README file.
INFO test_notebooks:test_notebooks.py:55 Running katib-integration.ipynb...
ERROR test_notebooks:test_notebooks.py:69 Cell In[9], line 8, in assert_experiment_succeeded(client, experiment)
1 @retry(
2 wait=wait_exponential(multiplier=2, min=1, max=10),
3 stop=stop_after_attempt(30),
4 reraise=True,
5 )
6 def assert_experiment_succeeded(client, experiment):
7 """Wait for the Katib Experiment to complete successfully."""
----> 8 assert client.is_experiment_succeeded(name=experiment), f"Katib Experiment was not successful."
AssertionError: Katib Experiment was not successful.
=========================== short test summary info ============================
FAILED test_notebooks.py::test_notebook[katib-integration] - Failed: AssertionError: Katib Experiment was not successful.
============ 1 failed, 3 passed, 5 deselected in 1341.19s (0:22:21) ============
##### git-sync initContainer logs #####
##### test-kubeflow container logs #####
FAILEDAdditional Context
No response