:bug: [BUG]: Failing CI tests from nbval output diff

[fmigneault]

Summary

Following the DACCS CI Jenkins update to handle JupyterHub v5 (#493), multiple tests are failing.

It seems to be related to output diffs, either from jupyter itself or underlying packages installed in the docker of workflow-tests.

Details

From http://daccs-jenkins.crim.ca/job/PAVICS-e2e-workflow-tests/job/master/29/consoleFull

19:48:05  + py.test --rootdir=. --nbval notebooks-auth/geoserver.ipynb notebooks-auth/test_cowbird_jupyter.ipynb notebooks-auth/test_thredds.ipynb pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb pavics-sdi-master/docs/source/notebooks/FAQ_dask_parallel.ipynb pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb pavics-sdi-master/docs/source/notebooks/climex.ipynb pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb pavics-sdi-master/docs/source/notebooks/forecasts.ipynb pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb pavics-sdi-master/docs/source/notebooks/opendap.ipynb pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb pavics-sdi-master/docs/source/notebooks/regridding.ipynb pavics-sdi-master/docs/source/notebooks/rendering.ipynb pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb pavics-sdi-master/docs/source/notebooks/subsetting.ipynb pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb finch-master/docs/source/notebooks/dap_subset.ipynb finch-master/docs/source/notebooks/finch-usage.ipynb PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb raven-main/docs/source/notebooks/Region_selection.ipynb raven-main/docs/source/notebooks/Subset_climate_data_over_watershed.ipynb RavenPy-master/docs/notebooks/00_Introduction_to_JupyterLab.ipynb RavenPy-master/docs/notebooks/01_Getting_watershed_boundaries.ipynb RavenPy-master/docs/notebooks/02_Extract_geographical_watershed_properties.ipynb RavenPy-master/docs/notebooks/03_Extracting_forcing_data.ipynb RavenPy-master/docs/notebooks/04_Emulating_hydrological_models.ipynb RavenPy-master/docs/notebooks/05_Advanced_RavenPy_configuration.ipynb RavenPy-master/docs/notebooks/06_Raven_calibration.ipynb RavenPy-master/docs/notebooks/07_Making_and_using_hotstart_files.ipynb RavenPy-master/docs/notebooks/08_Getting_and_bias_correcting_CMIP6_data.ipynb RavenPy-master/docs/notebooks/09_Hydrological_impacts_of_climate_change.ipynb RavenPy-master/docs/notebooks/10_Data_assimilation.ipynb RavenPy-master/docs/notebooks/11_Climatological_ESP_forecasting.ipynb RavenPy-master/docs/notebooks/12_Performing_hindcasting_experiments.ipynb RavenPy-master/docs/notebooks/Assess_probabilistic_flood_risk.ipynb RavenPy-master/docs/notebooks/Comparing_hindcasts_and_ESP_forecasts.ipynb RavenPy-master/docs/notebooks/Distributed_hydrological_modelling.ipynb RavenPy-master/docs/notebooks/HydroShare_integration.ipynb RavenPy-master/docs/notebooks/Hydrological_realtime_forecasting.ipynb RavenPy-master/docs/notebooks/Managing_Jupyter_Environments.ipynb RavenPy-master/docs/notebooks/Perform_Regionalization.ipynb RavenPy-master/docs/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb RavenPy-master/docs/notebooks/Sensitivity_analysis.ipynb RavenPy-master/docs/notebooks/time_series_analysis.ipynb RavenPy-master/docs/notebooks/paper/Perform_a_climate_change_impact_study_on_a_watershed.ipynb notebooks/hummingbird.ipynb notebooks/stress-tests.ipynb --nbval-sanitize-with notebooks/output-sanitize.cfg --dist=loadscope --numprocesses=0
19:48:08  ============================= test session starts ==============================
19:48:08  platform linux -- Python 3.11.10, pytest-8.3.3, pluggy-1.5.0
19:48:08  rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
19:48:08  plugins: anyio-4.6.2.post1, dash-2.18.1, nbval-0.11.0, tornasync-0.6.0.post2, xdist-3.6.1
19:48:08  collected 551 items
19:48:08  
19:48:16  notebooks-auth/geoserver.ipynb ..................                        [  3%]
19:49:27  notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  5%]
19:49:27  notebooks-auth/test_thredds.ipynb ...........                            [  7%]
19:51:09  pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [  8%]
20:33:00  pavics-sdi-master/docs/source/notebooks/FAQ_dask_parallel.ipynb .FFFFFFF [  9%]
20:33:00                                                                           [  9%]
20:33:00  pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [ 10%]
20:33:04  pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [ 11%]
20:42:08  pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 14%]
20:42:08  pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 14%]
20:42:16  ...............                                                          [ 17%]
20:42:24  pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 17%]
20:42:31  pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb .......           [ 19%]
20:42:46  pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 20%]
20:42:51  pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 21%]
20:42:56  pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 22%]
20:46:20  pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 25%]
20:47:27  .............                                                            [ 27%]
20:47:29  pavics-sdi-master/docs/source/notebooks/rendering.ipynb .....            [ 28%]
20:47:31  pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 29%]
20:47:52  .................                                                        [ 33%]
20:48:00  pavics-sdi-master/docs/source/notebooks/subsetting.ipynb ......          [ 34%]
20:48:02  pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 34%]
20:48:24  .........                                                                [ 35%]
20:48:34  finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 37%]
20:48:43  finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 39%]
20:48:44  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 39%]
20:48:49  .....                                                                    [ 40%]
20:49:56  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 40%]
20:50:15  ............                                                             [ 42%]
20:50:53  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 42%]
20:51:33  .....s.                                                                  [ 43%]
20:51:39  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 44%]
20:51:48  ..                                                                       [ 44%]
20:52:10  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 44%]
20:53:06  F........                                                                [ 46%]
20:53:28  PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 46%]
20:59:20  ....                                                                     [ 47%]
20:59:20  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 47%]
20:59:20  ....                                                                     [ 48%]
20:59:20  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 48%]
20:59:29  .....                                                                    [ 49%]
20:59:34  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 49%]
20:59:48  .............                                                            [ 51%]
20:59:53  PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 51%]
20:59:54  ......                                                                   [ 52%]
21:00:07  raven-main/docs/source/notebooks/Region_selection.ipynb .........        [ 54%]
21:00:09  raven-main/docs/source/notebooks/Subset_climate_data_over_watershed.ipynb . [ 54%]
21:00:35  ...F..                                                                   [ 55%]
21:00:37  RavenPy-master/docs/notebooks/00_Introduction_to_JupyterLab.ipynb ...... [ 56%]
21:00:37                                                                           [ 56%]
21:00:39  RavenPy-master/docs/notebooks/01_Getting_watershed_boundaries.ipynb .... [ 57%]
21:00:48  ....                                                                     [ 58%]
21:00:54  RavenPy-master/docs/notebooks/02_Extract_geographical_watershed_properties.ipynb . [ 58%]
21:01:00  ..............                                                           [ 61%]
21:01:32  RavenPy-master/docs/notebooks/03_Extracting_forcing_data.ipynb ......... [ 62%]
21:02:41  ..                                                                       [ 63%]
21:02:45  RavenPy-master/docs/notebooks/04_Emulating_hydrological_models.ipynb ... [ 63%]
21:02:55  .................                                                        [ 66%]
21:03:06  RavenPy-master/docs/notebooks/05_Advanced_RavenPy_configuration.ipynb .. [ 67%]
21:03:15  FFFF.......                                                              [ 69%]
21:03:28  RavenPy-master/docs/notebooks/06_Raven_calibration.ipynb ......          [ 70%]
21:03:33  RavenPy-master/docs/notebooks/07_Making_and_using_hotstart_files.ipynb . [ 70%]
21:03:38  .....                                                                    [ 71%]
21:03:40  RavenPy-master/docs/notebooks/08_Getting_and_bias_correcting_CMIP6_data.ipynb . [ 71%]
21:11:39  ................                                                         [ 74%]
21:11:42  RavenPy-master/docs/notebooks/09_Hydrological_impacts_of_climate_change.ipynb . [ 74%]
21:11:52  ....                                                                     [ 75%]
21:12:30  RavenPy-master/docs/notebooks/10_Data_assimilation.ipynb ........        [ 76%]
21:12:52  RavenPy-master/docs/notebooks/11_Climatological_ESP_forecasting.ipynb .. [ 77%]
21:13:17  ......                                                                   [ 78%]
21:13:35  RavenPy-master/docs/notebooks/12_Performing_hindcasting_experiments.ipynb . [ 78%]
21:13:46  .......                                                                  [ 79%]
21:13:53  RavenPy-master/docs/notebooks/Assess_probabilistic_flood_risk.ipynb .... [ 80%]
21:14:23  ....                                                                     [ 81%]
21:14:45  RavenPy-master/docs/notebooks/Comparing_hindcasts_and_ESP_forecasts.ipynb . [ 81%]
21:14:46  ..FFFFF                                                                  [ 82%]
21:14:52  RavenPy-master/docs/notebooks/Distributed_hydrological_modelling.ipynb . [ 82%]
21:15:14  ......                                                                   [ 83%]
21:15:18  RavenPy-master/docs/notebooks/HydroShare_integration.ipynb FFFF          [ 84%]
21:15:40  RavenPy-master/docs/notebooks/Hydrological_realtime_forecasting.ipynb .. [ 84%]
21:15:46  ....                                                                     [ 85%]
21:15:58  RavenPy-master/docs/notebooks/Managing_Jupyter_Environments.ipynb ...    [ 86%]
21:16:27  RavenPy-master/docs/notebooks/Perform_Regionalization.ipynb .......      [ 87%]
21:16:27  RavenPy-master/docs/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb . [ 87%]
21:16:49  .............                                                            [ 90%]
21:17:09  RavenPy-master/docs/notebooks/Sensitivity_analysis.ipynb .....           [ 90%]
21:17:17  RavenPy-master/docs/notebooks/time_series_analysis.ipynb ...........     [ 92%]
21:17:39  RavenPy-master/docs/notebooks/paper/Perform_a_climate_change_impact_study_on_a_watershed.ipynb . [ 93%]
21:24:27  ....................                                                     [ 96%]
21:24:30  notebooks/hummingbird.ipynb ............                                 [ 98%]
21:26:54  notebooks/stress-tests.ipynb ......                                      [100%]
21:26:54  
21:26:54  =================================== FAILURES ===================================

To Reproduce

Steps to reproduce the behavior:

1. Run the CI pipeline with everything by default (auto resolves all main/master branches, default docker tag, etc.)

Related issues

• Failing CI nbval output on single notebook Ouranosinc/PAVICS-landing#103

[tlvu]

FAQ_dask_parallel.ipynb is a brand new notebook since 4 days not "Jenkins proof" yet, I only have error with this one. Will try to fix that one.

PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb is a known weirdo error already tracked by Ouranosinc/PAVICS-landing#103.

Your other errors are Raven's notebooks? You intending to run them everytime now? They are known to not all pass. I don't think they are related to JupyterHub v5.

We never ran Raven notebooks during the regular pipeline before anyways. Any specific reason you want to activate them during the pipeline now?

[fmigneault]

I noticed from previous runs that many repos that were originally tested were omitted, so I decided to toggle the test switches dynamically according to the service/components enabled by EXTRA_CONFIG_DIRS in the CI. Since raven in included in the test configuration, its tests are run as well. I don't see a reason to skip it, since it is part of the stack. It was probably skipped just because of an undefined default TEST_RAVEN_REPO=false. From all tests available in https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/master/testall, only TEST_ESGF_COMPUTE_API_REPO should now be disabled because their notebooks don't actually point at the instance being tested.

[tlvu]

Raven nbs were not enabled by default because they were very unstable at the time. Now things are getting better. I could take some time to get them to pass in Jenkins.

[tlvu]

@fmigneault Quick update:

FAQ_dask_parallel.ipynb is a brand new notebook since 4 days not "Jenkins proof" yet, I only have error with this one. Will try to fix that one.

Was fixed in Ouranosinc/pavics-sdi#344

Your other errors are Raven's notebooks?

All except one will be fixed by
Ouranosinc/raven#554 (already done)
Ouranosinc/PAVICS-e2e-workflow-tests#142 (pending review)
Ouranosinc/jenkins-config#17 (pending review)

The only last Raven nb failure is Comparing_hindcasts_and_ESP_forecasts.ipynb. This one is a regression in the code. @Zeitsperre will be looking at this one.

[tlvu]

@fmigneault This issue should be completely fixed with this last PR CSHS-CWRA/RavenPy#463