jupyter: new image to add esgf-pyclient and xncml to fix Jenkins failure by tlvu · Pull Request #308 · bird-house/birdhouse-deploy

tlvu · 2023-04-04T17:35:24Z

Jupyter: new image to add esgf-pyclient and xncml to fix Jenkins failure

See PR Ouranosinc/PAVICS-e2e-workflow-tests#118 for more details.

Adds esgf-pyclient for esgf-dap.ipynb (Modify nb on ESGF access to remove authentication process Ouranosinc/pavics-sdi#269)
Adds xncml for gen_catalog refactoring (Define data model for catalog attributes using pydantic Ouranosinc/pavics-vdb#46)
Fixes annoying harmless error ERROR 1: PROJ: proj_create_from_database: Open of /opt/conda/envs/birdy/share/proj failed
Relevant changes (alphabetical order):

>   - esgf-pyclient=0.3.1=pyh1a96a4e_2

<   - gdal=3.5.3=py38h1f15b03_4
>   - gdal=3.6.0=py38h58634bd_13

>     - xncml==0.2

…ix Jenkins failure (#118) # Overview This is a build on top of an existing build. A fresh full build because it would pull a newer RavenPy which would requires a newer RavenWPS, which do not start at this moment. This incremental build is faster to test and will fix our failing Jenkins due to missing package `esgf-pyclient`. It is annoying to have Jenkins failing since a while just for a missing package. ## Changes - Adds `esgf-pyclient` for esgf-dap.ipynb (Ouranosinc/pavics-sdi#269) - Adds `xncml` for gen_catalog refactoring (Ouranosinc/pavics-vdb#46) - Fixes annoying harmless error `ERROR 1: PROJ: proj_create_from_database: Open of /opt/conda/envs/birdy/share/proj failed` - Jenkins: lower job retention duration since we now have separated jobs for nightly trigger and manual trigger - Existing `intake-esm` pin: #109 - Relevant changes (alphabetical order): ```diff > - esgf-pyclient=0.3.1=pyh1a96a4e_2 < - gdal=3.5.3=py38h1f15b03_4 > - gdal=3.6.0=py38h58634bd_13 > - xncml==0.2 ``` ## Test - Deployed as "beta" image in production for bokeh visualization performance regression testing. - Manual test notebook https://github.com/Ouranosinc/PAVICS-landing/blob/master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb for bokeh visualization performance and it looks fine. - Jenkins build: one known intermittent error [job-PAVICS-e2e-workflow-tests-quick-fix-production-image-3-consoleText.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/11151086/job-PAVICS-e2e-workflow-tests-quick-fix-production-image-3-consoleText.txt) ## Related Issue / Discussion - Matching notebook fixes: - Pavics-sdi: Ouranosinc/pavics-sdi#285 - Deployment to PAVICS: bird-house/birdhouse-deploy#308 ## Additional Information Full diff conda env export: [221130-221130-update230403-conda-env-export.diff.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/11151115/221130-221130-update230403-conda-env-export.diff.txt) Full new conda env export: [221130-update230403-conda-env-export.yml.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/11151116/221130-update230403-conda-env-export.yml.txt) DockerHub build logs: [Dockerhub-buildlogs-pavics-workflow-tests-221130-update230403.txt](https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/files/11151131/Dockerhub-buildlogs-pavics-workflow-tests-221130-update230403.txt)

tlvu · 2023-04-05T03:01:09Z

@matprov The pipeline seems frozen again !

mishaschwartz · 2023-04-05T12:56:25Z

CHANGES.md

+  - Adds `esgf-pyclient` for esgf-dap.ipynb (https://github.com/Ouranosinc/pavics-sdi/pull/269)
+  - Adds `xncml` for gen_catalog refactoring (https://github.com/Ouranosinc/pavics-vdb/pull/46)
+  - Fixes annoying harmless error `ERROR 1: PROJ: proj_create_from_database: Open of /opt/conda/envs/birdy/share/proj failed`
+  - Relevant changes (alphabetical order):


Where are these changes actually made? I see the esgf-pyclient and xncml install in Ouranosinc/PAVICS-e2e-workflow-tests#118 but these specific versions aren't mentioned there. And I don't see the gdal change, unless it's a dependency that's not explicitly updated?

Hehe, let me quickly introduce you to our Jupyter env building process.

Incremental builds are builds on top of previously released Jupyter env. DockerHub will use the Dockerfile.testing to build for incremental builds. DockerHub knows which Dockerfile to use depending on the tag prefix. For incremental build, the tag prefix is dockerupdate-.

In the Dockerfile.testing here https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/pull/118/files#diff-b44bae84e9671e9e5f0a9f5a7f41cfb09c368daacc0873bab1cf3a57da8afdee you see I updated to the previous released version and I added esgf-client and xncml. The rest of the changes (gdal and others), are pull indirectly.

If you look at the PR description, you see at the bottom there are 221130-221130-update230403-conda-env-export.diff.txt which is the full diff vs the previous released Jupyter env. You'll see more than just gdal but I only report gdal because it can potentially cause problem.

Full conda env export like 221130-update230403-conda-env-export.yml.txt are generated during each Jenkins builds so we know exactly what runtime env produce the error during the build. See https://github.com/Ouranosinc/PAVICS-e2e-workflow-tests/blob/597529c35f0a9eabac92374b3797bff0f30509db/testall#L109-L114

The "Relevant changes" section is very "subjective". It's what I considered "of interest" or "can cause trouble".

Full DockerHub builds will use regular Dockerfile and triggered by tag prefix docker-.

The name of our Jupyter env is funny too, it's pavics/workflow-tests. Historically it was meant only for Jenkins to test the notebooks but then I realized I can deploy exactly the same image to production so I kill 2 birds with 1 stone. I test the notebooks and I guaranty the runtime image works because both the notebooks and the runtime image are tested together. I basically deploy exactly what I test.

UofT can have your own Jupyter env too. CRIM actually have 2 more Jupyter env (EO and NLP at https://github.com/crim-ca/pavics-jupyter-images) that they deploy alongside our image for Climate.

The Jupyterhub can be configured to display several choices of Jupyter env image for use to choose, see

birdhouse-deploy/birdhouse/env.local.example

Lines 232 to 260 in 60116e2

# Jupyter single-user server images

#export DOCKER_NOTEBOOK_IMAGES="pavics/workflow-tests:210216 \

# pavics/crim-jupyter-eo:0.3.0 \

# pavics/crim-jupyter-nlp:0.4.0"

# Name of the images displayed on the JupyterHub image selection page

# The name order must correspond to the order of the DOCKER_NOTEBOOK_IMAGES variable,

# and both variables should have the same number of entries.

# Note that the selection names are also used as directory names for the tutorial-notebooks directories mounted when

# starting the corresponding image. The name can use the '<name>' or the '<name>:<version>' format. The version will be

# excluded when mounting the corresponding directory.

#export JUPYTERHUB_IMAGE_SELECTION_NAMES="pavics \

# eo-crim:0.3.0 \

# nlp-crim"

# allow jupyterhub user selection of which notebook image to run

# see https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html

#export ENABLE_JUPYTERHUB_MULTI_NOTEBOOKS="

#c.DockerSpawner.image_whitelist = {os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split()[0]: os.environ['DOCKER_NOTEBOOK_IMAGES'].split()[0],

# os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split()[1]: os.environ['DOCKER_NOTEBOOK_IMAGES'].split()[1],

# os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split()[2]: os.environ['DOCKER_NOTEBOOK_IMAGES'].split()[2],

# 'jupyter/scipy-notebook': 'jupyter/scipy-notebook',

# 'jupyter/r-notebook': 'jupyter/r-notebook',

# 'jupyter/tensorflow-notebook': 'jupyter/tensorflow-notebook',

# 'jupyter/datascience-notebook': 'jupyter/datascience-notebook',

# 'jupyter/pyspark-notebook': 'jupyter/pyspark-notebook',

# 'jupyter/all-spark-notebook': 'jupyter/all-spark-notebook',

# }

#"

Thank you for the detailed explanation!

crim-jenkins-bot · 2023-04-05T15:40:47Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1416/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : new-jupyter-env
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-69.rdext.crim.ca

Infrastructure deployment failed. Instance has not been destroyed. @matprov

crim-jenkins-bot · 2023-04-05T17:02:52Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac/61/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : new-jupyter-env
DACCS_CONFIGS_BRANCH :
PAVICS_E2E_WORKFLOW_TESTS_BRANCH :
PAVICS_SDI_BRANCH :

DESTROY_INFRA_ON_EXIT :
PAVICS_HOST : https://

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL :

NOTEBOOK TEST RESULTS

    
</code>

matprov · 2023-04-05T17:25:05Z

The pipeline seems frozen again !

@tlvu Thanks for noticing me of this misbehaviour. I have taken a look but haven't found the issue. Restarting jenkins made queued job finish. I'll continue investigation when problem arise in the future but for now I think the pipeline can accept new jobs.

mishaschwartz · 2023-04-10T14:43:52Z

@matprov

It looks like the netcat command is hanging (I'm guessing it's not supposed to run for 13+ hours 😬):

Is it possible that netcat doesn't think that stdin is closed? We could add a timeout with netcat -w 5 ... (for a 5 second timeout).

mishaschwartz · 2023-04-10T15:46:43Z

run tests

crim-jenkins-bot · 2023-04-10T17:06:34Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1426/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : new-jupyter-env
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-36.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1058/

NOTEBOOK TEST RESULTS

    
[2023-04-10T16:08:01.679Z] ============================= test session starts ==============================
[2023-04-10T16:08:01.679Z] platform linux -- Python 3.8.16, pytest-7.2.0, pluggy-1.0.0
[2023-04-10T16:08:01.679Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master@2
[2023-04-10T16:08:01.679Z] plugins: anyio-3.6.2, dash-2.7.0, nbval-0.9.6, tornasync-0.6.0.post2
[2023-04-10T16:08:01.679Z] collected 268 items
[2023-04-10T16:08:01.679Z] 
[2023-04-10T16:08:11.172Z] notebooks-auth/test_thredds.ipynb ...........                            [  4%]
[2023-04-10T16:08:19.787Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb .......        [  6%]
[2023-04-10T16:08:28.964Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb ......         [  8%]
[2023-04-10T16:08:32.551Z] pavics-sdi-master/docs/source/notebooks/WMS_example.ipynb ......FF       [ 11%]
[2023-04-10T16:08:35.665Z] pavics-sdi-master/docs/source/notebooks/WPS_example.ipynb ..........     [ 15%]
[2023-04-10T16:16:25.446Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ............        [ 20%]
[2023-04-10T16:16:25.446Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 20%]
[2023-04-10T16:16:25.446Z] ...............                                                          [ 26%]
[2023-04-10T16:16:31.706Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 27%]
[2023-04-10T16:17:17.097Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ......            [ 30%]
[2023-04-10T16:17:36.359Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 32%]
[2023-04-10T16:17:37.754Z] pavics-sdi-master/docs/source/notebooks/jupyter_extensions.ipynb .       [ 32%]
[2023-04-10T16:17:42.573Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 35%]
[2023-04-10T16:17:49.354Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 37%]
[2023-04-10T16:20:23.294Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 42%]
[2023-04-10T16:21:40.174Z] .............                                                            [ 47%]
[2023-04-10T16:21:42.715Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 49%]
[2023-04-10T16:21:45.030Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 52%]
[2023-04-10T16:22:13.233Z] .................                                                        [ 58%]
[2023-04-10T16:22:19.467Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 60%]
[2023-04-10T16:22:21.373Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 60%]
[2023-04-10T16:22:37.441Z] .........                                                                [ 64%]
[2023-04-10T16:22:49.183Z] finch-master/docs/source/notebooks/dap_subset.ipynb ...........          [ 68%]
[2023-04-10T16:22:59.890Z] finch-master/docs/source/notebooks/finch-usage.ipynb ......              [ 70%]
[2023-04-10T16:23:38.199Z] finch-master/docs/source/notebooks/subset.ipynb ....................ss.  [ 79%]
[2023-04-10T16:23:40.744Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 79%]
[2023-04-10T16:23:50.858Z] ......                                                                   [ 81%]
[2023-04-10T16:26:57.409Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 82%]
[2023-04-10T16:28:07.270Z] .............                                                            [ 86%]
[2023-04-10T16:28:25.360Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 87%]
[2023-04-10T16:33:48.163Z] ....s.                                                                   [ 89%]
[2023-04-10T16:33:56.300Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 89%]
[2023-04-10T16:34:01.774Z] ...                                                                      [ 91%]
[2023-04-10T16:34:16.677Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 91%]
[2023-04-10T16:34:41.429Z] ......                                                                   [ 93%]
[2023-04-10T16:34:44.177Z] notebooks/hummingbird.ipynb ............                                 [ 98%]
[2023-04-10T16:38:29.775Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2023-04-10T16:38:29.775Z] 
[2023-04-10T16:38:29.775Z] =================================== FAILURES ===================================

crim-jenkins-bot · 2023-04-12T02:34:16Z

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/1442/
Result : failure

BIRDHOUSE_DEPLOY_BRANCH : new-jupyter-env
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-154.rdext.crim.ca

Infrastructure deployment failed. Instance has not been destroyed. @matprov

jupyter: new image to add esgf-pyclient and xncml to fix Jenkins failure

bf38f16

tlvu requested a review from mishaschwartz April 4, 2023 17:35

tlvu mentioned this pull request Apr 4, 2023

Jupyter docker: incremental build to add esgf-pyclient and xncml to fix Jenkins failure Ouranosinc/PAVICS-e2e-workflow-tests#118

Merged

mishaschwartz reviewed Apr 5, 2023

View reviewed changes

matprov mentioned this pull request Apr 5, 2023

Enforce the load order of components defined in env.local #304

Merged

mishaschwartz approved these changes Apr 10, 2023

View reviewed changes

tlvu added 2 commits April 11, 2023 22:12

Merge remote-tracking branch 'origin/master' into new-jupyter-env

76aadba

Bump version: 1.25.1 → 1.25.2

e8e7cfe

github-actions bot added ci/operations Continuous Integration components documentation Improvements or additions to documentation labels Apr 12, 2023

tlvu merged commit 19ba8b8 into master Apr 12, 2023

tlvu deleted the new-jupyter-env branch April 12, 2023 02:14

tlvu mentioned this pull request Apr 12, 2023

Canarie-api should not be a mandatory component #310

Merged

bird-house deleted a comment from crim-jenkins-bot Apr 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

jupyter: new image to add esgf-pyclient and xncml to fix Jenkins failure#308

jupyter: new image to add esgf-pyclient and xncml to fix Jenkins failure#308
tlvu merged 3 commits intomasterfrom
new-jupyter-env

tlvu commented Apr 4, 2023

Uh oh!

tlvu commented Apr 5, 2023

Uh oh!

mishaschwartz Apr 5, 2023

Uh oh!

tlvu Apr 5, 2023

Uh oh!

mishaschwartz Apr 6, 2023

Uh oh!

crim-jenkins-bot commented Apr 5, 2023

Uh oh!

crim-jenkins-bot commented Apr 5, 2023

Uh oh!

matprov commented Apr 5, 2023

Uh oh!

mishaschwartz commented Apr 10, 2023

Uh oh!

mishaschwartz commented Apr 10, 2023

Uh oh!

crim-jenkins-bot commented Apr 10, 2023

Uh oh!

crim-jenkins-bot commented Apr 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	# Jupyter single-user server images
	#export DOCKER_NOTEBOOK_IMAGES="pavics/workflow-tests:210216 \
	# pavics/crim-jupyter-eo:0.3.0 \
	# pavics/crim-jupyter-nlp:0.4.0"

	# Name of the images displayed on the JupyterHub image selection page
	# The name order must correspond to the order of the DOCKER_NOTEBOOK_IMAGES variable,
	# and both variables should have the same number of entries.
	# Note that the selection names are also used as directory names for the tutorial-notebooks directories mounted when
	# starting the corresponding image. The name can use the '<name>' or the '<name>:<version>' format. The version will be
	# excluded when mounting the corresponding directory.
	#export JUPYTERHUB_IMAGE_SELECTION_NAMES="pavics \
	# eo-crim:0.3.0 \
	# nlp-crim"

	# allow jupyterhub user selection of which notebook image to run
	# see https://jupyter-docker-stacks.readthedocs.io/en/latest/using/selecting.html
	#export ENABLE_JUPYTERHUB_MULTI_NOTEBOOKS="
	#c.DockerSpawner.image_whitelist = {os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split()[0]: os.environ['DOCKER_NOTEBOOK_IMAGES'].split()[0],
	# os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split()[1]: os.environ['DOCKER_NOTEBOOK_IMAGES'].split()[1],
	# os.environ['JUPYTERHUB_IMAGE_SELECTION_NAMES'].split()[2]: os.environ['DOCKER_NOTEBOOK_IMAGES'].split()[2],
	# 'jupyter/scipy-notebook': 'jupyter/scipy-notebook',
	# 'jupyter/r-notebook': 'jupyter/r-notebook',
	# 'jupyter/tensorflow-notebook': 'jupyter/tensorflow-notebook',
	# 'jupyter/datascience-notebook': 'jupyter/datascience-notebook',
	# 'jupyter/pyspark-notebook': 'jupyter/pyspark-notebook',
	# 'jupyter/all-spark-notebook': 'jupyter/all-spark-notebook',
	# }
	#"

Conversation

tlvu commented Apr 4, 2023

Uh oh!

tlvu commented Apr 5, 2023

Uh oh!

mishaschwartz Apr 5, 2023

Choose a reason for hiding this comment

Uh oh!

tlvu Apr 5, 2023

Choose a reason for hiding this comment

Uh oh!

mishaschwartz Apr 6, 2023

Choose a reason for hiding this comment

Uh oh!

crim-jenkins-bot commented Apr 5, 2023

E2E Test Results

DACCS-iac Pipeline Results

Uh oh!

crim-jenkins-bot commented Apr 5, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

Uh oh!

matprov commented Apr 5, 2023

Uh oh!

mishaschwartz commented Apr 10, 2023

Uh oh!

mishaschwartz commented Apr 10, 2023

Uh oh!

crim-jenkins-bot commented Apr 10, 2023

E2E Test Results

DACCS-iac Pipeline Results

PAVICS-e2e-workflow-tests Pipeline Results

Uh oh!

crim-jenkins-bot commented Apr 12, 2023

E2E Test Results

DACCS-iac Pipeline Results

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants