Small updates by PeterSomkuti · Pull Request #3 · US-GHG-Center/custom-images

PeterSomkuti · 2025-05-18T22:56:16Z

I've added/modified the following:

minimal notebook file to make the precompile/build step within Jupyter (not sure why it's needed, but it is)
I made the two repos public now and they are added here
download the JulioMono font, which makes some text renderings in plots much nicer

Again - tested this in my local machine, but not yet on the hub.

chuckwondo · 2025-05-19T12:00:03Z

Dockerfile

+RUN julia -e 'using Pkg; Pkg.add(url="https://github.com/US-GHG-Center/RetrievalToolbox.jl");'
+
+# Add the demo notebook
+RUN git clone https://github.com/US-GHG-Center/IWGGMS21-Demo ~/IWGGMS21-Demo


Recall that we cannot put thing into the home dir. That's part of the issue we're having to deal with. Nothing placed into the home directory shows up in the hub sessions.

Does this demo repo need to be in the home dir? If so, then you simply need to clone it after your initial hub session starts.

Anything you place in your home directory during a hub session will remain in your home directory from session to session. We simply cannot populate the home directory during building of the image.

The idea for this image is that users at the conference can dig straight in, there should be no additional user commands needed to get into the notebook and execute the cells in the demo.
The location of the repo (I think) is not important; ideally the users thrown into this will see the opened notebook already. I'll investigate.

@PeterSomkuti we're going to achieve that through Binderhub launch link, not via the docker configuration. The Docker just needs the right environment. If the notebook repo is now public we should move this dockerfile into that repo and test with Binder.

@wildintellect I can move the Dockerfile (and the additional files needed) to the IWGGMS21-Demo repo. Does the Binder environment keep the AWS credentials as present in the "other" environment? The demo currently pulls the 3 big files from the S3 bucket right at the start.

Yes bucket access should work.

chuckwondo · 2025-05-19T12:02:35Z

Dockerfile

+RUN <<EOT
+   curl -OL https://github.com/cormullion/juliamono/releases/download/v0.060/JuliaMono-ttf.tar.gz
+   mkdir ~/.fonts
+   cd ~/.fonts
+   tar -xvzf ~/JuliaMono-ttf.tar.gz
+   rm ~/JuliaMono-ttf.tar.gz
+EOT


Again, we cannot place anything in the home dir. I suggest determining where else you can install fonts in the system so that they can be discovered. This may also require moving this block before the line USER ${NB_USER}, as I suspect installing fonts in some recognized system location may require root permissions.

chuckwondo · 2025-05-19T12:04:12Z

Dockerfile

+
+# Run the precompile/build within Jupyter (somehow that is necessary)
+COPY precompile_build.ipynb ./
+RUN jupyter execute precompile_build.ipynb && rm precompile_build.ipynb


Why do you need a notebook file to do this? Won't the following work?

RUN julia -e 'using Pkg; Pkg.precompile(); Pkg.build();'

That's similar to what I added only a few lines above.

I couldn't find out the reason why this is needed, but I'm not the only one seeing this link. The tiny notebook seems to solve the problem, though. Without this, another precompilation is triggered when users load any modules from a fresh session, so this cuts down the initial runtime of the whole demo notebook quite significantly.

I'm trying to see this impact, but can't. We can keep this for now, but I would like to explore this a bit more later, because I suspect there's a better way to address this.

chuckwondo · 2025-05-19T12:04:59Z

precompile_build.ipynb

I don't think this file is necessary. See my comment in the Dockerfile about adding the command there.

This step is required: one can easily test this by spawning a new session with custom image from chuckwondo/julia/retrievaltoolkit-docker and then typing using Plots in a notebook (not from the REPL!). One is greeted with an error related to "world age", that must be fixed through re-compilation within a notebook (though I do not know why), and this little notebook takes care of that.

chuckwondo · 2025-05-19T12:06:24Z

Dockerfile

+# Comment Peter: this is no longer necessary, I fixed my repo
+#    git -C xrtm reset --hard b3aeace


Nice to see you fixed your repo. You can remove these comment lines.

Suggested change

# Comment Peter: this is no longer necessary, I fixed my repo

# git -C xrtm reset --hard b3aeace

PeterSomkuti · 2025-05-20T16:06:25Z

@chuckwondo
Made some updates that move the git clone operation and the fonts into /opt; and while the image builds successfully, the NOTEBOOK_ARGS environment variable does not seem to work as intended (it does on a local run).
The idea is that for this image, the work directory will be /opt/IWGGMS21-Demo so that users only see the one notebook file when they run this. Is this achievable with the US-GHG-Center configuration at all?

chuckwondo · 2025-05-20T21:31:42Z

Dockerfile

 EOT
+
+# Add the RetrivalToolbox package
+RUN julia -e 'using Pkg; Pkg.add(url="https://github.com/US-GHG-Center/RetrievalToolbox.jl");'


Let's just add Pkg.add(url="https://github.com/US-GHG-Center/RetrievalToolbox.jl"); to the preceding julia command. No need to use 2 separate julia commands. Combining with the other will also avoid needlessly adding another layer to the Docker image.

chuckwondo · 2025-05-20T21:34:25Z

Dockerfile

+# Add the demo notebook into /opt/IWGGMS21-Demo
+USER root
+RUN git clone https://github.com/US-GHG-Center/IWGGMS21-Demo /opt/IWGGMS21-Demo && chown -R jovyan /opt/IWGGMS21-Demo


No need to switch back to root again. Simply move this git clone command up in the file, right after the block dealing with xrtm, before switching to NB_USER, since the root user will already be the current user.

Also, use ${NB_USER} instead of hard-coding jovyan.

We can also use git clone --depth 1 ... to reduce space requirements, unless you expect users to commit changes to the repo.

And the same can be done on the clone of the xrtm repo as well.

chuckwondo · 2025-05-20T21:35:19Z

Dockerfile

+RUN git clone https://github.com/US-GHG-Center/IWGGMS21-Demo /opt/IWGGMS21-Demo && chown -R jovyan /opt/IWGGMS21-Demo
+
+# Run the precompile/build within Jupyter (somehow that is necessary)
+USER ${NB_USER}


No need to switch back to NB_USER once you remove the preceeding USER root line, based upon my preceding comment.

chuckwondo · 2025-05-20T21:37:06Z

Dockerfile

+
+# Run the precompile/build within Jupyter (somehow that is necessary)
+COPY precompile_build.ipynb ./
+RUN jupyter execute precompile_build.ipynb && rm precompile_build.ipynb


I'm trying to see this impact, but can't. We can keep this for now, but I would like to explore this a bit more later, because I suspect there's a better way to address this.

chuckwondo · 2025-05-20T21:38:47Z

Dockerfile

+ENV NOTEBOOK_ARGS=/opt/IWGGMS21-Demo/IWGGMS-demonstration.ipynb
+
+# Clean up conda cache to reduce image size a little bit
+RUN conda clean --all -f -y


No sense in adding another layer to the image. Please just add this conda command up into the RUN block near the top of the file, where the other conda command is. Reducing the number of layers also helps reduce image size.

chuckwondo · 2025-05-20T21:42:33Z

Dockerfile

+# NOTEBOOK_ARGS set to this path will cause the Jupyter Lab environment 
+# to start in this folder, so users will see only this Notebook and thus
+# know which one to use.
+ENV NOTEBOOK_ARGS=/opt/IWGGMS21-Demo/IWGGMS-demonstration.ipynb


Let's move this up where the rest of the ENV directives are at the top of the file.

chuckwondo · 2025-05-20T22:11:57Z

Dockerfile

+# Download and install JuliaMono font for good rendering of plotting text
+RUN <<EOT
+   curl -OL https://github.com/cormullion/juliamono/releases/download/v0.060/JuliaMono-ttf.tar.gz
+   cd /usr/share/fonts
+   tar -xvzf /opt/JuliaMono-ttf.tar.gz
+   rm /opt/JuliaMono-ttf.tar.gz
+   chmod ugo+r /usr/share/fonts/*
+EOT


Suggested change

# Download and install JuliaMono font for good rendering of plotting text

RUN <<EOT

curl -OL https://github.com/cormullion/juliamono/releases/download/v0.060/JuliaMono-ttf.tar.gz

cd /usr/share/fonts

tar -xvzf /opt/JuliaMono-ttf.tar.gz

rm /opt/JuliaMono-ttf.tar.gz

chmod ugo+r /usr/share/fonts/*

EOT

# Download and install JuliaMono font for good rendering of plotting text

RUN <<EOT

#!/bin/bash

set -e

fonts_dir=/usr/share/fonts/truetype/julia

curl -OLs https://github.com/cormullion/juliamono/releases/download/v0.060/JuliaMono-ttf.tar.gz

mkdir -p "${fonts_dir}"

tar -xzf JuliaMono-ttf.tar.gz -C "${fonts_dir}"

rm JuliaMono-ttf.tar.gz

EOT

PeterSomkuti · 2025-05-26T20:16:57Z

Re-shuffled the Dockerfile, not tested in the US-GHG-Center jhub environment which seems to be different enough that the NOTEBOOK_ARGS env variable does not do what it's supposed to. Any ideas to sort this out?

Peter Somkuti and others added 2 commits May 18, 2025 17:29

Small changes, added JuliaMono font.

521e305

Added repos.

bb3edf0

chuckwondo requested changes May 19, 2025

View reviewed changes

Restructuring and moving deployments into /opt

69e630c

chuckwondo requested changes May 20, 2025

View reviewed changes

Re-shuffled Dockerfile.

2df0982

		# Comment Peter: this is no longer necessary, I fixed my repo
		# git -C xrtm reset --hard b3aeace

Conversation

PeterSomkuti commented May 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PeterSomkuti commented May 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PeterSomkuti commented May 26, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

PeterSomkuti commented May 20, 2025 •

edited

Loading