Running Jupyter Notebook from within the VENV produces error

[SamAhnert]

Running "jupyter notebook" to try and run the overview notebook after activating the venv gives me an error:

(firedrake) firedrake@bf0146359d08:/home/hydrogym/notebooks$ jupyter notebook
[I 2022-05-30 22:23:18.342 LabApp] JupyterLab extension loaded from /home/firedrake/firedrake/lib/python3.8/site-packages/jupyterlab
[I 2022-05-30 22:23:18.343 LabApp] JupyterLab application directory is /home/firedrake/firedrake/share/jupyter/lab
[I 22:23:18.362 NotebookApp] Serving notebooks from local directory: /home/hydrogym/notebooks
[I 22:23:18.362 NotebookApp] Jupyter Notebook 6.4.11 is running at:
[I 22:23:18.362 NotebookApp] http://localhost:8888/?token=1d0557ea0bb002ed0f3e6332ad9339742affd4ec77abb3e8
[I 22:23:18.362 NotebookApp] or http://127.0.0.1:8888/?token=1d0557ea0bb002ed0f3e6332ad9339742affd4ec77abb3e8
[I 22:23:18.363 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[W 22:23:18.378 NotebookApp] No web browser found: could not locate runnable browser.
[C 22:23:18.379 NotebookApp]

To access the notebook, open this file in a browser:
    file:///home/firedrake/.local/share/jupyter/runtime/nbserver-3742-open.html
Or copy and paste one of these URLs:
    http://localhost:8888/?token=1d0557ea0bb002ed0f3e6332ad9339742affd4ec77abb3e8
 or http://127.0.0.1:8888/?token=1d0557ea0bb002ed0f3e6332ad9339742affd4ec77abb3e8

then when trying to access the url, I am "unable to connect" (not sure if this is just a mac thing). I think this is to be expected when trying to run a Jupyter notebook on a Docker/virtual machine and then trying to access it locally since the docker file isn't exactly running locally (probably has it's own IP somewhere although I'm really not familiar with this sort of thing).

Not a priority type issue, but something I ran into as I'm trying to get more familiar to run a RL algorithm on here. I could try to work on it, but it's more of a "it would be nice to have this" if this were a polished product. For now it might just sidetrack from the main stuff, so I'll try to continue learning about the project another way (this thread may be promising for future reference: https://www.digitalocean.com/community/tutorials/how-to-install-run-connect-to-jupyter-notebook-on-remote-server).

[SamAhnert]

update: must execute the following command to run the container locally:

docker run --shm-size=1g -ti -v $PWD:/home/fenics/shared -p 8888:8888 hydrogym:latest

then must boot Jupiter notebook using the local address (need to activate the VENV before running this command to):

jupyter notebook --ip 0.0.0.0 --no-browser

A lot of outdated massages pop up, but at least the notebook boots.

(source: https://stackoverflow.com/questions/38830610/access-jupyter-notebook-running-on-docker-container)

[SamAhnert]

new problem arose in this thread trying to reboot the docker container from the same port. after closing the previous terminal, the port is still allocated for the the previous boot of the docker container even though nothing is running there. Might have to manually "terminate" the process or something. Will try a machine reboot as a potential short-term solution.

[SamAhnert]

solution is to manually kill the previous container before relaunching (makes me think that I might be able to just launch into a container without the "docker run..." command every time).

Example of this working:

(source: https://stackoverflow.com/questions/46176584/docker-bind-for-0-0-0-04000-failed-port-is-already-allocated)

[ludgerpaehler]

@SamAhnert @jcallaham wouldn't a conda environment install be a good alternative option for the dynamic dev environment?

Edit: conda is apparently deprecated by Firedrake itself, so pip it is...

@SamAhnert have you attempted whether you can launch a gitpod with the provided docker image?

[ludgerpaehler]

Also looking through the Dockerfile, I see you are on a MBPro Sam - right now the env is single-core due to this line in the Dockerfile:

NUM_THREADS=1

If you want to use a larger slice of your M1 (Max/Ultra/etc) you need to adjust that flag.

[ludgerpaehler]

If you want to reattach to the container, as you would reattach to a tmux session, then you need the docker exec command see here.

[SamAhnert]

@ludgerpaehler I have not tried to launch a gitpod (I've never actually used this before). Would be happy to learn more about it if you and @jcallaham think using this IDE would be advantageous for the project (my preliminary understanding is that gitpod is an online IDE).

[jcallaham]

@ludgerpaehler Definitely open to alternative or multiple development environments - I just set it up in Docker initially based on the firedrake images, but we could add some instructions/support for installing with the firedrake-install script and pip for this package. Is that more or less what you have in mind?

Also the container can run with multiple processors even with the OMP_NUM_THREADS=1 flag, since the parallelization is with MPI and not OpenMP. There's just a warning generated from Firedrake if you don't set that environment variable to 1, though I'm not totally sure why (something to do with BLAS, looks like?)

@SamAhnert Working with the ports and containers right now is definitely not very smooth, but that's basically what I do - one container with the port forwarding for Jupyter, and then attach to it with other terminals as Ludger mentioned.

[ludgerpaehler]

@jcallaham maybe we should consider to disentangle the two levels of computation from each other sooner rather than later to more easily advance development.

I.e. the gym environment and its API as a separate package, which then launches Firedrake application containers when prompted to do so. Would be easy to test on a single machine, and we could then slowly look towards scaling it to proper reinforcement learning training sizes.

Re a source-install, the easiest option would probably be a spack-install, especially with spack's recent upgrade with a package server running in the background. For quick development this might be the best option, and then we could downgrade a version of the container to an application container, which takes in as arguments the necessary commands to launch the simulation itself.