Enable Jupyter kernels for batch execution#61
Conversation
|
Thanks a lot @TuturBaba. Is your hypothesis that the conda package of ipykernel is broken? Or just too old? |
Dockerfile
Outdated
| 'r-tidyverse' \ | ||
| 'unixodbc' && \ | ||
| conda create -n ansible-kernel --yes && \ | ||
| conda run -n ansible-kernel pip install ipykernel bioblend galaxy-ie-helpers && \ |
There was a problem hiding this comment.
why do you install the ipykernel kernel in the ansible env?
There was a problem hiding this comment.
I just noticed that there are quite a few mistakes and that I didn’t fully understand what I had done. So, I’m reviewing my work and will get back to it later.
The main issue is with the metadata, as shown in the image. Since I focused on R and Python, I saw that I could modify the metadata using ipkernel. Without thinking too much, I applied the same method to other languages, but this only changed the name while the language remained Python.
So, I need to find a solution to ensure that when running jupyter kernelspec list, the different kernels are correctly configured. At the moment, with the current setup, only the kernels that come with the base image work: python3 and julia-1.10
There was a problem hiding this comment.
And I forgot, but even the kernel that come with base image don't work by batch
There was a problem hiding this comment.
Isn't that a problem https://github.com/anaconda/nb_conda_kernels tries to solve?
There was a problem hiding this comment.
Maybe, but I haven't found a complete solution, however, I used this
|
Hello, |
|
One RUN command is recommended or you need to use a multi-stage Docker build. This is all to reduce the size of the final image. |
Dockerfile
Outdated
| 'r-tidymodels' \ | ||
| 'r-tidyverse' \ | ||
| 'unixodbc' && \ | ||
| conda run -n rlang-kernel R -e 'IRkernel::installspec(user = TRUE)' && \ |
There was a problem hiding this comment.
You can also add here something like (name = 'rlang-kernel', displayname = 'R')
There was a problem hiding this comment.
and I guess for the other kernels as well
There was a problem hiding this comment.
Yes u can : conda run -n rlang-kernel R -e "IRkernel::installspec(user = TRUE, name = 'rlang-kernel', displayname = 'R')"
But it's also possible without specifying a name: conda run -n python-kernel-3.12 python -m ipykernel install --user
|
I have a question, Jupyter version 1.0.0 does not use a Conda envs. So I was wondering why you chose to create environments. I did a quick test, and it seems to work fine (without any complications). |
Before this change, there was an issue with the Dockerfile when trying to install R libraries using commands like system("conda install r-terra"). While the libraries were technically installed, they were not placed in the correct location. I confirmed this by checking with .libPaths().
After discovering the problem, I searched for a solution and found that by adding a .Rprofile file with the correct paths, the installation now works as expected. This modification sets the CONDA_PREFIX and updates the PATH in the .Rprofile to ensure R libraries are installed in the appropriate environment
|
Because people wanted to install new packages for R, python, c++ whatever. The only package manager that enables all this and does not waste resources on compling is conda (in our community) |
|
I understand the benefit of using Conda to manage dependencies, but why create a new environment ( PS: By the way, I made a commit that fixes my issue, so I think this version is working fine. |
|
If you mix many different languages and many different packages my assumption was that if you install new packages over time it gets more complicated and brittle. Maybe I'm wrong and this is overengeenered. Is now everything working for you? Can you maybe outline how you tested this locally and maybe add it to the readme? |
bgruening
left a comment
bgruening
left a comment
There was a problem hiding this comment.
Will merge in the next days if no one else has comments and release a new version.
|
Thank you Björn! I think the PR is not ready to merge for now, the issue for R is solved, but not for Python (in batch mode, when a conda package is installed, it works fine BUT when the code is trying to use the installed conda package, Arthur has an error as the PATH is not the good one). Arthur is trying to look at it more deeply and try to apply the best patch he can if possible for all envs. |
|
Hello Bjorn, |
|
Ok, just checking, there are no new commits, the current version is working for you? |
Hello, I’m going to explain my work during the testing phase. First, I use Conda packages for different languages. I use the python -m command to access certain package options. When I use this method to install the kernel, I don’t need nb_conda_kernels. So, I can remove nb_conda_kernels, and this allows the base image kernels (Julia and Python) to get the correct path and metadata again. That’s why I didn’t create a new Python environment, but I can create one if needed. However, like with R, the path won’t be correct. I think I can define a solution to fix that if you want. To test my Dockerfile, I use three scripts, which you can find on my GitHub: https://github.com/TuturBaba/Antarctic/tree/main/script_test_dockerfile . I only test R, Python, and Bash because I don’t have experience with other languages.I used basic commands, like doing math operations, writing to a file, and installing packages. To test this, I simply added the following line at the end of my Dockerfile: ADD ./R.ipynb /import/R.ipynb Then I run: RUN chmod -R 777 /import
There was a problem with "proj", the path after download was not the right one. Because of the way we manage R, it means that we can find other problems like
|
Hi Arthur, Björn. Seems to me this is ok to merge isn't it ? For links, related issues are usegalaxy-eu/galaxy#244 and usegalaxy-eu/galaxy#266 |
|
I will create a new release and a new Galaxy tool version. Thanks everyone! |
|
Ok, it is deployed: https://usegalaxy.eu/root?tool_id=interactive_tool_jupyter_notebook make sure to use the latest 1.0.2 version. |
|
@TuturBaba it seems this has broken some other functionality. Can you try to create a new env with some new packages? I think its not possible to change into this new env anymore via the UI.
Can you maybe create a small bash script that is testing your usecase, independent of Galaxy, just this container and a notebook? That would help me I guess to make sure your use-case is working but still not breaking the venv selector. |
|
@bgruening I didn’t fully understand what you want to do. conda create -n test-kernel python=3.10 ipykernel -y Also, running without Galaxy works for me because I use the container directly on my computer. |
3 participants
Hello,
I am a master's student doing my internship with @yvanlebras . I need to create a workflow on Galaxy and noticed an issue with the latest Jupyter tool in Galaxy. Since it cannot be used in batch mode, I looked for a simple way to fix it. From what I implemented, it seems to work. I used pip install inside Conda and, most importantly, ipykernel, which allows Jupyter to correctly recognize the kernel paths and manually name them.