You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/hpc/08_ood/open_on_demand.md
+20-16Lines changed: 20 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,9 +9,13 @@ This page describes how to use your Singularity with conda environment in Open O
9
9
The following commands must be run from the terminal. Information on accessing via the terminal can be found at the [Connecting to the HPC page](../02_connecting_to_hpc/01_connecting_to_hpc.md).
10
10
11
11
### Preinstallation Warning
12
-
If you have initialized Conda in your base environment (your prompt on Greene may show something like `(base) [NETID@log-1 ~]$)` then you must first comment out or remove this portion of your ~/.bashrc file:
13
-
12
+
If you have initialized Conda in your base environment, your prompt on Greene may show something like:
13
+
```sh
14
+
(base) [NETID@log-1 ~]$
14
15
```
16
+
then you must first comment out or remove this portion of your `~/.bashrc` file:
17
+
18
+
```bash
15
19
# >>> conda initialize >>>
16
20
# !! Contents within this block are managed by 'conda init' !!
The above code automatically makes your environment look for the default shared installation of Conda on the cluster and will sabotage any attempts to install packages to a Singularity environment. Once removed or commented out, log out and back into the cluster for a fresh environment.
Above we used the overlay file "overlay-15GB-500K.ext3.gz" which will contain all of the installed packages. There are more optional overlay files. You can find instructions on the following pages: [Singularity with Conda](./singularity_with_conda.md), [Squash File System and Singularity](./squash_file_system_and_singularity.md).
41
45
42
46
### Launch Singularity Environment for Installation
Above we used the Singularity OS image "cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif " which provides the base operating system environment for the conda environment. There are other Singularity OS images available at `/scratch/work/public/singularity`
47
51
48
52
Launching Singularity with the --overlay flag mounts the overlay file to a new directory: /ext3 - you will notice that when not using Singularity /ext3 is not available. Be sure that you have the Singularity prompt (Singularity>) and that /ext3 is available before the next step:
The wrapper script will activate your conda environment, to which you will be installing your packages and dependencies.
79
83
80
84
Next, activate your conda environment with the following:
81
-
```
85
+
```sh
82
86
source /ext3/env.sh
83
87
```
84
88
85
89
### Install Packages to Miniforge Environment
86
90
Now that your environment is activated, you can update and install packages
87
-
```
91
+
```sh
88
92
conda config --remove channels defaults
89
93
conda update -n base conda -y
90
94
conda clean --all --yes
91
95
conda install pip --yes
92
96
conda install ipykernel --yes # Note: ipykernel is required to run as a kernel in the Open OnDemand Jupyter Notebooks
93
97
```
94
98
To confirm that your environment is appropriately referencing your Miniforge installation, try out the following:
95
-
```
99
+
```sh
96
100
unset which
97
101
which conda
98
102
# output: /ext3/miniforge3/bin/conda
@@ -110,7 +114,7 @@ which pip
110
114
Now use either conda install or pip to install your required python packages to the Miniforge environment.
111
115
112
116
To install larger packages, like Tensorflow, you must first start an interactive job with adequate compute and memory resources to install packages. The login nodes restrict memory to 2GB per user, which may cause some large packages to crash.
***WARNING:*** If you used a different overlay (/scratch/$USER/my_env/overlay-15GB-500K.ext3 shown above) or .sif file (/scratch/work/public/singularity/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif shown above), you MUST change those lines in the command above to the files you used.
150
154
151
155
Edit the default kernel.json file by setting PYTHON_LOCATION and KERNEL_DISPLAY_NAME using a text editor like nano/vim.
152
-
```
156
+
```json
153
157
{
154
158
"argv": [
155
159
"PYTHON_LOCATION",
@@ -163,7 +167,7 @@ Edit the default kernel.json file by setting PYTHON_LOCATION and KERNEL_DISPLAY_
0 commit comments