Skip to content

Commit f941ff1

Browse files
committed
added lang spec to code blocks except julia which needs to be installed
1 parent 4c40025 commit f941ff1

File tree

2 files changed

+70
-66
lines changed

2 files changed

+70
-66
lines changed

docs/hpc/08_ood/open_on_demand.md

Lines changed: 20 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,13 @@ This page describes how to use your Singularity with conda environment in Open O
99
The following commands must be run from the terminal. Information on accessing via the terminal can be found at the [Connecting to the HPC page](../02_connecting_to_hpc/01_connecting_to_hpc.md).
1010

1111
### Preinstallation Warning
12-
If you have initialized Conda in your base environment (your prompt on Greene may show something like `(base) [NETID@log-1 ~]$)` then you must first comment out or remove this portion of your ~/.bashrc file:
13-
12+
If you have initialized Conda in your base environment, your prompt on Greene may show something like:
13+
```sh
14+
(base) [NETID@log-1 ~]$
1415
```
16+
then you must first comment out or remove this portion of your `~/.bashrc` file:
17+
18+
```bash
1519
# >>> conda initialize >>>
1620
# !! Contents within this block are managed by 'conda init' !!
1721
__conda_setup="$('/share/apps/anaconda3/2020.07/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
@@ -31,7 +35,7 @@ unset __conda_setup
3135
The above code automatically makes your environment look for the default shared installation of Conda on the cluster and will sabotage any attempts to install packages to a Singularity environment. Once removed or commented out, log out and back into the cluster for a fresh environment.
3236

3337
### Prepare Overlay File
34-
```
38+
```sh
3539
mkdir /scratch/$USER/my_env
3640
cd /scratch/$USER/my_env
3741
cp -rp /scratch/work/public/overlay-fs-ext3/overlay-15GB-500K.ext3.gz .
@@ -40,26 +44,26 @@ gunzip overlay-15GB-500K.ext3.gz
4044
Above we used the overlay file "overlay-15GB-500K.ext3.gz" which will contain all of the installed packages. There are more optional overlay files. You can find instructions on the following pages: [Singularity with Conda](./singularity_with_conda.md), [Squash File System and Singularity](./squash_file_system_and_singularity.md).
4145

4246
### Launch Singularity Environment for Installation
43-
```
47+
```sh
4448
singularity exec --overlay /scratch/$USER/my_env/overlay-15GB-500K.ext3:rw /scratch/work/public/singularity/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif /bin/bash
4549
```
4650
Above we used the Singularity OS image "cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif " which provides the base operating system environment for the conda environment. There are other Singularity OS images available at `/scratch/work/public/singularity`
4751

4852
Launching Singularity with the --overlay flag mounts the overlay file to a new directory: /ext3 - you will notice that when not using Singularity /ext3 is not available. Be sure that you have the Singularity prompt (Singularity>) and that /ext3 is available before the next step:
49-
```
53+
```sh
5054
Singularity> ls -lah /ext3
5155
total 8.5K
5256
drwxrwxr-x. 2 root root 4.0K Oct 19 10:01 .
5357
drwx------. 29 root root 8.0K Oct 19 10:01 ..
5458
```
5559

5660
### Install Miniforge to Overlay File
57-
```
61+
```sh
5862
wget --no-check-certificate https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
5963
sh Miniforge3-Linux-x86_64.sh -b -p /ext3/miniforge3
6064
```
6165
Next, create a wrapper script at /ext3/env.sh
62-
```
66+
```sh
6367
touch /ext3/env.sh
6468
echo '#!/bin/bash' >> /ext3/env.sh
6569
echo 'unset -f which' >> /ext3/env.sh
@@ -68,7 +72,7 @@ echo 'export PATH=/ext3/miniforge3/bin:$PATH' >> /ext3/env.sh
6872
echo 'export PYTHONPATH=/ext3/miniforge3/bin:$PATH' >> /ext3/env.sh
6973
```
7074
Your /ext3/env.sh file should now contain the following:
71-
```
75+
```bash
7276
#!/bin/bash
7377
unset -f which
7478
source /ext3/miniforge3/etc/profile.d/conda.sh
@@ -78,21 +82,21 @@ export PYTHONPATH=/ext3/miniforge3/bin:$PATH
7882
The wrapper script will activate your conda environment, to which you will be installing your packages and dependencies.
7983

8084
Next, activate your conda environment with the following:
81-
```
85+
```sh
8286
source /ext3/env.sh
8387
```
8488

8589
### Install Packages to Miniforge Environment
8690
Now that your environment is activated, you can update and install packages
87-
```
91+
```sh
8892
conda config --remove channels defaults
8993
conda update -n base conda -y
9094
conda clean --all --yes
9195
conda install pip --yes
9296
conda install ipykernel --yes # Note: ipykernel is required to run as a kernel in the Open OnDemand Jupyter Notebooks
9397
```
9498
To confirm that your environment is appropriately referencing your Miniforge installation, try out the following:
95-
```
99+
```sh
96100
unset which
97101
which conda
98102
# output: /ext3/miniforge3/bin/conda
@@ -110,7 +114,7 @@ which pip
110114
Now use either conda install or pip to install your required python packages to the Miniforge environment.
111115

112116
To install larger packages, like Tensorflow, you must first start an interactive job with adequate compute and memory resources to install packages. The login nodes restrict memory to 2GB per user, which may cause some large packages to crash.
113-
```
117+
```sh
114118
srun --cpus-per-task=2 --mem=10GB --time=04:00:00 --pty /bin/bash
115119

116120
# wait to be assigned a node
@@ -125,7 +129,7 @@ After it is running, you’ll be redirected to a compute node. From there, run s
125129

126130
### Configure iPython Kernels
127131
To create a kernel named my_env copy the template files to your home directory.
128-
```
132+
```sh
129133
mkdir -p ~/.local/share/jupyter/kernels
130134
cd ~/.local/share/jupyter/kernels
131135
cp -R /share/apps/mypy/src/kernel_template ./my_env # this should be the name of your Singularity env
@@ -140,7 +144,7 @@ To set the conda environment, edit the file named 'python' in /.local/share/jupy
140144
The python file is a wrapper script that the Jupyter notebook will use to launch your Singularity container and attach it to the notebook.
141145

142146
At the bottom of the file we have the template singularity command.
143-
```
147+
```sh
144148
singularity exec $nv \
145149
--overlay /scratch/$USER/my_env/overlay-15GB-500K.ext3:ro \
146150
/scratch/work/public/singularity/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif \
@@ -149,7 +153,7 @@ singularity exec $nv \
149153
***WARNING:*** If you used a different overlay (/scratch/$USER/my_env/overlay-15GB-500K.ext3 shown above) or .sif file (/scratch/work/public/singularity/cuda12.3.2-cudnn9.0.0-ubuntu-22.04.4.sif shown above), you MUST change those lines in the command above to the files you used.
150154

151155
Edit the default kernel.json file by setting PYTHON_LOCATION and KERNEL_DISPLAY_NAME using a text editor like nano/vim.
152-
```
156+
```json
153157
{
154158
"argv": [
155159
"PYTHON_LOCATION",
@@ -163,7 +167,7 @@ Edit the default kernel.json file by setting PYTHON_LOCATION and KERNEL_DISPLAY_
163167
}
164168
```
165169
to
166-
```
170+
```json
167171
{
168172
"argv": [
169173
"/home/<Your NetID>/.local/share/jupyter/kernels/my_env/python",

0 commit comments

Comments
 (0)