minor cleanup

s-sajid-ali · s-sajid-ali · commit 946161a0130c · 2025-02-14T17:10:28.000-05:00
renamed:    docs/hpc/04_datasets/squash_file_system_and_singularity.md -&gt; docs/hpc/04_datasets/02_squash_file_system_and_singularity.md
	new file:   docs/hpc/06_tools_and_software/01_intro.md
	renamed:    docs/hpc/06_tools_and_software/conda_environments.md -&gt; docs/hpc/06_tools_and_software/02_conda_environments.md
	new file:   docs/hpc/07_containers/01_intro.md
	renamed:    docs/hpc/07_containers/singularity_with_conda.md -&gt; docs/hpc/07_containers/02_singularity_with_conda.md
	new file:   docs/hpc/07_containers/03_squash_file_system_and_singularity.md
	renamed:    docs/hpc/08_ood/open_on_demand.md -&gt; docs/hpc/08_ood/01_open_on_demand.md
	modified:   docs/hpc/09_tutorials/02_hpc_foundations.mdx
diff --git a/docs/hpc/04_datasets/02_squash_file_system_and_singularity.md b/docs/hpc/04_datasets/02_squash_file_system_and_singularity.md
@@ -1,6 +1,6 @@
 # Squash File System and Singularity
 
-View available datasets on the [Datasets page](./datasets.md).
+View available datasets on the [Datasets page](../04_datasets/01_intro.md).
 
 ## Working with Datasets
 Writable ext3 overlay images have conda environments installed inside, Singularity can work with squashFS for fixed datasets, such as the coco datasets.
diff --git a/docs/hpc/06_tools_and_software/01_intro.md b/docs/hpc/06_tools_and_software/01_intro.md
@@ -0,0 +1 @@
+# Tools and Software
diff --git a/docs/hpc/06_tools_and_software/02_conda_environments.md b/docs/hpc/06_tools_and_software/02_conda_environments.md
diff --git a/docs/hpc/07_containers/01_intro.md b/docs/hpc/07_containers/01_intro.md
@@ -0,0 +1 @@
+# Apptainer/Singularity 
diff --git a/docs/hpc/07_containers/02_singularity_with_conda.md b/docs/hpc/07_containers/02_singularity_with_conda.md
@@ -287,7 +287,7 @@ pip install tensorboard
 ```
 
 :::note
-[Click here](./conda_environments.md) for information on how to configure your conda environment.
+[Click here](../06_tools_and_software/02_conda_environments.md) for information on how to configure your conda environment.
 :::
 
 Please also keep in mind that once the overlay image is opened in default read-write mode, the file will be locked. You will not be able to open it from a new process. Once the overlay is opened either in read-write or read-only mode, it cannot be opened in RW mode from other processes either. For production jobs to run, the overlay image should be open in read-only mode. You can run many jobs at the same time as long as they are run in read-only mode. In this ways, it will protect the computation software environment, software packages are not allowed to change when there are jobs running. 
diff --git a/docs/hpc/07_containers/03_squash_file_system_and_singularity.md b/docs/hpc/07_containers/03_squash_file_system_and_singularity.md
@@ -0,0 +1,64 @@
+# Squash File System and Singularity
+
+View available datasets on the [Datasets page](../04_datasets/01_intro.md).
+
+## Working with Datasets
+Writable ext3 overlay images have conda environments installed inside, Singularity can work with squashFS for fixed datasets, such as the coco datasets.
+
+```sh
+/scratch/work/public/ml-datasets/coco/coco-2014.sqf
+/scratch/work/public/ml-datasets/coco/coco-2015.sqf
+/scratch/work/public/ml-datasets/coco/coco-2017.sqf
+
+
+singularity exec \
+--overlay /scratch/wang/zzz/pytorch1.8.0-cuda11.1.ext3:ro \
+--overlay /scratch/work/public/ml-datasets/coco/coco-2014.sqf:ro \
+--overlay /scratch/work/public/ml-datasets/coco/coco-2015.sqf:ro \
+--overlay /scratch/work/public/ml-datasets/coco/coco-2017.sqf:ro \
+/scratch/work/public/singularity/cuda11.1-cudnn8-devel-ubuntu18.04.sif /bin/bash
+```
+
+If you have many tiny files as fixed datasets, please make squashFS files to work with Singularity. Here is an example
+
+1.  Make a temporary folder in /state/partition1, it is a folder in local hard drive on each computer node
+```sh
+mkdir -p /state/partition1/sw77
+cd /state/partition1/sw77
+```
+
+2.  Unzip files there, for example
+```sh
+tar -vxzf /scratch/work/public/examples/squashfs/imagenet-example.tar.gz
+```
+
+3.  Change access permissions in case we'll share files with others
+```sh
+find imagenet-example -type d -exec chmod 755 {} \;
+find imagenet-example -type f -exec chmod 644 {} \;
+```
+
+4.  Convert to a single squashFS file on host
+```sh
+mksquashfs imagenet-example imagenet-example.sqf  -keep-as-directory
+```
+For more details on working with squashFS, please see [this page from the SquashFS documentation](http://www.iitk.ac.in/LDP/HOWTO/SquashFS-HOWTO/mksqoverview.html). 
+
+5.  Copy this file to /scratch
+```sh
+cp -rp /state/partition1/sw77/imagenet-example.sqf /scratch/sw77/.
+```
+
+6.  To test, files are in /imagenet-example inside Singularity container 
+```sh
+singularity exec --overlay /scratch/sw77/imagenet-example.sqf:ro /scratch/work/public/singularity/ubuntu-20.04.1.sif /bin/bash
+Singularity> find /imagenet-example | wc -l
+1303
+Singularity> find /state/partition1/sw77/imagenet-example | wc -l
+1303
+```
+
+7.  To delete the tempoary folder on host
+```sh
+rm -rf /state/partition1/sw77
+```
diff --git a/docs/hpc/08_ood/01_open_on_demand.md b/docs/hpc/08_ood/01_open_on_demand.md
@@ -43,7 +43,7 @@ cd /scratch/$USER/my_env
 cp -rp /scratch/work/public/overlay-fs-ext3/overlay-15GB-500K.ext3.gz .
 gunzip overlay-15GB-500K.ext3.gz
 ```
-Above we used the overlay file "overlay-15GB-500K.ext3.gz" which will contain all of the installed packages. There are more optional overlay files. You can find instructions on the following pages: [Singularity with Conda](./singularity_with_conda.md), [Squash File System and Singularity](./squash_file_system_and_singularity.md). 
+Above we used the overlay file "overlay-15GB-500K.ext3.gz" which will contain all of the installed packages. There are more optional overlay files. You can find instructions on the following pages: [Singularity with Conda](../07_containers/02_singularity_with_conda.md), [Squash File System and Singularity](../07_containers/03_squash_file_system_and_singularity.md). 
 
 ### Launch Singularity Environment for Installation
 ```sh
diff --git a/docs/hpc/09_tutorials/02_hpc_foundations.mdx b/docs/hpc/09_tutorials/02_hpc_foundations.mdx
@@ -109,7 +109,7 @@ Similar to `/home`, users have access to multiple filesystems that are :
 | /scratch   | /scratch/**Net_ID**/ | General Storage | $SCRATCH
 | /archive   | /archive/**Net_ID**/ | Cold Storage    | $ARCHIVE
 
-You will find more details about these filesystems at [Greene Storage Types page](../06_spec_sheet.mdx).
+You will find more details about these filesystems at [Greene Storage Types page](../11_spec_sheet.mdx).
 
 You can jump to your `/scratch` directory at `/scratch/Net_ID/` with the `cd` command as `cd /scratch/Net_ID`, Or you could simple use the `$SCRATCH` environment variable as:
 
