reorganize sections

s-sajid-ali · s-sajid-ali · commit 04e103b83dcd · 2025-02-14T17:19:58.000-05:00
modified:   docs/hpc/01_getting_started/_category_.json
	modified:   docs/hpc/02_connecting_to_hpc/_category_.json
	deleted:    docs/hpc/03_navigating_the_cluster/_category_.json
	new file:   docs/hpc/03_storage/01_intro.md
	new file:   docs/hpc/03_storage/02_available_storage_systems.md
	new file:   docs/hpc/03_storage/03_data_transfers.md
	new file:   docs/hpc/03_storage/04_globus.md
	new file:   docs/hpc/03_storage/_category_.json
	new file:   docs/hpc/04_datasets/01_intro.md
	renamed:    docs/hpc/08_ood/squash_file_system_and_singularity.md -&gt; docs/hpc/04_datasets/02_squash_file_system_and_singularity.md
	new file:   docs/hpc/04_datasets/_category_.json
	deleted:    docs/hpc/05_slurm/_category_.json
	deleted:    docs/hpc/05_slurm/slurm.md
	renamed:    docs/hpc/05_slurm/slurm_submitting_jobs.md -&gt; docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md
	renamed:    docs/hpc/05_slurm/slurm_tutorial.md -&gt; docs/hpc/05_submitting_jobs/02_slurm_tutorial.md
	renamed:    docs/hpc/05_slurm/slurm_main_commands.md -&gt; docs/hpc/05_submitting_jobs/03_slurm_main_commands.md
	new file:   docs/hpc/05_submitting_jobs/_category_.json
	new file:   docs/hpc/06_tools_and_software/01_intro.md
	renamed:    docs/hpc/08_ood/conda_environments.md -&gt; docs/hpc/06_tools_and_software/02_conda_environments.md
	new file:   docs/hpc/06_tools_and_software/_category_.json
	new file:   docs/hpc/07_containers/01_intro.md
	new file:   docs/hpc/07_containers/02_containers.md
	renamed:    docs/hpc/08_ood/singularity_with_conda.md -&gt; docs/hpc/07_containers/03_singularity_with_conda.md
	new file:   docs/hpc/07_containers/04_squash_file_system_and_singularity.md
	renamed:    docs/hpc/08_ood/open_on_demand.md -&gt; docs/hpc/08_ood/01_open_on_demand.md
	deleted:    docs/hpc/08_ood/datasets.md
	renamed:    docs/hpc/03_navigating_the_cluster/01_linux_tutorial.mdx -&gt; docs/hpc/09_tutorials/01_linux_tutorial.mdx
	renamed:    docs/hpc/03_navigating_the_cluster/02_hpc_foundations.mdx -&gt; docs/hpc/09_tutorials/02_hpc_foundations.mdx
	new file:   docs/hpc/09_tutorials/_category_.json
	renamed:    docs/hpc/04_training_and_support/01_training_at_nyu.mdx -&gt; docs/hpc/10_training_and_support/01_training_at_nyu.mdx
	renamed:    docs/hpc/04_training_and_support/02_training_outside.mdx -&gt; docs/hpc/10_training_and_support/02_training_outside.mdx
	renamed:    docs/hpc/04_training_and_support/03_support.mdx -&gt; docs/hpc/10_training_and_support/03_support.mdx
	renamed:    docs/hpc/04_training_and_support/_category_.json -&gt; docs/hpc/10_training_and_support/_category_.json
	renamed:    docs/hpc/06_spec_sheet.mdx -&gt; docs/hpc/11_spec_sheet.mdx
	renamed:    docs/hpc/07_system_status.mdx -&gt; docs/hpc/12_system_status.mdx
diff --git a/docs/hpc/01_getting_started/_category_.json b/docs/hpc/01_getting_started/_category_.json
@@ -1,4 +1,3 @@
 {
   "label": "Getting Started",
-  "position": 1,
 }
diff --git a/docs/hpc/02_connecting_to_hpc/_category_.json b/docs/hpc/02_connecting_to_hpc/_category_.json
@@ -1,4 +1,3 @@
 {
-  "label": "Connecting to HPC cluster",
-  "position": 1,
+  "label": "Setup",
 }
diff --git a/docs/hpc/03_navigating_the_cluster/_category_.json b/docs/hpc/03_navigating_the_cluster/_category_.json
diff --git a/docs/hpc/03_storage/01_intro.md b/docs/hpc/03_storage/01_intro.md
@@ -0,0 +1 @@
+# Available storage systems
diff --git a/docs/hpc/03_storage/02_available_storage_systems.md b/docs/hpc/03_storage/02_available_storage_systems.md
@@ -0,0 +1 @@
+# Available storage systems
diff --git a/docs/hpc/03_storage/03_data_transfers.md b/docs/hpc/03_storage/03_data_transfers.md
@@ -0,0 +1 @@
+# Data transfers
diff --git a/docs/hpc/03_storage/04_globus.md b/docs/hpc/03_storage/04_globus.md
@@ -0,0 +1 @@
+# Globus
diff --git a/docs/hpc/03_storage/_category_.json b/docs/hpc/03_storage/_category_.json
@@ -0,0 +1,3 @@
+{
+  "label": "Storage & Data transfers",
+}
diff --git a/docs/hpc/04_datasets/01_intro.md b/docs/hpc/04_datasets/01_intro.md
@@ -0,0 +1 @@
+# Datasets available
diff --git a/docs/hpc/04_datasets/02_squash_file_system_and_singularity.md b/docs/hpc/04_datasets/02_squash_file_system_and_singularity.md
@@ -1,6 +1,6 @@
 # Squash File System and Singularity
 
-View available datasets on the [Datasets page](./datasets.md).
+View available datasets on the [Datasets page](../04_datasets/01_intro.md).
 
 ## Working with Datasets
 Writable ext3 overlay images have conda environments installed inside, Singularity can work with squashFS for fixed datasets, such as the coco datasets.
diff --git a/docs/hpc/04_datasets/_category_.json b/docs/hpc/04_datasets/_category_.json
@@ -0,0 +1,3 @@
+{
+  "label": "Datasets",
+}
diff --git a/docs/hpc/05_slurm/_category_.json b/docs/hpc/05_slurm/_category_.json
diff --git a/docs/hpc/05_slurm/slurm.md b/docs/hpc/05_slurm/slurm.md
diff --git a/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md b/docs/hpc/05_submitting_jobs/01_slurm_submitting_jobs.md
@@ -1,23 +1,5 @@
----
-sidebar_position: 3
-toc_max_heading_level: 2
----
-
 # Slurm - Submitting Jobs
 
-- [Batch vs Interactive Jobs](#batch-vs-interactive-jobs)
-
-- [Batch Jobs](#batch-jobs)
-
-- [R Job Example](#r-job-example)
-
-- [Array Jobs](#array-jobs)
-
-- [More Examples](#more-examples)
-
-- [GPU Jobs](#gpu-jobs)
-
-- [Interactive Jobs](#interactive-jobs)
 
 ## Batch vs Interactive Jobs
 
diff --git a/docs/hpc/05_submitting_jobs/02_slurm_tutorial.md b/docs/hpc/05_submitting_jobs/02_slurm_tutorial.md
@@ -1,28 +1,5 @@
----
-sidebar_position: 2 
----
-
 # Slurm Tutorial
 
-[getting and renewing an account page]: ../01_getting_started/02_getting_and_renewing_an_account.md
-
-- [Introduction to High Performance Computing Clusters](#introduction-to-high-performance-computing-clusters)
-
-- [Slurm Commands](#slurm-commands)
-
-- [Software and environment modules](#software-and-environment-modules)
-
-- [Batch Job Example](#batch-job-example)
-
-- [Interactive Job Example](#interactive-job-example)
-
-- [MPI Job Example](#mpi-job-example)
-
-- [GPU Job Example](#gpu-job-example)
-
-- [Array Job Example](#array-job-example)
-
-
 ## Introduction to High Performance Computing Clusters
 
 In a High Performance Computing Cluster, such as the NYU-IT HPC Greene cluster, there are hundreds of computing nodes interconnected by high-speed networks.
diff --git a/docs/hpc/05_submitting_jobs/03_slurm_main_commands.md b/docs/hpc/05_submitting_jobs/03_slurm_main_commands.md
@@ -1,8 +1,3 @@
----
-sidebar_position: 3
-toc_max_heading_level: 3
----
-
 # Slurm: Main Commands
 
 Slurm offers many utility commands to work with, some of the most popularly used commands are:
diff --git a/docs/hpc/05_submitting_jobs/_category_.json b/docs/hpc/05_submitting_jobs/_category_.json
@@ -0,0 +1,3 @@
+{
+  "label": "Submitting jobs",
+}
diff --git a/docs/hpc/06_tools_and_software/01_intro.md b/docs/hpc/06_tools_and_software/01_intro.md
@@ -0,0 +1 @@
+# Tools and Software
diff --git a/docs/hpc/06_tools_and_software/02_conda_environments.md b/docs/hpc/06_tools_and_software/02_conda_environments.md
diff --git a/docs/hpc/06_tools_and_software/_category_.json b/docs/hpc/06_tools_and_software/_category_.json
@@ -0,0 +1,3 @@
+{
+  "label": "Tools & Software available",
+}
diff --git a/docs/hpc/07_containers/01_intro.md b/docs/hpc/07_containers/01_intro.md
@@ -0,0 +1 @@
+# Apptainer/Singularity 
diff --git a/docs/hpc/07_containers/02_containers.md b/docs/hpc/07_containers/02_containers.md
@@ -0,0 +1 @@
+# Basics of containers
diff --git a/docs/hpc/07_containers/03_singularity_with_conda.md b/docs/hpc/07_containers/03_singularity_with_conda.md
@@ -287,7 +287,7 @@ pip install tensorboard
 ```
 
 :::note
-[Click here](./conda_environments.md) for information on how to configure your conda environment.
+[Click here](../06_tools_and_software/02_conda_environments.md) for information on how to configure your conda environment.
 :::
 
 Please also keep in mind that once the overlay image is opened in default read-write mode, the file will be locked. You will not be able to open it from a new process. Once the overlay is opened either in read-write or read-only mode, it cannot be opened in RW mode from other processes either. For production jobs to run, the overlay image should be open in read-only mode. You can run many jobs at the same time as long as they are run in read-only mode. In this ways, it will protect the computation software environment, software packages are not allowed to change when there are jobs running. 
diff --git a/docs/hpc/07_containers/04_squash_file_system_and_singularity.md b/docs/hpc/07_containers/04_squash_file_system_and_singularity.md
@@ -0,0 +1,64 @@
+# Squash File System and Singularity
+
+View available datasets on the [Datasets page](../04_datasets/01_intro.md).
+
+## Working with Datasets
+Writable ext3 overlay images have conda environments installed inside, Singularity can work with squashFS for fixed datasets, such as the coco datasets.
+
+```sh
+/scratch/work/public/ml-datasets/coco/coco-2014.sqf
+/scratch/work/public/ml-datasets/coco/coco-2015.sqf
+/scratch/work/public/ml-datasets/coco/coco-2017.sqf
+
+
+singularity exec \
+--overlay /scratch/wang/zzz/pytorch1.8.0-cuda11.1.ext3:ro \
+--overlay /scratch/work/public/ml-datasets/coco/coco-2014.sqf:ro \
+--overlay /scratch/work/public/ml-datasets/coco/coco-2015.sqf:ro \
+--overlay /scratch/work/public/ml-datasets/coco/coco-2017.sqf:ro \
+/scratch/work/public/singularity/cuda11.1-cudnn8-devel-ubuntu18.04.sif /bin/bash
+```
+
+If you have many tiny files as fixed datasets, please make squashFS files to work with Singularity. Here is an example
+
+1.  Make a temporary folder in /state/partition1, it is a folder in local hard drive on each computer node
+```sh
+mkdir -p /state/partition1/sw77
+cd /state/partition1/sw77
+```
+
+2.  Unzip files there, for example
+```sh
+tar -vxzf /scratch/work/public/examples/squashfs/imagenet-example.tar.gz
+```
+
+3.  Change access permissions in case we'll share files with others
+```sh
+find imagenet-example -type d -exec chmod 755 {} \;
+find imagenet-example -type f -exec chmod 644 {} \;
+```
+
+4.  Convert to a single squashFS file on host
+```sh
+mksquashfs imagenet-example imagenet-example.sqf  -keep-as-directory
+```
+For more details on working with squashFS, please see [this page from the SquashFS documentation](http://www.iitk.ac.in/LDP/HOWTO/SquashFS-HOWTO/mksqoverview.html). 
+
+5.  Copy this file to /scratch
+```sh
+cp -rp /state/partition1/sw77/imagenet-example.sqf /scratch/sw77/.
+```
+
+6.  To test, files are in /imagenet-example inside Singularity container 
+```sh
+singularity exec --overlay /scratch/sw77/imagenet-example.sqf:ro /scratch/work/public/singularity/ubuntu-20.04.1.sif /bin/bash
+Singularity> find /imagenet-example | wc -l
+1303
+Singularity> find /state/partition1/sw77/imagenet-example | wc -l
+1303
+```
+
+7.  To delete the tempoary folder on host
+```sh
+rm -rf /state/partition1/sw77
+```
diff --git a/docs/hpc/08_ood/01_open_on_demand.md b/docs/hpc/08_ood/01_open_on_demand.md
@@ -43,7 +43,7 @@ cd /scratch/$USER/my_env
 cp -rp /scratch/work/public/overlay-fs-ext3/overlay-15GB-500K.ext3.gz .
 gunzip overlay-15GB-500K.ext3.gz
 ```
-Above we used the overlay file "overlay-15GB-500K.ext3.gz" which will contain all of the installed packages. There are more optional overlay files. You can find instructions on the following pages: [Singularity with Conda](./singularity_with_conda.md), [Squash File System and Singularity](./squash_file_system_and_singularity.md). 
+Above we used the overlay file "overlay-15GB-500K.ext3.gz" which will contain all of the installed packages. There are more optional overlay files. You can find instructions on the following pages: [Singularity with Conda](../07_containers/03_singularity_with_conda.md), [Squash File System and Singularity](../07_containers/04_squash_file_system_and_singularity.md). 
 
 ### Launch Singularity Environment for Installation
 ```sh
diff --git a/docs/hpc/08_ood/datasets.md b/docs/hpc/08_ood/datasets.md
diff --git a/docs/hpc/09_tutorials/01_linux_tutorial.mdx b/docs/hpc/09_tutorials/01_linux_tutorial.mdx
diff --git a/docs/hpc/09_tutorials/02_hpc_foundations.mdx b/docs/hpc/09_tutorials/02_hpc_foundations.mdx
@@ -109,7 +109,7 @@ Similar to `/home`, users have access to multiple filesystems that are :
 | /scratch   | /scratch/**Net_ID**/ | General Storage | $SCRATCH
 | /archive   | /archive/**Net_ID**/ | Cold Storage    | $ARCHIVE
 
-You will find more details about these filesystems at [Greene Storage Types page](../06_spec_sheet.mdx).
+You will find more details about these filesystems at [Greene Storage Types page](../11_spec_sheet.mdx).
 
 You can jump to your `/scratch` directory at `/scratch/Net_ID/` with the `cd` command as `cd /scratch/Net_ID`, Or you could simple use the `$SCRATCH` environment variable as:
 
diff --git a/docs/hpc/09_tutorials/_category_.json b/docs/hpc/09_tutorials/_category_.json
@@ -0,0 +1,3 @@
+{
+  "label": "Tutorials from old website",
+}
diff --git a/docs/hpc/10_training_and_support/01_training_at_nyu.mdx b/docs/hpc/10_training_and_support/01_training_at_nyu.mdx
diff --git a/docs/hpc/10_training_and_support/02_training_outside.mdx b/docs/hpc/10_training_and_support/02_training_outside.mdx
diff --git a/docs/hpc/10_training_and_support/03_support.mdx b/docs/hpc/10_training_and_support/03_support.mdx
diff --git a/docs/hpc/10_training_and_support/_category_.json b/docs/hpc/10_training_and_support/_category_.json
@@ -1,4 +1,3 @@
 {
   "label": "Training and Support",
-  "position": 4,
 }
diff --git a/docs/hpc/11_spec_sheet.mdx b/docs/hpc/11_spec_sheet.mdx
diff --git a/docs/hpc/12_system_status.mdx b/docs/hpc/12_system_status.mdx

Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,3 @@`
`1`	`1`	`{`
`2`	`2`	`"label": "Getting Started",`
`3`		`- "position": 1,`
`4`	`3`	`}`
Original file line number	Diff line number	Diff line change
`@@ -1,4 +1,3 @@`
`1`	`1`	`{`
`2`		`- "label": "Connecting to HPC cluster",`
`3`		`- "position": 1,`
	`2`	`+ "label": "Setup",`
`4`	`3`	`}`
Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,3 @@`
	`1`	`+{`
	`2`	`+ "label": "Storage & Data transfers",`
	`3`	`+}`