make default containers directory alongside modules for faster loading (#490)

* make default containers directory alongside modules for faster loading

Signed-off-by: vsoch <[email protected]>

* tweak release workflow

Signed-off-by: vsoch <[email protected]>

* Tweak settings container base test

Co-authored-by: vsoch <[email protected]>

Author: vsoch

.github/workflows/release.yaml

@@ -19,11 +19,13 @@ jobs:
         export PATH="/usr/share/miniconda/bin:$PATH"
         source activate shpc
         pip install -e .
-        pip install setuptools wheel
+        pip install setuptools wheel twine
     - name: Build and publish
       env:
         TWINE_USERNAME: ${{ secrets.PYPI_USER }} 
         TWINE_PASSWORD: ${{ secrets.PYPI_PASS }}
       run: |
+        export PATH="/usr/share/miniconda/bin:$PATH"
+        source activate shpc
         python setup.py sdist bdist_wheel
         twine upload dist/*

CHANGELOG.md

@@ -13,7 +13,8 @@ and **Merged pull requests**. Critical items to know are:
 
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
-## [0.0.x](https://github.scom/singularityhub/singularity-hpc/tree/master) (0.0.x)
+## [0.0.x](https://github.scom/singularityhub/singularity-hpc/tree/main) (0.0.x)
+ - default containers directory should be separate from modules (0.0.43)
  - remove redundancy of manual bindpaths and SINGULARITY_BINDPATH (0.0.42)
  - Do not unset container environment vars if they are already set (0.0.41)
  - Central config should be read first (and updated with user config) (0.0.4)

docs/getting_started/developer-guide.rst

@@ -112,7 +112,7 @@ this:
             └── container.yaml
 
 
-And this is what gets installed to the modules folder, where each is kept in
+And this is what gets installed to the modules and containers directories, where each is kept in
 a separate directory based on version.
 
 .. code-block:: console
@@ -121,10 +121,14 @@ a separate directory based on version.
     modules/
     └── python
         └── 3.9.2
-            ├── module.lua
+            └── module.lua
+
+    $ tree containers/
+    containers/
+    └── python
+        └── 3.9.2
             └── python-3.9.2.sif
 
-    2 directories, 2 files
 
 So different versions could exist alongside one another.
 
@@ -468,7 +472,7 @@ or ask for help directly!
 
     Container:
 
-     - /home/vanessa/Desktop/Code/singularity-hpc/modules/python/3.9.2-slim/python-3.9.2-slim-sha256:85ed629e6ff79d0bf796339ea188c863048e9aedbf7f946171266671ee5c04ef.sif
+     - /home/vanessa/Desktop/Code/singularity-hpc/containers/python/3.9.2-slim/python-3.9.2-slim-sha256:85ed629e6ff79d0bf796339ea188c863048e9aedbf7f946171266671ee5c04ef.sif
 
     Commands include:
 
@@ -500,5 +504,4 @@ commands to shell, run, exec, and inspect. The custom commands (e.g., Python) ar
 Make sure to write to files outside of the container so you don't muck with permissions.
 Since we are using module use, this means that you can create module files as a user
 or an admin - it all comes down to who has permission to write to the modules
-folder, and of course use it. Note that I have not tested this on an HPC system
-but plan to shortly.
+and containers folder, and of course use it.

docs/getting_started/user-guide.rst

@@ -147,7 +147,7 @@ that you might set, the following variables are available to you:
 
 Additionally, the variables ``module_base``, ``container_base``, and ``registry``
 can be set with environment variables that will be expanded at runtime. You cannot
-use the protected set of substitution variables (``$install_dir`` and ``$install_root``)
+use the protected set of substitution variables (``$install_dir`` and ``$root_dir``)
 as environment variables, as they will be subbed in by shpc before environment
 variable replacement. A summary table of variables is included below, and then further discussed in detail.
 
@@ -169,8 +169,8 @@ variable replacement. A summary table of variables is included below, and then f
      - The install directory for modules
      - $root_dir/modules
    * - container_base
-     - Where to install containers. If not defined, they are installed alongside modules.
-     - null
+     - Where to install containers. If not defined, they are installed in "containers" in the install root
+     - $root_dir/containers
    * - container_tech
      - The container technology to use (singularity or podman)
      - singularity
@@ -287,29 +287,32 @@ your install:
     $ shpc config set module_base:\$root_dir/modules
 
 
-This directory will be the base where lua files are added, and container are stored.
-For example, if you were to add a container with unique resource identifier `python/3.8`
-you would see:
+This directory will be the base where lua files are added, and containers are stored
+in a directory alongside it. For example, if you were to add a container with unique 
+resource identifier `python/3.8` you would see:
 
 .. code-block:: console
 
     $install_dir/modules/
     └── python
         └── 3.9.2
-            ├── module.lua
+            └── module.lua
+
+    $install_dir/containers/
+    └── python
+        └── 3.9.2
             └── python-3.9.2.sif
 
-Although your module path might have multiple locations, Singularity Registry HPC 
-assumes this one location to install container modules to in order to ensure
+Singularity Registry HPC uses this simple directory structure to ensure
 a unique namespace. 
 
 
 Container Images Folder
 -----------------------
 
-If you don't want your container images (sif files) to live alongside your
-module files, then you should define the ``container_base`` to be something
-non-null (a path that exists). For example:
+If you don't want your container images (sif files) to live in the root of shpc
+in a directory called "containers," then you should define the ``container_base`` to be something
+different. For example:
 
 .. code-block:: console
 
@@ -318,7 +321,9 @@ non-null (a path that exists). For example:
 
 
 The same hierarchy will be preserved as to not put all containers in the same
-directory.
+directory. It's strongly recommended to keep modules separate from containers
+for faster loading (applies to container technologies like Singularity that
+pull binary files directly).
 
 
 Registry
@@ -567,10 +572,13 @@ and then a message to indicate that the module was created.
     modules/
     └── python
         └── 3.9.2
-            ├── module.lua
-            └── python-3.9.2.sif
+            └── module.lua
 
-    2 directories, 2 files
+    $ tree containers/
+    containers/
+    └── python
+        └── 3.9.2
+            └── python-3.9.2.sif
     
 
 You can also install a specific tag (as shown in list).
@@ -1015,10 +1023,10 @@ If you want to quickly get the path to a container binary, you can use get.
 .. code-block:: console
 
     $ shpc get vanessa/salad:latest
-    /home/vanessa/Desktop/Code/singularity-hpc/modules/vanessa/salad/latest/vanessa-salad-latest-sha256:8794086402ff9ff9f16c6facb93213bf0b01f1e61adf26fa394b78587be5e5a8.sif
+    /home/vanessa/Desktop/Code/singularity-hpc/containers/vanessa/salad/latest/vanessa-salad-latest-sha256:8794086402ff9ff9f16c6facb93213bf0b01f1e61adf26fa394b78587be5e5a8.sif
 
     $ shpc get tensorflow/tensorflow:2.2.2
-    /home/vanessa/Desktop/Code/singularity-hpc/modules/tensorflow/tensorflow/2.2.2/tensorflow-tensorflow-2.2.2-sha256:e2cde2bb70055511521d995cba58a28561089dfc443895fd5c66e65bbf33bfc0.sif
+    /home/vanessa/Desktop/Code/singularity-hpc/containers/tensorflow/tensorflow/2.2.2/tensorflow-tensorflow-2.2.2-sha256:e2cde2bb70055511521d995cba58a28561089dfc443895fd5c66e65bbf33bfc0.sif
 
 If you select a higher level module directory or there is no sif, you'll see:
 

shpc/settings.yml

@@ -30,7 +30,8 @@ module_name: '{{ tool }}'
 default_version: true
 
 # store containers separately from module files
-container_base:
+# It's recommended to do this for faster loading
+container_base: $root_dir/containers
 
 # if defined, add to lmod script to load this Singularity module first
 singularity_module:

shpc/tests/test_settings.py

@@ -32,7 +32,7 @@ def test_set_get(tmp_path):
     """Test variable set/get"""
     settings_file = os.path.join(root, "settings.yml")
     settings = Settings(settings_file)
-    assert not settings.container_base
+    assert settings.container_base.endswith("containers")
     settings.set("container_base", "/tmp/containers")
     settings.set("container_features", "gpu:amd")
     assert settings.container_base == "/tmp/containers"

shpc/version.py

@@ -2,7 +2,7 @@
 __copyright__ = "Copyright 2021-2022, Vanessa Sochat"
 __license__ = "MPL 2.0"
 
-__version__ = "0.0.42"
+__version__ = "0.0.43"
 AUTHOR = "Vanessa Sochat"
 NAME = "singularity-hpc"
 PACKAGE_URL = "https://github.com/singularityhub/singularity-hpc"