Add support for singularity_options and docker_options (#440)

* add support for singularity_options and docker_options
* missing code block yaml in developer guide!

Signed-off-by: vsoch <[email protected]>
Co-authored-by: vsoch <[email protected]>

Author: vsoch

CHANGELOG.md

@@ -14,6 +14,9 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.scom/singularityhub/singularity-hpc/tree/master) (0.0.x)
+ - Adding singularity and docker specific options (0.0.32)
+   - Adding more documentation on aliases
+   - Bugfix to output error message if path does not exist for inspect
  - Fixing bug with using variables in recipe options (0.0.31)
  - Removing un-used defaults and lmod_base (0.0.30)
  - Allow environment variables in settings (0.0.29)

docs/getting_started/developer-guide.rst

@@ -169,14 +169,30 @@ Each recipe has an optional section for defining aliases in the modulefile; ther
     aliases:
       python: /usr/local/bin/python
 
-A second form is allowed, using dicts, in those cases where the command requires to specify custom options for the container runtime. For instance, suppose the python interpreter above requires an isolated shell environment (``--cleanenv`` in Singularity):
+This format is container technology agnostic, because the command (``python``) and executable it targets (``/usr/local/bin/python``) would be consistent between
+Podman and Singularity, for example. A second form is allowed, using dicts, in those cases where the command requires to specify custom options for the container runtime. For instance, suppose the python interpreter above requires an isolated shell environment (``--cleanenv`` in Singularity):
 
 .. code-block:: yaml
 
     aliases:
     - name: python
       command: /usr/local/bin/python
-      options: --cleanenv
+      singularity_options: --cleanenv
+
+
+Or perhaps the container required the docker options ``-it`` because it was an interactive, terminal session:
+
+.. code-block:: yaml
+
+    aliases:
+    - name: python
+      command: /usr/local/bin/python
+      docker_options: -it
+
+
+For each of the above, depending on the prefix of options that you choose, it will write them into the module files for Singularity and Docker, respectively.
+This means that if you design a new registry recipe, you should consider how to run it for both kinds of technology. Also note that ``docker_options`` are
+those that will also be used for Podman.
 
 
 Environment Variables

registry/jupyter/datascience-notebook/container.yaml

@@ -10,5 +10,4 @@ tags:
 aliases:
 - name: run-notebook
   command: jupyter notebook --no-browser --port=$(shuf -i 2000-65000 -n 1) --ip 0.0.0.0
-  options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp
-    -d):/run/user
+  singularity_options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp -d):/run/user

registry/jupyter/minimal-notebook/container.yaml

@@ -10,5 +10,4 @@ tags:
 aliases:
 - name: run-notebook
   command: jupyter notebook --no-browser --port=$(shuf -i 2000-65000 -n 1) --ip 0.0.0.0
-  options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp
-    -d):/run/user
+  singularity_options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp -d):/run/user

registry/jupyter/pyspark-notebook/container.yaml

@@ -10,5 +10,4 @@ tags:
 aliases:
 - name: run-notebook
   command: jupyter notebook --no-browser --port=$(shuf -i 2000-65000 -n 1) --ip 0.0.0.0
-  options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp
-    -d):/run/user
+  singularity_options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp -d):/run/user

registry/jupyter/r-notebook/container.yaml

@@ -10,5 +10,4 @@ tags:
 aliases:
 - name: run-notebook
   command: jupyter notebook --no-browser --port=$(shuf -i 2000-65000 -n 1) --ip 0.0.0.0
-  options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp
-    -d):/run/user
+  singularity_options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp -d):/run/user

registry/jupyter/scipy-notebook/container.yaml

@@ -10,5 +10,4 @@ tags:
 aliases:
 - name: run-notebook
   command: jupyter notebook --no-browser --port=$(shuf -i 2000-65000 -n 1) --ip 0.0.0.0
-  options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp
-    -d):/run/user
+  singularity_options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp -d):/run/user

registry/jupyter/tensorflow-notebook/container.yaml

@@ -12,5 +12,4 @@ filter:
 aliases:
 - name: run-notebook
   command: jupyter notebook --no-browser --port=$(shuf -i 2000-65000 -n 1) --ip 0.0.0.0
-  options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp
-    -d):/run/user
+  singularity_options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp -d):/run/user

shpc/main/container/base.py

@@ -119,7 +119,10 @@ def installed_tags(self, module_name):
         """
         Get a list of installed tags.
         """
-        return os.listdir(os.path.join(self.settings.module_base, module_name))
+        module_dir = os.path.join(self.settings.module_base, module_name)
+        if not os.path.exists(module_dir):
+            logger.exit("%s does not exist." % module_dir)
+        return os.listdir(module_dir)
 
     def get_environment_file(self, module_name):
         """

shpc/main/modules/templates/docker.lua

@@ -25,7 +25,7 @@ Commands include:
        {{ command }} inspect <container>
 
 {% if aliases %}{% for alias in aliases %} - {{ alias.name }}:
-       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ alias.entrypoint }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %} -v ${PWD} -w ${PWD} <container> "{{ alias.args }}"
+       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ alias.entrypoint }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.docker_options %}{{ alias.docker_options }} {% endif %} -v ${PWD} -w ${PWD} <container> "{{ alias.args }}"
 {% endfor %}{% endif %}
 
 For each of the above, you can export:
@@ -58,7 +58,7 @@ conflict(myModuleName(){% if aliases %}{% for alias in aliases %}{% if alias.nam
 
 -- exec functions to provide "alias" to module commands
 {% if aliases %}{% for alias in aliases %}
-set_shell_function("{{ alias.name }}", execCmd .. {% if alias.options %} "{{ alias.options }} " .. {% endif %} " --entrypoint {{ alias.entrypoint }} " .. containerPath .. " {{ alias.args }} $@", execCmd .. {% if alias.options %} "{{ alias.options }} " .. {% endif %} " --entrypoint {{ alias.entrypoint }} " .. containerPath .. " {{ alias.args }}")
+set_shell_function("{{ alias.name }}", execCmd .. {% if alias.docker_options %} "{{ alias.docker_options }} " .. {% endif %} " --entrypoint {{ alias.entrypoint }} " .. containerPath .. " {{ alias.args }} $@", execCmd .. {% if alias.docker_options %} "{{ alias.docker_options }} " .. {% endif %} " --entrypoint {{ alias.entrypoint }} " .. containerPath .. " {{ alias.args }}")
 {% endfor %}{% endif %}
 
 {% if aliases %}