Add/wrapper scripts (#496)

* reverting string to parse config params
* config get: changed behaviour to work with dicts
* reverting config get impl for non-dict, to try and fix tests
* adding wrapper script variables
* fix to updated settings.yml
* new default: containers_base distinct from modules_base
* 1st pass at wrapper_scripts for singularity
* wrapper_script: added tcl for singularity
* wrapper script: added script template for real (was in gitignore)
* wrapper script: fix for add case
* wrapper scripts now created as executable
* wrapper scripts: templates for docker/podman
* wrapper scripts: bin directory under modules
* wrapper scripts: docker/podman support
* reduced newlines in module templates
* all shell settings nearby in settings.yml
* utils/fileio: make executable using stat rather than subprocess methods
* wrapper script templates: protecting argument passing
* wrapper script templates: protecting argument passing for docker, too
* container: defined wrapper_subdir attribute
* modules: defined wrappertemplatefile property
* wrapper scripts: relative PATH in modulefile templates
* wrapper_scripts: new function _generate_wrapper_scripts in container/base.py
* first shot at adding wrapper scripts

this will support two kinds of wrapper scripts - global that live in the settings.yaml,
and container.yaml specific that go alongside that file. We basically will then generate
aliases for any executable names that were not created as wrappers. we will want to follow
this up with a command to list wrappers, and also include descriptions for the interested
user, and a flag to install to select on the fly

warning - parsed_names (tool, registry, etc) are now prefixed with parsed_name.<variable> and
the user will need to update settings
* Add/wrapper scripts - minor polishing (#497)
* fixed typos in module/templates/docs.md
* minor polishing in settings.yml, wrappers_subdir
* docs: edits to user and developer guides, for wrapper scripts
* reverted a couple of shell occurrences (docker/podman) in modules/templates/docs.md
* one more edit for shell in modules/templates/docs.md
* Update docs/getting_started/user-guide.rst
* adding new design for templates: bases and snippets!
* wrapper/templates/bases: csh always needs "set" for variables
* wrappers/templates: using variable moduleDir
* modules/templates lua: moduleDir always replaces {{ module_dir }}
* modules/templates tcl: moduleDir always replaces {{ module_dir }}
* main/container: no more need to pass module_dir to template.render for modulefiles
* registry/vanessa/salad: updated wrapper scripts with moduleDir var
* registry/vanessa/salad: typo in singularity_fork.sh

Signed-off-by: vsoch <[email protected]>

Co-authored-by: Marco De La Pierre <[email protected]>
Co-authored-by: vsoch <[email protected]>

Author: vsoch

CHANGELOG.md

@@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/main) (0.0.x)
+ - Adding support for wrapper scripts for global and container.yaml (0.0.45)
  - Lua and tcl module file bug fixes for shell functions and aliases (0.0.44)
   - restoring -B for Singularity bindpaths over using envar
  - default containers directory should be separate from modules (0.0.43)

docs/getting_started/developer-guide.rst

@@ -198,6 +198,215 @@ For each of the above, depending on the prefix of options that you choose, it wi
 This means that if you design a new registry recipe, you should consider how to run it for both kinds of technology. Also note that ``docker_options`` are
 those that will also be used for Podman.
 
+Wrapper Script
+--------------
+
+Singularity HPC allows exposure of two kinds of wrapper scripts:
+
+1. A global level wrapper intended to replace aliases. E.g., if an alias "samtools" is typically a direct container call, enabling a wrapper will generate an executable script "samtools" in a "bin" directory associated with the container, added to the path, to call instead. This is desired when MPI ("mpirun")  or scheduler (e.g. "srun" with Slurm) utilities are needed to run the scripts. This global script is defined in settings.yml and described in the user guide.
+2. A container level wrapper that is specific to a container, described here.
+
+For container specific scripts, you can add sections to a ``container.yaml`` to specify the script (and container type)
+and the scripts must be provided alongside the container.yaml to install.
+
+.. code-block:: yaml
+
+    docker_scripts:
+      fork: docker_fork.sh
+    singularity_scripts:
+      fork: singularity_fork.sh
+
+The above says "given generation of a docker or podman container, write a script named "fork" that uses "docker_fork.sh" as a template"
+and the same for Singularity. And then I (the developer) would provide the custom scripts alongside container.yaml:
+
+.. code-block:: console
+
+    registry/vanessa/salad/
+    ├── container.yaml
+    ├── docker_fork.sh
+    └── singularity_fork.sh
+
+You can look at ``registry/vanessa/salad`` for an example that includes Singularity
+and Docker wrapper scripts. For example, when generating for a singularity container with
+the global wrapped scripts enabled, we get one wrapper script for the alias "salad" and one for
+the custom container script "fork":
+
+.. code-block:: console
+
+    $ tree modules/vanessa/salad/
+    modules/vanessa/salad/
+    └── latest
+        ├── 99-shpc.sh
+        ├── bin
+        │   ├── fork
+        │   └── salad
+        └── module.lua
+
+If we disable all wrapper scripts, the bin directory would not exist. If we set the default wrapper
+scripts for singularity and docker in settings.yml and left enable to true, we would only see "fork."
+
+How to write an alias wrapper script
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+First, decide if you want a global script (to replace or wrap aliases) OR a custom container script. For an alias derived (global) script, you should:
+
+1. Write the new script file into shpc/main/wrappers.
+2. Add an entry to shpc/main/wrappers/scripts referencing the script.
+
+For these global scripts, the user can select to use it in their settings.yaml.
+We will eventually write a command to list global wrappers available, so if you add a new one future users will know
+about it. For alias wrapper scripts, the following variables are passed for rendering:
+
+.. list-table:: Title
+   :widths: 15 15 40 30
+   :header-rows: 1
+
+   * - Name
+     - Type
+     - Description
+     - Example
+   * - alias
+     - dictionary
+     - The entire alias in question, including subfields name, command, singularity_options or docker_options, and args
+     - ``{{ alias.name }}`` 
+   * - settings
+     - dictionary
+     - Everything referenced in the user settings
+     - ``{{ settings.wrapper_shell }}``
+   * - container
+     - dictionary
+     - The container technology
+     - ``{{ container.command }}`` renders to docker, singularity, or podman
+   * - config
+     - dictionary
+     - The entire container config (container.yaml) structured the same
+     - ``{{ config.docker }}``
+   * - image
+     - string
+     - The name of the container binary (SIF) or unique resource identifier
+     - ``{{ image }}``
+   * - module_dir
+     - string
+     - The name of the module directory
+     - ``{{ module_dir }}``
+   * - features
+     - dictionary
+     - A dictionary of parsed features
+     - ``{{ features.gpu }}``
+
+     
+
+How to write an container wrapper script
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you want to write a custom container.yaml script:
+
+1. Add either (or both) of singularity_scripts/docker_scripts in the container.yaml, including an alias command and an associated script.
+2. Write the script with the associated name into that folder.
+
+The following variables are passed for rendering.
+
+.. list-table:: Title
+   :widths: 15 15 40 30
+   :header-rows: 1
+
+   * - Name
+     - Type
+     - Description
+     - Example
+   * - alias
+     - string
+     - The alias name defined under singularity_scripts or docker_scripts
+     - ``{{ alias }}`` 
+   * - settings
+     - dictionary
+     - Everything referenced in the user settings
+     - ``{{ settings.wrapper_shell }}``
+   * - container
+     - dictionary
+     - The container technology
+     - ``{{ container.command }}`` renders to docker, singularity, or podman
+   * - config
+     - dictionary
+     - The entire container config (container.yaml) structured the same
+     - ``{{ config.docker }}``
+   * - image
+     - string
+     - The name of the container binary (SIF) or unique resource identifier
+     - ``{{ image }}``
+   * - module_dir
+     - string
+     - The name of the module directory
+     - ``{{ module_dir }}``
+   * - features
+     - dictionary
+     - A dictionary of parsed features
+     - ``{{ features.gpu }}``
+
+
+Templating for both wrapper script types
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Note that you are free to use "snippets" and "bases" either as an inclusion or "extends" meaning you can
+easily re-use code. For example, if we have the following registered directories under ``shpc/main/wrappers/templates`` 
+for definition of bases and templates:
+
+.. code-block:: console
+
+    main/wrappers/templates/
+    
+    # These are intended for use with "extends"
+    ├── bases
+    │   ├── __init__.py
+    │   └── shell-script-base.sh
+    
+    # These are top level template files, as specified in the settings.yml
+    ├── docker.sh
+    ├── singularity.sh
+
+    # A mostly empty directory ready for any snippets!
+    └── snippets
+
+For example, a "bases" template to define a shell and some special command that might look like this:
+
+.. code-block:: console
+
+    #!{{ settings.wrapper_shell }}
+
+    script=`realpath $0`
+    wrapper_bin=`dirname $script`
+    {% if '/csh' in settings.wrapper_shell %}set moduleDir=`dirname $wrapper_bin`{% else %}export moduleDir=$(dirname $wrapper_bin){% endif %}
+
+    {% block content %}{% endblock %}
+
+
+And then to use it for any container- or global- wrapper we would do the following in the wrapper script:
+
+.. code-block:: console
+
+    {% extends "bases/my-base-shell.sh" %}
+
+    # some custom wrapper before stuff here
+
+    {% block content %}{% endblock %}
+
+    # some custom wrapper after stuff here
+
+
+For snippets, which are intended to be more chunks of code you can throw in one spot
+on the fly, you can do this:
+
+
+.. code-block:: console
+
+    {% include "snippets/export-envars.sh" %}
+    # some custom wrapper after stuff here
+
+
+Finally, if you want to add your own custom templates directory for which you
+can refer to templates relatively, define ``wrapper_scripts`` -> ``templates`` as a full path
+in your settings.
+
 
 Environment Variables
 ---------------------
@@ -337,7 +546,7 @@ Fields include:
      - A list of patterns to use for adding new tags. If not defined, all are added 
      - false
    * - aliases
-     - Named entrypoints for container (dict)
+     - Named entrypoints for container (dict) as described above
      - false
    * - url
      - Documentation or other url for the container uri
@@ -351,7 +560,12 @@ Fields include:
    * - features
      - Optional key, value paired set of features to enable for the container. Currently allowed keys: *gpu* *home* and *x11*.
      - varies
-
+   * - singularity_scripts
+     - key value pairs of wrapper names (e.g., executable called by user) and local container script for Singularity
+     - false
+   * - docker_scripts
+     - key value pairs of wrapper names (e.g., executable called by user) and local container script for Docker or Singularity
+     - false
 
 A complete table of features is shown here. The
 

docs/getting_started/user-guide.rst

@@ -201,6 +201,21 @@ variable replacement. A summary table of variables is included below, and then f
    * - test_shell
      - The shell used for the test.sh file
      - /bin/bash
+   * - wrapper_shell
+     - The shell used for wrapper scripts
+     - /bin/bash
+   * - wrapper_scripts:enabled
+     - enable or disable generation of wrapper scripts, instead of module aliases
+     - false
+   * - wrapper_scripts:docker
+     - The name of the generic wrapper script template for docker
+     - docker.sh
+   * - wrapper_scripts:podman
+     - The name of the generic wrapper script template for podman
+     - docker.sh
+   * - wrapper_scripts:singularity
+     - The name of the generic wrapper script template for singularity
+     - singularity.sh
    * - namespace
      - Set a default module namespace that you want to install from.
      - null
@@ -446,6 +461,95 @@ If you would like support for a different container technology that has not been
 mentioned, please also `open an issue <https://github.com/singularityhub/singularity-hpc>`_ and
 provide description and links to what you have in mind.
 
+Wrapper Scripts
+---------------
+
+Singularity HPC allows for global definition of wrapper scripts, meaning that instead of writing a module alias to run a container for some given alias,
+we generate a wrapper script of the same name instead. Since the settings.yml is global, all wrapper scripts defined here are specific to replacing aliases.
+Container-specific scripts you'll want to include in the container.yaml are described in the developer docs. Let's take a look at the settings:
+
+
+.. code-block:: yaml
+
+    wrapper_scripts:
+
+      # Enable wrapper scripts, period. If enabled, generate scripts for aliases instead of commands
+      # if enabled, we also allow container-specific wrapper scripts.
+      enabled: false
+
+      # use for docker aliases
+      docker: docker.sh
+
+      # use for podman aliases
+      podman: docker.sh
+
+      # use for singularity aliases
+      singularity: singularity.sh 
+
+
+Since different container technologies might expose different environment variables (e.g., ``SINGULARITY_OPTS`` vs ``PODMAN_OPTS``)
+they are organized above based on the container technology. If you want to customize the wrapper script, simply replace the relative paths
+above (e.g., ``singularity.sh``) with an absolute path to a file that will be used instead. For global alias scripts such as these, 
+Singularity HPC will look for:
+
+1. An absolute path first, if found is used first.
+2. Then a script name in the shpc/main/wrappers directory
+
+Here is an example of using wrapper scripts for the "python" container, which doesn't have container specific wrappers. What you see
+is the one entrypoint, `python`, being placed in a "bin" subdirectory that the module will see instead of defining the alias.
+
+
+.. code-block:: console
+
+    modules/python/
+    └── 3.9.10
+        ├── 99-shpc.sh
+        ├── bin
+        │   └── python
+        └── module.lua
+
+For container specific scripts, you can add sections to a ``container.yaml`` to specify the script (and container type)
+and the scripts must be provided alongside the container.yaml to install.
+
+.. code-block:: yaml
+
+    docker_scripts:
+      fork: docker_fork.sh
+    singularity_scripts:
+      fork: singularity_fork.sh
+
+The above says "given generation of a docker or podman container, write a script named "fork" that uses "docker_fork.sh" as a template"
+and the same for Singularity. And then I (the developer) would provide the custom scripts alongside container.yaml:
+
+.. code-block:: console
+
+    registry/vanessa/salad/
+    ├── container.yaml
+    ├── docker_fork.sh
+    └── singularity_fork.sh
+
+And here is what those scripts look like installed. Since we are installing for just one container technology, we are seeing the alias wrapper for salad as "salad" and the container-specific wrapper for fork as "fork."
+
+
+.. code-block:: console
+
+    modules/vanessa/salad/
+    └── latest
+        ├── 99-shpc.sh
+        ├── bin
+        │   ├── fork
+        │   └── salad
+        └── module.lua
+
+
+We currently don't have a global argument to enable alias wrappers but not container wrappers. If you see a need for this please let us know.
+
+Where are wrapper scripts stored?
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Since we don't allow overlap
+of the name of an alias wrapper script (e.g., ``bin/python`` as a wrapper to a python entrypoint) from a custom container wrapper script (e.g., a wrapper script with name "python" under a container.yaml) we can keep them both in the modules directory. If you see a need to put them elsewhere please let us know. 
+
 .. _getting_started-commands:
 
 

registry/vanessa/salad/container.yaml

@@ -8,5 +8,13 @@ tags:
   latest: sha256:e8302da47e3200915c1d3a9406d9446f04da7244e4995b7135afd2b79d4f63db
 aliases:
   salad: /code/salad
+
+# An example of a custom wrapper script, stored here, will generate "fork" alias
+# for each of a docker and singularity container
+docker_scripts:
+  fork: docker_fork.sh
+singularity_scripts:
+  fork: singularity_fork.sh
+
 env:
   maintainer: vsoch

registry/vanessa/salad/docker_fork.sh

@@ -0,0 +1,9 @@
+{% extends "bases/shell-script-base.sh" %}
+
+# See shpc/main/wrappers/snippets for what is added via this extend
+
+# This is an example of a custom wrapper script for a docker container
+# Note that we are showing an example of all the default args provided to the script!
+# You don't need to use any of this syntax
+
+{% block content %}{{ container.command }} ${PODMAN_OPTS} run ${PODMAN_COMMAND_OPTS} -i{% if settings.enable_tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if settings.environment_file %}--env-file $moduleDir/{{ settings.environment_file }}{% endif %} {% if settings.bindpaths %}-v {{ settings.bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v ${PWD} -w ${PWD} --entrypoint /code/salad {{ image }} fork {% if '/sh' in settings.wrapper_shell or '/bash' in settings.wrapper_shell %}"$@"{% elif '/csh' in settings.wrapper_shell %}$argv:q{% endif %}{% endblock %}

registry/vanessa/salad/singularity_fork.sh

@@ -0,0 +1,9 @@
+{% extends "bases/shell-script-base.sh" %}
+
+# See shpc/main/wrappers/snippets for what is added via this extend
+
+# This is an example of a singularity wrapper script, under singularity_scripts
+# Note that we are showing an example of all the default args provided to the script!
+# You don't need to use any of this syntax
+
+{% block content %}singularity ${SINGULARITY_OPTS} exec ${SINGULARITY_COMMAND_OPTS} {% if features.gpu %}{{ features.gpu }} {% endif %}{% if features.home %}-B {{ features.home }} --home {{ features.home }} {% endif %}{% if features.x11 %}-B {{ features.x11 }} {% endif %}{% if settings.environment_file %}-B $moduleDir/{{ settings.environment_file }}:/.singularity.d/env/{{ settings.environment_file }}{% endif %} {% if settings.bindpaths %}-B {{ settings.bindpaths }}{% endif %} {{ image }} /code/salad fork {% if '/sh' in settings.wrapper_shell or '/bash' in settings.wrapper_shell %}"$@"{% elif '/csh' in settings.wrapper_shell %}$argv:q{% endif %}{% endblock %}