adding support for custom environment file

Signed-off-by: vsoch <[email protected]>

Author: vsoch

CHANGELOG.md

@@ -15,6 +15,7 @@ The versions coincide with releases on pip. Only major versions will be released
 
 ## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/master) (0.0.x)
  - adding namespaces to make module install, show, inspect easier (0.0.24)
+ - added support for env section to render to bound environment file
  - container url does not render in docs (0.0.23)
  - addition of tcl modules, removal of un-needed database (0.0.22)
  - first update of all containers, and bugfix to pull (0.0.21)

docs/getting_started/user-guide.rst

@@ -209,6 +209,9 @@ A summary table of variables is included below, and then further discussed in de
    * - namespace
      - Set a default module namespace that you want to install from.
      - null
+   * - environment_file
+     - The name of the environment file to generate and bind to the container.
+     - 99-shpc.sh
 
 
 
@@ -721,6 +724,7 @@ bypython (per what is available on your system). To start a shell:
 
     $ shpc shell
 
+
 or with a specific interpreter:
 
 .. code-block:: console
@@ -938,6 +942,51 @@ that since we have a list, you technically could provide duplicate commands.
 However, the list is parsed into a dictionary, so only unique values are 
 enforced. If you accidentally have a repeated value, a warning will be printed.
 
+Environment Variables
+---------------------
+
+
+Finally, each recipe has an optional section for environment variables. For
+example, the container ``vanessa/salad`` shows definition of one environment
+variable:
+
+.. code-block:: yaml
+
+    docker: vanessa/salad
+    url: https://hub.docker.com/r/vanessa/salad
+    maintainer: '@vsoch'
+    description: A container all about fork and spoon puns.
+    latest:
+      latest: sha256:e8302da47e3200915c1d3a9406d9446f04da7244e4995b7135afd2b79d4f63db
+    tags:
+      latest: sha256:e8302da47e3200915c1d3a9406d9446f04da7244e4995b7135afd2b79d4f63db
+    aliases:
+      salad: /code/salad
+    env:
+      maintainer: vsoch
+
+And then during build, this variable is written to a ``99-shpc.sh`` file that
+is mounted into the countainer. For the above, the following will be written:
+
+.. code-block:: console
+
+    export maintainer=vsoch
+
+If a recipe does not have environment variables in the container.yaml, you have
+two options for adding a variable after install. For a more permanent solution,
+you can update the container.yaml file and install again. The container won't
+be re-pulled, but the environment file will be re-generated. If you want to 
+manually add them to the container, each module folder will have an environment
+file added regardless of having this section or not, so you can export them there.
+When you shell, exec, or run the container (all but inspect) you should be able
+to see your environment variables:
+
+.. code-block:: console
+
+    $ echo $maintainer
+    vsoch
+
+
 Singularity Deploy
 ------------------
 

registry/vanessa/salad/container.yaml

@@ -8,3 +8,5 @@ tags:
   latest: sha256:e8302da47e3200915c1d3a9406d9446f04da7244e4995b7135afd2b79d4f63db
 aliases:
   salad: /code/salad
+env:
+  maintainer: vsoch

shpc/main/container.py

@@ -214,6 +214,12 @@ def validate(self):
         """
         jsonschema.validate(instance=self._config, schema=schemas.containerConfig)
 
+    def get_envars(self):
+        """
+        Return loaded environment variables.
+        """
+        return dict(self.env) if self.env else {}
+
     def get_aliases(self):
         """
         Return a consistently formatted list of aliases

shpc/main/lmod/template.lua

@@ -16,19 +16,19 @@ Container:
 Commands include:
 
  - {{ prefix }}{{ flatname }}-run:
-       singularity run {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container>
+       singularity run {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container>
  - {{ prefix }}{{ flatname }}-shell:
-       singularity shell -s {{ singularity_shell }} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container>
+       singularity shell -s {{ singularity_shell }} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container>
  - {{ prefix }}{{ flatname }}-exec:
-       singularity exec -s {{ singularity_shell }} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container> "$@"
+       singularity exec -s {{ singularity_shell }} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container> "$@"
  - {{ prefix }}{{ flatname }}-inspect-runscript:
        singularity inspect -r <container>
  - {{ prefix }}{{ flatname }}-inspect-deffile:
        singularity inspect -d <container>
 
 {% if aliases %}{% for alias in aliases %} - {{ alias.name }}:
-       singularity exec {% if bindpaths %}-B {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %}<container> {{ alias.command }}
-{% endfor %}{% else %} - {{ prefix }}{{ flatname }}: singularity run {% if bindpaths %}-B {{ bindpaths }}{% endif %}<container>{% endif %}
+       singularity exec {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %}<container> {{ alias.command }}
+{% endfor %}{% else %} - {{ prefix }}{{ flatname }}: singularity run {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }}{% endif %}<container>{% endif %}
 
 For each of the above, you can export:
 
@@ -47,9 +47,9 @@ setenv("SINGULARITY_SHELL", "{{ singularity_shell }}")
 
 -- interactive shell to any container, plus exec for aliases
 local containerPath = '{{ container_sif }}'
-local shellCmd = "singularity ${SINGULARITY_OPTS} shell ${SINGULARITY_COMMAND_OPTS} -s {{ singularity_shell }} {% if bindpaths %}-B {{ bindpaths }}{% endif %} " .. containerPath
-local execCmd = "singularity ${SINGULARITY_OPTS} exec ${SINGULARITY_COMMAND_OPTS} {% if bindpaths %}-B {{ bindpaths }}{% endif %} "
-local runCmd = "singularity ${SINGULARITY_OPTS} run ${SINGULARITY_COMMAND_OPTS} {% if bindpaths %}-B {{ bindpaths }}{% endif %} " .. containerPath .. " ${ SINGULARITY_COMMAND_ARGS }"
+local shellCmd = "singularity ${SINGULARITY_OPTS} shell ${SINGULARITY_COMMAND_OPTS} -s {{ singularity_shell }} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }}{% endif %} " .. containerPath
+local execCmd = "singularity ${SINGULARITY_OPTS} exec ${SINGULARITY_COMMAND_OPTS} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }}{% endif %} "
+local runCmd = "singularity ${SINGULARITY_OPTS} run ${SINGULARITY_COMMAND_OPTS} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }}{% endif %} " .. containerPath .. " ${ SINGULARITY_COMMAND_ARGS }"
 local inspectCmd = "singularity ${SINGULARITY_OPTS} inspect ${SINGULARITY_COMMAND_OPTS} " 
 
 -- set_shell_function takes bashStr and cshStr

shpc/main/modules.py

@@ -7,6 +7,7 @@
 from datetime import datetime
 import shpc.utils as utils
 import shpc.defaults as defaults
+import shpc.main.templates
 from jinja2 import Template
 
 from glob import glob
@@ -389,6 +390,7 @@ def install(self, name, tag=None):
 
         # prepare aliases
         aliases = config.get_aliases()
+        envars = config.get_envars()
 
         self._install(
             module_dir,
@@ -399,9 +401,20 @@ def install(self, name, tag=None):
             description=config.description,
             version=tag.name,
         )
+
+        # Write the environment file to be bound to the container
+        self._add_environment(module_dir, envars)
         logger.info("Module %s/%s was created." % (config.name, tag.name))
         return container_path
 
+    def _add_environment(self, module_dir, envars):
+        """
+        Given one or more environment variables in a dictionary, write to file
+        """
+        out = Template(shpc.main.templates.environment_file).render(envars=envars)
+        env_file = os.path.join(module_dir, self.settings.environment_file)
+        utils.write_file(env_file, out)
+
     def _install(
         self,
         module_dir,
@@ -461,5 +474,6 @@ def _install(
             creation_date=datetime.now(),
             name=name,
             flatname=name.replace("/", "-"),
+            envfile=self.settings.environment_file,
         )
         utils.write_file(module_path, out)

shpc/main/schemas.py

@@ -57,6 +57,7 @@
         "type": "array",
         "items": {"type": "string"},
     },
+    "env": aliases,
     "aliases": {
         "oneOf": [
             aliases,

shpc/main/tcl/template.tcl

@@ -14,19 +14,19 @@ proc ModulesHelp { } {
     puts stderr " - {{ container_sif }}"
     puts stderr "Commands include:"
     puts stderr " - {{ prefix }}{{ flatname }}-run:"
-    puts stderr "       singularity run {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container>"
+    puts stderr "       singularity run {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container>"
     puts stderr " - {{ prefix }}{{ flatname }}-shell:"
-    puts stderr "       singularity shell -s {{ singularity_shell }} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container>"
+    puts stderr "       singularity shell -s {{ singularity_shell }} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container>"
     puts stderr " - {{ prefix }}{{ flatname }}-exec:"
-    puts stderr "       singularity exec -s {{ singularity_shell }} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container> \"$@\""
+    puts stderr "       singularity exec -s {{ singularity_shell }} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }} {% endif %}<container> \"$@\""
     puts stderr " - {{ prefix }}{{ flatname }}-inspect-runscript:"
     puts stderr "       singularity inspect -r <container>"
     puts stderr " - {{ prefix }}{{ flatname }}-inspect-deffile:"
     puts stderr "       singularity inspect -d <container>"
 
 {% if aliases %}{% for alias in aliases %}    puts stderr " - {{ alias.name }}:"
-    puts stderr "       singularity exec {% if bindpaths %}-B {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %}<container> {{ alias.command }}"
-{% endfor %}{% else %}    puts stderr " - {{ prefix }}{{ flatname }}: singularity run {% if bindpaths %}-B {{ bindpaths }}{% endif %}<container>"{% endif %}
+    puts stderr "       singularity exec {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }} {% endif %}{% if alias.options %}{{ alias.options }} {% endif %}<container> {{ alias.command }}"
+{% endfor %}{% else %}    puts stderr " - {{ prefix }}{{ flatname }}: singularity run {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }}{% endif %}<container>"{% endif %}
 
     puts stderr "For each of the above, you can export:"
 
@@ -58,9 +58,9 @@ conflict {{ name }}
 setenv SINGULARITY_SHELL {{ singularity_shell }}
 
 # interactive shell to any container, plus exec for aliases
-set shellCmd "singularity \${SINGULARITY_OPTS} shell \${SINGULARITY_COMMAND_OPTS} -s {{ singularity_shell }} {% if bindpaths %}-B {{ bindpaths }}{% endif %} ${containerPath}" 
-set execCmd "singularity \${SINGULARITY_OPTS} exec \${SINGULARITY_COMMAND_OPTS} {% if bindpaths %}-B {{ bindpaths }}{% endif %} "
-set runCmd "singularity \${SINGULARITY_OPTS} run \${SINGULARITY_COMMAND_OPTS} {% if bindpaths %}-B {{ bindpaths }}{% endif %} ${containerPath} \${SINGULARITY_COMMAND_ARGS}"
+set shellCmd "singularity \${SINGULARITY_OPTS} shell \${SINGULARITY_COMMAND_OPTS} -s {{ singularity_shell }} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }}{% endif %} ${containerPath}" 
+set execCmd "singularity \${SINGULARITY_OPTS} exec \${SINGULARITY_COMMAND_OPTS} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }}{% endif %} "
+set runCmd "singularity \${SINGULARITY_OPTS} run \${SINGULARITY_COMMAND_OPTS} {% if envfile %}-B {{ module_dir }}/{{ envfile }}:/.singularity.d/env/{{ envfile }}{% endif %} {% if bindpaths %}-B {{ bindpaths }}{% endif %} ${containerPath} \${SINGULARITY_COMMAND_ARGS}"
 set inspectCmd "singularity \${SINGULARITY_OPTS} inspect \${SINGULARITY_COMMAND_OPTS} " 
 
 # set_shell_function takes bashStr and cshStr

shpc/main/templates.py

@@ -0,0 +1,11 @@
+__author__ = "Vanessa Sochat"
+__copyright__ = "Copyright 2021, Vanessa Sochat"
+__license__ = "MPL 2.0"
+
+
+environment_file = """#!/bin/sh
+
+# Add custom environment variables here with export
+{% for key, value in envars.items() %}export {{ key }}="{{ value }}"
+{% endfor %}
+"""

shpc/settings.yml

@@ -31,3 +31,7 @@ singularity_shell: /bin/bash
 # the module namespace you want to install from. E.g., if you have ghcr.io/autamus/clingo
 # and you set the namespace to ghcr.io/autamus, you can just do: shpc install clingo.
 namespace: null
+
+# THe name of the environment file to bind to the container.
+# This determines order of load
+environment_file: "99-shpc.sh"