Adding first shot at new features (x11 and fake home) (#444)

* adding first shot at new features

this first commit will add a new feature, x11, which will be null by default. If the
value is True, we default to binding ~/.Xauthority. if it is found to be a string, we use that
instead. This means if there needs to be some custom bind like ~/.special-Xauthority TO the ~/.Xauthority,
both of those need to be represented in the single string.

Signed-off-by: vsoch <[email protected]>

* adding support for home feature

Signed-off-by: vsoch <[email protected]>

* adding more docs about features to developer guide;

Signed-off-by: vsoch <[email protected]>

* small tweaks to docs and adding home feature to biocontainers beast2

Signed-off-by: vsoch <[email protected]>

Co-authored-by: vsoch <[email protected]>

Author: vsoch

CHANGELOG.md

@@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.scom/singularityhub/singularity-hpc/tree/master) (0.0.x)
+ - New features for X11 and custom home (0.0.33)
  - Adding singularity and docker specific options (0.0.32)
    - Adding more documentation on aliases
    - Bugfix to output error message if path does not exist for inspect

docs/getting_started/developer-guide.rst

@@ -314,9 +314,46 @@ Fields include:
      - A list of environment variables to be defined in the container (key value pairs, e.g. var: value)
      - false
    * - features
-     - Optional key, value paired set of features to enable for the container. Currently allowed keys: *gpu*. Allowed values: *true*, *false* (default)
-     - false
+     - Optional key, value paired set of features to enable for the container. Currently allowed keys: *gpu* *home* and *x11*.
+     - varies
+
+
+A complete table of features is shown here. The
+
+Fields include:
 
+.. list-table:: Title
+   :widths: 20 20 20 10 10 10
+   :header-rows: 1
+
+   * - Name
+     - Description
+     - Container.yaml Values
+     - Settings.yaml Values
+     - Default
+     - Supported
+   * - gpu
+     - Add flags to the container to enable GPU support (typically amd or nvidia)
+     - true or false
+     - null, amd, or nvidia
+     - null
+     - Singularity
+   * - x11
+     - Indicate to bind an Xauthority file to allow x11
+     - true or false
+     - null, true (uses default ~/.Xauthority) or bind path
+     - null
+     - Singularity
+   * - home
+     - Indicate a custom home to bind
+     - true or false
+     - null, or path to a custom home
+     - null
+     - Singularity, Docker
+
+
+For bind paths (e.g., home and x11) you can do a single path to indicate the same
+source and destination (e.g., /my/path) or a double for customization of that (e,g., /src:/dest).
 Other supported (but not yet developed) fields could include different unique
 resource identifiers to pull/obtain other kinds of containers. For this
 current version, since we are assuming HPC and Singularity, we will typically

docs/getting_started/user-guide.rst

@@ -251,7 +251,15 @@ option will be added. Currently, the following features are supported:
      - If the container technology supports it, add flags to indicate using gpu.
      - null
      - nvidia, amd, null
-     
+   * - x11
+     - Bind mount ~/.Xauthority or a custom path
+     - null
+     - true (uses default path ~/.Xauthority), false/null (do not enable) or a custom path to an x11 file
+   * - home
+     - Specify and bind mount a custom home path
+     - null
+     - custom path for the home, or false/null
+
 
 Modules Folder
 --------------

registry/jupyter/minimal-notebook/container.yaml

@@ -7,7 +7,9 @@ latest:
 tags:
   "4.0": sha256:6f9d189494fe091700681eb39b429dfe1420ad0717697c2e2293779fcb80cd1e
   latest: sha256:be2da9cd312d7a19438fc0410613b1a6078d67305f23083721b2a073bb7d8b85
+features:
+  home: true
 aliases:
 - name: run-notebook
   command: jupyter notebook --no-browser --port=$(shuf -i 2000-65000 -n 1) --ip 0.0.0.0
-  singularity_options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp -d):/run/user
+  singularity_options: --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local --bind $(mktemp -d):/run/user

registry/quay.io/biocontainers/beast2/container.yaml

@@ -13,3 +13,5 @@ aliases:
   treeannotator: /usr/local/bin/treeannotator
 env:
   DISPLAY: ""
+features:
+  home: true

shpc/main/container/base.py

@@ -159,12 +159,20 @@ def get_features(self, config_features, settings_features, extra=None):
             # If the container technology has the feature and is defined in settings
             if key in self.features and key in settings_features:
 
-                # And if the settings feature is known to the container technology
+                # Case 1: the feature is known to the container technology
                 if settings_features[key] in self.features[key]:
 
-                    # Add the feature to be given to the container!
                     features[key] = self.features[key][settings_features[key]]
 
+                # Case 2: the exact value isn't known, but the feature accepts a string
+                elif type(settings_features[key]) in self.features[key]:
+
+                    # Add the feature to be given to the container!
+                    value = self.features[key][type(settings_features[key])]
+                    if value == "[use-self]":
+                        value = settings_features[key]
+                    features[key] = value
+
         return features
 
     def __str__(self):

shpc/main/container/docker.py

@@ -22,7 +22,7 @@ class DockerContainer(ContainerTechnology):
     # The module technology adds extensions here
     templatefile = "docker"
     command = "docker"
-    features = {}
+    features = {"home": {str: "[use-self]"}}
 
     def __init__(self):
         if shpc.utils.which(self.command)["return_code"] != 0:

shpc/main/container/singularity.py

@@ -25,7 +25,11 @@ class SingularityContainer(ContainerTechnology):
     templatefile = "singularity"
 
     # Singularity container features
-    features = {"gpu": {"nvidia": "--nv", "amd": "--rocm"}}
+    features = {
+        "gpu": {"nvidia": "--nv", "amd": "--rocm"},
+        "x11": {True: "~/.Xauthority", str: "[use-self]"},
+        "home": {str: "[use-self]"},
+    }
 
     def __init__(self):
         try:

shpc/main/modules/templates/docker.lua

@@ -16,16 +16,16 @@ Container:
 Commands include:
 
  - {|module_name|}-run:
-       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}-v ${PWD} -w ${PWD} <container> "$@"
+       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %}-v ${PWD} -w ${PWD} <container> "$@"
  - {|module_name|}-shell:
-       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}--entrypoint {{ shell }} -v ${PWD} -w ${PWD}<container>
+       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %}--entrypoint {{ shell }} -v ${PWD} -w ${PWD}<container>
  - {|module_name|}-exec:
-       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint "" {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} <container> "$@"
+       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint "" {% if envfile %}--env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v ${PWD} -w ${PWD} <container> "$@"
  - {|module_name|}-inspect:
        {{ command }} inspect <container>
 
 {% if aliases %}{% for alias in aliases %} - {{ alias.name }}:
-       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ alias.entrypoint }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.docker_options %}{{ alias.docker_options }} {% endif %} -v ${PWD} -w ${PWD} <container> "{{ alias.args }}"
+       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ alias.entrypoint }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %}{% if alias.docker_options %}{{ alias.docker_options }} {% endif %} -v ${PWD} -w ${PWD} <container> "{{ alias.args }}"
 {% endfor %}{% endif %}
 
 For each of the above, you can export:
@@ -44,10 +44,10 @@ local MODULEPATH="{{ module_dir }}"
 -- interactive shell to any container, plus exec for aliases
 local containerPath = '{{ image }}'
 
-local shellCmd = "{{ command }} ${PODMAN_OPTS} run -i{% if tty %}t{% endif %} ${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm --entrypoint {{ shell }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} " .. containerPath
+local shellCmd = "{{ command }} ${PODMAN_OPTS} run -i{% if tty %}t{% endif %} ${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm --entrypoint {{ shell }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v ${PWD} -w ${PWD} " .. containerPath
 -- execCmd needs entrypoint to be the executor
-local execCmd = "{{ command }} ${PODMAN_OPTS} run ${PODMAN_COMMAND_OPTS} -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} "
-local runCmd = "{{ command }} ${PODMAN_OPTS} run ${PODMAN_COMMAND_OPTS} -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v ${PWD} -w ${PWD} " .. containerPath
+local execCmd = "{{ command }} ${PODMAN_OPTS} run ${PODMAN_COMMAND_OPTS} -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v ${PWD} -w ${PWD} "
+local runCmd = "{{ command }} ${PODMAN_OPTS} run ${PODMAN_COMMAND_OPTS} -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v ${PWD} -w ${PWD} " .. containerPath
 local inspectCmd = "{{ command }} ${PODMAN_OPTS} inspect ${PODMAN_COMMAND_OPTS} " .. containerPath 
 
 -- set_shell_function takes bashStr and cshStr

shpc/main/modules/templates/docker.tcl

@@ -15,15 +15,15 @@ proc ModulesHelp { } {
     puts stderr " - {{ image }}"
     puts stderr "Commands include:"
     puts stderr " - {|module_name|}-run:"
-    puts stderr "       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file  {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . <container>"
+    puts stderr "       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm {% if envfile %}--env-file  {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v . -w . <container>"
     puts stderr " - {|module_name|}-shell:"
-    puts stderr "       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ shell }} {% if envfile %} --env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . <container>"
+    puts stderr "       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint {{ shell }} {% if envfile %} --env-file {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v . -w . <container>"
     puts stderr " - {|module_name|}-exec:"
-    puts stderr "       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint \"\" {% if envfile %} --env-file  {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v . -w . <container> $*"
+    puts stderr "       {{ command }} run -i{% if tty %}t{% endif %} -u `id -u`:`id -g` --rm --entrypoint \"\" {% if envfile %} --env-file  {{ module_dir }}/{{ envfile }} {% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v . -w . <container> $*"
     puts stderr " - {|module_name|}-inspect:"
     puts stderr "       {{ command }} inspect <container>"
 {% if aliases %}{% for alias in aliases %}    puts stderr " - {{ alias.name }}:"
-    puts stderr "       {{ command }} run -i{% if tty %}t{% endif %} --rm -u `id -u`:`id -g` --entrypoint {{ alias.entrypoint | replace("$", "\$") }} {% if envfile %}--envfile  {{ module_dir }}/{{ envfile }} {% endif %}{% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if alias.docker_options %}{{ alias.docker_options | replace("$", "\$") }} {% endif %} -v . -w . <container> {{ alias.args | replace("$", "\$") }}"
+    puts stderr "       {{ command }} run -i{% if tty %}t{% endif %} --rm -u `id -u`:`id -g` --entrypoint {{ alias.entrypoint | replace("$", "\$") }} {% if envfile %}--envfile  {{ module_dir }}/{{ envfile }} {% endif %}{% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %}{% if alias.docker_options %}{{ alias.docker_options | replace("$", "\$") }} {% endif %} -v . -w . <container> {{ alias.args | replace("$", "\$") }}"
 {% endfor %}{% endif %}
 
     puts stderr "For each of the above, you can export:"
@@ -56,11 +56,11 @@ conflict {{ name }}
 {% endfor %}{% endif %}
 
 # interactive shell to any container, plus exec for aliases
-set shellCmd "{{ command }} \${PODMAN_OPTS} run \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm -i{% if tty %}t{% endif %} --entrypoint {{ shell }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v $workdir -w $workdir ${containerPath}" 
+set shellCmd "{{ command }} \${PODMAN_OPTS} run \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm -i{% if tty %}t{% endif %} --entrypoint {{ shell }} {% if envfile %}--env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v $workdir -w $workdir ${containerPath}" 
 
 # execCmd needs entrypoint to be the executor
 set execCmd "{{ command }} \${PODMAN_OPTS} run -i{% if tty %}t{% endif %} \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm {% if envfile %} --env-file {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }}{% endif %} -v $workdir -w $workdir"
-set runCmd "{{ command }} \${PODMAN_OPTS} run -i{% if tty %}t{% endif %} \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm {% if envfile %}--env-file  {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %} -v $workdir -w $workdir ${containerPath}"
+set runCmd "{{ command }} \${PODMAN_OPTS} run -i{% if tty %}t{% endif %} \${PODMAN_COMMAND_OPTS} -u `id -u`:`id -g` --rm {% if envfile %}--env-file  {{ module_dir }}/{{ envfile }}{% endif %} {% if bindpaths %}-v {{ bindpaths }} {% endif %}{% if features.home %}-v {{ features.home }} {% endif %} -v $workdir -w $workdir ${containerPath}"
 set inspectCmd "{{ command }} \${PODMAN_OPTS} inspect ${containerPath}" 
 
 # set_shell_function takes bashStr and cshStr