Adding namespaces to make module install easier (#366)

* adding namespaces to make module install easier
* add tests for show and namespace

Signed-off-by: vsoch <[email protected]>

Author: vsoch

.gitignore

@@ -1,6 +1,7 @@
 shpc.db
 singularity_hpc.egg-info/
 .env
+env
 build
 docs/_build
 release

CHANGELOG.md

@@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/master) (0.0.x)
+ - adding namespaces to make module install, show, inspect easier (0.0.24)
  - container url does not render in docs (0.0.23)
  - addition of tcl modules, removal of un-needed database (0.0.22)
  - first update of all containers, and bugfix to pull (0.0.21)

docs/getting_started/installation.rst

@@ -76,9 +76,9 @@ You'll next want to configure and create your registry, discussed next in
 .. code-black:: console
 
 
-    $ shpc config registry:/<DIR>
-    $ shpc config lmod_base:/<DIR> 
-    $ shpc config container_base:/<DIR> 
+    $ shpc config set registry:/<DIR>
+    $ shpc config set lmod_base:/<DIR> 
+    $ shpc config set container_base:/<DIR> 
 
 
 You can also easily (manually) update any settings in the ``shpc/settings.yaml`` file. 

docs/getting_started/user-guide.rst

@@ -206,6 +206,9 @@ A summary table of variables is included below, and then further discussed in de
    * - singularity_shell
      - exported to SINGULARITY_SHELL, defaults to /bin/bash.
      - /bin/bash
+   * - namespace
+     - Set a default module namespace that you want to install from.
+     - null
 
 
 
@@ -339,16 +342,29 @@ Config
 ------
 
 If you want to edit a configuration value, you can either edit the ``shpc/settings.yml``
-file directly, or you can use ``shpc config``. The following example shows changing
-the default module_base path from the install directory modules folder.
+file directly, or you can use ``shpc config``, which will accept:
+
+ - set to set a parameter and value
+ - get to get a parameter by name
+
+
+The following example shows changing the default module_base path from the install directory modules folder.
 
 .. code-block:: console
 
     # an absolute path
-    $ shpc config module_base:/opt/lmod/modules
+    $ shpc config set module_base:/opt/lmod/modules
 
     # or a path relative to the install directory, remember to escape the "$"
-    $ shpc config module_base:\$install_dir/modules
+    $ shpc config set module_base:\$install_dir/modules
+
+
+And then to get values:
+
+.. code-block:: console
+
+    $ shpc config get module_base
+
 
 
 Show and Install
@@ -420,6 +436,70 @@ the container technology.
 If you don't have any module software on your system, you can now test interacting
 with the module via the :ref:`getting_started-development` instructions.
 
+
+Namespace
+---------
+
+Let's say that you are exclusively using continers in the namespace ghcr.io/autamus.
+
+.. code-block:: console
+
+    registry/ghcr.io/
+    └── autamus
+        ├── abi-dumper
+        ├── abyss
+        ├── accumulo
+        ├── addrwatch
+        ...
+        ├── xrootd
+        ├── xz
+        └── zlib
+
+
+It can become arduous to type the entire namespace every time! For this purpose,
+you can set a namespace:
+
+.. code-block:: console
+
+    $ shpc namespace use ghcr.io/autamus
+
+And then instead of asking to install clingo as follows:
+
+.. code-block:: console
+
+    $ shpc install ghcr.io/autamus/clingo
+    
+
+You can simply ask for:
+
+
+.. code-block:: console
+
+    $ shpc install clingo
+    
+    
+And when you are done, unset the namespace.
+
+
+.. code-block:: console
+
+    $ shpc namespace unset
+
+
+Note that you can also set the namespace as any other setting:
+
+.. code-block:: console
+
+    $ shpc config set namespace:ghcr.io/autamus
+
+Namespaces currently work with:
+
+ - install
+ - uninstall
+ - show
+ - add
+ - check
+
 List
 ----
 

docs/index.rst

@@ -57,6 +57,9 @@ Resources
 `Singularity HPC Library <https://github.com/singularityhub/singularity-hpc/issues>`_
     Shows modules available to install as containers.
 
+`Autamus Registry <https://autamus.io>`_
+    Provides many of the shpc container modules, built directly from spack.
+
 
 .. toctree::
    :caption: Getting started

registry/vanessa/salad/container.yaml

@@ -0,0 +1,10 @@
+docker: vanessa/salad
+url: https://hub.docker.com/r/vanessa/salad
+maintainer: '@vsoch'
+description: A container all about fork and spoon puns.
+latest:
+  latest: sha256:e8302da47e3200915c1d3a9406d9446f04da7244e4995b7135afd2b79d4f63db
+tags:
+  latest: sha256:e8302da47e3200915c1d3a9406d9446f04da7244e4995b7135afd2b79d4f63db
+aliases:
+  salad: /code/salad

shpc/client/__init__.py

@@ -12,7 +12,10 @@
 
 
 def get_parser():
-    parser = argparse.ArgumentParser(description="Singularity Registry (HPC)")
+    parser = argparse.ArgumentParser(
+        description="Singularity Registry (HPC)",
+        formatter_class=argparse.RawTextHelpFormatter,
+    )
 
     # Global Variables
     parser.add_argument(
@@ -104,7 +107,10 @@ def get_parser():
     check = subparsers.add_parser("check", help="check if you have latest installed.")
     check.add_argument("module_name", help="module to check (module/version)")
 
-    config = subparsers.add_parser("config", help="update configuration settings.")
+    config = subparsers.add_parser(
+        "config",
+        help="update configuration settings. Use set or get to see or set information.",
+    )
     config.add_argument(
         "params",
         nargs="*",
@@ -198,6 +204,16 @@ def get_parser():
             default="singularity",
         )
 
+    namespace = subparsers.add_parser(
+        "namespace",
+        help="set or unset the install namespace. E.g.,:\n    shpc namespace set <namespace>\n    shpc namespace unset",
+    )
+    namespace.add_argument(
+        "namespace",
+        help="command (use/unset) and if use, the namespace to set",
+        nargs="*",
+    )
+
     show = subparsers.add_parser("show", help="show the config for a registry entry.")
     show.add_argument(
         "--versions", help="include versions", default=False, action="store_true"
@@ -277,6 +293,8 @@ def help(return_code=0):
         from .inspect import main
     elif args.command == "list":
         from .listing import main
+    elif args.command == "namespace":
+        from .namespace import main
     elif args.command == "shell":
         from .shell import main
     elif args.command == "show":

shpc/client/config.py

@@ -17,11 +17,31 @@ def main(args, parser, extra, subparser):
         print(subparser.format_help())
         sys.exit(0)
 
-    # For each new setting, update and save!
-    for param in args.params:
-        key, value = param.split(":", 1)
-        logger.info("Updating %s to be %s" % (key, value))
-        cli.settings.set(key, value)
+    # The first "param" is either set of get
+    command = args.params.pop(0)
 
-    # Save settings
-    cli.settings.save()
+    # For each new setting, update and save!
+    if command == "set":
+        for param in args.params:
+            if ":" not in param:
+                logger.warning(
+                    "Param %s is missing a :, should be key:value pair. Skipping."
+                    % param
+                )
+                continue
+            key, value = param.split(":", 1)
+            logger.info("Updating %s to be %s" % (key, value))
+            cli.settings.set(key, value)
+
+        # Save settings
+        cli.settings.save()
+
+    # For each get request, print the param pair
+    elif command == "get":
+        for key in args.params:
+            value = cli.settings.get(key)
+            value = value or "is unset"
+            logger.info("%s %s" % (key.ljust(30), value))
+
+    else:
+        logger.error("%s is not a recognized command." % command)

shpc/client/namespace.py

@@ -0,0 +1,35 @@
+__author__ = "Vanessa Sochat"
+__copyright__ = "Copyright 2021, Vanessa Sochat"
+__license__ = "MPL 2.0"
+
+
+from shpc.logger import logger
+import sys
+
+
+def main(args, parser, extra, subparser):
+
+    from shpc.main import get_client
+
+    cli = get_client(quiet=args.quiet, settings_file=args.settings_file)
+
+    # Case 1: we need to unset a namespace
+    if not args.namespace:
+        sys.exit("Please choose: shpc use <namespace> or shpc unset.")
+
+    command = args.namespace.pop(0)
+
+    # Unset a namespace, does not need or consider additional arguments
+    if command == "unset":
+        logger.info("Unsetting all namespaces.")
+        cli.settings.set("namespace", None)
+
+    # use a namespace requires another argument (the namespace)
+    elif command == "use":
+        if not args.namespace:
+            sys.exit("use must be followed by a namespace!")
+        namespace = args.namespace[0]
+        logger.info("Setting namespace %s" % namespace)
+        cli.settings.set("namespace", namespace)
+
+    cli.settings.save()

shpc/main/client.py

@@ -106,6 +106,14 @@ def get(self, module_name):
         """
         raise NotImplementedError
 
+    def add_namespace(self, name):
+        """
+        If a namespace is defined in settings, use it
+        """
+        if self.settings.namespace:
+            name = "%s/%s" % (self.settings.namespace.strip("/"), name)
+        return name
+
     def _load_container(self, name, tag=None):
         """
         Given a name and an optional tag to default to, load a package
@@ -225,6 +233,7 @@ def show(self, name, names_only=False, out=None):
         Show available packages
         """
         if name:
+            name = self.add_namespace(name)
             config = self._load_container(name)
             config.dump(out)
         else: