refactor of "add" to generate a container.yaml first to close #519 (#520)

* refactor of "add" to generate a container.yaml first to close #519
* this set of changes re-organizes the container template to be under the container
module, and also moves around some code in modules and container (base.py and config.py)
because in development I find the current locations not intuititve enough.
* add should only support maintaining the namespace for the docker:// uri
Signed-off-by: vsoch <[email protected]>

Author: vsoch

.github/workflows/main.yml

@@ -34,9 +34,28 @@ jobs:
         run: |
           export PATH="/usr/share/miniconda/bin:$PATH"
           source activate black
+          pyflakes shpc/utils/fileio.py
+          pyflakes shpc/utils/terminal.py
           pyflakes shpc/main/*.py
           pyflakes shpc/main/modules
           pyflakes shpc/main/container/base.py
           pyflakes shpc/main/container/podman.py
           pyflakes shpc/main/container/docker.py
           pyflakes shpc/main/container/singularity.py
+          pyflakes shpc/main/container/update/docker.py
+          pyflakes shpc/tests
+          pyflakes shpc/*.py
+          pyflakes shpc/client/add.py
+          pyflakes shpc/client/check.py
+          pyflakes shpc/client/config.py
+          pyflakes shpc/client/docgen.py
+          pyflakes shpc/client/get.py
+          pyflakes shpc/client/inspect.py
+          pyflakes shpc/client/install.py
+          pyflakes shpc/client/listing.py
+          pyflakes shpc/client/namespace.py
+          pyflakes shpc/client/pull.py
+          pyflakes shpc/client/show.py
+          pyflakes shpc/client/test.py
+          pyflakes shpc/client/uninstall.py
+          pyflakes shpc/main/wrappers

CHANGELOG.md

@@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/main) (0.0.x)
+ - refactor to "add" to generate a container.yaml first (0.0.49)
  - Properly cleanup empty module directories, and asking to remove a container that doesn't exist now logs a _warning_ (0.0.48)
  - wrapper script generation permissions error (0.0.47)
  - fixing but with stream command repeating output (0.0.46)

docs/getting_started/user-guide.rst

@@ -1114,19 +1114,87 @@ Add
 ---
 
 It might be the case that you have a container locally, and you want to
-make it available as a module (without pulling it from a registry). Although
-this is discouraged because it means you will need to manually maintain
-versions, shpc does support the "add" command to do this. You can simply provide
-the container path and the unique resource identifier:
+make it available as a module (without pulling it from a registry). You might also
+have a container on Docker Hub that you want to contribute to the registry! 
+shpc does support the "add" command to perform both of these functions. 
+The steps for adding a container are:
+
+1. Running ``shpc add`` to create a container.yaml in the registry namespace
+2. Customizing the container.yaml to your liking
+3. Running ``shpc install`` to formally install your new container.
+
+In the case of a docker image that is public (that you can share) you are encouraged
+to contribute your recipe directly to shpc for others to use, and once in the repository
+tags will also get updated automatically. 
+
+Add a Local Container
+^^^^^^^^^^^^^^^^^^^^^
+
+As an example, let's start with the container ``salad_latest.sif``. We have it
+on our local machine and cannot pull it from a registry. First, let's run ``shpc add``
+and tell shpc that we want it under the ``dinosaur/salad`` namespace.
+
+.. code-block:: console
+
+    $ shpc add salad_latest.sif dinosaur/salad:latest
+    Registry entry dinosaur/salad:latest was added! Before shpc install, edit:
+    /home/vanessa/Desktop/Code/shpc/registry/dinosaur/salad/container.yaml
+
+At this point, you should open up the container.yaml generated and edit to your liking.
+This usually means updating the description, maintainer, aliases, and possibly providing a url
+to find more information or support. Also notice we've provided the tag to be latest. If you update this registry
+entry in the future with a new version, you'll want to provide a new tag. If you provide
+an existing tag, you'll be asked to confirm before continuing. When you are happy, 
+it's time to install it, just as you would a regular container!
 
 .. code-block:: console
 
-    $ shpc add salad_latest.sif vanessa/salad:latest
+    $ shpc install dinosaur/salad:latest
+
+
+And this will generate the expected module and container in your respective directory bases:
+
+
+.. code-block:: console
+
+    $ tree modules/dinosaur/salad/
+    modules/dinosaur/salad/
+    └── latest
+        ├── 99-shpc.sh
+        └── module.lua
+
+    1 directory, 2 files
+
+    $ tree containers/dinosaur/salad/
+    containers/dinosaur/salad/
+    └── latest
+        └── sha256:77c7326e74d0e8b46d4e50d99e848fc950ed047babd60203e17449f5df8f39d4.sif
+
+    1 directory, 1 file
+
+
+Add a Registry Container
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+Let's say we want to generate a container.yaml recipe for a container on Docker Hub.
+Let's say we want to add `vanessa/pokemon <https://hub.docker.com/r/vanessa/pokemon>`_.
+First, let's run ``shpc add``. Note that we provide the ``docker://`` unique resource
+identifier to tell shpc it's from a Docker (OCI) registry.
+
+.. code-block:: console
+
+    $ shpc add docker://vanessa/pokemon
+    Registry entry vanessa/pokemon:latest was added! Before shpc install, edit:
+    /home/vanessa/Desktop/Code/shpc/registry/vanessa/pokemon/container.yaml
+
+
+And that's it! The container module will use the same namespace, ``vanessa/pokemon`` as the Docker image,
+and we do this purposefully as a design decision. Note that ``add`` previously would add the container directly to the module
+directory, and as of version 0.0.49 it's been updated to generate the container.yaml first.
+Also note that ``add`` is only supported for Singularity, as Docker and Podman containers are 
+typically provided via registries. If you are looking for support for add for another
+container technology, please `open a new issue <https://github.com/singularityhub/singularity-hpc/issues>`_.
 
-If the unique resource identifier corresponds with a registry entry, you
-will not be allowed to create it, as this would create a namespace conflict.
-Since we don't have a configuration file to define custom aliases, the container
-will just be exposed as it's command to run it.
 
 Get
 ---

registry/tensorflow/tensorflow/container.yaml

@@ -3,19 +3,18 @@ url: https://hub.docker.com/r/tensorflow/tensorflow
 maintainer: '@vsoch'
 description: An end-to-end open source platform for machine learning.
 latest:
-  2.8.0: sha256:7c01f75d58fadc2cd1109d5baac1925ed131e05925d840b1b49363c794d1c4db
+  2.7.1-gpu: sha256:581575fc3a736398f0dff9e950f57f2e6d808296267ac98325451a0b1d101dd0
 tags:
-  2.2.2: sha256:e2cde2bb70055511521d995cba58a28561089dfc443895fd5c66e65bbf33bfc0
   2.5.0-custom-op-gpu-ubuntu16: sha256:478bee6f0691b48d74adc3fcffe3e9ececf35df5c02860cc51a2c48b1d92c730
   2.5.0rc0-gpu-jupyter: sha256:9808e04142b09482bb6b3d1738430ae7472a214dd38e086d41e481b376fa9abd
   2.6.0: sha256:773d5ce09e4ce003db02740c6a372a8a9f43be2bac23544d8f452bfec5347c53
   2.6.0rc0-gpu-jupyter: sha256:358b5bf90aaf4e56813ff22f2981d86fab7ddc59552b0be6022ae04d6a9f43c3
   2.7.0: sha256:31e09cf438a41f12c759cc8cc79c6b0fbb0db5abfc3de8169e916c8c9ac38dc5
   2.7.0rc0: sha256:abbc457c9b7c0725d7d0db885dbb313db3d0ae25733b083900a508efb672af94
-  2.7.1-gpu: sha256:581575fc3a736398f0dff9e950f57f2e6d808296267ac98325451a0b1d101dd0
   2.8.0: sha256:7c01f75d58fadc2cd1109d5baac1925ed131e05925d840b1b49363c794d1c4db
   2.8.0rc0: sha256:11e5d21a786da523d2f7de530c083d5c72a06e02c8895c84595d107c579027a1
   latest-gpu: sha256:1e03623e335aac1610b1a3cfa6a96cf10156acb095287f9d6031df3980148663
+  2.7.1-gpu: sha256:581575fc3a736398f0dff9e950f57f2e6d808296267ac98325451a0b1d101dd0
 filter:
 - 2.*
 - latest-gpu

shpc/__init__.py

@@ -1 +1,3 @@
 from shpc.version import __version__
+
+assert __version__

shpc/client/__init__.py

@@ -129,9 +129,11 @@ def get_parser():
 
     # Add a container direcly
     add = subparsers.add_parser("add", description="add an image to modules manually")
-    add.add_argument("sif_path", help="full path to container image file", nargs=1)
+    add.add_argument("container_uri", help="full path to container image file")
     add.add_argument(
-        "module_id", help='desired identifier for module (e.g. "name/version")', nargs=1
+        "module_id",
+        help='desired identifier for module (e.g. "name/version"). Not required for docker)',
+        nargs="?",
     )
 
     check = subparsers.add_parser(
@@ -368,11 +370,11 @@ def help(return_code=0):
 
     # Pass on to the correct parser
     return_code = 0
-    try:
-        main(args=args, parser=parser, extra=extra, subparser=helper)
-        sys.exit(return_code)
-    except UnboundLocalError:
-        return_code = 1
+    #    try:
+    main(args=args, parser=parser, extra=extra, subparser=helper)
+    sys.exit(return_code)
+    #    except UnboundLocalError:
+    #        return_code = 1
 
     help(return_code)
 

shpc/client/add.py

@@ -13,4 +13,5 @@ def main(args, parser, extra, subparser):
         module=args.module,
         container_tech=args.container_tech,
     )
-    cli.add(args.sif_path[0], args.module_id[0])
+    # If we don't have a module name, we derive from container URI
+    cli.add(args.container_uri, args.module_id)