Add/views (#545)

* adding testing of a symlink tree install
* unset symlink_home
* missing module load
* adding symlink_tree setting to indicate to always set
also adding support for custom config on the fly with -c and for
removing symlinks on uninstall
* missing command argument
* symlink_home -> symlink_tree and --symlink-tree instead of --symlink
* forgot to add dest
* ensure that we cleanup symlink directory of .version and empty
* ensure we dont create symlink version unless tcl and default version is true
* do not require symlink_base to be defined
we can have a default of $root_dir/symlinks instead
* bugfix: without this, create_symlink would only be called based on the CLI argument, not the .yml setting (#509)
* verison bump
* Minor refactoring of `check_symlink` (#510)
* No need to create the symlink base directory here since it will be created by `create_symlink`
* Make this part of the code symmetric with self.create_symlink()
* Fixed a truncated sentence
* running black
* Skip `module` in the symlinks (#511)
* Implemented "default_version" for TCL
* Use write_version_file for the symlink tree too
* Skip `module.tcl` in the symlinks
This is done by symlinking `<software>/<version>` itself to
`<namespace>/<software>/<version>/module.tcl`.
For the directory of the wrapper scripts to be correctly found, the
symlink has to be resolved, but TCL's `file normalize` won't normalise
the filename. So, we need to use `file readlink` instead, but only on
real symlinks because it raises an error.
* Symlink to module.lua when possible
which is when default_version==True (default_version==False can't be
made to work with symlinks).
* Added a `--force` option to `shpc install` to force overwriting existing symlinks
The name `--force` is generic, so that other things could be forced
through it, not just overwriting symlinks.
Also added an info message if a symlink is overwritten, which can be
hidden with the `--quiet` flag.
* Made `force` optional
* Forgot the variable for substitution
* The "delete" command was superseded by "uninstall" in #6
* Added `--no-symlink-tree` to override the config file
`--symlink-tree` now also overrides the config file
* Make it explicit we are expecting yes or no
* add force and symlink arg back in
* do not pin black
* Completion of the symlink feature (#539)
* Removed leftover from an earlier implementation of `default_version`
* bugfix: True and False are deprecated (but still valid)
* Lmod symlinks can now skip "module" too thanks to @marcodelapierre
* Added the missing newline character at the end of the file
* bugfix: the return needs to happen after the creation of the symlink
* `symlink_base` should allow environment variable expansion, like the other directories
* Cannot consider using symlinks without the base defined
* The two classes are meant to address exactly this
* bugfix: the symlink needs to be cleaned regardless of where the modules are held
* rmdir_to_base does the upwards clean-up correctly
* No need to complain if symlinks are not enabled in the first place
* bugfix: the caller of write_version_file needs to build the directory path
All other calls were already doing it, this was the only exception.
The change is required because not all .version files are in
$module_base
* bugfix: .version needs to be updated in the symlink tree too
* bugfix: when uninstalling the last version of a tool there is no directory any more
* Make the settings object upgrade legacy values so that the code only needs to know about the current ones
* Expanded the function to deal with files and symlinks
* Adding we can just use a filesystem loader and then from_string instead
* Message update
cf #502 (comment)
* saving start of work on views

this adds an shpc view command and refactors the symlink nomenclature to instead b
about views! I need to write documentation and make a few additional commands to load
a view from file and write tests for views too.
* adding changes to docs and tests for views!
* spelling mistake and missing tests helpers file
* force needs to be specific for uninstall
* remove deprecated symlink-tree
* adding support to install from a file and list installed modules
* legacy values is broken - remove for now
* fixing bug with module version files - cannot do any funny business with creating module class on the fly!
* module .version in symlink only should exist for lmod
* fixing bug with uninstall and adding shpc view list
* adding support to generate a view module
a user can now do shpc view add <var> <val> to add customizations to a view! In
practice this means that installed modules need to make an attempt to load the view
and given that it is a symbolic link, it should find the file (at least we hope). This
means in practice one view should be loaded at once, otherwise you will have two conflicting
.view_module.
* ensure we only use try-load for modules >= 4.8.

I am also updating the testing to run for both an older and newer version (with
and without support)

* fixing bug with test
I had changed the client variable to be module_sys instead of module to match
the rest of the library, and forgot to update it in the tests helper script.
* adding support for shpc view remove <name> <var> <value>

and I am also adding better testing to look at both the config and file
content of view_module in both cases

Signed-off-by: vsoch <[email protected]>
Co-authored-by: vsoch <[email protected]>
Co-authored-by: Matthieu Muffato <[email protected]>
Co-authored-by: Matthieu Muffato <[email protected]>

Author: vsoch

.github/workflows/main.yml

@@ -57,5 +57,6 @@ jobs:
           pyflakes shpc/client/pull.py
           pyflakes shpc/client/show.py
           pyflakes shpc/client/test.py
+          pyflakes shpc/client/view.py
           pyflakes shpc/client/uninstall.py
           pyflakes shpc/main/wrappers

.github/workflows/test.yml

@@ -9,7 +9,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        module: ["lmod", "tcl"]
+        module: [["lmod", "8.4.27"] , ["tcl", "4.7.1"], ["tcl", "5.0.1"]]
         container_tech: ["podman", "singularity", "docker"]
     steps:
 
@@ -18,20 +18,23 @@ jobs:
           sudo apt-get install lua5.2 liblua5.2-dev lua-filesystem lua-posix tcl tcl-dev tcsh
 
       - name: Install Environment Modules
-        if: ${{ matrix.module == 'tcl' }}
+        if: ${{ matrix.module[0] == 'tcl' }}
+        env:
+          PKG_VERSION: ${{ matrix.module[1] }}
         run: |
-         curl -LJO https://github.com/cea-hpc/modules/releases/download/v4.7.1/modules-4.7.1.tar.gz
-         tar xfz modules-4.7.1.tar.gz
+         curl -LJO https://github.com/cea-hpc/modules/releases/download/v${PKG_VERSION}/modules-${PKG_VERSION}.tar.gz
+         tar xfz modules-${PKG_VERSION}.tar.gz
          ls
-         cd modules-4.7.1
+         cd modules-${PKG_VERSION}
          ./configure
          make
          sudo make install
 
       - name: Install Lmod
-        if: ${{ matrix.module == 'lmod' }}
+        if: ${{ matrix.module[0] == 'lmod' }}
+        env:
+          PKG_VERSION: ${{ matrix.module[1] }}
         run: |
-          PKG_VERSION=8.4.27
           PKG_URL="https://github.com/TACC/Lmod/archive/${PKG_VERSION}.tar.gz"
           wget $PKG_URL
           tar xfz ${PKG_VERSION}.tar.gz
@@ -47,7 +50,7 @@ jobs:
       - name: Create conda environment
         run: conda create --quiet -c conda-forge --name shpc spython
 
-      - uses: actions/checkout@v2
+      - uses: actions/checkout@v3
       - name: Install shpc
         run: |
           export PATH="/usr/share/miniconda/bin:$PATH"
@@ -56,8 +59,10 @@ jobs:
 
       - name: Run python module tests (bash)
         shell: bash
+        env:
+          module_sys: ${{ matrix.module[0] }}
         run: |
-          case "${{ matrix.module }}" in
+          case "${module_sys}" in
             lmod)
               . /usr/share/lmod/lmod/init/bash
               ;;
@@ -71,7 +76,7 @@ jobs:
           export PATH="/usr/share/miniconda/bin:$PATH"
           source activate shpc
           shpc config set container_tech:${{ matrix.container_tech }}
-          shpc config set module_sys:${{ matrix.module }}
+          shpc config set module_sys:${module_sys}
           shpc config set enable_tty:false
 
           printf "\n\shpc test ============================================\n"
@@ -100,13 +105,20 @@ jobs:
           cat test_output
           grep --quiet 'Python 3.9.5' test_output
           rm test_output
-          shpc uninstall --force python:3.9.5-alpine 
+          shpc uninstall --force python:3.9.5-alpine
 
-        
+          # Try creating views install
+          mkdir -p tmp-modules
+          shpc config set views_base:tmp-modules
+          shpc view create noodles
+          shpc install --view noodles python:3.9.5-alpine
+ 
       - name: Run python module tests (tcsh)
         shell: tcsh -e {0}
+        env:
+          module_sys: ${{ matrix.module[0] }}
         run: |
-          switch ("${{ matrix.module }}")
+          switch ("${module_sys}")
             case lmod:
               source /usr/share/lmod/lmod/init/tcsh
               breaksw
@@ -118,7 +130,7 @@ jobs:
           source /usr/share/miniconda/etc/profile.d/conda.csh
           conda activate shpc
           shpc config set container_tech:${{ matrix.container_tech }}
-          shpc config set module_sys:${{ matrix.module }}
+          shpc config set module_sys:${module_sys}
           shpc config set enable_tty:false
 
           shpc install python:3.9.5-alpine

.gitignore

@@ -15,5 +15,5 @@ __pycache__
 *.sif
 *.img
 /.eggs
-/symlinks
 /modules
+/views

CHANGELOG.md

@@ -14,6 +14,9 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/main) (0.0.x)
+ - support for views create, delete, edit, install, uninstall (0.0.54)
+  - also including cleanup of module in views on uninstall
+  - ability to set custom config variable on the fly with -c
  - add support for shpc update so container can be updated without binoc (0.0.53)
  - better error message if container tag does not exist on update (0.0.52)
  - minimum version of spython required is 0.2.0 to support apptainer (0.0.51)