Skip to content

Commit 54f0b25

Browse files
vsochmuffato
andauthored
Add/views (#545)
* adding testing of a symlink tree install * unset symlink_home * missing module load * adding symlink_tree setting to indicate to always set also adding support for custom config on the fly with -c and for removing symlinks on uninstall * missing command argument * symlink_home -> symlink_tree and --symlink-tree instead of --symlink * forgot to add dest * ensure that we cleanup symlink directory of .version and empty * ensure we dont create symlink version unless tcl and default version is true * do not require symlink_base to be defined we can have a default of $root_dir/symlinks instead * bugfix: without this, create_symlink would only be called based on the CLI argument, not the .yml setting (#509) * verison bump * Minor refactoring of `check_symlink` (#510) * No need to create the symlink base directory here since it will be created by `create_symlink` * Make this part of the code symmetric with self.create_symlink() * Fixed a truncated sentence * running black * Skip `module` in the symlinks (#511) * Implemented "default_version" for TCL * Use write_version_file for the symlink tree too * Skip `module.tcl` in the symlinks This is done by symlinking `<software>/<version>` itself to `<namespace>/<software>/<version>/module.tcl`. For the directory of the wrapper scripts to be correctly found, the symlink has to be resolved, but TCL's `file normalize` won't normalise the filename. So, we need to use `file readlink` instead, but only on real symlinks because it raises an error. * Symlink to module.lua when possible which is when default_version==True (default_version==False can't be made to work with symlinks). * Added a `--force` option to `shpc install` to force overwriting existing symlinks The name `--force` is generic, so that other things could be forced through it, not just overwriting symlinks. Also added an info message if a symlink is overwritten, which can be hidden with the `--quiet` flag. * Made `force` optional * Forgot the variable for substitution * The "delete" command was superseded by "uninstall" in #6 * Added `--no-symlink-tree` to override the config file `--symlink-tree` now also overrides the config file * Make it explicit we are expecting yes or no * add force and symlink arg back in * do not pin black * Completion of the symlink feature (#539) * Removed leftover from an earlier implementation of `default_version` * bugfix: True and False are deprecated (but still valid) * Lmod symlinks can now skip "module" too thanks to @marcodelapierre * Added the missing newline character at the end of the file * bugfix: the return needs to happen after the creation of the symlink * `symlink_base` should allow environment variable expansion, like the other directories * Cannot consider using symlinks without the base defined * The two classes are meant to address exactly this * bugfix: the symlink needs to be cleaned regardless of where the modules are held * rmdir_to_base does the upwards clean-up correctly * No need to complain if symlinks are not enabled in the first place * bugfix: the caller of write_version_file needs to build the directory path All other calls were already doing it, this was the only exception. The change is required because not all .version files are in $module_base * bugfix: .version needs to be updated in the symlink tree too * bugfix: when uninstalling the last version of a tool there is no directory any more * Make the settings object upgrade legacy values so that the code only needs to know about the current ones * Expanded the function to deal with files and symlinks * Adding we can just use a filesystem loader and then from_string instead * Message update cf #502 (comment) * saving start of work on views this adds an shpc view command and refactors the symlink nomenclature to instead b about views! I need to write documentation and make a few additional commands to load a view from file and write tests for views too. * adding changes to docs and tests for views! * spelling mistake and missing tests helpers file * force needs to be specific for uninstall * remove deprecated symlink-tree * adding support to install from a file and list installed modules * legacy values is broken - remove for now * fixing bug with module version files - cannot do any funny business with creating module class on the fly! * module .version in symlink only should exist for lmod * fixing bug with uninstall and adding shpc view list * adding support to generate a view module a user can now do shpc view add <var> <val> to add customizations to a view! In practice this means that installed modules need to make an attempt to load the view and given that it is a symbolic link, it should find the file (at least we hope). This means in practice one view should be loaded at once, otherwise you will have two conflicting .view_module. * ensure we only use try-load for modules >= 4.8. I am also updating the testing to run for both an older and newer version (with and without support) * fixing bug with test I had changed the client variable to be module_sys instead of module to match the rest of the library, and forgot to update it in the tests helper script. * adding support for shpc view remove <name> <var> <value> and I am also adding better testing to look at both the config and file content of view_module in both cases Signed-off-by: vsoch <[email protected]> Co-authored-by: vsoch <[email protected]> Co-authored-by: Matthieu Muffato <[email protected]> Co-authored-by: Matthieu Muffato <[email protected]>
1 parent da4dfe2 commit 54f0b25

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+1787
-270
lines changed

.github/workflows/main.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,5 +57,6 @@ jobs:
5757
pyflakes shpc/client/pull.py
5858
pyflakes shpc/client/show.py
5959
pyflakes shpc/client/test.py
60+
pyflakes shpc/client/view.py
6061
pyflakes shpc/client/uninstall.py
6162
pyflakes shpc/main/wrappers

.github/workflows/test.yml

Lines changed: 26 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ jobs:
99
strategy:
1010
fail-fast: false
1111
matrix:
12-
module: ["lmod", "tcl"]
12+
module: [["lmod", "8.4.27"] , ["tcl", "4.7.1"], ["tcl", "5.0.1"]]
1313
container_tech: ["podman", "singularity", "docker"]
1414
steps:
1515

@@ -18,20 +18,23 @@ jobs:
1818
sudo apt-get install lua5.2 liblua5.2-dev lua-filesystem lua-posix tcl tcl-dev tcsh
1919
2020
- name: Install Environment Modules
21-
if: ${{ matrix.module == 'tcl' }}
21+
if: ${{ matrix.module[0] == 'tcl' }}
22+
env:
23+
PKG_VERSION: ${{ matrix.module[1] }}
2224
run: |
23-
curl -LJO https://github.com/cea-hpc/modules/releases/download/v4.7.1/modules-4.7.1.tar.gz
24-
tar xfz modules-4.7.1.tar.gz
25+
curl -LJO https://github.com/cea-hpc/modules/releases/download/v${PKG_VERSION}/modules-${PKG_VERSION}.tar.gz
26+
tar xfz modules-${PKG_VERSION}.tar.gz
2527
ls
26-
cd modules-4.7.1
28+
cd modules-${PKG_VERSION}
2729
./configure
2830
make
2931
sudo make install
3032
3133
- name: Install Lmod
32-
if: ${{ matrix.module == 'lmod' }}
34+
if: ${{ matrix.module[0] == 'lmod' }}
35+
env:
36+
PKG_VERSION: ${{ matrix.module[1] }}
3337
run: |
34-
PKG_VERSION=8.4.27
3538
PKG_URL="https://github.com/TACC/Lmod/archive/${PKG_VERSION}.tar.gz"
3639
wget $PKG_URL
3740
tar xfz ${PKG_VERSION}.tar.gz
@@ -47,7 +50,7 @@ jobs:
4750
- name: Create conda environment
4851
run: conda create --quiet -c conda-forge --name shpc spython
4952

50-
- uses: actions/checkout@v2
53+
- uses: actions/checkout@v3
5154
- name: Install shpc
5255
run: |
5356
export PATH="/usr/share/miniconda/bin:$PATH"
@@ -56,8 +59,10 @@ jobs:
5659
5760
- name: Run python module tests (bash)
5861
shell: bash
62+
env:
63+
module_sys: ${{ matrix.module[0] }}
5964
run: |
60-
case "${{ matrix.module }}" in
65+
case "${module_sys}" in
6166
lmod)
6267
. /usr/share/lmod/lmod/init/bash
6368
;;
@@ -71,7 +76,7 @@ jobs:
7176
export PATH="/usr/share/miniconda/bin:$PATH"
7277
source activate shpc
7378
shpc config set container_tech:${{ matrix.container_tech }}
74-
shpc config set module_sys:${{ matrix.module }}
79+
shpc config set module_sys:${module_sys}
7580
shpc config set enable_tty:false
7681
7782
printf "\n\shpc test ============================================\n"
@@ -100,13 +105,20 @@ jobs:
100105
cat test_output
101106
grep --quiet 'Python 3.9.5' test_output
102107
rm test_output
103-
shpc uninstall --force python:3.9.5-alpine
108+
shpc uninstall --force python:3.9.5-alpine
104109
105-
110+
# Try creating views install
111+
mkdir -p tmp-modules
112+
shpc config set views_base:tmp-modules
113+
shpc view create noodles
114+
shpc install --view noodles python:3.9.5-alpine
115+
106116
- name: Run python module tests (tcsh)
107117
shell: tcsh -e {0}
118+
env:
119+
module_sys: ${{ matrix.module[0] }}
108120
run: |
109-
switch ("${{ matrix.module }}")
121+
switch ("${module_sys}")
110122
case lmod:
111123
source /usr/share/lmod/lmod/init/tcsh
112124
breaksw
@@ -118,7 +130,7 @@ jobs:
118130
source /usr/share/miniconda/etc/profile.d/conda.csh
119131
conda activate shpc
120132
shpc config set container_tech:${{ matrix.container_tech }}
121-
shpc config set module_sys:${{ matrix.module }}
133+
shpc config set module_sys:${module_sys}
122134
shpc config set enable_tty:false
123135
124136
shpc install python:3.9.5-alpine

.gitignore

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,5 +15,5 @@ __pycache__
1515
*.sif
1616
*.img
1717
/.eggs
18-
/symlinks
1918
/modules
19+
/views

CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,9 @@ and **Merged pull requests**. Critical items to know are:
1414
The versions coincide with releases on pip. Only major versions will be released as tags on Github.
1515

1616
## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/main) (0.0.x)
17+
- support for views create, delete, edit, install, uninstall (0.0.54)
18+
- also including cleanup of module in views on uninstall
19+
- ability to set custom config variable on the fly with -c
1720
- add support for shpc update so container can be updated without binoc (0.0.53)
1821
- better error message if container tag does not exist on update (0.0.52)
1922
- minimum version of spython required is 0.2.0 to support apptainer (0.0.51)

0 commit comments

Comments
 (0)