Skip to content

Commit 471c0f9

Browse files
vsochvsoch
authored
Add support for shpc update (#538)
* adding support for update * preview of changes done by shpc update (#541) * If the container is deleted, the module has to be deleted too. No need to ask the user (#528) * Only ask for confirmation once * The .version file is only in module_dir, not container_dir * When deleting containers, also delete the parent directories as long as they are empty * `shpc add` now supports Docker cf #520 (comment) * round 2 of attempted updates! I saved the original registry this time so I should not need to open a new PR for every attempt * remove debug ipython * Adding more clear error message when tag is not known (#543) * adding more clear error message when tag is not known Signed-off-by: vsoch <[email protected]>
1 parent 4d2ba4e commit 471c0f9

File tree

215 files changed

+1325
-548
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

215 files changed

+1325
-548
lines changed

‎.github/workflows/update.yaml‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
name: 'Update Containers'
2+
3+
on:
4+
schedule:
5+
# Once a month
6+
- cron: '0 0 1 * *'
7+
8+
jobs:
9+
auto-scan:
10+
runs-on: ubuntu-latest
11+
steps:
12+
- name: Checkout
13+
uses: actions/checkout@v3
14+
with:
15+
fetch-depth: '0'
16+
17+
- name: Create conda environment
18+
run: conda create --quiet -c conda-forge --name shpc spython
19+
20+
- name: Install shpc
21+
run: |
22+
export PATH="/usr/share/miniconda/bin:$PATH"
23+
source activate shpc
24+
pip install -e .
25+
for name in $(shpc show); do
26+
shpc update ${name}
27+
done
28+
29+
- name: Checkout Update branch
30+
env:
31+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
32+
BRANCH_AGAINST: "main"
33+
run: |
34+
printf "GitHub Actor: ${GITHUB_ACTOR}\n"
35+
export BRANCH_FROM="binoc/update-$(date '+%Y-%m-%d')"
36+
git remote set-url origin "https://x-access-token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git"
37+
BRANCH_EXISTS=$(git ls-remote --heads origin ${BRANCH_FROM})
38+
if [[ -z ${BRANCH_EXISTS} ]]; then
39+
printf "Branch does not exist in remote.\n"
40+
else
41+
printf "Branch already exists in remote.\n"
42+
exit 1
43+
fi
44+
git branch
45+
git checkout -b "${BRANCH_FROM}" || git checkout "${BRANCH_FROM}"
46+
git branch
47+
git config --global user.name "github-actions"
48+
git config --global user.email "[email protected]"
49+
git add registry/*
50+
git status
51+
if git diff-index --quiet HEAD --; then
52+
export OPEN_PULL_REQUEST=0
53+
printf "No changes\n"
54+
else
55+
export OPEN_PULL_REQUEST=1
56+
printf "Changes\n"
57+
git commit -a -m "Automated deployment to update containers $(date '+%Y-%m-%d')"
58+
git push origin "${BRANCH_FROM}"
59+
fi
60+
echo "OPEN_PULL_REQUEST=${OPEN_PULL_REQUEST}" >> $GITHUB_ENV
61+
echo "PULL_REQUEST_FROM_BRANCH=${BRANCH_FROM}" >> $GITHUB_ENV
62+
echo "PULL_REQUEST_TITLE=[bot] ${BRANCH_FROM}" >> $GITHUB_ENV
63+
echo "PULL_REQUEST_BODY=Update container modules pull request." >> $GITHUB_ENV
64+
- name: Open Pull Request
65+
uses: vsoch/[email protected]
66+
if: ${{ env.OPEN_PULL_REQUEST == '1' }}
67+
env:
68+
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
69+
PULL_REQUEST_BRANCH: "main"

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
1414
The versions coincide with releases on pip. Only major versions will be released as tags on Github.
1515

1616
## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/main) (0.0.x)
17+
- add support for shpc update so container can be updated without binoc (0.0.53)
1718
- better error message if container tag does not exist on update (0.0.52)
1819
- minimum version of spython required is 0.2.0 to support apptainer (0.0.51)
1920
- add support for TCL and LMOD default version, multiple variants (0.0.50)

‎docs/getting_started/user-guide.rst‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -817,6 +817,84 @@ each unique module name, just add ``--short``:
817817
ghcr.io/autamus/samtools: latest
818818
ghcr.io/autamus/clingo: 5.5.0
819819
820+
Update
821+
------
822+
823+
As of version 0.0.52, you can request on demand updates of container.yaml recipes,
824+
where an update means we ping the registry or resource for the module and find
825+
updated tags. An update generally means that:
826+
827+
- We start with the 50 latest tags of the container, as determined by `crane.ggcr.dev <https://crane.ggcr.dev/ls/quay.io/biocontainers/samtools>`_
828+
- We filter according to any recipe ``filters`` in the container.yaml
829+
- Given a convention of including a hash, we try to remove it and generate a loose version
830+
- Any versions (including latest) that cannot be sorted based on some semblance to a version are filtered out
831+
- We sort the list, and given duplicates of some major minor (ignoring the last part of): ``<major>.<minor>.<ignored>`` we take the first seen in the sorted list.
832+
- Then we take the top 5 newest to add.
833+
- We then filter down to not include any versions older than the current oldest in the container.yaml
834+
835+
This action is run automatically on CI for you, however it's just done once a month and you are welcome to run it on your own, and contribute
836+
changes to container.yaml files that you think are meaningful. To update one container
837+
module recipe in the registry:
838+
839+
.. code-block:: console
840+
841+
$ shpc update quay.io/biocontainers/samtools
842+
Looking for updated digests for quay.io/biocontainers/samtools
843+
>> quay.io/biocontainers/samtools
844+
>> Latest
845+
1.15--h3843a85_0:sha256:d68e1b5f504dc60eb9f2a02eecbac44a63f144e7d455b3fb1a25323c667ca4c4
846+
>> Tags
847+
+ 1.9--h8571acd_11:sha256:3883c91317e7b6b62e31c82e2cef3cc1f3a9862633a13f850a944e828dd165ec
848+
+ 1.8--h46bd0b3_5:sha256:e495550231927c4b9b23a9f5920906f608129bf470dc3409ef7c6eecf0fa6d8e
849+
+ 1.7--2:sha256:9b3e923c44aa401e3e2b3bff825d36c9b07e97ba855ca04a368bf7b32f28aa97
850+
+ 1.6--he673b24_3:sha256:42031f060cde796279c09e6328d72bbce70d83a8f96e161faee3380ab689246d
851+
+ 1.5--2:sha256:9a2f99c26cee798e3b799447a7cfa0fbb0c1ce27c42eef7a3c1289ba871f55cb
852+
1.12--h9aed4be_1:sha256:5fd5f0937adf8a24b5bf7655110e501df78ae51588547c8617f17c3291a723e1
853+
1.15--h3843a85_0:sha256:d68e1b5f504dc60eb9f2a02eecbac44a63f144e7d455b3fb1a25323c667ca4c4
854+
1.10--h2e538c0_3:sha256:84a8d0c0acec87448a47cefa60c4f4a545887239fcd7984a58b48e7a6ac86390
855+
1.14--hb421002_0:sha256:88632c41eba8b94b7a2a1013f422aecf478a0cb278740bcc3a38058c903d61ad
856+
1.13--h8c37831_0:sha256:04da5297386dfae2458a93613a8c60216d158ee7cb9f96188dad71c1952f7f72
857+
1.11--h6270b1f_0:sha256:141120f19f849b79e05ae2fac981383988445c373b8b5db7f3dd221179af382b
858+
859+
860+
or to ask for a dryrun, meaning we check for updates but don't perform them.
861+
862+
.. code-block:: console
863+
864+
$ shpc update quay.io/biocontainers/samtools --dryrun
865+
866+
867+
If you want to look for a specific string or pattern in the tags, just add ``--filter``
868+
869+
.. code-block:: console
870+
871+
$ shpc update redis --dryrun --filter alpine
872+
873+
Since no tags are deleted, this will add the latest set found with the term "alpine." You can also use this
874+
strategy to add a specific tag:
875+
876+
877+
.. code-block:: console
878+
879+
$ shpc update redis --dryrun --filter 6.0-rc-alpine
880+
881+
The current implementation just supports updating from a Docker / oras registry (others will come after)
882+
and we don't currently support updating all tags at once, because the feature is relatively know
883+
and we want to take a conservative approach until we've seen it in action. However, you can easily
884+
loop through your module names to accomplish this:
885+
886+
.. code-block:: console
887+
888+
889+
$ for name in $(shpc show); do
890+
shpc update ${name} --dryun
891+
done
892+
893+
894+
Let us know if there are other features you'd like for update! For specific recipes
895+
it could be that a different method of choosing or sorting tags (beyond the defaults mentioned above
896+
and filter) is needed.
897+
820898

821899
Inspect
822900
-------

‎registry/adminer/container.yaml‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,10 @@ url: https://hub.docker.com/_/adminer
33
maintainer: '@vsoch'
44
description: Database management in a single PHP file.
55
latest:
6-
4.8.1: sha256:bbecfea5b1bafbe2057ce53996463f349daaa01198215e14a498a4a9c428f0ff
6+
4.8.1: sha256:b9125cae28bf249d1d2e9c4ffa42143a39ce1edc0eccbaf83e48ad6fbb3171b6
77
tags:
88
4.8.0-fastcgi: sha256:5368f087fed03f49e9de8731ee3d9998d7e78391720d500309b5bcde2a401058
9-
4.8.1: sha256:bbecfea5b1bafbe2057ce53996463f349daaa01198215e14a498a4a9c428f0ff
10-
4.8.1-fastcgi: sha256:fccb435d21db4a4e07bdbf830b870870584e15e96bed1bcf7ca0f241c41a2839
11-
latest: sha256:bbecfea5b1bafbe2057ce53996463f349daaa01198215e14a498a4a9c428f0ff
9+
4.8.1: sha256:b9125cae28bf249d1d2e9c4ffa42143a39ce1edc0eccbaf83e48ad6fbb3171b6
10+
4.8.1-fastcgi: sha256:16281d71200d8d8035e257f4624eb09ea3d03b5b4802c04844448c8c8f18f978
11+
latest: sha256:b9125cae28bf249d1d2e9c4ffa42143a39ce1edc0eccbaf83e48ad6fbb3171b6
12+
'4': sha256:b9125cae28bf249d1d2e9c4ffa42143a39ce1edc0eccbaf83e48ad6fbb3171b6

‎registry/bids/aa/container.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
docker: bids/aa
22
# Most of the tags/digests on Docker Hub seem to be unknown
33
latest:
4-
"enh_various": "sha256:c5b2c733ee6475449066f7dfe7865cc4bf2c74fe3c1150fd4f61a6fed2a6f78b"
4+
enh_various: sha256:c5b2c733ee6475449066f7dfe7865cc4bf2c74fe3c1150fd4f61a6fed2a6f78b
55
tags:
6-
"enh_various": "sha256:c5b2c733ee6475449066f7dfe7865cc4bf2c74fe3c1150fd4f61a6fed2a6f78b"
6+
enh_various: sha256:c5b2c733ee6475449066f7dfe7865cc4bf2c74fe3c1150fd4f61a6fed2a6f78b
77
filter:
8-
- "enh_various"
9-
maintainer: "@vsoch"
10-
description: "BIDS App containing an instance of the Automatic Analysis. (https://github.com/BIDS-Apps/aa)"
8+
- enh_various
9+
maintainer: '@vsoch'
10+
description: BIDS App containing an instance of the Automatic Analysis. (https://github.com/BIDS-Apps/aa)
1111
url: https://hub.docker.com/r/bids/aa

‎registry/bids/baracus/container.yaml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,11 @@ url: https://hub.docker.com/r/bids/baracus
33
maintainer: '@vsoch'
44
description: Brain-Age Regression Analysis and Computation Utility Software (https://github.com/BIDS-Apps/baracus)
55
latest:
6-
v1.1.4: sha256:8e42305ab7321fcdb468c22a87b5a9b248866a5821e64721200f927c2019dcb4
6+
dev7: sha256:6b47fc45ec68e1cbcff5f844a076a36a272ee43fb13217f8f9832abf01a6906a
77
tags:
88
latest: sha256:8e42305ab7321fcdb468c22a87b5a9b248866a5821e64721200f927c2019dcb4
99
v1.1.4: sha256:8e42305ab7321fcdb468c22a87b5a9b248866a5821e64721200f927c2019dcb4
10+
dev7: sha256:6b47fc45ec68e1cbcff5f844a076a36a272ee43fb13217f8f9832abf01a6906a
11+
unstable: sha256:8e42305ab7321fcdb468c22a87b5a9b248866a5821e64721200f927c2019dcb4
1012
filter:
1113
- v*

‎registry/bids/brainiak-srm/container.yaml‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
docker: bids/brainiak-srm
22
latest:
3-
"latest": "sha256:5ff856420a178be4e8150e574251a44885451050e9803515269e5ff15cd67430"
3+
latest: sha256:5ff856420a178be4e8150e574251a44885451050e9803515269e5ff15cd67430
44
tags:
5-
"latest": "sha256:5ff856420a178be4e8150e574251a44885451050e9803515269e5ff15cd67430"
5+
latest: sha256:5ff856420a178be4e8150e574251a44885451050e9803515269e5ff15cd67430
66
filter:
7-
- "v*"
8-
maintainer: "@vsoch"
9-
description: "Shared Response Model (SRM) from the Brain Imaging Analysis Kit (BrainIAK) (https://github.com/BIDS-Apps/brainiak-srm)"
7+
- v*
8+
maintainer: '@vsoch'
9+
description: Shared Response Model (SRM) from the Brain Imaging Analysis Kit (BrainIAK)
10+
(https://github.com/BIDS-Apps/brainiak-srm)
1011
url: https://hub.docker.com/r/bids/brainiak-srm

‎registry/bids/brainsautoworkup/container.yaml‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
docker: bids/brainsautoworkup
22
latest:
3-
"latest": "sha256:39fb638ba68955e2964a2c6cdab6cf7857b248f2b30d6b69b6f0a9a3401a376e"
3+
latest: sha256:39fb638ba68955e2964a2c6cdab6cf7857b248f2b30d6b69b6f0a9a3401a376e
44
tags:
5-
"latest": "sha256:39fb638ba68955e2964a2c6cdab6cf7857b248f2b30d6b69b6f0a9a3401a376e"
5+
latest: sha256:39fb638ba68955e2964a2c6cdab6cf7857b248f2b30d6b69b6f0a9a3401a376e
66
filter:
7-
- "v*"
8-
maintainer: "@vsoch"
9-
description: "A BIDS App for running BRAINSAutoWorkup (https://github.com/BIDS-Apps/BRAINSAutoworkup)"
7+
- v*
8+
maintainer: '@vsoch'
9+
description: A BIDS App for running BRAINSAutoWorkup (https://github.com/BIDS-Apps/BRAINSAutoworkup)
1010
url: https://hub.docker.com/r/bids/brainsautoworkup

‎registry/bids/broccoli/container.yaml‎

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,13 @@
11
docker: bids/broccoli
22
# The rest of this container's tags are unknown
33
latest:
4-
"enh_v": "sha256:1ae7cc35e6299fbd6ee020fbe39379d65bc7103b853eae707b8f3f2581c3bee4"
4+
enh_various: sha256:64050b22ef4a843a6651b3d782cdf26d7c6e6994ff7994ae9707b7e359ad3602
55
tags:
6-
"enh_v": "sha256:1ae7cc35e6299fbd6ee020fbe39379d65bc7103b853eae707b8f3f2581c3bee4"
6+
enh_v: sha256:1ae7cc35e6299fbd6ee020fbe39379d65bc7103b853eae707b8f3f2581c3bee4
7+
enh_various: sha256:64050b22ef4a843a6651b3d782cdf26d7c6e6994ff7994ae9707b7e359ad3602
78
filter:
8-
- "enh_v"
9-
maintainer: "@vsoch"
10-
description: "BROCCOLI is a software for analysis of fMRI (functional magnetic resonance imaging) data and is written in OpenCL (Open Computing Language). (https://github.com/BIDS-Apps/BROCCOLI)"
9+
- enh_v
10+
maintainer: '@vsoch'
11+
description: BROCCOLI is a software for analysis of fMRI (functional magnetic resonance
12+
imaging) data and is written in OpenCL (Open Computing Language). (https://github.com/BIDS-Apps/BROCCOLI)
1113
url: https://hub.docker.com/r/bids/broccoli

‎registry/bids/freesurfer/container.yaml‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,13 @@ url: https://hub.docker.com/r/bids/freesurfer
33
maintainer: '@vsoch'
44
description: Surface reconstruction using Freesurfer
55
latest:
6-
v6.0.1-6.1: sha256:02237eda4a22bb5fd66a50a66d21b13e0f378840fa61a937dab8794ee527cd48
6+
V30-a43f1f: sha256:cb7aee6c7634d7e57530b6b72230009759fe2c020c573aad87e71a523ba075f3
77
tags:
88
latest: sha256:02237eda4a22bb5fd66a50a66d21b13e0f378840fa61a937dab8794ee527cd48
99
v6.0.1-6.1: sha256:02237eda4a22bb5fd66a50a66d21b13e0f378840fa61a937dab8794ee527cd48
10+
V30-a43f1f: sha256:cb7aee6c7634d7e57530b6b72230009759fe2c020c573aad87e71a523ba075f3
11+
V29-37bf1c: sha256:513303323f955365d05b5b5d3223a37102c6eb064cc46d2a658db8de33db06c9
12+
V28-ae189c: sha256:ebdd94e63210301f277c00ad0e3464f1b1baa5703f5dad19f094f299e74f86c0
13+
enh_QA: sha256:b9fe8cf6d6cc2912f1671ab009d8d8805eb22f8b260f598069822807507f02fd
1014
filter:
1115
- v*

0 commit comments

Comments
 (0)