Conversation
Collaborator
Author
|
Release process notes:
Step 1: setup/testingOn Bebop: cd ~/ez/zstash
git status
# On branch issue-374-refactor-tar-deletion
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 2
# 01fe023 Add Python 3.14 support by forcing parallelism to use fork (#434)
# 33f379d Merge pull request #416 from E3SM-Project/issue-374-refactor-tar-deletion
# Good, matches https://github.com/E3SM-Project/zstash/commits/main/
lcrc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zstash_main_20260406
conda activate zstash_main_20260406
pre-commit run --all-files
python -m pip install .
pytest tests/unit/test_*.py
# 25 passed, 46 warnings in 1.96s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py
# Ran 69 tests in 233.037s
# OK (skipped=32)
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 11.193s
# OK
cd ~/ez/zstash/tests/integration/bash_tests/run_from_chrysalis/
# If not done above, do the following:
# Log into globus.org
# Log into endpoints (LCRC Improv DTN, NERSC Perlmutter) at globus.org: File Manager > Add the endpoints in the "Collection" fields
# Reset completely:
# Revoke consents: https://auth.globus.org/v2/web/consents > Globus Endpoint Performance Monitoring > rescind all
# Run the following lines to set up the database_corruption test
rm ~/.zstash_globus_tokens.json
mkdir zstash_demo; echo 'file0 stuff' > zstash_demo/file0.txt
# NERSC_PERLMUTTER_ENDPOINT=6bdc7956-fc0f-4ad2-989c-7aa5ee643a79
zstash create --hpss=globus://6bdc7956-fc0f-4ad2-989c-7aa5ee643a79//global/homes/f/forsyth/zstash/tests/test_database_corruption_setup23 zstash_demo
# ERROR: /home/ac.forsyth2/.zstash.ini does not have the local Globus endpoint set nor could one be found in REGEX_ENDPOINT_MAP.
# That's because we're on Bebop, not Chrysalis....
cd ~/ez/zstash
git status
rm -rf tests/integration/bash_tests/run_from_chrysalis/zstash_demo/
git grep -in "chr" zstash
# zstash/extract.py:526: # This is to synchronize the print statements.
# zstash/globus_utils.py:34: r"chrlogin.*\.lcrc\.anl\.gov": "15288284-7006-4041-ba1a-6b52501e49f1",
# zstash/globus_utils.py:36: r"chr.*\.lcrc\.anl\.gov": "15288284-7006-4041-ba1a-6b52501e49f1",
# zstash/parallel.py:22: Used to synchronize the printing of the output between workers.
# zstash/parallel.py:37: self._cv: multiprocessing.synchronize.Condition = multiprocessing.Condition()
hostname
# beboplogin5.lcrc.anl.gov
# Edit zstash/globus_utils.py
git diff
# r"beboplogin\d+.*\.lcrc\.anl\.gov": "15288284-7006-4041-ba1a-6b52501e49f1",
git checkout -b add-bebop-support
git add -A
pre-commit run --all-files
git commit -m "Add Bebop support"
git log --oneline | head -n 3
# 18f8187 Add Bebop support
# 01fe023 Add Python 3.14 support by forcing parallelism to use fork (#434)
# 33f379d Merge pull request #416 from E3SM-Project/issue-374-refactor-tar-deletion
python -m pip install .
# Restart testing:
cd ~/ez/zstash/tests/integration/bash_tests/run_from_chrysalis/
# Run the following lines to set up the database_corruption test
mkdir zstash_demo; echo 'file0 stuff' > zstash_demo/file0.txt
# NERSC_PERLMUTTER_ENDPOINT=6bdc7956-fc0f-4ad2-989c-7aa5ee643a79
zstash create --hpss=globus://6bdc7956-fc0f-4ad2-989c-7aa5ee643a79//global/homes/f/forsyth/zstash/tests/test_database_corruption_20260406 zstash_demo
# Paste auth code
rm -rf zstash_demo/
# Then, set a unique_id below to avoid using an old directory.
time ./database_corruption.bash test_database_corruption_20260406v2
# Success count: 25
# Fail count: 0
# Review:
# real 8m34.807s
# user 0m5.272s
# sys 0m2.644s
time ./symlinks.sh
# real 0m2.010s
# No errors
# Make sure the new change didn't mess up the previously passing tests:
cd ~/ez/zstash
pytest tests/unit/test_*.py
# 25 passed, 46 warnings in 0.66s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py
# Ran 69 tests in 69.216s
# OK (skipped=32)
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 3.263s
# OK
cd ~/ez/zstash/tests/integration/bash_tests/run_from_any/
./test_globus_tar_deletion.bash run20260406 /home/ac.forsyth2/ez/zstash /home/ac.forsyth2/zstash_tests LCRC_IMPROV_DTN_ENDPOINT
# ==========================================
# TEST RESULTS
# ==========================================
# ✓ blocking_non-keep PASSED
# ✓ non-blocking_non-keep PASSED
# ✓ blocking_keep PASSED
# ✓ non-blocking_keep PASSED
# ✓ blocking_progressive_deletion PASSED
# ✓ non-blocking_progressive_deletion PASSED
# ==========================================
# TEST SUMMARY
# ==========================================
# Total tests: 6
# Passed: 6
# Failed: 0
# ==========================================
# All globus tar deletion tests completed successfully.
# We should be able to use chrysalis below, even though we're on Bebop:
./globus_auth.bash run20260406 chrysalis /home/ac.forsyth2/ez/zstash /home/ac.forsyth2/zstash_tests /global/homes/f/forsyth/zstash_tests /home/f/forsyth/zstash_tests /compyfs/fors729/zstash_tests
# All globus_auth tests completed successfully.
cd ~/ez/zstash
git status
git push upstream add-bebop-supportCreated and merged #437. On Perlmutter: cd ~/ez/zstash
git status
# On branch profile-refactored-tar-deletion
# nothing to commit, working tree clean
git fetch upstream main
git checkout main
git reset --hard upstream/main
git log --oneline | head -n 3
# 601eb4b Add Bebop support (#437)
# 01fe023 Add Python 3.14 support by forcing parallelism to use fork (#434)
# 33f379d Merge pull request #416 from E3SM-Project/issue-374-refactor-tar-deletion
# Good, matches https://github.com/E3SM-Project/zstash/commits/main/
nersc_conda
rm -rf build
conda clean --all --y
conda env create -f conda/dev.yml -n zstash_main_20260406
conda activate zstash_main_20260406
pre-commit run --all-files
python -m pip install .
pytest tests/unit/test_*.py
# 25 passed, 46 warnings in 0.35s
python -m unittest tests/integration/python_tests/group_by_command/test_*.py
# Ran 69 tests in 353.212s
# OK
python -m unittest tests/integration/python_tests/group_by_workflow/test_*.py
# Ran 4 tests in 3.127s
# OK
cd tests/integration/bash_tests/run_from_perlmutter/
time ./follow_symlinks.sh
# real 0m33.307s
# No errors
time ./test_update_non_empty_hpss.bash
# real 0m9.916s
# No errors
# Log into globus.org
# Log into endpoints (NERSC Perlmutter, Globus Tutorial Collection 1) at globus.org: File Manager > Add the endpoints in the "Collection" fields
time ./test_ls_globus.bash # Paste auth code
# real 0m41.239s
# No errors
cd ~/ez/zstash
git status
# No changes were madeStep 2: Bump versionsWhat is our new version? Nothing has been a breaking change, so we don't need to increment the major version. https://github.com/E3SM-Project/zstash/commits/main/:
So, we will increment the minor version. We'll go from Step 2a: Confluence
Step 2b: zstash repocd ~/ez/zstash
git fetch upstream main
git checkout -b v1.6.0rc1 upstream/main
tbump 1.6.0rc1 --no-tag
git diff HEAD^ HEAD | cat
# Yes, these match up with changes in https://github.com/E3SM-Project/zstash/pull/389
git push upstream v1.6.0rc1
# Create, and "Update version" label" to, and merge https://github.com/E3SM-Project/zstash/pull/438
git checkout main
git fetch upstream
git reset --hard upstream/main
git tag -a v1.6.0rc1 -m "v1.6.0rc1"
# Delete the branch from the tbump step. Otherwise, the push command won't work.
git branch -D v1.6.0rc1
git push upstream v1.6.0rc1
Step 2c: zstash-feedstock repocd /global/homes/f/forsyth/zstash-feedstock
curl -sL https://github.com/E3SM-Project/zstash/archive/v1.6.0rc1.tar.gz | openssl sha256
# SHA2-256(stdin)= 1f0d08b264dd8bf06266369b0377ea3a44519c674781f1e402626f0a2974173a
git status # Check for uncommitted changes
git fetch upstream dev
git checkout -b v1.6.0rc1 upstream/dev
emacs recipe/meta.yaml
# In `recipe/meta.yaml`, update the version and sha256 (and the build number if needed):
# {% set version = "1.6.0rc1" %}
# sha256: ... # The sha256 from the previous step
# number: 0 # build >>> number should always be 0
# From https://github.com/conda-forge/zppy-interfaces-feedstock/pull/5#pullrequestreview-3351366258:
# Please always check the diff between the tag you're making a package for and the previous one. Please look for changes e.g. in dev.yml
# Review:
# https://github.com/E3SM-Project/zstash/compare/v1.5.0...v1.6.0rc1
# conda/dev.yml changes:
# - python >=3.11,<3.14 =>
# - python >=3.11,<3.15
# - setuptools
# In recipe/meta.yaml we already have:
# requirements:
# host:
# - python {{ python_min }}
# - pip
# - setuptools
# run:
# - python >={{ python_min }}
# - sqlite
# - six >=1.16.0
# - globus-sdk >=3.15.0,<4.0
# So, it looks like there's nothing to change then.
git add -A
git commit -m "v1.6.0rc1"
git push forsyth2 v1.6.0rc1Created conda-forge/zstash-feedstock#25 |
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bump to 1.6.0rc1