Skip to content

Perform Slurm database upgrade if necessary #670

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 12 commits into from
May 15, 2025
Merged

Perform Slurm database upgrade if necessary #670

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
ca44f89
bump OpenHPC snapshots to v3.1.1 (slurm 24.11.5) and v2.9.1 (slurm 23…
sjpb May 13, 2025
929dde2
bump CI image
sjpb May 13, 2025
a77ef88
extend timeout for slurmdbd startup to cope with major version upgrad…
sjpb May 13, 2025
4bce66c
configure openhpc for slurmdbd backup/update
sjpb May 14, 2025
92a37f2
support mysql tasks in openhpc role
sjpb May 14, 2025
6b8f843
Merge branch 'main' into feat/slurmdbd-upgrade
sjpb May 14, 2025
860cb73
remove slurmdbd startup timeout increase - got borked during merge fr…
sjpb May 14, 2025
5a923b2
mysql package now installed separately in role from openhpc_packages
sjpb May 14, 2025
8d7371f
bump CI image to get mysql client installed
sjpb May 14, 2025
fb5e7cd
delete snapshot when cleaning up in CI
sjpb May 15, 2025
9422cdf
bump openhpc role to commit
sjpb May 15, 2025
e50f964
bump openhpc role to release
sjpb May 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/workflows/stackhpc.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -230,6 +230,16 @@ jobs:
env:
DEMO_USER_PASSWORD: ${{ secrets.TEST_USER_PASSWORD }}

- name: Delete possible volume snapshot from slurm upgrade
run: |
. venv/bin/activate
. environments/.stackhpc/activate
if [ -n "$SNAPSHOT" ]
then
echo Deleting $SNAPSHOT
openstack volume snapshot delete $SNAPSHOT
fi

- name: Delete infrastructure
run: |
. venv/bin/activate
Expand Down
4 changes: 2 additions & 2 deletions environments/.stackhpc/tofu/cluster_image.auto.tfvars.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"cluster_image": {
"RL8": "openhpc-RL8-250513-1045-ca44f898",
"RL9": "openhpc-RL9-250513-1046-ca44f898"
"RL8": "openhpc-RL8-250514-1502-5a923b2c",
"RL9": "openhpc-RL9-250514-1502-5a923b2c"
}
}
13 changes: 9 additions & 4 deletions environments/common/inventory/group_vars/all/openhpc.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,12 @@ ohpc_default_extra_repos:
"9": []
"8": []

# systemd.service.unit.TimeoutStartSec to wait for slurmdbd startup
# Set long enought to avoid problems with a major version upgrade
# Currently implemented in environments/common/inventory/group_vars/all/systemd.yml
openhpc_slurmdbd_timeout_start_sec: '45 minutes'
# configure slurm database pre-upgrade backups:
openhpc_slurm_accounting_storage_service: mysql
openhpc_slurm_accounting_storage_backup_cmd: >-
openstack volume snapshot create
--volume {{ openhpc_cluster_name }}-state
--force
{{ openhpc_cluster_name }}-state-{{ ansible_date_time.iso8601_basic_short }}
openhpc_slurm_accounting_storage_backup_host: localhost
openhpc_slurm_accounting_storage_backup_become: false
8 changes: 1 addition & 7 deletions environments/common/inventory/group_vars/all/systemd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,13 +14,7 @@ systemd_dropins:
content: "{{ _systemd_requiresmount_statedir }}"
slurmdbd:
group: openhpc
content: |
{{ _systemd_requiresmount_statedir }}

[Service]
# Allow slurmdbd to complete major version upgrades
TimeoutStartSec={{ openhpc_slurmdbd_timeout_start_sec }}

content: "{{ _systemd_requiresmount_statedir }}"
slurmctld:
group: openhpc
content: "{{ _systemd_requiresmount_statedir }}"
Expand Down
2 changes: 1 addition & 1 deletion requirements.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ roles:
version: v25.3.2
name: stackhpc.nfs
- src: https://github.com/stackhpc/ansible-role-openhpc.git
version: v0.28.0
version: v0.30.0
name: stackhpc.openhpc
- src: https://github.com/stackhpc/ansible-node-exporter.git
version: stackhpc
Expand Down
Loading