Skip to content

Commit a2ce511

Browse files
khk-globuskhk-globus
authored
Update GH actions runner per GH image retirement (#3844)
# Description GitHub retired the Ubuntu 20.04 image per their warning that we had been ignoring. In order to unbreak CI, update to use the current LTS image, Ubuntu 24.04. In so doing, the Radical Pilot tests broke, so take the opportunity to do some housekeeping in two key changes: - Install MPI and other external dependencies from Conda -- no more manual download and installation - Move the Radical, WorkQueue, and TaskVine tests to separate GitHub actions so that optional extra packages can be tested separately. This allows us to 1) identify test failures originating from core parsl vs external packages and 2) test different components in parallel. # Changed Behaviour No user-facing behavior, but CI tests are now a bit more decoupled. # Fixes - Broken CI - Fixes: #3544 ## Type of change - Code maintenance/cleanup
1 parent a282d79 commit a2ce511

File tree

10 files changed

+119
-75
lines changed

10 files changed

+119
-75
lines changed

‎.github/workflows/ci.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ jobs:
1212
strategy:
1313
matrix:
1414
python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
15-
runs-on: ubuntu-20.04
15+
runs-on: ubuntu-24.04
1616
timeout-minutes: 60
1717

1818
steps:

‎.github/workflows/gce_test.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ env:
1111

1212
jobs:
1313
main-test-suite:
14-
runs-on: ubuntu-20.04
14+
runs-on: ubuntu-24.04
1515
timeout-minutes: 60
1616

1717
steps:

‎.github/workflows/parsl+cctools.yaml‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
name: WorkQueue and TaskVine tests
2+
on:
3+
pull_request:
4+
types:
5+
- opened
6+
- synchronize
7+
8+
jobs:
9+
build-linux:
10+
runs-on: ubuntu-24.04
11+
timeout-minutes: 60
12+
strategy:
13+
max-parallel: 5
14+
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Set up Python 3.11
19+
uses: actions/setup-python@v3
20+
with:
21+
python-version: '3.11'
22+
23+
- name: Add conda to system path
24+
run: |
25+
# $CONDA is an environment variable pointing to the root of the miniconda directory
26+
echo $CONDA/bin >> $GITHUB_PATH
27+
echo "CONDA BIN: $CONDA/bin"
28+
29+
- name: Install parsl and dependencies
30+
run: |
31+
conda install --channel=conda-forge ndcctools mpich mpi4py openssl
32+
make deps
33+
pip install .[workqueue]
34+
export PATH=/usr/share/miniconda/bin/:$PATH
35+
36+
- name: Sanity check
37+
run: |
38+
echo "PATH: $PATH"
39+
echo "Python: "; which python3
40+
python3 -c "import ndcctools; print(ndcctools.__file__)"
41+
python3 -c "from ndcctools import work_queue; print(work_queue.__version__)"
42+
python3 -c "import parsl; print(parsl.__version__)"
43+
which parsl_coprocess.py
44+
which python3
45+
find /usr/share/miniconda | grep libssl
46+
47+
- name: Run pytest suite for WorkQueue and TaskVine
48+
run: |
49+
make wqex_local_test
50+
make vineex_local_test

‎.github/workflows/parsl+radical.yaml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Radical tests
2+
on:
3+
pull_request:
4+
types:
5+
- opened
6+
- synchronize
7+
8+
jobs:
9+
build-linux:
10+
runs-on: ubuntu-24.04
11+
timeout-minutes: 60
12+
strategy:
13+
max-parallel: 5
14+
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Set up Python 3.11
19+
uses: actions/setup-python@v3
20+
with:
21+
python-version: '3.11'
22+
23+
- name: Add conda to system path
24+
run: |
25+
# $CONDA is an environment variable pointing to the root of the miniconda directory
26+
echo $CONDA/bin >> $GITHUB_PATH
27+
echo "CONDA BIN: $CONDA/bin"
28+
29+
- name: Install parsl and dependencies
30+
run: |
31+
conda install --channel=conda-forge mpich mpi4py openssl
32+
make deps
33+
pip3 install ".[radical-pilot]"
34+
export PATH=/usr/share/miniconda/bin/:$PATH
35+
36+
- name: Sanity check
37+
run: |
38+
echo "PATH: $PATH"
39+
echo "Python: "; which python3
40+
mpiexec --version
41+
mpiexec -n 4 python3 -c "from mpi4py import MPI; assert MPI.COMM_WORLD.Get_size() == 4, 'Expected 4 ranks'; print(f'Rank {MPI.COMM_WORLD.Get_rank()} of {MPI.COMM_WORLD.Get_size()}')"
42+
43+
- name: Run pytest suite for Radical
44+
run: |
45+
export RADICAL_CI=1
46+
make radical_local_test

‎Makefile‎

Lines changed: 8 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -3,11 +3,8 @@ SHELL := $(shell which bash) # Use bash instead of bin/sh as shell
33
GIT := $(shell which git || echo ".git_is_missing")
44
CWD := $(shell pwd)
55
DEPS := .deps
6-
CCTOOLS_INSTALL := /tmp/cctools
76
MPICH=mpich
87
OPENMPI=openmpi
9-
export PATH := $(CCTOOLS_INSTALL)/bin/:$(PATH)
10-
export CCTOOLS_VERSION=7.8.0
118
export HYDRA_LAUNCHER=fork
129
export OMPI_MCA_rmaps_base_oversubscribe=yes
1310
MPI=$(MPICH)
@@ -69,27 +66,25 @@ htex_local_alternate_test: ## run all tests with htex_local config
6966
pip3 install ".[monitoring]"
7067
pytest parsl/tests/ -k "not cleannet" --config parsl/tests/configs/htex_local_alternate.py --random-order --durations 10
7168

72-
$(CCTOOLS_INSTALL): #CCtools contains both taskvine and workqueue so install only once
73-
parsl/executors/taskvine/install-taskvine.sh
74-
7569
.PHONY: vineex_local_test
76-
vineex_local_test: $(CCTOOLS_INSTALL) ## run all tests with taskvine_ex config
77-
PYTHONPATH=/tmp/cctools/lib/python3.8/site-packages pytest parsl/tests/ -k "not cleannet" --config parsl/tests/configs/taskvine_ex.py --random-order --durations 10
70+
vineex_local_test:
71+
pytest parsl/tests/ -k "not cleannet" --config parsl/tests/configs/taskvine_ex.py --random-order --durations 10
7872

7973
.PHONY: wqex_local_test
80-
wqex_local_test: $(CCTOOLS_INSTALL) ## run all tests with workqueue_ex config
81-
PYTHONPATH=/tmp/cctools/lib/python3.8/site-packages pytest parsl/tests/ -k "not cleannet" --config parsl/tests/configs/workqueue_ex.py --random-order --durations 10
74+
wqex_local_test:
75+
pytest parsl/tests/ -k "not cleannet" --config parsl/tests/configs/workqueue_ex.py --random-order --durations 10
8276

8377
.PHONY: radical_local_test
8478
radical_local_test:
8579
pip3 install ".[radical-pilot]"
8680
mkdir -p ~/.radical/pilot/configs && echo '{"localhost": {"virtenv_mode": "local"}}' > ~/.radical/pilot/configs/resource_local.json
8781
pytest parsl/tests/ -k "not cleannet and not issue3328 and not executor_supports_std_stream_tuples" --config parsl/tests/configs/local_radical.py --random-order --durations 10
82+
pytest parsl/tests/ -m "radical" --config local --random-order --durations 10
8883

8984
.PHONY: config_local_test
90-
config_local_test: $(CCTOOLS_INSTALL)
85+
config_local_test:
9186
pip3 install ".[monitoring,visualization,proxystore,kubernetes]"
92-
PYTHONPATH=/tmp/cctools/lib/python3.8/site-packages pytest parsl/tests/ -k "not cleannet" --config local --random-order --durations 10
87+
pytest parsl/tests/ -k "not cleannet" --config local --random-order --durations 10
9388

9489
.PHONY: site_test
9590
site_test:
@@ -101,7 +96,7 @@ perf_test:
10196
parsl-perf --time 5 --config parsl/tests/configs/local_threads.py
10297

10398
.PHONY: test ## run all tests with all config types
104-
test: clean_coverage isort lint flake8 mypy local_thread_test htex_local_test htex_local_alternate_test wqex_local_test vineex_local_test radical_local_test config_local_test perf_test ## run all tests
99+
test: clean_coverage isort lint flake8 mypy local_thread_test htex_local_test htex_local_alternate_test config_local_test perf_test ## run all tests
105100

106101
.PHONY: tag
107102
tag: ## create a tag in git. to run, do a 'make VERSION="version string" tag

‎parsl/executors/taskvine/install-taskvine.sh‎

Lines changed: 0 additions & 21 deletions
This file was deleted.

‎parsl/executors/workqueue/executor.py‎

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,16 +40,18 @@
4040

4141
from .errors import WorkQueueFailure, WorkQueueTaskFailure
4242

43+
IMPORT_EXCEPTION = None
4344
try:
44-
import work_queue as wq
45-
from work_queue import (
45+
from ndcctools import work_queue as wq
46+
from ndcctools.work_queue import (
4647
WORK_QUEUE_ALLOCATION_MODE_MAX_THROUGHPUT,
4748
WORK_QUEUE_DEFAULT_PORT,
4849
WorkQueue,
4950
)
50-
except ImportError:
51+
except ImportError as e:
5152
_work_queue_enabled = False
5253
WORK_QUEUE_DEFAULT_PORT = 0
54+
IMPORT_EXCEPTION = e
5355
else:
5456
_work_queue_enabled = True
5557

@@ -257,7 +259,7 @@ def __init__(self,
257259
BlockProviderExecutor.__init__(self, provider=provider,
258260
block_error_handler=True)
259261
if not _work_queue_enabled:
260-
raise OptionalModuleMissing(['work_queue'], "WorkQueueExecutor requires the work_queue module.")
262+
raise OptionalModuleMissing(['work_queue'], f"WorkQueueExecutor requires the work_queue module. More info: {IMPORT_EXCEPTION}")
261263

262264
self.scaling_cores_per_worker = scaling_cores_per_worker
263265
self.label = label

‎parsl/executors/workqueue/install-workqueue.sh‎

Lines changed: 0 additions & 16 deletions
This file was deleted.

‎parsl/tests/test_monitoring/test_basic.py‎

Lines changed: 1 addition & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -54,26 +54,8 @@ def htex_filesystem_config():
5454
return c
5555

5656

57-
def workqueue_config():
58-
from parsl.tests.configs.workqueue_ex import fresh_config
59-
c = fresh_config()
60-
c.monitoring = MonitoringHub(
61-
hub_address="localhost",
62-
resource_monitoring_interval=1)
63-
return c
64-
65-
66-
def taskvine_config():
67-
c = Config(executors=[TaskVineExecutor(manager_config=TaskVineManagerConfig(port=9000),
68-
worker_launch_method='provider')],
69-
70-
monitoring=MonitoringHub(hub_address="localhost",
71-
resource_monitoring_interval=1))
72-
return c
73-
74-
7557
@pytest.mark.local
76-
@pytest.mark.parametrize("fresh_config", [htex_config, htex_filesystem_config, htex_udp_config, workqueue_config, taskvine_config])
58+
@pytest.mark.parametrize("fresh_config", [htex_config, htex_filesystem_config, htex_udp_config])
7759
def test_row_counts(tmpd_cwd, fresh_config):
7860
# this is imported here rather than at module level because
7961
# it isn't available in a plain parsl install, so this module

‎parsl/tests/test_radical/test_mpi_funcs.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
import os
2+
13
import pytest
24

35
import parsl
@@ -18,6 +20,10 @@ def some_mpi_func(msg, sleep, comm=None, parsl_resource_specification={}):
1820

1921
@pytest.mark.local
2022
@pytest.mark.radical
23+
@pytest.mark.skipif(
24+
os.environ.get("RADICAL_CI") != "1",
25+
reason="Only runs in Radical CI workflow"
26+
)
2127
def test_radical_mpi(n=7):
2228
# rank size should be > 1 for the
2329
# radical runtime system to run this function in MPI env

0 commit comments

Comments
 (0)