Skip to content

Implementation of proposed syntax changes #768

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 512 commits into from
Apr 1, 2025
Merged

Implementation of proposed syntax changes #768

Show file tree
Hide file tree
Changes from 250 commits
Commits
Show all changes
512 commits
Select commit Hold shift + click to select a range
f1a6cb3
working on singularity errors
tclose Feb 23, 2025
d8696f2
more debugging of singularity
tclose Feb 23, 2025
b30d982
more debugging
tclose Feb 23, 2025
bc5397d
attempting to fix singularity tests
tclose Feb 23, 2025
b1184d4
fixed up environments import
tclose Feb 24, 2025
c8bdd3b
shell.define can take a list of tokens as well as a single string
tclose Feb 24, 2025
2fc135f
debugging docker unittests
tclose Feb 24, 2025
a42cc51
test_dockertask tests pass
tclose Feb 24, 2025
fdd5e52
fixed initialisation of environment in Submitter
tclose Feb 24, 2025
44d0d67
debugged test_dockertask
tclose Feb 24, 2025
28f1bc5
fixed merge error
tclose Feb 25, 2025
3f397e0
fixed merging error in test_shell
tclose Feb 25, 2025
4080298
debugging unittests
tclose Feb 25, 2025
ffeece0
debugged singularity tests
tclose Feb 25, 2025
3b8f818
made imports in pydra.design.* modules absolute
tclose Feb 25, 2025
6aaf1b7
added fileformats-extras to test deps
tclose Feb 25, 2025
e6724c1
fetch tags in github actions so installed version is correct
tclose Feb 25, 2025
4b97ed6
Merge pull request #769 from nipype/test_environments
tclose Feb 25, 2025
f7021e5
fixed bug in state depth calculation where, single, combined states w…
tclose Feb 25, 2025
b2034d5
moved resolved lazy inputs into NodeExecution class from TaskDef
tclose Feb 25, 2025
db8f799
finished debugging test_specs
tclose Feb 26, 2025
b5a0455
Merge pull request #770 from nipype/test_specs
tclose Feb 26, 2025
33bb999
debugging test_specs
tclose Feb 26, 2025
bb75d94
deleted workflow to pick up changes in workflow graph error message a…
tclose Feb 26, 2025
a5f6e45
Merge pull request #771 from nipype/test_submitter
tclose Feb 26, 2025
bda7ea2
debugging test_task
tclose Feb 26, 2025
dfbd989
fixed up remaining test_task tests
tclose Feb 26, 2025
85a3d95
fixed test_task_files
tclose Feb 26, 2025
bf22f78
started debugging test_node_tasks
tclose Feb 26, 2025
f2bf301
updated tests in test_node_task (most are passing)
tclose Feb 27, 2025
9c7d57e
debugging splitting and combining
tclose Feb 28, 2025
e3fe8e0
reverted change to LazyField._get_value
tclose Feb 28, 2025
2e3f5ed
reworking test_shelltask_inputspec
tclose Feb 28, 2025
110bfb5
debugging test_shell_inputspec
tclose Feb 28, 2025
d88acfe
debugging requires for bool fields
tclose Mar 2, 2025
d199981
reworked test_shelltask to new syntax
tclose Mar 2, 2025
753f0af
removing shelly.result\(\)
tclose Mar 2, 2025
0dba80b
more cleaning of test_shelltask
tclose Mar 2, 2025
b60a2fb
fixed error message in missing base class
tclose Mar 3, 2025
76c769b
debugging test_shelltask*
tclose Mar 3, 2025
739e8f3
updating test_workflow to new syntax
tclose Mar 3, 2025
9ba0e42
cleaned up all linting errors and warnings in test_workflow
tclose Mar 3, 2025
afe64e1
assert results not errored
tclose Mar 3, 2025
cc8646b
added node names
tclose Mar 3, 2025
bddb1f1
debugging test_workflow
tclose Mar 3, 2025
35c1594
Merge pull request #772 from nipype/test_task
tclose Mar 3, 2025
75a2983
fixed check for missing fields
tclose Mar 4, 2025
7b3aeb5
fixed splitting of tasks over states where value comes from lazy field
tclose Mar 4, 2025
9344e1c
debugging hashing of function AST
tclose Mar 4, 2025
8a29e1e
debugging optional shell outputs
tclose Mar 4, 2025
b33bbdf
debugging shell task tests
tclose Mar 5, 2025
090baa5
debugged 'xor' arg attribute
tclose Mar 6, 2025
af65aab
fixed up 'xor' arg functionality
tclose Mar 6, 2025
4a77081
debugged test_shelltask_inputspec
tclose Mar 6, 2025
7f1b259
fixed up issue with optional xor
tclose Mar 6, 2025
2664509
handle empty state arrays, i.e. nodes that don't run any jobs
tclose Mar 6, 2025
a52b748
renamed NodeExecution.task to NodeExecution.get_tasks
tclose Mar 6, 2025
9df1191
moved xor into *.define decorators from *.arg fields
tclose Mar 7, 2025
1cc4074
debugging combining states to preserve nested lists over staggered co…
tclose Mar 7, 2025
c5da03b
debugging node state operations
tclose Mar 7, 2025
5611651
fixed bug in _set_state
tclose Mar 7, 2025
7b64080
implemented state depth() implementation by handling the RPN represen…
tclose Mar 7, 2025
dfa195c
replaced 'after_combine=False' with before_combine=True for state.dep…
tclose Mar 7, 2025
8fa2d94
removed task definition attribute from state
tclose Mar 7, 2025
0b01701
cleaned up comment
tclose Mar 7, 2025
1455c2e
debugging issues with nested output types
tclose Mar 7, 2025
6980cf0
debugging removal of definition from state
tclose Mar 7, 2025
a1e3268
debugging test_shelltask
tclose Mar 7, 2025
b741563
debugging test_shelltask
tclose Mar 10, 2025
183782b
added pympler to test dependencies
tclose Mar 10, 2025
022c02a
fixed issue in doc-string passing
tclose Mar 10, 2025
ee88000
changed additional_args to be MultiInputObj[str] from list[str] type
tclose Mar 10, 2025
f62fd0d
debugging test_shelltask, reworking resolve of lazy inputs
tclose Mar 10, 2025
da92150
dropped StateIndex object in favour of simple int index
tclose Mar 11, 2025
b0db082
debugging shelltask
tclose Mar 11, 2025
3a9e35f
test_shelltask passes
tclose Mar 12, 2025
3287bea
reverted all plugin specifications in test_shelltask to the plugin fi…
tclose Mar 12, 2025
83ac1e8
modified bytes_repr_type so that the module location drops any sub-mo…
tclose Mar 12, 2025
8f593fd
reverted argstr with "..." in them to use field sep (instead of " " h…
tclose Mar 12, 2025
c7ad6b2
fixed up list of stdlib modules and is_stdlib
tclose Mar 12, 2025
5295e46
fixed up LazyField._resolve_value to handle combine states after swit…
tclose Mar 12, 2025
128c224
minor fixups
tclose Mar 12, 2025
54bbd47
fixed all unittests outside of test_workflow that were broken in rece…
tclose Mar 12, 2025
4e331e5
debugging states and workflows
tclose Mar 13, 2025
ce860d2
reimplementing lazy-field split
tclose Mar 13, 2025
d8321c0
attempting to debug state splitting/combining logic
tclose Mar 13, 2025
32931d3
debugging state preparation and lazy value resolution
tclose Mar 14, 2025
f51063e
moved Node.cont_dim into State
tclose Mar 14, 2025
b03307d
backed out of trying to allow execution of downstream nodes before/wh…
tclose Mar 15, 2025
4a51195
debugging test_workflow
tclose Mar 15, 2025
5c101a8
debugging test_workflow
tclose Mar 16, 2025
e0d4a8c
Update new-docs/source/tutorial/1-getting-started.ipynb
tclose Mar 17, 2025
e276608
cleaning up broken unittests
tclose Mar 16, 2025
3de0fa0
debugged workflow graph plotting
tclose Mar 17, 2025
3ac52a1
Update new-docs/source/tutorial/5-shell.ipynb
tclose Mar 17, 2025
49af485
removed stray text files generated by tests, and changed tests so tha…
tclose Mar 17, 2025
3e5090e
Update new-docs/source/tutorial/1-getting-started.ipynb
tclose Mar 17, 2025
7245524
require all jobs in predecessor nodes to have completed successfully …
tclose Mar 17, 2025
29456be
debugged cache and rerunning in test_workflow
tclose Mar 17, 2025
9c27f08
debugged error rerun
tclose Mar 18, 2025
7e1428c
test_workflow passes (except funny behaviour with rerun error)
tclose Mar 18, 2025
16a3423
fixed direct output setting in workflow
tclose Mar 18, 2025
68a2b6e
cleaned up rebroken unittests
tclose Mar 18, 2025
95759b7
added "cf" to test matrix
tclose Mar 18, 2025
66bcfaa
attempting to fix remaining errors showing up in GHA
tclose Mar 18, 2025
6d2e6ff
fixed task def call error message
tclose Mar 18, 2025
5360c73
fixed up testslurm pytest call
tclose Mar 18, 2025
107c9e5
reverted get_runnable_tasks and added guard against adding task futur…
tclose Mar 18, 2025
f6ff11f
cleaning up test failures
tclose Mar 18, 2025
e226c1c
added debugging statement into testslurm
tclose Mar 18, 2025
d23c510
dropped problematic hard-coded hashs from file hash tests
tclose Mar 18, 2025
a086db9
mark file hash tests as flaky
tclose Mar 18, 2025
302683c
formatted tutorial notebook
tclose Mar 18, 2025
bfdf152
touched up slurm ci script
tclose Mar 18, 2025
fbb13dd
fixed up error message
tclose Mar 18, 2025
ad01174
restricted testslurm to test_workflow test module
tclose Mar 18, 2025
bb545c1
formatted notebook
tclose Mar 18, 2025
c8cb96c
debugged and formatted tutorial notebooks
tclose Mar 18, 2025
3a7bbdf
relaxed slurm tests to whole package
tclose Mar 18, 2025
1b4cb46
marked test_typing_cast as flaky
tclose Mar 18, 2025
932fc3f
touched up error message for consistent matching
tclose Mar 18, 2025
1ec1933
added guards for arguments name clashes with 'function', 'executable'…
tclose Mar 18, 2025
6b1feb5
fixed handling of reserved attributes in python, shell and workflow t…
tclose Mar 18, 2025
01a66f4
hopefully fixed issue with raising errors from cf worker
tclose Mar 19, 2025
f20fc23
fixed issue with worker.run not awaited
tclose Mar 19, 2025
9d817a5
ensure completed tasks retrieve exceptions
tclose Mar 19, 2025
937e170
collect and handle exceptions from workflows much more explicitly
tclose Mar 19, 2025
e0b0e00
fixed up syntax error
tclose Mar 19, 2025
33014fa
fixed up error in exc regex match
tclose Mar 19, 2025
030b9ab
fixed up error messages to match tests
tclose Mar 19, 2025
3ea1336
use cp.load instead of cp.loads with read_bytes
tclose Mar 19, 2025
3cd6581
fixed pickle load
tclose Mar 19, 2025
7f213cd
debugging multioutput type
tclose Mar 19, 2025
12d46ba
added comprehensive ci-cd workflow to replace other tests/deploy
tclose Mar 19, 2025
8ebd43e
moved ci-cd out of workflows dir temporarily
tclose Mar 19, 2025
2bad6e1
replaced independent tests with unified test workflow
tclose Mar 19, 2025
965fd59
marked xfail for Py3.11 ubuntu test
tclose Mar 19, 2025
8281c59
uncommented passing tests from ci-cd
tclose Mar 19, 2025
4ab9408
added assertion to help debug slurm
tclose Mar 19, 2025
c4ba773
added bash scripts to help test slurm (via docker) locally
tclose Mar 19, 2025
a0c7b1e
reenabled test matrix in ci-cd
tclose Mar 19, 2025
50e8e37
added bulid dep to test-psijslurm
tclose Mar 19, 2025
e4c1f18
touched up docker test scripts
tclose Mar 19, 2025
0b6d8dd
updated plugins fixture so that psij=* disables other plugin tests (n…
tclose Mar 19, 2025
5686862
changed all imports to absolute imports
tclose Mar 19, 2025
953827a
added pydra module path to the sys path in run_pickled to avoid impor…
tclose Mar 19, 2025
60e4bf8
added arg to run-docker-slurm tests
tclose Mar 19, 2025
fd2e966
made all imports absolute in tests to avoid issues with namespace pac…
tclose Mar 19, 2025
f1b4ebc
reinstated exception catching in submitter
tclose Mar 19, 2025
5dcf577
fixed up test_shelltask to make it work with psijlocal
tclose Mar 19, 2025
2174457
fixes for psijlocal tests
tclose Mar 19, 2025
5aad70d
fix up to tests_shelltask
tclose Mar 19, 2025
37356b4
added docker scripts to help debug slurm worker
tclose Mar 20, 2025
7508bd1
restricted slurm tests to test_workflow test_submitter and test_node_…
tclose Mar 20, 2025
cde9d13
marked test_rerun_errored as flaky
tclose Mar 20, 2025
4176284
simplifying test job structure, avoiding duplicate test runs between …
tclose Mar 24, 2025
a8fdafd
added '--only-slurm' option to conftest to avoid duplicating tests in…
tclose Mar 24, 2025
6bfdb0d
reinstated separate singularity test job
tclose Mar 24, 2025
38ef69b
[skip ci] added back in test-singularity dep to deploy job
tclose Mar 24, 2025
29de260
updating pydra slurm docker
tclose Mar 24, 2025
df55125
debugging ci-cd simplification
tclose Mar 24, 2025
940b436
added pytest rootidr to ci-cd
tclose Mar 24, 2025
7b9ff71
debugging ci-cd changes
tclose Mar 24, 2025
2bade6e
cleaned up fixture conftest
tclose Mar 24, 2025
8de6ed4
renamed worker.run_async to worker.submit
tclose Mar 24, 2025
be2d496
created submitter.submit method
tclose Mar 24, 2025
d18df93
fixed infinite loop in submitting slurm jobs
tclose Mar 24, 2025
32d6976
updated expected outputs of BasicWorkflow in slurm tests
tclose Mar 24, 2025
5ae5574
drop stderr from all test comparisoins in test_singularity as it may …
tclose Mar 24, 2025
b7249d9
debugged test-slurm ci job
tclose Mar 25, 2025
ce5d18f
split psij-slurm and slurm into separate test jobs
tclose Mar 25, 2025
bcdbbf2
split out psij worker into separate ci job
tclose Mar 25, 2025
dc12f4f
renamed ci job
tclose Mar 25, 2025
9d1ca95
fixed test job deps
tclose Mar 25, 2025
6162ae8
fix up any_worker fixture
tclose Mar 25, 2025
d94a85a
[skip ci] removed stray print statements
tclose Mar 25, 2025
1dab419
redecorated test_slurm_max_jobs
tclose Mar 25, 2025
7a1d86d
added a couple of tests, removed files to increase test coverage
tclose Mar 25, 2025
3e96c58
removed references to pydra.engine.task
tclose Mar 25, 2025
ae1922e
added files arg to code-cov upload for slurm jobs
tclose Mar 25, 2025
425592c
fixing up codecov upload for slurm
tclose Mar 25, 2025
4d47d00
dropped boutiques from main repo (to be added in later as a plugin)
tclose Mar 25, 2025
7ec167d
disabling readthedocs and circleCI
tclose Mar 25, 2025
d7b8eac
removed unused helper function is_local_file
tclose Mar 25, 2025
cec22d9
upped fileformats-extras deps
tclose Mar 25, 2025
11bfd2e
debugging readthedocs build
tclose Mar 25, 2025
a2cc4d6
debugging readthedocs build again
tclose Mar 25, 2025
44041fe
renamed Task to Job and *Def to *Task
tclose Mar 26, 2025
5c2af82
renamed TaskDef to Task
tclose Mar 26, 2025
0833ea8
fixed up psij worker after renaming
tclose Mar 26, 2025
9c9410e
split base and shell compose modules into packages
tclose Mar 27, 2025
379f126
fixed up all import errors
tclose Mar 27, 2025
fe8f078
renamed list_fields to task_fields
tclose Mar 27, 2025
4f42f70
cleaned up engine.__init__ so it only contains Submitter class
tclose Mar 27, 2025
b0586e9
split up the workers module into sub-modules for each worker type
tclose Mar 27, 2025
a649dbc
split up environments module
tclose Mar 27, 2025
d897fea
fixed up errors locally
tclose Mar 28, 2025
a483660
updated version location
tclose Mar 28, 2025
82b4962
split out dask and psij workers into separate plugin packages
tclose Mar 28, 2025
8f1b031
added getstate and setsate for sge and slurm workers
tclose Mar 28, 2025
b98dd2b
fixed up test_pickle
tclose Mar 28, 2025
9d22796
removed relative import above namespace
tclose Mar 28, 2025
d7569ee
fixed load_and_run import location
tclose Mar 28, 2025
4fde3c7
moved fetch_finished into Submitter from Worker
tclose Mar 28, 2025
abd52c4
set rootdir of pytest
tclose Mar 28, 2025
3023abe
updated version import in docs
tclose Mar 28, 2025
763e7f6
renamed base_environment and base_worker to avoid clash in tests with…
tclose Mar 28, 2025
c6f556a
fixed up references to base_worker and base_environment modules
tclose Mar 28, 2025
dcc23f8
fixed up stray import
tclose Mar 28, 2025
ab5bf22
fixed up import locations in tutorials
tclose Mar 28, 2025
23e187b
fixed up import
tclose Mar 28, 2025
e9019e5
Update docs/source/tutorial/2-advanced-execution.ipynb
effigies Mar 28, 2025
4c3d993
Update docs/source/tutorial/2-advanced-execution.ipynb
effigies Mar 28, 2025
45d2877
Update docs/source/tutorial/3-troubleshooting.ipynb
effigies Mar 28, 2025
1094194
debugged script dir names
tclose Mar 29, 2025
e8d6cf0
Update docs/source/tutorial/3-troubleshooting.ipynb
tclose Mar 29, 2025
f510157
Update docs/source/tutorial/5-shell.ipynb
tclose Mar 29, 2025
9bdf784
Update docs/source/tutorial/7-canonical-form.ipynb
tclose Mar 29, 2025
a7b23f4
Update docs/source/explanation/typing.rst
tclose Mar 29, 2025
dce557e
Update docs/source/tutorial/7-canonical-form.ipynb
tclose Mar 29, 2025
94cb29a
Update pydra/compose/python.py
tclose Mar 29, 2025
e4ac56c
Update pydra/compose/python.py
tclose Mar 29, 2025
0203105
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Mar 29, 2025
2da3a9b
reverted setting of defaults in python tasks to fix test failure
tclose Mar 29, 2025
bb255b5
favour using `task_help` method instead of pprint(fields_dict(...))
tclose Mar 29, 2025
52a5187
fixed up print help and optional outputs
tclose Mar 30, 2025
16fa35a
added show_workflow utils function and use it in the tutorial
tclose Mar 30, 2025
b4687b6
made task_fields return a dict-like object
tclose Mar 30, 2025
53cc36c
dropped fields_dict in favour of task_fields
tclose Mar 30, 2025
ec79a99
fixed up task_help
tclose Mar 30, 2025
34061f4
expanded print_help to be more helpful and updated related tests
tclose Mar 31, 2025
8df2898
use full import paths for tests/utils.py files
tclose Mar 31, 2025
f16343e
split-out unittests into new sub-packages/modules
tclose Mar 31, 2025
4164e93
updated singularity tests
tclose Mar 31, 2025
eca83ad
moved run_pickled into a separate scripts dir
tclose Mar 31, 2025
9925e9c
handle partially written result files and lockfile getting deleted be…
tclose Mar 31, 2025
47e9155
renamed cache_dir -> cache_root
tclose Mar 31, 2025
1425627
fixed platformdirs.user_cache_dir ref
tclose Mar 31, 2025
4282382
made run_start_time more robust to lockfile deletion
tclose Mar 31, 2025
55f0f8c
removed comment
tclose Mar 31, 2025
c538929
renamed cache_locations to readonly_caches
tclose Mar 31, 2025
89a474a
renamed cont_dim to container_ndim
tclose Mar 31, 2025
ba08ee9
renamed additional_args to append_args
tclose Mar 31, 2025
ac19f96
renamed output_dir to cache_dir
tclose Mar 31, 2025
dbf4bb1
cleaning up other references to output_dir
tclose Mar 31, 2025
b148052
expanded glossary
tclose Mar 31, 2025
1c49336
finished glossary
tclose Mar 31, 2025
15a0fc1
added splitter and combiner to glossary
tclose Mar 31, 2025
4ae54a7
added link to glossary on index page
tclose Mar 31, 2025
bc6b0a9
changed Native.__doc__
tclose Mar 31, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
122 changes: 0 additions & 122 deletions .circleci/config.yml
View file
Open in desktop

This file was deleted.

Loading
Loading