Adding new choice to --on-error#1974

Open

AlexTate wants to merge 30 commits intocommon-workflow-language:mainfrom

AlexTate:on-error-abort

Commits on Nov 13, 2024

Adding inputs for new "kill" mode of the on-error parameter

AlexTate
authored and
mr-c
committed
Adding kill switch trigger. If a job's processStatus != "success" and runtimeContext.on_error = "kill", then the switch is activated. WorkflowKillSwitch is raised so it can be handled at the workfl…

AlexTate
authored and
mr-c
committed
Actively running jobs respond to the kill switch by checking the switch's status in the monitor function. The monitor function, up to this point, has been for gathering memory usage statistics via …

AlexTate
authored and
mr-c
committed
When the WorkflowKillSwitch exception reaches the TaskQueue, try to prevent pending tasks from starting by simply draining the queue. This is a very loose policy, but since kill switch response is …

AlexTate
authored and
mr-c
committed
Propagation of the WorkflowKillSwitch exception stops once it reaches an executor. The workflow_eval_lock release had to be moved to the finally block in MultithreadedJobExecutor.run_jobs(). Otherw…

AlexTate
authored and
mr-c
committed
Don't create runtime_context.kill_switch by default
mr-c
committed
This commit brings kill switch initialization and monitoring to the TaskQueue. This helps to better synchronize the kill switch event and avoid adding/executing tasks after the switch has been set.

AlexTate
authored and
mr-c
committed
JobBase._execute() previously skipped some important post-job actions when exiting due to kill switch. Those actions have been placed under a `finally` block so that they are executed by both the "…

AlexTate
authored and
mr-c
committed
Adding a unit test. Two outcomes are measured:

AlexTate
authored and
mr-c
committed
Formatting corrections provided by `make cleanup` which are relevant to this issue. Other changes were offered by the tool, but they are outside the scope of this issue.

AlexTate
authored and
mr-c
committed
cleanups
mr-c
committed
DO NOT MERGE, let all tests run
mr-c
committed
Removing irrelevant debug line from test body

AlexTate
authored and
mr-c
committed
Removing CPU resource constraints for the test. My intention is to have MultithreadedJobExecutor ignore allocated resources when deciding whether to run the next parallel job. The steps in this wor…

AlexTate
authored and
mr-c
committed
Using a different approach to remove the "one job per core" resource constraint. The current ResourceRequirement implementation doesn't allow {coresMin: 0}. However, this can still be achieved with…

AlexTate
authored and
mr-c
committed
Adding suggested changes to the --on-error help string per @mr-c. Formatting compliance updates in test_parallel.py

AlexTate
authored and
mr-c
committed
Missed running `make mypy` before pushing last commit. Opting to disable type checking on selectResources() because it's just an implementation detail for the test

AlexTate
authored and
mr-c
committed
don't cancel jobs; typecheck the test just in case
mr-c
committed
test_on_error_kill: allow for some longer runtime due to busy systems
mr-c
committed
Fixing ReceiveScatterOutput to allow outputs to be collected from successful steps when 1) these steps are upstream from a scattered subworkflow, 2) the workflow kill switch is activated by one of …

AlexTate
authored and
mr-c
committed
Simplifying value checks for RuntimeContext.on_error in parallel_steps() and WorkflowJob.job().

AlexTate
authored and
mr-c
committed
Adding __repr__ to WorkflowStep, WorkflowJob, and WorkflowJobStep classes. This can be VERY helpful while debugging, particularly when unraveling callback chains.

AlexTate
authored and
mr-c
committed
It is critical for WorkflowJob.processStatus to update in WorkflowJob.receive_output(). Otherwise, MultithreadedJobExecutor.run_jobs() will likely stop iterating over the topmost WorkflowJob.job() …

AlexTate
authored and
mr-c
committed
Simplifying the approach in on-error_kill.cwl. It now activates the kill switch via a ToolTimeLimit requirement. It also uses a much longer timeout which will hopefully be sufficient for the CI ser…

AlexTate
authored and
mr-c
committed
Reformatting and type hint updates

AlexTate
authored and
mr-c
committed
Formatting mistake fix

AlexTate
authored and
mr-c
committed
Adding instance ID to repr strings, improving readability of WorkflowStep's repr string, and adding docstrings.

AlexTate
authored and
mr-c
committed
Changing TaskQueue constructor to accept the kill_switch threading.Event as an argument rather than an entire RuntimeContext, per @mr-c

AlexTate
authored and
mr-c
committed
codecov: don't fail CI if there is an error
mr-c
committed

Commits on Nov 20, 2024

Merge branch 'main' into on-error-abort
mr-c
authored

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Adding new choice to --on-error#1974

Adding new choice to --on-error#1974
AlexTate wants to merge 30 commits intocommon-workflow-language:mainfrom
AlexTate:on-error-abort

Commits on Nov 13, 2024

Adding inputs for new "kill" mode of the on-error parameter

Adding kill switch trigger. If a job's processStatus != "success" and runtimeContext.on_error = "kill", then the switch is activated. WorkflowKillSwitch is raised so it can be handled at the workfl…

Actively running jobs respond to the kill switch by checking the switch's status in the monitor function. The monitor function, up to this point, has been for gathering memory usage statistics via …

When the WorkflowKillSwitch exception reaches the TaskQueue, try to prevent pending tasks from starting by simply draining the queue. This is a very loose policy, but since kill switch response is …

Propagation of the WorkflowKillSwitch exception stops once it reaches an executor. The workflow_eval_lock release had to be moved to the finally block in MultithreadedJobExecutor.run_jobs(). Otherw…

Don't create runtime_context.kill_switch by default

This commit brings kill switch initialization and monitoring to the TaskQueue. This helps to better synchronize the kill switch event and avoid adding/executing tasks after the switch has been set.

JobBase._execute() previously skipped some important post-job actions when exiting due to kill switch. Those actions have been placed under a `finally` block so that they are executed by both the "…

Adding a unit test. Two outcomes are measured:

Formatting corrections provided by `make cleanup` which are relevant to this issue. Other changes were offered by the tool, but they are outside the scope of this issue.

cleanups

DO NOT MERGE, let all tests run

Removing irrelevant debug line from test body

Removing CPU resource constraints for the test. My intention is to have MultithreadedJobExecutor ignore allocated resources when deciding whether to run the next parallel job. The steps in this wor…

Using a different approach to remove the "one job per core" resource constraint. The current ResourceRequirement implementation doesn't allow {coresMin: 0}. However, this can still be achieved with…

Adding suggested changes to the --on-error help string per @mr-c. Formatting compliance updates in test_parallel.py

Missed running `make mypy` before pushing last commit. Opting to disable type checking on selectResources() because it's just an implementation detail for the test

don't cancel jobs; typecheck the test just in case

test_on_error_kill: allow for some longer runtime due to busy systems

Fixing ReceiveScatterOutput to allow outputs to be collected from successful steps when 1) these steps are upstream from a scattered subworkflow, 2) the workflow kill switch is activated by one of …

Simplifying value checks for RuntimeContext.on_error in parallel_steps() and WorkflowJob.job().

Adding repr to WorkflowStep, WorkflowJob, and WorkflowJobStep classes. This can be VERY helpful while debugging, particularly when unraveling callback chains.

It is critical for WorkflowJob.processStatus to update in WorkflowJob.receive_output(). Otherwise, MultithreadedJobExecutor.run_jobs() will likely stop iterating over the topmost WorkflowJob.job() …

Simplifying the approach in on-error_kill.cwl. It now activates the kill switch via a ToolTimeLimit requirement. It also uses a much longer timeout which will hopefully be sufficient for the CI ser…

Reformatting and type hint updates

Formatting mistake fix

Adding instance ID to repr strings, improving readability of WorkflowStep's repr string, and adding docstrings.

Changing TaskQueue constructor to accept the kill_switch threading.Event as an argument rather than an entire RuntimeContext, per @mr-c

codecov: don't fail CI if there is an error

Commits on Nov 20, 2024

Merge branch 'main' into on-error-abort

Uh oh!

Adding new choice to --on-error#1974AlexTate wants to merge 30 commits intocommon-workflow-language:maincommon-workflow-language/cwltool:mainfrom AlexTate:on-error-abortAlexTate/cwltool:on-error-abortCopy head branch name to clipboard

Commits

Commits on Nov 13, 2024

Commits on Nov 20, 2024

Adding new choice to --on-error#1974
AlexTate wants to merge 30 commits intocommon-workflow-language:mainfrom
AlexTate:on-error-abort