Fix AWS Batch task polling forever when the job is missing from the DescribeJobs response#7302
Open
item084 wants to merge 1 commit into
Open
Fix AWS Batch task polling forever when the job is missing from the DescribeJobs response#7302item084 wants to merge 1 commit into
item084 wants to merge 1 commit into
Conversation
…escribeJobs response When an AWS Batch job can no longer be resolved via DescribeJobs -- the response is empty or the job is missing from it, e.g. it aged out of the Batch job history -- AwsBatchTaskHandler.checkIfCompleted() treated the null describeJob() result as 'not done yet' and polled forever, hanging the workflow indefinitely. Bound the condition with the existing executor.exitReadTimeout config option (default 270s), mirroring how GridTaskHandler already bounds the analogous condition on grid executors: latch a timestamp when the status first becomes unresolvable, reset it on any successful describe, and fail the task once the timeout elapses. Fixes nextflow-io#7301 Signed-off-by: Yuan Yao <item084@users.noreply.github.com>
✅ Deploy Preview for nextflow-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7301
Problem
When an AWS Batch job can no longer be resolved via
DescribeJobs— the response is empty, or the job is missing from it (e.g. it aged out of the Batch job history, which retains jobs for ~24 hours) —AwsBatchTaskHandler.checkIfCompleted()treats thenullfromdescribeJob()as "not done yet" and polls forever. The head process never exits and the workflow hangs indefinitely, with the only signal being DEBUG-level log lines. See #7301 for the full analysis and a self-contained local reproduction (fake Batch API viaAWS_ENDPOINT_URL_BATCH, no AWS account needed).The grid executors already bound the analogous condition (job gone from the queue with no
.exitcodefile) viaexecutor.exitReadTimeoutinGridTaskHandler.readExitStatus(). This PR applies the same pattern to AWS Batch.Changes
AwsBatchTaskHandler: whendescribeJob()returnsnullincheckIfCompleted(), latch a timestamp on the first occurrence and reset it whenever the status becomes resolvable again. Once the condition persists beyondexecutor.exitReadTimeout(same config option as grid, default270s), fail the task (exitStatus = Integer.MAX_VALUE,ProcessException) instead of polling forever — mirroring both the mechanism and the error semantics ofGridTaskHandler.config.mdxthatexecutor.exitReadTimeoutnow also applies to AWS Batch.Transient describe failures are unaffected: a single successful describe resets the window, so only a persistently unresolvable status (longer than
exitReadTimeout) trips it.Note:
checkIfRunning()has a similar unbounded path for a job that vanishes while stillSUBMITTED; this PR intentionally keeps the scope to theRUNNING-phase wedge reported in #7301, but the same latch could be extended there if desired.Tests
Three new cases in
AwsBatchTaskHandlerTest:Integer.MAX_VALUE/ProcessExceptionmentioningexitReadTimeout;Also validated end-to-end against the reproduction harness from #7301 (fake Batch API + s3proxy, no AWS account): with
executor.exitReadTimeout = '45s', the patched build fails the task withand the head exits non-zero — where the unpatched build polls the vanished job indefinitely (observed 10+ minutes with no error and no exit).