Skip to content

Fix AWS Batch task polling forever when the job is missing from the DescribeJobs response#7302

Open
item084 wants to merge 1 commit into
nextflow-io:masterfrom
item084:fix-awsbatch-lost-job-timeout
Open

Fix AWS Batch task polling forever when the job is missing from the DescribeJobs response#7302
item084 wants to merge 1 commit into
nextflow-io:masterfrom
item084:fix-awsbatch-lost-job-timeout

Conversation

@item084

@item084 item084 commented Jul 3, 2026

Copy link
Copy Markdown

Fixes #7301

Problem

When an AWS Batch job can no longer be resolved via DescribeJobs — the response is empty, or the job is missing from it (e.g. it aged out of the Batch job history, which retains jobs for ~24 hours) — AwsBatchTaskHandler.checkIfCompleted() treats the null from describeJob() as "not done yet" and polls forever. The head process never exits and the workflow hangs indefinitely, with the only signal being DEBUG-level log lines. See #7301 for the full analysis and a self-contained local reproduction (fake Batch API via AWS_ENDPOINT_URL_BATCH, no AWS account needed).

The grid executors already bound the analogous condition (job gone from the queue with no .exitcode file) via executor.exitReadTimeout in GridTaskHandler.readExitStatus(). This PR applies the same pattern to AWS Batch.

Changes

  • AwsBatchTaskHandler: when describeJob() returns null in checkIfCompleted(), latch a timestamp on the first occurrence and reset it whenever the status becomes resolvable again. Once the condition persists beyond executor.exitReadTimeout (same config option as grid, default 270s), fail the task (exitStatus = Integer.MAX_VALUE, ProcessException) instead of polling forever — mirroring both the mechanism and the error semantics of GridTaskHandler.
  • Docs: note in config.mdx that executor.exitReadTimeout now also applies to AWS Batch.

Transient describe failures are unaffected: a single successful describe resets the window, so only a persistently unresolvable status (longer than exitReadTimeout) trips it.

Note: checkIfRunning() has a similar unbounded path for a job that vanishes while still SUBMITTED; this PR intentionally keeps the scope to the RUNNING-phase wedge reported in #7301, but the same latch could be extended there if desired.

Tests

Three new cases in AwsBatchTaskHandlerTest:

  • unresolvable status within the timeout → task keeps waiting (no failure);
  • unresolvable status beyond the timeout → task fails with Integer.MAX_VALUE / ProcessException mentioning exitReadTimeout;
  • status becomes resolvable again → the unresolved window resets.

Also validated end-to-end against the reproduction harness from #7301 (fake Batch API + s3proxy, no AWS account): with executor.exitReadTimeout = '45s', the patched build fails the task with

Task 'LOST_TASK' status could not be resolved -- the job was missing from the DescribeJobs
response for longer than `exitReadTimeout` (45000 ms). Likely it has been terminated by the
external system

and the head exits non-zero — where the unpatched build polls the vanished job indefinitely (observed 10+ minutes with no error and no exit).

…escribeJobs response

When an AWS Batch job can no longer be resolved via DescribeJobs -- the
response is empty or the job is missing from it, e.g. it aged out of the
Batch job history -- AwsBatchTaskHandler.checkIfCompleted() treated the
null describeJob() result as 'not done yet' and polled forever, hanging
the workflow indefinitely.

Bound the condition with the existing executor.exitReadTimeout config
option (default 270s), mirroring how GridTaskHandler already bounds the
analogous condition on grid executors: latch a timestamp when the status
first becomes unresolvable, reset it on any successful describe, and fail
the task once the timeout elapses.

Fixes nextflow-io#7301

Signed-off-by: Yuan Yao <item084@users.noreply.github.com>
@item084 item084 requested a review from a team as a code owner July 3, 2026 21:06
@netlify

netlify Bot commented Jul 3, 2026

Copy link
Copy Markdown

Deploy Preview for nextflow-docs ready!

Name Link
🔨 Latest commit 293eaa8
🔍 Latest deploy log https://app.netlify.com/projects/nextflow-docs/deploys/6a4824425fd8b80008f61004
😎 Deploy Preview https://deploy-preview-7302--nextflow-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AWS Batch: task polls forever when the job is missing from the DescribeJobs response — no bound, unlike grid's exitReadTimeout

1 participant