Fix S3 delete throwing DirectoryNotEmptyException due to eventual consistency

[pditommaso]

Summary

• Remove the DirectoryNotEmptyException check from S3FileSystemProvider.delete() — S3 directories are virtual (marker objects or implied key prefixes), so enforcing POSIX-like emptiness semantics is unreliable due to eventual consistency and unnecessary because deleting a directory marker does not affect its children.

Problem

When PublishDir copies a directory to S3 and the target already exists, Nextflow's retry logic deletes the existing directory first via FileHelper.deletePath() → Files.walkFileTree(). The walk deletes all child objects, then postVisitDirectory calls Files.delete(dir) on the parent.

However, S3FileSystemProvider.delete() lists the directory's children to enforce a DirectoryNotEmptyException check. Due to S3 eventual consistency, just-deleted objects may still appear in the listing, causing the check to fail. The exception propagates through Failsafe retry, which eventually hangs — blocking ThreadPerTaskExecutor.awaitTermination() on the main thread indefinitely.

Observed in production: a pipeline with 2835 succeeded tasks and 1 failed task hung for 35+ minutes waiting for file transfers to complete, with 2 PublishDir virtual threads stuck trying to copy a quast output directory to S3.

Fix

Remove the directory emptiness check from S3FileSystemProvider.delete(). On S3, "directories" are either marker objects (key/) or implied by child key prefixes — there is no real directory to protect. The emptiness check was both unreliable (eventual consistency) and semantically wrong for object storage.

Test plan

• Added integration test 'should delete a non-empty directory on S3 without throwing' in AwsS3NioTest — confirmed RED (fails before fix), GREEN (passes after fix)

🤖 Generated with Claude Code

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	6b9c45f
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/6994798d823a3d0008c0395e