AWS Batch: What happens after maxTransferAttempts?

[arareko]

Hi,

We're using 21.08.0-edge and have a configuration with the following parameters for AWS Batch:

aws {
    batch {
        // Retry S3 downloads if they fail, waiting 20 seconds to retry
        maxTransferAttempts = 5
        delayBetweenAttempts = 20

        // Lower I/O load to try and prevent failures with multiple jobs
        // on a single instance
        maxParallelTransfers = 5
    }
}

I'd like to know what happens after the specified number of attempts is exhausted? For our case, the pipelines continue to run even when downloads were incomplete, obviously producing inconsistent results. Does Nextflow have a configuration/option for triggering a fatal error and stopping the pipeline when these things happen?

[mribeirodantas]

I'm not fully aware of the AWS Batch plugin for Nextflow. Having said that, you could create a process to check if all files were appropriately downloaded and throw an exception if they are not. The throw will terminate the pipeline.

[delagoya]

NF uses the AWS CLI to download files via a Bash script that is created on the fly, and these options are passed to it.

I would have to dig into the code to see if the exit status is captured from that Bash script.

[arareko]

@delagoya Are these the same options that were fixed in this commit, which was added after the version of NF that we currently use?

[pditommaso]

Uploads and downloads and executed in parallel by the snippet below. If the transfer fail, the error code is ignored however the corresponding that will fail for the lack of the expected file.

nextflow/modules/nextflow/src/main/groovy/nextflow/executor/BashFunLib.groovy

Lines 84 to 113 in b93634e

	nxf_parallel() {
	IFS=\$'\\n'
	local cmd=("\$@")
	local cpus=\$(nproc 2>/dev/null || < /proc/cpuinfo grep '^process' -c)
	local max=\$(if (( cpus>$maxParallelTransfers )); then echo $maxParallelTransfers; else echo \$cpus; fi)
	local i=0
	local pid=()
	(
	set +u
	while ((i<\${#cmd[@]})); do
	local copy=()
	for x in "\${pid[@]}"; do
	[[ -e /proc/\$x ]] && copy+=(\$x)
	done
	pid=("\${copy[@]}")
	if ((\${#pid[@]}>=\$max)); then
	nxf_sleep 0.2
	else
	eval "\${cmd[\$i]}" &
	pid+=(\$!)
	((i+=1))
	fi
	done
	for p in "\${pid[@]}"; do
	wait \$p
	done
	)
	unset IFS
	}

Said that, S3 CLI approach has serious scalability issues. Recently we have introduced the Fusion file system to tackle this problem

[arareko]

@pditommaso @delagoya Thanks for the replies and the link to the Fusion file system blog post, it looks great and really worth investigating updating all of our pipelines to make use of it.

With regards to the original issue we were experiencing, we were able to fix after updating to a slightly newer version of Nextflow than the one we had (21.08.0-edge -> 21.10.6-all), which contains the commit I mentioned before and not a lot more (for now we only wanted to introduce very little change to not break other things).