Can't using curl/wget to fetch file from ENA in a process

[DongzeHE]

Bug report

I have a list of paired-end read FASTQ URLs stored in a CSV file. All of them are from ENA. The Read1 and Read2 FASTQ files are passed to my process using path(URL) as two paths.

Also, I tried to use wget and curl to download those files within the process. However, I got an error curl: (56) Recv failure: Connection reset by peer.
The command was

curl -o SRR8513799_1.fastq.gz 'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR851/009/SRR8513799/SRR8513799_1.fastq.gz'

If possible, would you mind giving me some advice on how to fetch files from ENA within a process?

Expected behavior and actual behavior

Be able to fetch file from ENA.

Steps to reproduce the problem

process fetch {

"""


curl -o SRR8513799_1.fastq.gz 'ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR851/009/SRR8513799/SRR8513799_1.fastq.gz'

"""

}

Program output

curl: (56) Recv failure: Connection reset by peer.

Environment

• Nextflow version: 23.10.0
• Java version: openjdk 20-internal 2023-03-21
• Operating system: Linux
• Bash version: GNU bash, version 4.4.20(1)-release (x86_64-redhat-linux-gnu)

Additional context

(Add any other context about the problem here)

[adamrtalbot]

Given this is a question about how to use Nextflow rather than a bug or feature, I would recommend posting on https://community.seqera.io/c/nextflow/5 instead of here. When you do, be sure to share the code and relevant logs to make it easier for people to work out what is going on.

For what it's worth, this looks like an error with CURL or with the connection between you and the SRA.

[pditommaso]

Indeed, this looks like more a problem with curl. There should be a retry option, https://everything.curl.dev/usingcurl/downloads/retry