cannot copy file from s3 bucket

[lpembleton]

I am receiving the following error when running my nextflow pipeline on aws.

cp: cannot stat 's3://bucket-name/*_sample-1_*fastq.gz': No such file or directory

relevant section in the .nf file:

params.index = 'samples.csv'

Channel
    .fromPath(params.index)
    .splitCsv(header:true)
    .map{ row-> tuple(row.ID, row.Sample, row.SeqCode, row.LibraryLayout, row.SeqType, row.LibraryType, row.Platform, row.Model, row.PreTrimmed, row.ReadPath) }
    .set { samples_ch }


/*
 * Step 1. Source raw fastq gzip seq reads and quality & adapter trim
 */
process preprocess {

    input:
    set ID, Sample, SeqCode, LibraryLayout, SeqType, LibraryType, Platform, Model, PreTrimmed, ReadPath from samples_ch

    output:
    tuple val(Sample), val(SeqCode), val(SeqType), val(LibraryLayout), val(LibraryType), file("*-pass.fq") into trimmed_reads_ch
    file "*_fastqc.{zip,html}"  into fastqc_results

    script:
    """

    cp ${ReadPath}/*_${SeqCode}_*fastq.gz . 
    gunzip *fastq.gz

    reads=(\$(ls *fastq))
    fastqc \${reads[@]}

    quadtrim -m 3 -p 5 -l 35 -q 20 \${reads[@]}

    reads=(\$(ls *pass.fq))
    fastqc \${reads[@]} 

    """
}

This process works locally when ${ReadPath} is a local dir. My assumption was that nextflow had built in capability for s3 paths and I wouldn't need to change anything in the process code.

Any suggestions would be greatly appreciated.

[lpembleton]

I have determined the issue. As the s3 path is passed through to the process from a channel using splitCsv it is not seen as a true 's3' path and therefore internally nextflow does not invoke the aws cli to access the file. If I add a column containing the fastq file name into samples.csv and define the parameter params.fastq_dir='s3://bucket/data-dir/' at the start of the nextflow script and then within the process add the file name to fastq_dir I can access the relevant file, for example:

params.index = 'samples.csv'
params.fastq_dir1='s3://bucket/data-dir/'

Channel
    .fromPath(params.index)
    .splitCsv(header:true)
    .map{ row-> tuple(row.ID, row.Sample, row.SeqCode, row.LibraryLayout, row.SeqType, row.LibraryType, row.Platform, row.Model, row.PreTrimmed, row.Read1, row.ReadPath) }
    .set { samples_ch }


/*
 * Step 1. Source raw fastq gzip seq reads and quality & adapter trim
 */
process preprocess {

    input:
    set ID, Sample, SeqCode, LibraryLayout, SeqType, LibraryType, Platform, Model, PreTrimmed, Read1, ReadPath from samples_ch
    path fastq_dir1 from params.fastq_dir1

    output:
    tuple val(Sample), val(SeqCode), val(SeqType), val(LibraryLayout), val(LibraryType), file("*-pass.fq") into trimmed_reads_ch
    file "*-pass.fq" into trimmed_reads_ch

    script:
    """

    cp ${fastq_dir1}/${Read1} .

    gunzip ${Read1}

    reads=(\$(ls *fastq))

    echo \${reads[@]}

    quadtrim -m 3 -p 5 -l 35 -q 20 \${reads[@]}

    """
}

However, it would be nice to be able to source fastq files from multiple different s3 buckets defined by the sample.csv file rather than being restricted to the s3 bucket set by params.fastq_dir1='s3://bucket/data-dir/'. Is there a workaround for this? Thanks.

[mattdmem]

I am seeing the same behaviour

version 21.10.6 build 5660

Can we change this to a bug?

[ScottNortonPhD]

I am observing the same behavior on 22.10.4 build 5836.

[ScottNortonPhD]

It appears the problem is related to the string representation of the path, which drops the s3:/ prefix. Double-wrapping an S3 URI in file or path results in the observed incorrect behavior.