Nextflow is attempting to copy S3 objects in place?

[amanda-mp]

Hi there, I have a pipeline that was developed and successfully tested locally which I'm now trying to run with the awsbatch executor. I followed all steps to the letter, including correct IAM permissions. When I run the pipeline, the first process fails with Essential container in task exited.

The process:

process runAdapterTrimming {
    container 'ourdocker/ngs:bbtools'

    publishDir "${params.s3dir}/fastq/trimmed", mode: 'copy'

    input:
    tuple idPatient, status, file(R1), file(R2) from trimming_ch
    path params.datadir
    file params.adapters
    val params.ktrim

    output:
    tuple idPatient, status, "${idPatient}_${status}_R1.fastq", "${idPatient}_${status}_R2.fastq" into (pairsFastQCTrimmed, pairsAligning)

    script:
    template 'trimbbduk.sh'
}

The error:

copy failed: s3://ourbucket/data/adapters.fa to s3://ourbucket/data/adapters.fa 
An error occurred (InvalidRequest) when calling the CopyObject operation: This copy request is illegal 
because it is trying to copy an object to itself without changing the object's metadata, storage class, 
website redirect location or encryption attributes.

I know this error is occurring due to having path params.datadir as an input (So the Docker container can find the params.adapters file, which is inside the datadir). But what I don't understand is why Nextflow is trying to copy the contents of the directory to the same directory? Shouldn't a symlink to that directory be created inside the /work directory? When I check the work directory on S3 in contains the command.* files, but none of the input files for the process.

nextflow info
  Version: 21.10.6 build 5660
  Created: 21-12-2021 16:55 UTC
  System: Linux 4.14.256-197.484.amzn2.x86_64
  Runtime: Groovy 3.0.9 on OpenJDK 64-Bit Server VM 1.8.0_312-b07
  Encoding: UTF-8 (UTF-8)

Any help or advice would be appreciated, thank you!

[pditommaso]

    path params.datadir
    file params.adapters
    val params.ktrim

This it's not how inputs are expected to be defined. The input name should be either an identified or a string value representing the name used to stage the file. The actual input value will be provided when invoking the process as an argument

[amanda-mp]

Ah, I think I misunderstood the docs and as this worked locally I stuck with it. So in this case, would this be the correct input definition?

Channel
    .of(params.ktrim)
    .set { ktrimChannel }
Channel
    .fromPath(params.adapters)
    .set { adaptChannel }
...
...
input:
tuple file(adapters), ktrim from adaptChannel.combine(ktrimChannel)

I'll try that for now, thank you for your feedback!

[pditommaso]

happy for that