Missing output file (s) expected by process on k8s pipeline while using S3

[luisovallex]

Hello everyone,
I've started to use Nextflow a couple of weeks back, I'm trying to execute a pipeline with multiples output and input files using 'k8s' as the executor.

For the architecture we are using AWS EKS for the cluster, EFS as the nfs system file and S3 to push the result files. Some processes should push files to S3 and I have add the storeDir directive in the processes that require this. After the process completes I get this error:

Missing output file (s) 'file name' expected by process 'process name':

Apart from that I'm getting an 0 exit code, that means that the process works correctly but if I check my S3 bucket, I cannot see the file there.

If I try with the publishDir directive I'm able to see the files, but as other processes require those files, we should use storeDir.

Please let me know if the problem could be because of Kubernetes and S3 or something it's missing. And also let me know if you want to check some files or the scripts that Im running.

Thanks.

Example of the processes:

process alignment {
        echo true
        tag { sample_name + ":" + lane_id }

        pod = [nodeSelector: "work=worker-r5.2xlarge"]
        cpus 4
        memory "40 GB"
        maxRetries 3

        storeDir { "${params.results}/$runid/$sample_name/align/$lane_id" }
        //publishDir { "${params.results}/$runid/$sample_name/align/$lane_id" }
        input:
                file reference
                set sample_name, sample_id, lane_id, file(read1), file(read2) from fastqs
                val k from kickoff

        output:
                set sample_name, file ("${lane_id}.Aligned.sortedByCoord.out.bam")  into alignments
                set sample_name, file ("${lane_id}.Chimeric.out.sam")  into fusion_sams
                set sample_name, file ("${lane_id}.Chimeric.out.junction")  into fusion_jxns

                set sample_name, file ("${lane_id}.Log.final.out")  into star_logs

                set sample_name, sample_id, lane_id into meta

        script:
                template "alignment.sh"
}

Environment:

Nextflow version: 21.06.0-edge
Java version: 1.8
Operating system: Linux

[bentsherman]

This issue has been addressed by the following update -- 49ff02a -- which is available in the latest edge release.