Publishing large file (1.9TB) after process takes very long (6 days) with publish mode copy.

[kobelavaerts]

Nextflow information

nextflow version 21.10.6.5660
Executor: Local.
Run with docker.
System used:

• OS: ubuntu 20.04 lts
• CPU: 16 threads (for the specific process)
• RAM: 64GB (for the specific process)
• network interface: 10Gb/s

Problem

We have a process in our pipeline that creates a tar archive. Nextflow takes a very long time (6 days) to publish large tar file (1.9TB) when using the publishDir directive with publish mode: copy.
The nextflow command terminated with the following warning message:

WARN: Exiting before FileTransfer thread pool complete -- Some files maybe lost

At this point, we didn’t get our command line prompt back and the file transfer was still ongoing.

Questions

Which command and parameters are used by Nextflow to copy the files when specifying copy option in publishDir directive?

From what we saw of the source code and the .command.run file we assume it’s just a cp -fRL command on the host system.
When manually copying the same source file to the same destination, the copying is done in 3 hours, when it took nextflow several days. What would be the reason for this?

Extra information

Cp from inside NextFlow (cp -fRL ?) LocalSource (work directory) remote share (7200RPM HDD)
Bandwith Out +/- 45 Mb/s

Cp -fRL LocalSource (work directory) RemoteShare (7200RPM HDD)
Bandwith Out +/- 1.5Gb/s

If you have any questions or need more information from the log files, please ask.

[kobelavaerts]

Update

I did some more digging into the source code and it seems the earlier mentioned cp -fRL is only used to stage input files and not to publish the output files.
For publishing the output files, it seems like the java FileSystemProvide class is used to copy the output files when using

publishDir "output", mode: "copy"

Since I don't know how to test / improve copying speed in this case, I decided to create a workaround myself.

workaround solution

First, let me show the publishDir directive for the process in the main script.

process wrapdemux {
    label "process_medium"

    publishDir "${params.tosenddir}", pattern: "*.{tar,txt}", saveAs: {filename -> params.splitprojectsarchive && params.hassampleproject? "$project_name/$filename": filename}, mode: "copy"
    publishDir "${params.logdir}/${task.process}/${task.hash}", pattern: ".*", mode: "copy"


    input:
    tuple val(project_name), val(samples), path(fastq_files)

    output:
    tuple val(project_name), val(samples), path("*.{tar,txt}")
    path(".*")

    script:
    """
    # code to generate .tar and .txt files
    """
}

I decided to mimic the publishDir functionality by making use of an afterScript block where I create cp -fRL commands for copying the files to their output folder instead of using the java API.

process {
    withName: wrapdemux {
        afterScript = {
            if(params.hassampleproject && params.splitprojectsarchive) {
                copylocation = params.tosenddir + "/" +project_name
            } else {
                copylocation = params.tosenddir
            }
            """
            mkdir -p ${copylocation}
            find . -name "*.txt" -exec cp -fRL '{}' ${copylocation} \\;
            find . -name "*.tar" -exec cp -fRL '{}' ${copylocation} \\;
            """
        }
    }
}

This afterScript block works when using the local executer, even when run with docker. The problem now is that before executing the afterScript block, the process tries to publish the files using the publishDir directive as well.

We don't want to completely remove the publishDir directive however, because we want to keep using it's functionality for when we run the pipeline in the cloud (azure batch).

To solve this, we make use of how nextflow prioritizes publishDir directives. When you only define a publishDir directive in the main script, Nextflow uses this to publish the files. If you also have a publishDir directive defined in the nextflow.config file however (under the process directive), the publishDir directive of the main script gets overwritten and the one from the nextflow.config file is used.

Since we publish the .command.* log files for each process, we can use this to add a publishDir directive in the .nextflow.config file, which will overwrite the directive of the main script.

process {
    withName: wrapdemux {
        afterScript = {
            if(params.hassampleproject && params.splitprojectsarchive) {
                copylocation = params.tosenddir + "/" +project_name
            } else {
                copylocation = params.tosenddir
            }
            """
            mkdir -p ${copylocation}
            find . -name "*.txt" -exec cp -fRL '{}' ${copylocation} \\;
            find . -name "*.tar" -exec cp -fRL '{}' ${copylocation} \\;
            """
        }
        publishDir = [
            path: {"${params.logdir}/${task.process}/${task.hash}"},
            pattern: ".*",
            mode: "copy"
        ]
    }
}

Now, I added the above code in a separate config file named send-tar.config. I can now include this config file in the user profiles that use the local executor. For example

profiles {
    local_server {
        includeConfig "conf/send-tar.config"
    }
}

I hope this can be helpful if someone runs into a similar problem.