How to decompress a set of archives in a directory with nextflow ?

[alimatai]

Hello,

I'm new to nextflow and struggling with a simple command.

One of my processes generates a set of .tar.gz files in a single folder. I have been trying to create a process which decompresses all those archives.

My last attempt is with:

params.path = "/the/path/of/the/whole/project"

process fastgz_to_fastq {

    input:
        val targz

    script:
        """
        tar -xf $targz -C $params.run_path/demultiplexed_unzip/
        """

}

workflow {
     def ZipChannel = Channel.fromPath("$params.path/directory_of_archives/*.fastq.gz")
     fastgz_to_fastq(ZipChannel)
}

I guess various things are wrong there. Nextflow does not crash, but nothing seem to happen once the run is over.

[bentsherman]

You need to:

1. change the input targz to be a path instead of val, so that the file is staged into the task directory
2. declare an output path in your process so that it captures the output files, then use publishDir to publish those files from the work directory to the desired output directory

That way your decompress command can just be tar -xf $targz and it will extract to the task directory. Nextflow processes are designed to be self-contained, so Nextflow will manage the input and output files without you having to specify absolute paths everywhere.

[alimatai]

Thank you, it worked ! But is it possible to delete the input archive after decompression in the same process ?

params.path = "/the/path/of/the/whole/project"

process fastgz_to_fastq {

  input:
          path targz
  
  publishDir "$params.path/directory/"
  
  output:
          path "*.fastq"
  
  script:
          """
          tar -xzf $targz &&  rm $targz
          """
}

workflow {
     def ZipChannel = Channel.fromPath("$params.path/directory/*.fastq.gz")
     fastgz_to_fastq(ZipChannel)
}

[bentsherman]

You can do that, but it will break the resumability of your pipeline. So if something goes wrong and you have to resume your pipeline, you will lose all of that progress.

Consider setting cleanup = true in your config file instead (see the docs). It will delete all of those intermediate files once the workflow successfully completes.