How to implement a "run if file not existing in variable publishDir" process without passing duplicate arguments

[VasLem]

Hi all, I am facing a problem understanding what is the best way to have a process that:

1. If the output file, which is saved in a variable directory, exists, then soft-link that output file and return it to the output
2. If the file does not exist, perform the operation.
   I need this script to run in google cloud, so passing the variable directory only as val data type will, but if I pass it as path, only the parent directory name will be kept.

I have ended up using this minimum reproducible example, which I think is quit ugly, any suggestions are welcome:

process runIfNotExists {
input:
 path input
 val out_dir1
 path out_dir2 // exactly the same as out_dir1

publishDir "$out_dir1"

output:
 path "output", includeInputs: false

script:
"""
if [ -f $out_dir2/output ]
then
 ln -sf $out_dir2/output output
else
 touch output
fi
"""
}

[bentsherman]

This is something that we intend to address under the hood, see #3849 and #4729. These features are in development but they should allow you to run a workflow, Nextflow will automatically delete intermediate outputs, and on a resume, Nextflow will skip the task as long as the published outputs are still present.

[VasLem]

In a real case scenario, there is a folder with samples, that are to be processed. The process would skip the already processed samples, if the output file exists. How to approach this scenario within Nextflow? I am not asking for a caching mechanism for Nextflow, more like how to "correctly" define the input, output and publishDir parameters. To make it even clearer, I am asking for a way to ensure that when a step is skipped (similar to using the When clause), the next steps downstream are not affected.

[VasLem]

An easy way out (but not the prettiest) would be to be able to expose the true supplied path front the parent workflow to the process script somehow, instead of the transformed one, when defining a "path" input variable, but I don't think this is currently feasible, any "x/y/z" argument provided by the workflow is being seen as "z" directory inside the process.

[bentsherman]

I guess the way I would do it is to filter the list of sample ids based on whether the corresponding output is present in the published directory. Because these results would be from a previous run, checking the publish directory would be safe. Then you only forward the remaining sample ids for downstream processing.

[VasLem]

Although indeed this solves the problem for a single process, it is not scalable to multiple ones. In case I want to implement that same logic all over my pipeline, I would have to have repetitive code. If I extracted the logic into a process, I would not be able to call the same process twice within the workflow, by design, unless I keep renaming it to match the process tested. I don't know, don't you agree it is not straightforward and quite ugly as well? Having a workflow, dynamic enough to allow insertion of new processes and the processing of new samples as they arrive, or old ones if not yet completely processed, is that something that could be done with the caching upgrades you are working with? I guess yes, however the disk requirements overhead to save the intermediate results would be painful, particularly in a gcloud setting...

[bentsherman]

This is a typical trade-off with caching. The more fine-grained your caching, the more glue logic you must add to your code. It is a trade-off that you must navigate based on your needs

Nextflow can cache things within its own little world of past executions, but it doesn't know anything about the outside world