Allow to reuse hash after process fail/cancellation

[xpastor]

Hi,

I was looking for a way to reuse a hash in case a process fails or is cancelled (i.e. due to resources limitations). I find it convenient for tools like cellranger, where after a cancellation it could resume the process where it was left, saving a lot of time in tasks like the aggregation of multiple samples, where it can take several days and after cancellation it will restart from scratch.

I could not find anything in the documentation that allowed something like this. I tried the different options in the cache directive but it did not work. I was wondering if it would be possible to define additional values to the directive to allow such behavior, or maybe even a new directive, if possible at all.

Thanks a lot.

[pditommaso]

The task ID (hash) is automatically computed given the task inputs and compute. If there's no change, the same ID is assigned. There's no way to assign or reuse a specific one

[xpastor]

Hi Paolo,

Unfortunately this is not what I am experiencing. After manually cancelling the process, resuming the workflow without any changes neither in the inputs nor in the code of the process, causes the task to be run in a different directory. I use dolphinnext to develop and submit workflows. Not sure if this could be part of the problem.

I will continue researching on this to try to find a solution.

Best,
Xavier

[bobamess]

I think that a sub-workDir is only created with exactly the same path on resume (without any edits to the process) if you rename or delete the original sub-workDir

[xpastor]

I see. That's a pity because actually I want to reuse the contents of the failed process, since cellranger can resume the process where it was left. If I cannot find a native way in nextflow I will implement some strategy that runs the task outside nextflow's workDir.

Thanks for clarifying this :)