A few clean up questions

[rongcuid]

nextflow clean only takes run ID or run names. Neither means anything to me:

f0a-bb75-4d32-93ad-af01ce635724 nextflow run scripts/analysis.nf -resume
2024-02-08 18:18:44     1h 7m 41s       compassionate_picasso   ERR     b119978e80      e46b5f0a-bb75-4d32-93ad-af01ce635724 nextflow run scripts/analysis.nf -resume
2024-02-08 19:56:50     -               pensive_kirch           -       e2f5313f0e      e46b5f0a-bb75-4d32-93ad-af01ce635724 nextflow run scripts/analysis.nf -resume
2024-02-09 15:35:23     30.5s           compassionate_payne     ERR     1651ac7d33      e46b5f0a-bb75-4d32-93ad-af01ce635724 nextflow run scripts/analysis.nf -resume
2024-02-09 15:36:04     30.4s           curious_perlman         ERR     1651ac7d33      e46b5f0a-bb75-4d32-93ad-af01ce635724 nextflow run scripts/analysis.nf -resume
2024-02-09 15:36:55     35.1s           serene_fourier          OK      1651ac7d33      e46b5f0a-bb75-4d32-93ad-af01ce635724 nextflow run scripts/analysis.nf -resume

There's just no way I can identify which one did what. Is there a way to do the following?

• Clean up the output of one process so I can rerun it (perhaps due to updating the program to run it)
• Clean up all stale runs
• Clean up error runs

[bentsherman]

• If you modify the process definition, it will invalidate the cache and cause the process to be re-executed
• Depends on how you define "stale"
• You should be able to write a bash script to (1) query the list of failed runs using nextflow log and some filtering and (2) delete those runs using nextflow clean

[rongcuid]

I found that if I execute a python script with python3 script.py, nextflow doesn't seem to recognize that the script is being modified. Is that the case?

For staleness, say I run nextflow once. Then, I updated the scripts/process definition, etc., and run again. Any cached data from first run is now stale. I would like to remove those as they will take up disk space.

[bentsherman]

I think what you have to do is put your scripts in the bin directory (following these best practices) and then the script will be included in the task hash. That way if you change the script but not the process definition, it will still be re-executed.

As for staleness, I'm not sure if it can be automated because it might be subjective whether a run is stale to you. I guess you could run nextflow log to browse your previous runs (it also shows the command you ran) and then use nextflow clean to delete the runs that are stale to you. Maybe if you do that manually for a while you will pick up on a pattern that you can use to automate the selection of stale runs

[rongcuid]

I will probably restructure it. However, nextflow does track executables I directly specify the path to.

I'll see what info nextflow log can provide.