On support for HPC pipelines with fine core control and MPI

[nabw]

Hi everyone,

First of all a wanted to congratulate the developers at large. This software fills that small but fundamental gap that most of us have always filled with endless and unsustainable bash scripts.

I work in HPC and would like more control with respect to the number of cores each task in a process uses, for example to study scalability. I couldn't find anything like this, so if not present maybe with some help I would gladly add this feature.

Same thing holds for singularity with MPI parallelism, where currently the script under the 'script:' portion is run as

singularity exec $script

whereas it should be run as

srun -n $TASKS singularity exec "script"

If anyone has found a workaround, please let me know.

Best,
Nicolas