Ability to run parallel process in a specific order

[jmlivingstone]

Inputs within a process should be able to run in the ordered they are received. Currently documentation states, 'The channel guarantees that items are delivered in the same order as they have been sent - but - since the process is executed in a parallel manner, there is no guarantee that they are processed in the same order as they are received.'

It would be beneficial to be able to process inputs in the order they are received so that inputs that will take longer can be started before shorter inputs. This will have an impact on the overall run time of the pipeline.

Usage scenario

This would be beneficiialy when running a process in parallel per chromosome. Larger chromosomes (ie 1-4) will take longer than smaller chromosomes (ie 20-22). You would want to be able to give an ordered list to the process of which chromosomes (inputs) to run first. If there are only enough resources to run x number of inputs in parallel than you would want to start the longer jobs first.

[bentsherman]

HI @jmlivingstone , the scheduling of tasks is generally handled by the underlying executor -- SLURM, AWS Batch, etc. Especially in the world of HPC schedulers there are all sorts of scheduling algorithms that try to "fit" jobs in the most efficient way.

In your example, you might set the time directive dynamically based on the chromosome (longer wall time for 1-4, shorter walltime for 20-22, etc). An HPC scheduler will use the requested walltime (among other factors such as cpus, memory) to schedule jobs. Just make sure your queueSize is sufficiently large so that Nextflow submits all the jobs upfront, leaving the scheduler to decide how to place them.