is there a way to process n number of files at a time?

[complexgenome]

Hello there,

I've 11 samples (11 tumors, 11 normals) that I'd like to process using nextflow pipeline I've put. At the moment all 11 are processed in parallel. Is there a way that can restrict the number of samples to process in parallel? For example process 4 at a time, then next 4 then remaining, in this case 3.

I have code as:
read_pairs_ch = Channel.fromFilePairs( params.raw_files).take(3)

But it only takes 3 sample, remaining are not processed. I'd like to have three-three at a time.
How do I enable processing three samples at a time?

[bentsherman]

You can use the collate operator to group the samples into groups of 4 (including the remainder group):

read_pairs_ch = Channel.fromFilePairs( params.raw_files).collate(4)

Alternatively, you can use the maxForks directive to limit the number of parallel tasks.

[complexgenome]

Thanks for your reply. When I put collate, I run into error for WARN: Input tuple does not match input set cardinality declared by process

Please see attached process, main.nf scripts along with error.

[bentsherman]

Yes, this is because your process expects a single sample, whereas collate groups multiple samples into a tuple. This is why I suggested maxForks as the better way to limit the parallelism.

[complexgenome]

Understood, thank you 🙏 so much.