Set the number of cores (cpus). Is this normal behaviour?

[Subject-13]

Expected behaviour and actual behaviour

I have written a workflow under Ubuntu, which I limit to 4 cpus, for some processes even a bit more per directive. If I use the same workflow under macOS with M1 chip, it is no longer parallelized as usual.

If I assign 8 (cpus = 8) to a process with 10 cores available on the machine, it uses only one core. If I assign 1 (cpus = 1) core to the process, ALL cores are used. With cpus = 2, 5 are actively used. With cpus = 3, 3 are used.

It seems like my available cores are divided by the value I specify in cpus (and cast an integer from that? Like 10 / 3 = 3.33 => 3 cores used). I can't find anything in the documentation about this behaviour. Is it supposed to do that under 21.10.6? With Ubuntu it works as usual mentioned in the documentation.

Steps to reproduce the problem

I simply created a channel containing 10 values and passing it to a process with the described directive (cpus), which calls a python script that just execute time.sleep(5).

Environment

• Nextflow version: 21.10.6
• Java version: 11.0.13
• Operating system: macOS M1 Pro
• Bash version: GNU bash, version 3.2.57(1)-release (arm64-apple-darwin22)

[mribeirodantas]

Hello, @Subject-13 !

I think it would be good if you can share how you're measuring CPU usage. I have seen people using different approaches to do that, and they're not always correct if I remember correctly.

[Subject-13]

Hello @mribeirodantas,

Sorry for the late respond, but sure thing. All I do is using this little nextflow-script (main.nf) which is calling a python-script. No extra config file. See code below:

main.nf

#! /usr/bin/env nextflow
nextflow.enable.dsl=2

workflow {
    ch = Channel.from(1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

    main:
    sleeper(ch)       
}

process sleeper {
    cpus 10
    tag "${name}"

    input:
    val x

    script:
    """
    sleeping.py
    """
}

sleeping.py

#! /usr/bin/env python3

import time

time.sleep(5)

Next, I call the nextflow script with nextflow main.nf in my terminal and watch how processes get completed. Output for cpus 10 looks like this:

N E X T F L O W ~ version 21.10.6
Launching main.nf [tiny_ekeblad] - revision: 9b35fe23aa
executor > local (2)
[76/5856c2] process > sleeper (null) [10%] 1 of 10

With 10 the available cores it completes each task separately in this case. With cpus 2 it completes 5 processes at once.

Edit: Typos

[mribeirodantas]

Oh, now I understand what you mean.

Let's say you have access to 8 logical cores. If you set cpus to 8 in your process, you tell Nextflow that you need 8 logical cpus to deal with every task of this process. The effect of that is the all the cpus will be busy for every task and your process will be run sequentially. All cores focused for the first element of the channel (first task), then all cores focused for the second element of the channel (second task) and so on. If you set cpus to 1, 1 will be used for the first element, but because the second also needs 1 and you have 7 left it will start right away, what we call implicit parallelization. In my machine, it started and pretty soon 80% of the 10 tasks had finished because I set cpus to 1 and I have 8 logical cpus available. Is it clear to you now? 😄

[Subject-13]

Yes, that is very clear. Thanks for that! So I define how many cores a single process call can see. This one claims all the ones it sees and blocks them for further process calls (no matter if he needs all of them). If they aren't enough cores available no more process will be started until the other cores are free. So defined from the inside out. Right?

But is there any way to define that a process can be called a maximum of 4? So I say cpus 1 for a process (containing a single-core script) with 10 available cores on the machine, but I want this process to be parallelized at most 4 times and during this time the remaining cores will be split to other processes?

[mribeirodantas]

I don't want to be pedantic but maybe it helps to understand what every word means 😄. The process is the definition, so process X is only one process, regardless of how many elements are in the channels. If you have 10 elements in the input channel that your process X expects to receive, you'll have 10 tasks (process instances), so what you define in the process definition is the resources requested for every single task. The idea in your first paragraph is correct.

As for your second paragraph, yes, there is a way. You can use a process directive called maxForks. You can find more information about this directive here. It does exactly what you asked 😄

[Subject-13]

Thank you for the more precise wording. Then everything is clear!