Azure batch : Tasks == Cores

[ossonts]

nextflow/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy

Line 516 in 904c940

throw new IllegalStateException("Azure Batch pool '${pool.id()}' slots per node does not match the VM num cores (slots: ${pool.taskSlotsPerNode()}, cores: ${spec.vmType.numberOfCores})")

Anyone know why this is enforced? Surely you would not want the tasks to be the same as the core count?

[pditommaso]

Because nextflow assumes 1 cpus == 1 slot to determine the slots requirement for each task

nextflow/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy

Lines 362 to 372 in 904c940

	final slots = computeSlots(task, pool)
	log.trace "[AZURE BATCH] Submitting task: $taskId, cpus=${task.config.getCpus()}, mem=${task.config.getMemory()?:'-'}, slots: $slots"
	return new TaskAddParameter()
	.withId(taskId)
	.withUserIdentity(userIdentity(pool.opts.privileged, pool.opts.runAs))
	.withContainerSettings(containerOpts)
	.withCommandLine("sh -c 'bash ${TaskRun.CMD_RUN} 2>&1 | tee ${TaskRun.CMD_LOG}'")
	.withResourceFiles(resourceFileUrls(task,sas))
	.withOutputFiles(outputFileUrls(task, sas))
	.withRequiredSlots(slots)

[ossonts]

Thanks, I am hoping to alter this, as I don't want to use all cores, say 10 of 12 the cores used for tasks -- are then implications to changing this?

[pditommaso]

You can specify in the task (or in the config file) how many cpus you want to use using the cpus directive. You don't to hack in the code.

[ossonts]

yes but if I have a machine with 120 cores how do I specify that I only want 100 of the cores to be running my tasks?

[pditommaso]

Nextflow is designed to request resources per task level. If you are planning to cap the amount of cpus per node is not possible.

Instead, if your requirement is to cap the top amount of cpus used by a workflow execution you can achieve it using the configuration setting executor.queueSize.

For example, having a task using 2 cpus, you can set executor.queueSize = 50.

[ossonts]

Sorry, I am not sure I follow. I have 10 machines with 120 cores (total of 1200 cores) -- Lets say all my tasks are 1 core each. In batch I make my own pool and limit the slots per node to 100 (leaving 20 cores for other stuff). So I can run 1000 jobs in parallel.
Is it not possible to read the slots per node (100) X nodes (10) and tell nextflow to schedule appropriately? (are you around for a chat sometime?)

[ossonts]

So I ran a quick test....

I pre-made a pool with: (Azure portal)
2 x standard_d15_v2 -- these are 20 cores each
on the pool I set the number of tasks per VM to 16 (leaving a few cores spare)

Then in the nextflow azure config,

 vmType = 'Standard_D14'

Note this is a 16 core machine -- NOT the actual one I am using -- I set this as the cores on this VM == 16

Then I ran 3 jobs with 1000 tasks each.

VM count ==1 and then part way through set the VM count to 2.

So with nextflow thinking I have 16 cores VMs and azure limiting to 16 per VM, I see the expected 32 tasks.

Somehow I feel that for a predefined pool (one created in the Azure Portal) -- I should just be able to set the batch credentials and then on each nextflow process set the number of cores that it should use (my case 1) -- having to specify a VM type for a predefined pool seems a bit obsolete -- especially as there is no validation.....

happy to take any thoughts....

[ossonts]

how about this:

If the pool is preconfigured (azure portal , with a predefined number of tasks per node)

Can nextflow not just use this task count ?