Tower strange autoscaling behaviour on Azure Batch

[wikiselev]

Bug report

Expected behavior and actual behavior

Expected: run a full pipeline on a single compute node using Tower and Azure Batch.

Actual: in addition to a single node 10 more nodes are added to a pool, then immediately transition to idle state and stay idle until the pipeline is finished.

Steps to reproduce the problem

1. Create a compute environment using Tower Forge with 60 nodes that have 16 cores and 64 GB of RAM each and enable autoscaling.
   Tower configuration:
   
   Batch pool configuration:
   
   
   
2. Start a pipeline from Tower with 10 processes (1 cpu and 4 GB of RAM each). So 10 processes should all fit into a single node of a newly created pool.

Program output

1. The pipeline starts with 1 node and submits all 10 processes on that node.
2. After 5 mins 10 more nodes without any tasks are added to the pool. They immediately get into idle state and are not deallocated from the pool until the pipeline is finished. So the total number of nodes after 5 mins becomes 11 instead of 1.
3. Once the pipeline is finished (10 original processes are finished) the pool starts to scale down from 11 to 0.

Environment

• Nextflow version: 22.10.7 build 5853
• Java version: openjdk version "11.0.18" 2023-01-17
• Operating system: ubuntu
• Bash version: GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

Additional context

The main problem with this behaviour is that I believe I am still charged for the idle nodes, so I pay for 11 nodes instead of 1.

[bentsherman]

From my limited knowledge of Azure Batch, I wonder if it has to do with the autoscaling formula:

// Get pool lifetime since creation.
lifespan = time() - time("{{poolCreationTime}}");
interval = TimeInterval_Minute * {{scaleInterval}};

// Compute the target nodes based on pending tasks.
// $PendingTasks == The sum of $ActiveTasks and $RunningTasks
$samples = $PendingTasks.GetSamplePercent(interval);
$tasks = $samples < 70 ? max(0, $PendingTasks.GetSample(1)) : max( $PendingTasks.GetSample(1), avg($PendingTasks.GetSample(interval)));
$targetVMs = $tasks > 0 ? $tasks : max(0, $TargetDedicatedNodes/2);
targetPoolSize = max(0, min($targetVMs, {{maxVmCount}}));

// For first interval deploy 1 node, for other intervals scale up/down as per tasks.
$TargetDedicatedNodes = lifespan < interval ? {{vmCount}} : targetPoolSize;
$NodeDeallocationOption = taskcompletion;

It doesn't seem to be accounting for task packing, it looks like the formula just scales up the nodes to match the number of pending tasks.

@abhi18av do you have any insights here?

[abhi18av]

Expected: run a full pipeline on a single compute node using Tower and Azure Batch.

Hi @wikiselev , yes @bentsherman is spot-on regarding this behavior.

If you'd like to use a single node for the entire pipeline then you could

1. Either modify the default scaling formula as per the Azure Batch autoscaling DSL ( https://learn.microsoft.com/en-us/azure/batch/batch-automatic-scaling ) to create a new static formula and put it in the your config file.

2. Or (simply) disable the autoscale setting on a pre-deployed pool. You can also do this (overridding the autoscale formula) from the Azure Batch portal even after the pool has been created

[wikiselev]

Thanks, @abhi18av ! But I do want the pool to scale. I expected all of my processes to land on one node only because it was big enough for all 10 of them. This corresponds to the Pack strategy. But if I have more processes like that I will need to autoscale.

So, from what you say - NF Azure autoscaling formula always start a node per process? I don't think I've seen this behaviour before when I run some of the nf-core pipelines. It is only now that I observe this with my own pipeline written from scratch.

What I still don't understand is why it was autoscaling and adding 10 more nodes after 5 minutes if all 10 processes had already landed on the first original node. At the moment the autoscaling description is:

The above formula initialises a pool with the number of VMs specified by the vmCount option, it scales up the pool on-demand, based on the number of pending tasks up to maxVmCount nodes. If no jobs are submitted for execution, it scales down to zero nodes automatically.

So, my tasks (processes) were not pending, they were already running on the first node. What was the reason for autoscaling in this case?

[abhi18av]

Ah, okay - thanks for the clarification @wikiselev. I think that the setting you're looking for in this case is azure.batch.pools.<name>.schedulePolicy which has the default value of spread.

I think that pack is the strategy you're looking for in this case and this would instructure the Azure Batch scheduler to try and put in all the tasks within a single node first before deploying another node

You can also specify a fill type, which determines whether Batch spreads the tasks evenly across all nodes in a pool, or packs each node with the maximum number of tasks before assigning tasks to another node.

From the official Azure Batch docs here https://learn.microsoft.com/en-us/azure/batch/nodes-and-pools#task-scheduling-policy