Fix: Reporting the correct number of utilized CPUs

[JosuaCarl]

As stated in #6743, a problem with flexible scheduling of tasks is that the rules for execution constraints upon logical cores is not enforced, or enforced with wiggle room.

In the instance of a Docker executor, --cpu-shared 2048 is used instead of --cpus 2 which prioritizes resource utilization, but is no hard cutoff (see https://docs.docker.com/engine/containers/resource_constraints/#configure-the-default-cfs-scheduler).

nextflow/modules/nextflow/src/main/groovy/nextflow/container/DockerBuilder.groovy

Lines 128 to 129 in 887443e

	if( cpus && !legacy )
	result << "--cpu-shares ${cpus * 1024} "

This behavior has many upsides, but the number of cpus in the trace is often reported incorrectly, as it takes the value set by the user in process.cpus for granted, even if it was not enforced.

The PR adds a sampling script which utilizes breadth-first search to extract the tasks and its children's last used logical CPU and record them in a list. In the end, this is summed to the total value of utilized CPUs.

The current sampling rate is at 0.1s. I tried 1.0s, but this missed several short processes in the demo pipeline, although I suspect this effect may vanish for longer processes. I could see no large changes in execution time and memory utilization for the tasks for either option. Sampling was chosen, because other ways either require root permission (tracking at kernel level with perf) or only reported minima (ceil(%cpu / 100)) and maxima (grep Cpus_allowed_list /proc/$pid/status) for the number of utilized CPUs. In practice I observed, that the number of actually used CPUs was most of the time close to the maximum, so if an approximation was to be used that does not rely on sampling, I would suggest grep Cpus_allowed_list /proc/$pid/status, which was my original solution in the first commit (have a look if you want).

• Closes cpus trace value does not correspond to used logical cores of the task #6743

Edits:

• Changed the parameter that is reported from cpus to a new TraceRecord field used_cpus.
• Replaced trace writing with already existing method nxf_write_trace to align nxf_trace_linux with nxf_trace_mac and save a few lines. This could also be put into a new PR, if deemed too much deviation from the main task.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	f6b45de
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/6985abb3c1eabe0008539c11

[muffato]

(I'm not in the Nextflow team, I'm just a Nextflow user.)

I would like to plead to leave the current cpus as it is, because together with memory and time (and disk) they report what the workflow had defined and requested for each task. The values can then be compared with the actual usage %cpu, peak_rss, and realtime to then optimise resource requests within the pipeline. Making resource usage match resource requests is a very important step when deploying a pipeline in production, to reduce scheduling time and wastage.

I see the metrics you're trying to add as an "advanced" trace that gives valuable insights, alongside the existing scheduler and runtime traces, like duration, %mem, vmem, write_bytes , and many others, but doesn't replace cpus.

[JosuaCarl]

@muffato So would you rather opt for an additional trace value, like used_cpus, so the reporting of user set cpus is not lost?

[muffato]

Personally, yes, but again I'm just a mere (moderately advanced) user. The call should come from Seqera.

[bentsherman]

Closing for the same reason as #6731 -- this seems like a lot of added complexity for not much added value, consider doing it through a plugin instead.