Calculation methods for several metrics

[vktrrdk]

Due to its traceability, nextflow offers very good possibilities to observe the efficiency of workflow executions.
A list can be found here: https://www.nextflow.io/docs/latest/tracing.html#trace-report

If one uses the weblog (https://www.nextflow.io/docs/latest/tracing.html#weblog-via-http) with the with-weblog flag, this data can be collected and processed programmatically.
However, for some of the metrics it is not clear to me how these are calculated.

• %mem: How is this value obtained? Is this a simple relation of the metric rss with the available RAM on the machine of execution?

• peak_rss and rss: In my observations, these two values are always identical! How come this is the case?

Does anyone have any insights into this, or sources that break down the calculation?

[bentsherman]

%mem is the ratio of rss to the total available memory (MemTotal in /proc/meminfo)

peak_rss is the maximum, rss is the average. So if the memory usage is constant for most of the runtime, these two will be nearly equal

These metrics are computed in the task wrapper script .command.run. Aside from the docs, the template for this code is the best way to see how the metrics are computed, although it's a bit ugly.

[vktrrdk]

First of all, thank you for the last answer!

I had another look and once again noticed that for me the two values are always identical.

I used the -with-trace flag this time to create a trace file and capture the whole thing.
For this I created the following repository - there is a worfklow which uses the tool stress-ng to simulate different RAM usages:
https://github.com/vktrrdk/nextflow_peak_rss

This repository also serves the purpose of making the problem reproducible.

To run the workflow i used the command nextflow run stress_ram_single.nf --memoryLimit "24 GB" -with-trace

Is there anything I'm missing? The average (supposedly represented by rss) can't really be identical to the peak (`peak_rss) for this case of variable memory usage, can it?

The nextflow version used is 23.04.3

[bentsherman]

I don't know... rss comes from VmRSS and peak_rss comes from VmPeak in /proc/<pid>/status. Nextflow simply passes these values along, so you'll have to take it up with /proc.