Consider an optional flag to collect and dump GPU metrics

[jsign]

It might be useful to add some flag e.g. --collect-metrics, which dumps in one (or more) files hardware metrics during the proving process.

The most obvious one are GPU metrics. Prob the easiest way to do this is to leverage nvidia-smi which already have watch flags and output formats, etc.

Example:

$ nvidia-smi --query-gpu=timestamp,name,utilization.gpu,utilization.memory,memory.used,memory.total,temperature.gpu --format=csv -l 1 > gpu_log.csv

And tailing the .csv file:

2026/01/06 15:13:03.389, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 49
2026/01/06 15:13:03.389, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 48
2026/01/06 15:13:04.389, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 41
2026/01/06 15:13:04.390, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 47
2026/01/06 15:13:04.390, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 48
2026/01/06 15:13:04.390, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 49
2026/01/06 15:13:04.390, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 47
2026/01/06 15:13:04.390, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 47
2026/01/06 15:13:04.390, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 49
2026/01/06 15:13:04.390, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 48
2026/01/06 15:13:05.391, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 41
2026/01/06 15:13:05.391, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 47
2026/01/06 15:13:05.391, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 48
2026/01/06 15:13:05.391, NVIDIA GeForce RTX 5090, 0 %, 0 %, 2 MiB, 32607 MiB, 48

Some considerations:

• We could name the file accordingly with some zkvm name, start-timestamp, end-timestamp
• See which --query-gpu stuff makes sense to collect, and maybe support ERE_GPU_METRICS to allow the user to customize
• Not sure about CSV format by default, but I guess sounds reasonable.
• Apparently, there is a -lms to set the frequency at ms level if that's useful, but not sure.

All points above are thinking out loud.

[jsign]

Dropping here another command that sounds useful to see PCIe usage:

 ubuntu@worker04:~/zkevm-benchmark-workload$ nvidia-smi dmon -s tp

# gpu  rxpci  txpci    pwr  gtemp  mtemp  
# Idx   MB/s   MB/s      W      C      C  
   0   6030     66    292     47      -  
   1      0    807    233     55      -  
   2      0      1    201     55      -  
   3      0      0    160     51      -  
   4      0      0    145     48      -  
   5      0      0    107     49      -  
   6      0      0     57     48      -  
   7      0      0     64     48      -  
   0   1699      1    287     51      -  
   1   1951    488    246     52      -  
   2      0      0    217     53      -  
   3      0      0    167     51      -  
   4      0      0     91     48      -  
   5      0      0     74     47      -  
   6      0      1     57     48      -  
   7      0      0     64     48      -
...