Slurm parallel run: compute node CPU usage alternates between ~100% and ~0% while submission node stays ~100%

[yamane89]

Hello,

I am running EGSnrc (egs++ application) in parallel on a Slurm cluster.

When I launch the job from the submission/login node (where I run the exb command), the CPU usage on that node stays around ~100% (see screenshot 1).
However, on the compute node(s), CPU usage alternates between ~100% and ~0% repeatedly during the run (see screenshot 2).

Could you explain what might cause this behavior?
Are there any recommended solutions or configuration changes to avoid this behavior?

Thank you!

[yamane89]

I tried setting both nbatch and nchunk to 1, and with this configuration the issue disappeared (CPU usage on the compute node stayed close to 100%).

Is this the correct or recommended way to address this behavior when running EGSnrc with Slurm, or are there any potential drawbacks to this approach?

[rtownson]

Was this a simulation where you were scoring the dose to many regions? This could be caused by writing out the data to save a checkpoint after every batch - the CPU usage will drop while it writes a large data file. By reducing the number of batches to one, you no longer write these intermediate data files. This is recommended if you're not too worried about needing to resume crashed or interrupted simulations.

[yamane89]

Yes, exactly. I was scoring a 3ddose over 61 × 61 × 150 regions.

Thank you very much for the clear explanation — that makes a lot of sense. I learned a lot from this.