Understanding py-spy>Ns behind in sampling results may be inaccurate. Try reducing the sampling rate

[leuchthelp]

As the headline would imply I would like to understand how bad exactly any form of delay can be when profiling. This isn't specifically an issue yet - I do seem to have some issues, most likely by design of my application - but I first would like to understand the inner workings of py-spy more before I make a call.

Currently my code does the following figure out file format -> gather code and dependencies -> assemble some form of benchmark -> run in separate shell entirely using sbatch. This task is performed on the DKRZs Levante HPC cluster which runs slurm.

Run with py-spy record --rate 1 --full-filenames -F -n --format speedscope -o <profile_path>.json -- python.

What I'm currently seeing is that even a run performed with a sampling rate of 1 results in some form of delay. This spikes at about 10 seconds. Running with a sampling rate of 10 peaks at about 40 seconds. The default 100 can sometimes even go to 300 seconds or more. So I would like to understand what the drawback might be of just running with something like --rate=10, taking the 40s penalty and being fine with it - from a purely technical standpoint. I haven't actually evaluated the results yet as I'm still in the process of testing the py-spy integration.

The bigger issue seems to be the Error: No child process (os error 10 that gets throw when the HDF5 format is used through the h5py python package. So I would also like to understand that.

Any help is greatly appreciated. If there is another channel for such question, that I have missed, I apologize in advance.