Documentation and tooling for detecting multithreaded scaling issues and regressions #117
Description
Over in numpy/numpy#27786 we had a NumPy user report a parallel scaling issue and I went through some effort to debug it.
The debugging section in the guide should explain some techniques for identifying scaling issues and then diagnosing their source. We should explain how to run native python extensions under native performance profiling tools like perf to generate a flame graph. I didn't do this over in NumPy, but we should also explore other tooling like Intel's VTune and valgrind that can detect multithreaded scaling issues. samply might also be worth looking into, since it works on both MacOS and Linux.
Longer term, it would be nice to have tooling to help with this. This is simpler than other kinds of benchmarking since you don't need to compare across commits - just detect that the parallel scaling is bad in a single commit.