Feature Request: Profiling

[siboehm]

I believe many projects (and many cluster quotas) would profit from having easy access to profiling for their experiments. In my seml projects I used this code: https://gist.github.com/siboehm/bf69a17cc9bca71c37a2fae0214a1eeb
It profiles the whole experiment using py-spy. It works well even for long running experiments (many hours). The size of the profile is a few MB and the CPU overhead of the profiler is pretty low. It runs in a subprocess and doesn't require changes to the main experiment.

Would you be interested in getting a PR that integrates profiling via py-spy into seml directly? If not, should I add the Gist as an example or can we link to it somehow?

[gasteigerjo]

Sure, this sounds pretty useful! Utility functions for the experiment go into seml/experiment.py.