Devise a test harness for testing performance

[godlygeek]

We know there's low hanging fruit available for optimizing PyStack, but we currently have no good way to benchmark our performance and quantify any improvements. Design some sort of a test harness that can be used for measuring the performance impact of our changes, possibly using https://asv.readthedocs.io/en/stable/

[Helithumper]

I'll look into this today and tomorrow.

[Helithumper]

Wasn't able to look into this. Feel free to unsassign

[ms2892]

How is the impact of the performance impact of changes defined? Is it meant to be a time and space benchmark against a predefined use?
Also is it meant to be integrated into a github workflow or is it meant to be a separate module of its own?

[sarahmonod]

How is the impact of the performance impact of changes defined? Is it meant to be a time and space benchmark against a predefined use? Also is it meant to be integrated into a github workflow or is it meant to be a separate module of its own?

My understanding is that we want to have something similar to the CodeCov integration, where we can easily tell, for any given PR, if a number of predefined measurements differ when measured on main vs on the PR branch. This way, we can both have a better understanding on the current time and space performance of PyStack, and be aware if we are significantly changing them with a PR.

Any PR whose goal is to change performance would be close to meaningless without a way of measuring the difference.

[ms2892]

How is the impact of the performance impact of changes defined? Is it meant to be a time and space benchmark against a predefined use? Also is it meant to be integrated into a github workflow or is it meant to be a separate module of its own?

My understanding is that we want to have something similar to the CodeCov integration, where we can easily tell, for any given PR, if a number of predefined measurements differ when measured on main vs on the PR branch. This way, we can both have a better understanding on the current time and space performance of PyStack, and be aware if we are significantly changing them with a PR.

Any PR whose goal is to change performance would be close to meaningless without a way of measuring the difference.

Hi Gus,

I implemented it in the PR #165. Please have a look