Feature description
In order to use benchmarks as performance regression tests, we need a way to assert on a performance regression. To do that, we need a way to summarize benchmark results, compare these benchmark summaries, and fail a benchmark when the comparison suggests a problem.
Implementation considerations
Summaries should be user-defined if at all possible (it really depends on the application which metrics matter and which don't). This ticket will likely require extending RobotFramework libraries in lambkin.shepherd.
Feature description
In order to use benchmarks as performance regression tests, we need a way to assert on a performance regression. To do that, we need a way to summarize benchmark results, compare these benchmark summaries, and fail a benchmark when the comparison suggests a problem.
Implementation considerations
Summaries should be user-defined if at all possible (it really depends on the application which metrics matter and which don't). This ticket will likely require extending
RobotFrameworklibraries inlambkin.shepherd.