Is it possible to use Skyline for performance test analisys

[domik82]

Hi,

I randomly found this project and started to dig into a bit. I'm looking for a tool that would be able to compare two 1h test runs between each other.

I have plenty of data in our InfluxDB but this is done in a way that we run a Load test lets say 8-9 AM and then there are no samples. Next day we do another regression running at same time.

Today we used Keptn with SLO definitions based on p99 and we do comparision between current build and last 14 runs.

I wonder - would it be possible to "hack" the tool to do a comparison analysis between Run A lets say we consider successful vs Run B "latest". So instead of writing more and more SLO's for perf tests this toold would collect it and compare.

[earthgecko]

Hi @domik82 I think I understand what you are trying to achieve and the short answer is no, not really. Although anything is possible, a "hack" suggests a fairly simple change that will achieve the goal. What you are describing here would not fit easily into any existing Skyline module or functionality I am sorry to say.

This type of use case would require entirely new functionality with specific workflow and logic. That raises other questions pertaining to the logic itself. What are the comparison parameters between A and B? Are all the comparisons subject to the same parameters or do they differ depending on what is being compared? What dimensions of A and B need to be compared, etc?

I think that this might be quite a complicated problem, in terms of defining workflows, logic, configuration, etc.

It would be much simpler to submit the p99 values as metrics to Skyline each day and monitor those metrics themselves, which could be achieved via algorithmic analysis or via a boundary check (those would require metrics based on the delta between the previous p99 value and the current p99 value, e.g. what is today's delta).

Perhaps if you add some anonymised example data I could make more concrete recommendations.