Benchmarking performance

[sole]

Oftentimes contributors suggest changes that 'improve performance'. Other times they suggest changes but we are reluctant to merge them just in case they 'worsen performance'.

But right now we have no way of measuring performance other than looking at FPS graphs and trying to make sure "it doesn't look worse", but this isn't very accurate, and it's also a bit time consuming.

We cannot tell for sure whether something is better or worse, but we should!

In the past I ran manual tests for comparing between changes but as anything manual, it puts you off.

The ideal benchmark...

• can be run in an automated way (no user intervention required other than initiating the benchmarks, via tests or just the benchmark itself)
• can run on travis (so it's part of the tests before PRs are accepted, and if things are slower than a certain baseline we should reject a PR?)
• could aggregate historical records so we can see if we're going slower or faster (or compare between platforms and engines - some engines optimise things differently) - this would be a stretch goal

So far I have been looking at:

• https://benchmarkjs.com/ which looked good, but it had some strange incompatibilities with node.js.
• related to it, https://jsperf.com/ could aggregate records but it's sort of on pause at the moment.

Any other ideas?

[digitarald]

http://openwebgames.com/ is a benchmark that penalizes dropped frames, which seems very applicable here.

The live test could be similar to openwebgames, just putting the test up on the web and collecting the numbers from real users. A continues design could run the tests on a daily basis on either cloud or a local machine.