Rperform: Performance analysis and visualization for R
There exists various tools and software to help developers across different languages test the performance of their code. This analysis can be performed in terms of quantifiable metrics such as time, memory, etc. Rperform aims to be a standard tool for performance testing R packages and code, in general. Rperform had started as a GSoC 2015 project and was again accepted in GSoC 2016. From its README,
Rperform is a package that allows R developers to track quantitative performance metrics of their code. It focuses on providing changes in a package’s performance metrics, related to runtime and memory, over different git versions and across git branches. Rperform can be integrated with Travis-CI to do performance testing during Travis builds by making changes to the repo's .travis.yml file. It can prove to be particularly useful while measuring the possible changes which can be introduced by a pull request (PR).
What other R packages with similar functionality already exist? Why aren't they good enough?
There are several focus areas which will be worked upon on this project. They include:
-
Allow for efficient and easier performance testing of standalone files and commits; make performance testing of package more intuitive and easier.
-
Improvement of the visualization functions: One of the most prominent and helpful feature of Rperform are the visualization functions it provides. Here's an example plot generated by Rperform after analyzing the runtime performance of a unit test from Hadley Wickham's stringr package.
Details about the same can be found on the Rperform Wiki. The visualization functions need to be improved upon and made interactive using packages such as animint. The direction this aspect of the project takes will heavily depend on how the UI implementation is done (see below).
Associated issue: https://github.com/analyticalmonk/Rperform/issues/15
- Provision to create and maintain database of metrics: There should be a method to create and maintain a database of the results obtained using Rperform at various points of time. This would not only be helpful to the users but would also help in identifying inconsistencies in Rperform's measurements, if any. Inclusion of such a database in a UI (next objective) would be very useful.
Associated issue: https://github.com/analyticalmonk/Rperform/issues/34
- Changepoint detection: Include feature to perform changepoint detection in context of a code-base's performance. Something like the changepoint package could be used.
Associated issue: https://github.com/analyticalmonk/Rperform/issues/35
-
Creating a coherent and useful user interface: Currently, Rperform does have a function which generates a webpage comprised of multiple plots obtained after analyzing the package code. However, there is a need for development of a proper user interface for the package developer to be able to interact meaningfully with the results. Inspiration for the same can be taken from projects such as asv and codespeed.
Some potentially useful features are:
- Option to display plots for various files using a dropdown menu. Also, being able to display multiple plots at the same time for comparison purposes.
- Function to find a commit that produces a large regression (inspired from asv).
- Make the plots interactive. Hovering over a datapoint provides details about the commit such as date, author, etc.
Associated issue: https://github.com/analyticalmonk/Rperform/issues/33
-
Improve test suite and increase code coverage: Rperform currently lacks adequate number of unit tests. With increased functionality, working on this aspect will become even more important.
-
Make package CRAN-ready: The package, at the end of the project, should pass R CMD CHECK.
Associated issue: https://github.com/analyticalmonk/Rperform/issues/16
Mentors, please explain how this project will produce a useful package for the R community.
- Akash Tandon [email protected] had worked on the package during GSoC 2015, 2016 and will be a mentor for the project, if selected, this year.
- Joshua Ulrich [email protected] was a mentor for the project during GSoC 2016 and has agreed to mentor the project this year.
- Easy: Install Rperform package and recreate this example from Rperform's wiki page for a fresh fork of the stringr package.
Students, please post a link to your test results here.