Optimizing mlpack neural networks and adding a finetuning example

[andrewfurey21]

Project

mlpack

Summary

Our main objective with the last SDG was revamping the neural network library in mlpack as a proof of concept. With the success of the previous grant, the goal is to optimize the neural network library and provide a finetuning example so that users can train neural networks specific to their usecase rather than just using pretrained weights.

submitter

Andrew Furey

project lead

@rcurtin

Community benefit

Problem

mlpack has an artificial neural network codebase that allows users to build their own neural networks from scratch. With the last small development grant, we successfully revamped the neural network library to allow arbitrary computational graphs. This allows more complicated and modern neural networks to be implemented and used. We also implemented a family of object detection models (YOLOv3) as an example of using the library for inference.

However, training and finetuning models for a specific purpose is the main source of utility from these models. Training is currently infeasible because of the slowness of some of the layers needed for vision models. For example, both the darknet library and pytorch run yolov3-tiny on an Intel i7 runs in about one second, where mlpack takes 10. We have not been working on optimizing mlpack recently and more focused on proof of concept, and because of this, we know there is a lot of progress we can make. We believe there is atleast a 10x gain to be made.

Solution

The goals for this grant are:

1. Optimizing mlpack layers
2. Implement a finetuning example for the bdd100k dataset
3. Get bandicoot working for GPU accelerated training for these models

How it helps the community

Users of mlpack now have the ability to design and implement modern neural networks. However, for users of any library, speed is extremely important. Without it, building and testing models becomes cumbersome, while also making training on anything impossible. With this SDG, we would be able to make mlpack faster and more useful for its users but also the C++ scientific computing community in general.

Amount requested (USD)

5000

Execution plan

Plan

We have a 16 week plan to reach our goals with this project.

• Weeks 1-8: Optimize mlpack layers. This includes convolutions, activation functions, normalization layers, pooling layers, etc. We'll be using many profiling tools and flame graphs to make progress here.

• Weeks 8-12: Optimize mlpack's computational graph code. With the last SDG we implemented a proof of concept, and did not spend time on optimizing. We already know of optimizations to get rid of pointless copies.

• Weeks 12-15: Developing the finetuning example using the bdd100k dataset as our example. bdd100k is a large and diverse driving dataset, and uses a similar format to COCO which is a popular large scale vision training set. This makes writing a finetuning example with bdd100k very useful for others, as most of the code will be easy to adapt to their usecase. This will also include using bandicoot (mlpack's new GPU backend) and getting it working with an example training run, which will also demonstrate bandicoot and using the GPU with mlpack.

• Weeks 15-16: Writing documentation and final benchmarking.

Team

Andrew Furey

I successfully completed a Small Development Grant from NumFocus this year for upgrading the neural network infrastructure in mlpack as well as adding example models (such as the YOLOv3 family), and I am excited to further improve the capabilities of mlpack's neural network code.

[dpshelio]

🎉 Congratulations, this project has been selected this time (by the random script).