NewFeatureIdeas

Ideas

New features:

1. Extend the set of inference frameworks and models in regular benchmarks. Support for inference using OpenCV (DNN module, with / without Inference Engine backend), MXNet, PyTorch.
2. Check models for the following frameworks: TensorFlow lite and ONNXRuntime.
3. Develop docker containers for TensorFlow lite and ONNXRuntime.
4. Check docker containers for the following frameworks: Intel® Optimization for Caffe, Intel® Optimization for TensorFlow.
5. Provide collecting the performance metrics and the accuracy metrics for the following frameworks: Intel® Optimization for Caffe, Intel® Optimization for TensorFlow, TensorFlow lite and ONNXRuntime.
6. Test prepared step-by-step manual.
7. Automate the regular testing of the main scenario correctness, the final collecting of the performance metrics and the accuracy of the available deep models using generally accepted CI/CD tools.
8. Develop a demo to demonstrate the main scenario of the benchmarking system DLI.
9. Develop and/or integrate utilities for converting models from one storage format to another one. Providing the ability to convert models between framework formats supported by the benchmarking system DLI, or to the ONNX format.
10. Extend the set of hardware platforms for regular performance measurements (RaspberryPi 4 8GB) (if we will have access to hardware).

Papers:

1. An overview of frameworks for inferring deep neural network models and comparing their functionality.