Merge pull request #1639 from fzhar/object_detection_demo_ssd_async_common_pipeline

Unify C++ object detection demos

Author: vladimir-dudnik

demos/README.md

@@ -1,7 +1,6 @@
 # Open Model Zoo Demos
 
-The Open Model Zoo demo applications are console applications that demonstrate how you can use the Inference Engine in your applications to solve specific use-cases.
-
+The Open Model Zoo demo applications are console applications that demonstrate how you can use the Inference Engine in your applications to solve specific use-cases
 The Open Model Zoo includes the following demos:
 
 - [3D Human Pose Estimation Python\* Demo](./python_demos/human_pose_estimation_3d_demo/README.md) - 3D human pose estimation demo.
@@ -19,6 +18,7 @@ The Open Model Zoo includes the following demos:
 - [Image Inpainting Python\* Demo](./python_demos/image_inpainting_demo/README.md) - Demo application for GMCNN inpainting network.
 - [Image Retrieval Python\* Demo](./python_demos/image_retrieval_demo/README.md) - The demo demonstrates how to run Image Retrieval models using OpenVINO™.
 - [Image Segmentation C++ Demo](./segmentation_demo/README.md) - Inference of image segmentation networks like FCN8 (the demo supports only images as inputs).
+- [Image Segmentation Asynchronous C++ Demo](./segmentation_demo_async/README.md) - Inference of image segmentation networks like FCN8, async API showcase, simple OpenCV interoperability (supports video and camera inputs).
 - [Image Translation Python\* Demo](./python_demos/image_translation_demo/README.md) - Demo application to synthesize a photo-realistic image based on exemplar image.
 - [Instance Segmentation Python\* Demo](./python_demos/instance_segmentation_demo/README.md) - Inference of instance segmentation networks trained in `Detectron` or `maskrcnn-benchmark`.
 - [Interactive Face Detection C++ Demo](./interactive_face_detection_demo/README.md) - Face Detection coupled with Age/Gender, Head-Pose, Emotion, and Facial Landmarks detectors. Supports video and camera inputs.
@@ -30,8 +30,7 @@ The Open Model Zoo includes the following demos:
 - [Object Detection for CenterNet Python\* Demo](./python_demos/object_detection_demo_centernet/README.md) - Demo application for CenterNet object detection network.
 - [Object Detection for Faster R-CNN C++ Demo](./object_detection_demo_faster_rcnn/README.md) - Inference of object detection networks like Faster R-CNN (the demo supports only images as inputs).
 - [Object Detection for RetinaFace Python\* Demo](./python_demos/object_detection_demo_retinaface/README.md) - Demo application for RetinaFace face detection model.
-- [Object Detection for SSD C++ Demo](./object_detection_demo_ssd_async/README.md) - Demo application for SSD-based Object Detection networks, new Async API performance showcase, and simple OpenCV interoperability (supports video and camera inputs).
-- [Object Detection for YOLO V3 C++ Demo](./object_detection_demo_yolov3_async/README.md) - Demo application for YOLOV3-based Object Detection networks, new Async API performance showcase, and simple OpenCV interoperability (supports video and camera inputs).
+- [Object Detection C++ Demo](./object_detection_demo/README.md) - Demo application for Object Detection networks (different models architectures are supproted), async API showcase, simple OpenCV interoperability (supports video and camera inputs).
 - [Pedestrian Tracker C++ Demo](./pedestrian_tracker_demo/README.md) - Demo application for pedestrian tracking scenario.
 - [Security Barrier Camera C++ Demo](./security_barrier_camera_demo/README.md) - Vehicle Detection followed by the Vehicle Attributes and License-Plate Recognition, supports images/video and camera inputs.
 - [Speech Recognition Python\* Demo](./python_demos/speech_recognition_demo/README.md) - Speech recognition demo: takes audio file with an English phrase on input, and converts it into text.
@@ -118,7 +117,6 @@ The table below shows the correlation between models, demos, and supported plugi
 | vehicle-detection-adas-0002                      | any demo that supports SSD\*-based models, above                                                               | Supported | Supported | Supported   | Supported       |
 | yolo-v2-tiny-vehicle-detection-0001              | [Object Detection for YOLO V3 Python\* Demo](./python_demos/object_detection_demo_yolov3_async/README.md) | Supported |           |             |                 |
 
-
 Notice that the FPGA support comes through a [heterogeneous execution](https://docs.openvinotoolkit.org/latest/_docs_IE_DG_supported_plugins_HETERO.html), for example, when the post-processing is happening on the CPU.
 
 ## Build the Demo Applications

demos/common/CMakeLists.txt

@@ -6,6 +6,8 @@ find_package(InferenceEngine 2.0 REQUIRED)
 find_package(OpenCV REQUIRED COMPONENTS core imgcodecs videoio)
 
 add_subdirectory(monitors)
+add_subdirectory(models)
+add_subdirectory(pipelines)
 
 file(GLOB_RECURSE HEADERS "${CMAKE_CURRENT_SOURCE_DIR}/include/*")
 file(GLOB_RECURSE SOURCES "${CMAKE_CURRENT_SOURCE_DIR}/src/*")

demos/common/include/samples/performance_metrics.hpp

@@ -36,6 +36,20 @@ class PerformanceMetrics {
                 double fontScale = 0.75,
                 cv::Scalar color = {200, 10, 10},
                 int thickness = 2);
+    void update(TimePoint lastRequestStartTime);
+
+    /// Paints metrics over provided mat
+    /// @param frame frame to paint over
+    /// @param position left top corner of text block
+    /// @param fontScale font scale
+    /// @param color font color
+    /// @param thickness font thickness
+    void paintMetrics(cv::Mat& frame,
+        cv::Point position = { 15, 30 },
+        double fontScale = 0.75,
+        cv::Scalar color = { 200, 10, 10 },
+        int thickness = 2) const;
+
     Metrics getLast() const;
     Metrics getTotal() const;
     void printTotal() const;

demos/common/models/CMakeLists.txt

@@ -0,0 +1,18 @@
+# Copyright (C) 2018-2019 Intel Corporation
+# SPDX-License-Identifier: Apache-2.0
+#
+
+find_package(OpenCV REQUIRED COMPONENTS core imgproc)
+find_package(ngraph REQUIRED)
+
+FILE(GLOB SOURCES ./src/*.cpp)
+FILE(GLOB HEADERS ./include/models/*.h)
+
+# Create named folders for the sources within the .vcproj
+# Empty name lists them directly under the .vcproj
+source_group("src" FILES ${SOURCES})
+source_group("include" FILES ${HEADERS})
+
+add_library(models STATIC ${SOURCES} ${HEADERS})
+target_include_directories(models PUBLIC "${CMAKE_CURRENT_SOURCE_DIR}/include")
+target_link_libraries(models PRIVATE ngraph::ngraph gflags ${InferenceEngine_LIBRARIES} common opencv_core opencv_imgproc)

demos/common/models/include/models/detection_model.h

@@ -0,0 +1,46 @@
+/*
+// Copyright (C) 2018-2020 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+*/
+#pragma once
+#include "models/model_base.h"
+#include "opencv2/core.hpp"
+
+class DetectionModel : public ModelBase {
+public:
+    /// Constructor
+    /// @param modelFileName name of model to load
+    /// @param confidenceThreshold - threshold to eleminate low-confidence detections.
+    /// Any detected object with confidence lower than this threshold will be ignored.
+    /// @param useAutoResize - if true, image will be resized by IE.
+    /// Otherwise, image will be preprocessed and resized using OpenCV routines.
+    /// @param labels - array of labels for every class. If this array is empty or contains less elements
+    /// than actual classes number, default "Label #N" will be shown for missing items.
+    DetectionModel(const std::string& modelFileName, float confidenceThreshold, bool useAutoResize, const std::vector<std::string>& labels);
+
+    virtual std::shared_ptr<InternalModelData> preprocess(const InputData& inputData, InferenceEngine::InferRequest::Ptr& request) override;
+
+    static std::vector<std::string> loadLabels(const std::string& labelFilename);
+
+protected:
+    std::vector<std::string> labels;
+
+    size_t netInputHeight = 0;
+    size_t netInputWidth = 0;
+
+    bool useAutoResize;
+    float confidenceThreshold;
+
+    std::string getLabelName(int labelID) { return (size_t)labelID < labels.size() ? labels[labelID] : std::string("Label #") + std::to_string(labelID); }
+};

demos/common/models/include/models/detection_model_ssd.h

@@ -0,0 +1,40 @@
+/*
+// Copyright (C) 2018-2020 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+*/
+
+#pragma once
+#include "detection_model.h"
+class ModelSSD : public DetectionModel {
+public:
+    /// Constructor
+    /// @param modelFileName name of model to load
+    /// @param confidenceThreshold - threshold to eleminate low-confidence detections.
+    /// Any detected object with confidence lower than this threshold will be ignored.
+    /// @param useAutoResize - if true, image will be resized by IE.
+    /// Otherwise, image will be preprocessed and resized using OpenCV routines.
+    /// @param labels - array of labels for every class. If this array is empty or contains less elements
+    /// than actual classes number, default "Label #N" will be shown for missing items.
+    ModelSSD(const std::string& modelFileName,
+        float confidenceThreshold, bool useAutoResize,
+        const std::vector<std::string>& labels = std::vector<std::string>());
+
+    std::shared_ptr<InternalModelData> preprocess(const InputData& inputData, InferenceEngine::InferRequest::Ptr& request);
+    virtual std::unique_ptr<ResultBase> postprocess(InferenceResult& infResult) override;
+
+protected:
+    virtual void prepareInputsOutputs(InferenceEngine::CNNNetwork& cnnNetwork) override;
+    size_t maxProposalCount = 0;
+    size_t objectSize = 0;
+};

demos/common/models/include/models/detection_model_yolo.h

@@ -0,0 +1,72 @@
+/*
+// Copyright (C) 2018-2020 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+*/
+
+#pragma once
+#include "detection_model.h"
+
+namespace ngraph {
+    namespace op {
+        namespace v0 {
+            class RegionYolo;
+        }
+        using v0::RegionYolo;
+    }
+}
+
+class ModelYolo3 : public DetectionModel {
+protected:
+    class Region {
+    public:
+        int num = 0;
+        int classes = 0;
+        int coords = 0;
+        std::vector<float> anchors;
+
+        Region(const std::shared_ptr<ngraph::op::RegionYolo>& regionYolo);
+    };
+
+public:
+    /// Constructor.
+    /// @param modelFileName name of model to load
+    /// @param confidenceThreshold - threshold to eleminate low-confidence detections.
+    /// Any detected object with confidence lower than this threshold will be ignored.
+    /// @param useAutoResize - if true, image will be resized by IE.
+    /// Otherwise, image will be preprocessed and resized using OpenCV routines.
+    /// @param useAdvancedPostprocessing - if true, an advanced algorithm for filtering/postprocessing will be used
+    /// (with better processing of multiple crossing objects). Otherwise, classic algorithm will be used.
+    /// @param boxIOUThreshold - threshold to treat separate output regions as one object for filtering
+    /// during postprocessing (only one of them should stay). The default value is 0.4
+    /// @param labels - array of labels for every class. If this array is empty or contains less elements
+    /// than actual classes number, default "Label #N" will be shown for missing items.
+    ModelYolo3(const std::string& modelFileName, float confidenceThreshold, bool useAutoResize,
+        bool useAdvancedPostprocessing = false, float boxIOUThreshold = 0.4, const std::vector<std::string>& labels = std::vector<std::string>());
+
+    std::unique_ptr<ResultBase> postprocess(InferenceResult& infResult) override;
+
+protected:
+    virtual void prepareInputsOutputs(InferenceEngine::CNNNetwork& cnnNetwork) override;
+
+    void parseYOLOV3Output(const std::string& output_name, const InferenceEngine::Blob::Ptr& blob,
+        const unsigned long resized_im_h, const unsigned long resized_im_w, const unsigned long original_im_h,
+        const unsigned long original_im_w, std::vector<DetectedObject>& objects);
+
+    static int calculateEntryIndex(int side, int lcoords, int lclasses, int location, int entry);
+    static double intersectionOverUnion(const DetectedObject& o1, const DetectedObject& o2);
+
+    std::map<std::string, Region> regions;
+    double boxIOUThreshold;
+    bool useAdvancedPostprocessing;
+};

demos/common/models/include/models/input_data.h

@@ -0,0 +1,39 @@
+/*
+// Copyright (C) 2018-2020 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+*/
+
+#pragma once
+#include <opencv2/core.hpp>
+
+struct InputData {
+    virtual ~InputData() {}
+
+    template<class T> T& asRef() {
+        return dynamic_cast<T&>(*this);
+    }
+
+    template<class T> const T& asRef() const {
+        return dynamic_cast<const T&>(*this);
+    }
+};
+
+struct ImageInputData : public InputData {
+    cv::Mat inputImage;
+
+    ImageInputData() {}
+    ImageInputData(const cv::Mat& img) {
+        inputImage = img;
+    }
+};

demos/common/models/include/models/internal_model_data.h

@@ -0,0 +1,38 @@
+/*
+// Copyright (C) 2018-2020 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+*/
+
+#pragma once
+struct InternalModelData {
+   virtual ~InternalModelData(){}
+
+   template<class T> T& asRef() {
+        return dynamic_cast<T&>(*this);
+    }
+
+    template<class T> const T& asRef() const {
+        return dynamic_cast<const T&>(*this);
+    }
+};
+
+struct InternalImageModelData : public InternalModelData {
+    InternalImageModelData(int width, int height) {
+        inputImgWidth = width;
+        inputImgHeight = height;
+    }
+
+    int inputImgWidth;
+    int inputImgHeight;
+};

demos/common/models/include/models/model_base.h

@@ -0,0 +1,41 @@
+/*
+// Copyright (C) 2018-2020 Intel Corporation
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//      http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+*/
+
+#pragma once
+#include "input_data.h"
+#include "results.h"
+
+class ModelBase {
+public:
+    ModelBase(const std::string& modelFileName) { this->modelFileName = modelFileName; }
+    virtual ~ModelBase() {}
+
+    virtual void prepareInputsOutputs(InferenceEngine::CNNNetwork& cnnNetwork) = 0;
+    virtual std::shared_ptr<InternalModelData> preprocess(const InputData& inputData, InferenceEngine::InferRequest::Ptr& request) = 0;
+    virtual std::unique_ptr<ResultBase> postprocess(InferenceResult& infResult) = 0;
+        virtual void onLoadCompleted(InferenceEngine::ExecutableNetwork* execNetwork, const std::vector<InferenceEngine::InferRequest::Ptr>& requests) {
+        this->execNetwork = execNetwork; }
+    const std::vector<std::string>& getOutputsNames() const { return outputsNames; }
+    const std::vector<std::string>& getInputsNames() const { return inputsNames; }
+
+    std::string getModelFileName() { return modelFileName; }
+
+protected:
+    std::vector<std::string> inputsNames;
+    std::vector<std::string> outputsNames;
+    InferenceEngine::ExecutableNetwork* execNetwork;
+    std::string modelFileName;
+};