Azure Custom Vision support (#16)

PranavDhulipala · Pranav Dhulipala · lilustga · web-flow · commit f2a0519d9dc2 · 2021-06-15T14:47:45.000-07:00
* Modified the node to enable building code using catkin_make * made the requested changes to the CMakeLists.txt and deleted the config files since they are no longer required * minor changes to the readme file * Added support to run trained onnx models from customvision.ai * Deleting onnx model * updated readme * adding dynamic reconfigure to switch onnx models at runtime * deleting redundant headers from main.cpp * moved the anchors to a yaml file and updated the readme * updated the readme * updating to onnx runtime version 1.7.0 for cpu and 1.7.1 for gpu * Updates to readme and camera node selection based on OS. (#17) * Updates to readme and camera node selection based on OS. * removed the engine block stl and added instructions to download. * Updated date. * README fixes for Linux * c_str to fix build error on linux. * Fixed link in README * Made path to STL explicit in README. * Switched to OTL cv_camera for linux and formatted link in README * Modified the node to enable building code using catkin_make * made the requested changes to the CMakeLists.txt and deleted the config files since they are no longer required * minor changes to the readme file * Added support to run trained onnx models from customvision.ai * Deleting onnx model * updated readme * adding dynamic reconfigure to switch onnx models at runtime * deleting redundant headers from main.cpp * moved the anchors to a yaml file and updated the readme * updated the readme * updating to onnx runtime version 1.7.0 for cpu and 1.7.1 for gpu * rebased with the latest commit, made changes in a few files Co-authored-by: Pranav Dhulipala <prdhulip@microsoft.com> Co-authored-by: Lior Lustgarten <45467030+lilustga@users.noreply.github.com>
diff --git a/README.md b/README.md
@@ -61,10 +61,10 @@ catkin_make -DCUDA_SUPPORT=ON
 
 ## Running the samples
 There are two launch files included as samples in the launch folder.
-An object tracking demo and a deep pose detection demo.
+An object tracking demo and a deep pose detection demo. 
 
-### Person Tracker Demo
- `tracker.launch` demonstrates tracking people in images/video.
+### Object Tracker Demo
+ `tracker.launch` demonstrates tracking of up to 20 classes icluding people in images/video.
 
 To run the person tracking detection demo, source the workspace and then roslaunch the launch file.
 
@@ -78,6 +78,15 @@ In another command prompt or terminal, run rviz and add the `/tracked_objects/im
 rosrun rviz rviz
 ```
 
+For a project trained using customvision.ai at runtime you can change the parameters using rqt_reconfigure.
+
+```Batchfile
+rosrun rqt_reconfigure rqt_reconfigure
+```
+Please follow this order, start by placing the files of interest i.e. the onnx model(.onnx) file and labels.txt obtained from the relevant onnx zip file downloaded from customvision.ai to `ros_msft_onnx/testdata/` or a known location. Change the anchor values in the cfg/anchors.yaml file by commenting the first line and uncommenting the second line, finally update `input_node_name` to "data", `output_node_name` to "model_outputs0" and update any other relevant parameters shown below:
+
+![Rqt Reconfigure](./ros_msft_onnx/testdata/rqt_reconfigure.PNG)
+
 ### Deep Pose Detection Demo
 `pose.launch` demonstrates estimating the position and rotation of an engine block from images\video.In preperation for running the engine pose demo:
 * Copy [Engine pose ONNX model](https://github.com/ms-iot/ros_msft_onnx_demo/releases/download/0.0/engine.onnx) to `ros_msft_onnx/testdata/`.
@@ -88,57 +97,27 @@ To run the engine pose detection demo, source the workspace and then roslaunch t
 roslaunch ros_msft_onnx pose.launch
 ```
 
-For your own project, you can create a launch file in the following format:
-
-```xml
-<launch>
-  <arg name="onnx_model_path_arg" default="$(find ros_msft_onnx)/testdata/model.onnx"/>
-  <node pkg="ros_msft_onnx" type="ros_msft_onnx_node" name="ros_msft_onnx" output="screen">
-    <param name="onnx_model_path" value="$(arg onnx_model_path_arg)"/>
-    <param name="confidence" value="0.5"/>
-    <param name="tensor_width" value="416"/>
-    <param name="tensor_height" value="416"/>
-    <param name="tracker_type" value="yolo"/>
-    <param name="image_processing" value="resize"/>
-    <param name="debug" value="true"/>
-    <param name="image_topic" value="/camera/image_raw" />
-  </node>
-  
-  <!-- NOTE: The image properties need to be valid for the camera, or the node will auto select the closest values -->
-  <node pkg="ros_msft_camera" type="ros_msft_camera_node" name="camera">
-    <param name="camera_info_url" value="file://$(find ros_msft_camera)/config/default_calibration.yaml" />
-    <param name="frame_id" value="camera" />
-    <param name="image_width" value="1280" />
-    <param name="image_height" value="720" />
-    <param name="frame_rate" value="30.0" />
-  </node>
-
-  <node pkg="tf" type="static_transform_publisher" name="onnx_link"
-    args="0 -0.02  0 0 0 0 map base_link 100" />  
-
-</launch>
-```
 
 > While 'Pose' processing is enabled, the service required to generate the model has not been published as of April 2021
 
 ## Property Descriptions
 
-| Property | Description |
-|----------| ------------|
-| onnx_model_path | Path to the model.onnx file | 
-| confidence | Minimum confidence before publishing an event. 0 to 1 |
-| tensor_width| The Width of the input to the model. |
-| tensor_height| The Height of the input to the model. |
-| tracker_type| Currently enabled - `yolo` or `pose`. |
-| image_processing| `resize`, `scale` or `crop` |
-| debug| `true` or `false` determines if a debug image is published |
-| image_topic| The image topic to subscribe to |
-| label | used to filter the found object to a specific label |
-| mesh_rotation| The orientation of the mesh when debug rendering pose |
-| mesh_scale| The scale of the mesh when debug rendering pose |
-| mesh_resource| The mesh used for debug rendering pose |
-| model_bounds| 9 coordinates used to perform the point in perspective caluclation for pose |
-| calibration | Path to the OpenCV calibration file for point in persective |
+| Property         | Description                                                                 |
+| ---------------- | --------------------------------------------------------------------------- |
+| onnx_model_path  | Path to the model.onnx file                                                 |
+| confidence       | Minimum confidence before publishing an event. 0 to 1                       |
+| tensor_width     | The Width of the input to the model.                                        |
+| tensor_height    | The Height of the input to the model.                                       |
+| tracker_type     | Currently enabled - `yolo` or `pose`.                                       |
+| image_processing | `resize`, `scale` or `crop`                                                 |
+| debug            | `true` or `false` determines if a debug image is published                  |
+| image_topic      | The image topic to subscribe to                                             |
+| label            | used to filter the found object to a specific label                         |
+| mesh_rotation    | The orientation of the mesh when debug rendering pose                       |
+| mesh_scale       | The scale of the mesh when debug rendering pose                             |
+| mesh_resource    | The mesh used for debug rendering pose                                      |
+| model_bounds     | 9 coordinates used to perform the point in perspective caluclation for pose |
+| calibration      | Path to the OpenCV calibration file for point in persective                 |
 
 ## Subscriptions
 Onnx subscribes to the topic listed in the `image_topic` property, or `/camera/image_raw`
@@ -167,4 +146,4 @@ provided by the bot. You will only need to do this once across all repos using o
 
 This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or
-contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
+contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.
diff --git a/ros_msft_onnx/CMakeLists.txt b/ros_msft_onnx/CMakeLists.txt
@@ -23,20 +23,28 @@ find_package(Eigen3 REQUIRED)
 
 find_package(OpenCV REQUIRED)
 
+find_package(yaml-cpp CONFIG REQUIRED)
+
 find_package(catkin REQUIRED COMPONENTS
   std_msgs
   geometry_msgs
   visualization_msgs
   ros_msft_onnx_msgs
   image_transport
+  dynamic_reconfigure
+  yaml-cpp
   roscpp
   cv_bridge
   tf
 )
 
+generate_dynamic_reconfigure_options(
+  cfg/reconfig.cfg
+)
+
 catkin_package(
   INCLUDE_DIRS include
-  CATKIN_DEPENDS std_msgs geometry_msgs visualization_msgs ros_msft_onnx_msgs image_transport roscpp cv_bridge tf
+  CATKIN_DEPENDS std_msgs geometry_msgs visualization_msgs ros_msft_onnx_msgs image_transport dynamic_reconfigure roscpp cv_bridge tf
   CFG_EXTRAS  "onnxruntime_vendor-extras.cmake"
 )
 
@@ -47,19 +55,19 @@ include_directories(
 )
 
 add_executable(${PROJECT_NAME}_node src/ros_msft_onnx.cpp src/main.cpp src/yolo_box.cpp src/pose_parser.cpp)
-add_dependencies(${PROJECT_NAME}_node ${catkin_EXPORTED_TARGETS})
-target_link_libraries(${PROJECT_NAME}_node ${catkin_LIBRARIES} ${OpenCV_LIBRARIES} ${EIGEN3_LIBS})
+add_dependencies(${PROJECT_NAME}_node ${PROJECT_NAME}_gencfg ${catkin_EXPORTED_TARGETS})
+target_link_libraries(${PROJECT_NAME}_node ${catkin_LIBRARIES} ${OpenCV_LIBRARIES} ${EIGEN3_LIBS} ${YAML_CPP_LIBRARIES})
 
 message("Installing onnxruntime_vendor Nuget package")
 
 if(CUDA_SUPPORT)
-  set(ONNX_RUNTIME "Microsoft.ML.OnnxRuntime.Gpu.1.4.0")
-  set(PACKAGE_URL "https://www.nuget.org/api/v2/package/Microsoft.ML.OnnxRuntime.Gpu/1.4.0")
-  set(PACKAGE_SHA512 "c9c2ba5c594c92c1e426e9c53f9909e8851a41c99f48f8a369e082f8047d521b236f2fbb943e73975cbb45bd9957f20139c25959e50e1679dca9eeac08f73b31")
+  set(ONNX_RUNTIME "Microsoft.ML.OnnxRuntime.Gpu.1.7.1")
+  set(PACKAGE_URL "https://www.nuget.org/api/v2/package/Microsoft.ML.OnnxRuntime.Gpu/1.7.1")
+  set(PACKAGE_SHA512 "41112118007aae34fcc38100152df6e6fa5fc567e61aa4ded42a26d39751f1be7ec225c0d73799f065015e284f0fb9bd7e0835c733e9abad5b0243a391411f8d")
 else()
-  set(ONNX_RUNTIME "Microsoft.ML.OnnxRuntime.1.4.0")
-  set(PACKAGE_URL "https://www.nuget.org/api/v2/package/Microsoft.ML.OnnxRuntime/1.4.0")
-  set(PACKAGE_SHA512 "cbff106bb1f114ee1d510b594abb487e9f965b7b7a3f37b92846013eb086126a4cd69eb4564717fe1acf04d4399d1fcd0a52c3ca508f330e91a3e5d0fe560ca3")
+  set(ONNX_RUNTIME "Microsoft.ML.OnnxRuntime.1.7.0")
+  set(PACKAGE_URL "https://www.nuget.org/api/v2/package/Microsoft.ML.OnnxRuntime/1.7.0")
+  set(PACKAGE_SHA512 "1fc15386bdfa455f457e50899e3c9c454aafbdc345799dcf4ecfd6990a9dbd8cd7f0b1f3bf412c47c900543c535f95aa1cb1e14e9851cb9b600c60a981f38a50")
 endif()
 
 file(DOWNLOAD
@@ -95,8 +103,8 @@ if(MSVC)
   configure_file(${CMAKE_CURRENT_BINARY_DIR}/${ONNX_RUNTIME}/runtimes/${ARCH}/native/onnxruntime.lib ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/onnxruntime.lib COPYONLY)
   configure_file(${CMAKE_CURRENT_BINARY_DIR}/${ONNX_RUNTIME}/runtimes/${ARCH}/native/onnxruntime.pdb ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/onnxruntime.pdb COPYONLY)
 else()
-  configure_file(${CMAKE_CURRENT_BINARY_DIR}/${ONNX_RUNTIME}/runtimes/${ARCH}/native/libonnxruntime.so ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${PROJECT_NAME}/libonnxruntime.so.1.4.0 COPYONLY)
-  target_link_libraries(${PROJECT_NAME}_node ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/${PROJECT_NAME}/libonnxruntime.so.1.4.0)
+  target_link_libraries(${PROJECT_NAME}_node ${CMAKE_CURRENT_BINARY_DIR}/${ONNX_RUNTIME}/runtimes/${ARCH}/native/libonnxruntime.so)
+  configure_file(${CMAKE_CURRENT_BINARY_DIR}/${ONNX_RUNTIME}/runtimes/${ARCH}/native/libonnxruntime.so ${CMAKE_RUNTIME_OUTPUT_DIRECTORY}/libonnxruntime.so COPYONLY)
 endif()
 
 # The node expects to use the Tiny YOLO model available in the ONNX model zoo.
@@ -106,4 +114,4 @@ file(DOWNLOAD
   ${CMAKE_CURRENT_SOURCE_DIR}/testdata/model.onnx
   SHOW_PROGRESS
 )
-endif()
+endif()
diff --git a/ros_msft_onnx/cfg/anchors.yaml b/ros_msft_onnx/cfg/anchors.yaml
@@ -0,0 +1,2 @@
+anchors: [1.08, 1.19, 3.42, 4.41, 6.63, 11.38, 9.42, 5.11, 16.62, 10.52]
+# anchors: [0.573, 0.677, 1.87, 2.06, 3.34, 5.47, 7.88, 3.53, 9.77, 9.17]
diff --git a/ros_msft_onnx/cfg/reconfig.cfg b/ros_msft_onnx/cfg/reconfig.cfg
@@ -0,0 +1,25 @@
+#!/usr/bin/env python
+PACKAGE = "ros_msft_onnx"
+ 
+from dynamic_reconfigure.parameter_generator_catkin import *
+import os
+
+os.chdir('../../../')
+path = os.getcwd()
+
+newPath = path.replace(os.sep, '/') + "/src/ros_msft_onnx/ros_msft_onnx/"
+gen = ParameterGenerator()
+ 
+gen.add("tensor_height",    int_t,    0, "The height of the tensor", 416, 0, 3072)
+gen.add("tensor_width",    int_t,    0, "The width of the tensor", 416, 0, 4096)
+gen.add("confidence", double_t, 0, "The confidence parameter",    .5, 0, 1)
+gen.add("onnx_model_path",    str_t,    0, "The path of the onnx model",  newPath + "testdata/model.onnx")
+gen.add("onnx_label_path",    str_t,    0, "The path of the class labels",  newPath + "testdata/labels.txt")
+gen.add("anchors_path", str_t,    0, "The path of the anchors.yaml file", newPath + "cfg/anchors.yaml")
+gen.add("tracker_type",    str_t,    0, "Name of the tracker",  "yolo")
+gen.add("image_processing",    str_t,    0, "Changes the image processing type",  "resize")
+gen.add("input_node_name",    str_t,    0, "Name of the input for onnx model",  "image")
+gen.add("output_node_name",    str_t,    0, "Name of the output for onnx model",  "grid")
+gen.add("debug",   bool_t,   0, "Sets the debug value",  True)
+
+exit(gen.generate(PACKAGE, "ros_msft_onnx", "reconfig"))
diff --git a/ros_msft_onnx/include/ros_msft_onnx/ros_msft_onnx.h b/ros_msft_onnx/include/ros_msft_onnx/ros_msft_onnx.h
@@ -1,7 +1,9 @@
 #pragma once
 
 #include <onnxruntime_cxx_api.h>
-
+#include <dynamic_reconfigure/server.h>
+#include <ros_msft_onnx/reconfigConfig.h>
+#include <yaml-cpp/yaml.h>
 class OnnxProcessor
 {
 public:
@@ -39,65 +41,43 @@ class OnnxProcessor
 
     std::string _linkName;
     std::string _onnxModel;
-    std::string _calibration;
-
-    cv::Mat _camera_matrix;
-    cv::Mat _dist_coeffs;
 
-    float _confidence;
-
-    bool _debug;
-    bool _normalize;
-
-    ros::Publisher _detect_pub;
-    image_transport::Publisher _image_pub;
-    image_transport::Publisher _debug_image_pub;
-    image_transport::Subscriber _cameraSub;
-/* TODO (lilustga): remove this
-    bool _fake;
-    winrt::hstring _inName;
-    winrt::hstring _outName;
-    std::string frame_id_;
-    std::string _onnxModel;
+    std::string _imageProcessingType;
     std::string _calibration;
 
     cv::Mat _camera_matrix;
     cv::Mat _dist_coeffs;
 
-    
     float _confidence;
 
     bool _debug;
     bool _normalize;
 
-    uint _tensorWidth;
-    uint _tensorHeight;
-
-    int _channelCount;
-    int _rowCount;
-    int _colCount;
-    winrt::Windows::AI::MachineLearning::LearningModel _model = nullptr;
-    winrt::Windows::AI::MachineLearning::LearningModelSession _session = nullptr;
 
     ros::Publisher _detect_pub;
     image_transport::Publisher _image_pub;
     image_transport::Publisher _debug_image_pub;
     image_transport::Subscriber _cameraSub;
-*/
 
 };
 
 class OnnxTracker
 {
     ros::NodeHandle _nh;
     ros::NodeHandle _nhPrivate;
+    dynamic_reconfigure::Server<ros_msft_onnx::reconfigConfig> server;
+    dynamic_reconfigure::Server<ros_msft_onnx::reconfigConfig>::CallbackType f;
+    bool _status;
 
     std::shared_ptr<OnnxProcessor> _processor;
 
 public:
     OnnxTracker() { };
 
     bool init(ros::NodeHandle& nh, ros::NodeHandle& nhPrivate);
+    void callback(ros_msft_onnx::reconfigConfig &config, uint32_t level);
+    void startProcessor(ros_msft_onnx::reconfigConfig &config); 
+    void stopProcessor(); 
     bool shutdown();
 };
 
diff --git a/ros_msft_onnx/include/ros_msft_onnx/yolo_box.h b/ros_msft_onnx/include/ros_msft_onnx/yolo_box.h
@@ -7,11 +7,6 @@
 
 namespace yolo
 {
-    struct YoloInitOptions
-    {
-        std::string modelFullPath;
-    };
-
     struct YoloBox
     {
     public:
@@ -21,7 +16,15 @@ namespace yolo
 
     class YoloProcessor : public OnnxProcessor
     {
-        std::string _label;
+        std::string _labelPath;
+        std::vector<float> _anchors;
+        std::vector<std::string> _labels;
+        std::string _anchorsPath;
+        int _class_count;
+        int _row_count;
+        int _col_count;
+        std::string _inputName;
+        std::string _outputName;
     public:
         YoloProcessor();
         
@@ -30,7 +33,7 @@ namespace yolo
         std::vector<YoloBox> GetRecognizedObjects(std::vector<float> modelOutputs, float threshold = 0.3f);
         virtual void ProcessOutput(std::vector<float> output, cv::Mat& image);
     private:
-        static int GetOffset(int x, int y, int channel);
+        int GetOffset(int x, int y, int channel);
         static float IntersectionOverUnion(YoloBox a, YoloBox b);
         static float Sigmoid(float value);
         static void Softmax(std::vector<float> &values);
diff --git a/ros_msft_onnx/launch/tracker.launch b/ros_msft_onnx/launch/tracker.launch
@@ -1,16 +1,8 @@
 <launch>
-  <arg name="onnx_model_path_arg" default="$(find ros_msft_onnx)/testdata/model.onnx"/>
   <arg name="os_windows_arg" value="$(eval 'false' if not optenv('OS', 'unknown').lower().startswith('windows') else 'true')" />
 
   <node pkg="ros_msft_onnx" type="ros_msft_onnx_node" name="ros_msft_onnx" output="screen">
-    <param name="onnx_model_path" value="$(arg onnx_model_path_arg)"/>
-    <param name="confidence" value="0.5"/>
-    <param name="tensor_width" value="416"/>
-    <param name="tensor_height" value="416"/>
-    <param name="tracker_type" value="yolo"/>
-    <param name="image_processing" value="resize"/>
     <param name="image_topic" value="$(eval '/camera/image_raw' if os_windows_arg else '/cv_camera/image_raw')"/>
-    <param name="debug" value="true"/>
   </node>
   
   <!-- The camera node will be selected based on os. ros_msft_camera for Windows and cv_camera for others. -->
diff --git a/ros_msft_onnx/package.xml b/ros_msft_onnx/package.xml
@@ -16,6 +16,7 @@
   <depend>cv_bridge</depend>
   <depend>roscpp</depend>
   <depend>tf</depend>
+  <depend>dynamic_reconfigure</depend>
   <depend>image_transport</depend>
   <depend>onnxruntime_vendor</depend>
 
diff --git a/ros_msft_onnx/src/main.cpp b/ros_msft_onnx/src/main.cpp
@@ -24,6 +24,7 @@ int main(int argc, char **argv)
     ros::NodeHandle nhPrivate("~");
 
     OnnxTracker tracker;
+     
 
     if (tracker.init(nh, nhPrivate))
     {
@@ -37,4 +38,6 @@ int main(int argc, char **argv)
     {
         return 1;
     }
+
+
 }
diff --git a/ros_msft_onnx/src/ros_msft_onnx.cpp b/ros_msft_onnx/src/ros_msft_onnx.cpp
diff --git a/ros_msft_onnx/src/yolo_box.cpp b/ros_msft_onnx/src/yolo_box.cpp
diff --git a/ros_msft_onnx/testdata/labels.txt b/ros_msft_onnx/testdata/labels.txt
diff --git a/ros_msft_onnx/testdata/rqt_reconfigure.PNG b/ros_msft_onnx/testdata/rqt_reconfigure.PNG

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,2 @@`
	`1`	`+anchors: [1.08, 1.19, 3.42, 4.41, 6.63, 11.38, 9.42, 5.11, 16.62, 10.52]`
	`2`	`+# anchors: [0.573, 0.677, 1.87, 2.06, 3.34, 5.47, 7.88, 3.53, 9.77, 9.17]`
Original file line number	Diff line number	Diff line change
`@@ -24,6 +24,7 @@ int main(int argc, char **argv)`
`24`	`24`	`ros::NodeHandle nhPrivate("~");`
`25`	`25`
`26`	`26`	`OnnxTracker tracker;`
	`27`	`+`
`27`	`28`
`28`	`29`	`if (tracker.init(nh, nhPrivate))`
`29`	`30`	`{`
`@@ -37,4 +38,6 @@ int main(int argc, char **argv)`
`37`	`38`	`{`
`38`	`39`	`return 1;`
`39`	`40`	`}`
	`41`	`+`
	`42`	`+`
`40`	`43`	`}`