Dev (#197)

* readme

* add openvino

* refactor format

Author: CoinCheung

.gitignore

@@ -115,5 +115,7 @@ datasets/coco/train.txt
 datasets/coco/val.txt
 pretrained/*
 dist_train.sh
+openvino/build/*
+openvino/output*
 
 tvm/

README.md

@@ -1,6 +1,6 @@
 # BiSeNetV1 & BiSeNetV2
 
-My implementation of [BiSeNetV1](https://arxiv.org/abs/1808.00897) and [BiSeNetV2](https://arxiv.org/abs/1808.00897).
+My implementation of [BiSeNetV1](https://arxiv.org/abs/1808.00897) and [BiSeNetV2](https://arxiv.org/abs/2004.02147).
 
 
 mIOUs and fps on cityscapes val set:
@@ -9,33 +9,39 @@ mIOUs and fps on cityscapes val set:
 | bisenetv1 | 75.44 | 76.94 | 77.45 | 78.86 | 68/23 | [download](https://github.com/CoinCheung/BiSeNet/releases/download/0.0.0/model_final_v1_city_new.pth) |
 | bisenetv2 | 74.95 | 75.58 | 76.53 | 77.08 | 59/21 | [download](https://github.com/CoinCheung/BiSeNet/releases/download/0.0.0/model_final_v2_city.pth) |
 
-
 mIOUs on cocostuff val2017 set:
 | none | ss | ssc | msf | mscf | link |
 |------|:--:|:---:|:---:|:----:|:----:|
 | bisenetv1 | 31.49 | 31.42 | 32.46 | 32.55 | [download](https://github.com/CoinCheung/BiSeNet/releases/download/0.0.0/model_final_v1_coco_new.pth) |
 | bisenetv2 | 30.49 | 30.55 | 31.81 | 31.73 | [download](https://github.com/CoinCheung/BiSeNet/releases/download/0.0.0/model_final_v2_coco.pth) |
 
 Tips: 
+
 1. **ss** means single scale evaluation, **ssc** means single scale crop evaluation, **msf** means multi-scale evaluation with flip augment, and **mscf** means multi-scale crop evaluation with flip evaluation. The eval scales and crop size of multi-scales evaluation can be found in [configs](./configs/).
 
 2. The fps is tested in different way from the paper. For more information, please see [here](./tensorrt).
 
-3. For cocostuff dataset: The authors of the paper `bisenetv2` used the "old split" of 9k train set and 1k val set, while I used the "new split" of 118k train set and 5k val set. Thus the above results on cocostuff does not match the paper. The authors of bisenetv1 did not report their results on cocostuff, so here I simply provide a "make it work" result. Following the tradition of object detection, I used "1x"(90k) and "2x"(180k) schedule to train bisenetv1(1x) and bisenetv2(2x) respectively. Maybe you can have a better result by picking up hyper-parameters more carefully.
+3. The authors of bisenetv2 used cocostuff-10k, while I used cocostuff-123k(do not know how to say, just same 118k train and 5k val images as object detection). Thus the results maybe different from paper. 
 
 4. The model has a big variance, which means that the results of training for many times would vary within a relatively big margin. For example, if you train bisenetv2 for many times, you will observe that the result of **ss** evaluation of bisenetv2 varies between 73.1-75.1. 
 
 
 ## deploy trained models
+
 1. tensorrt  
-You can go to [tensorrt](./tensorrt) for details.
+You can go to [tensorrt](./tensorrt) for details.  
 
 2. ncnn  
-You can go to [ncnn](./ncnn) for details.
+You can go to [ncnn](./ncnn) for details.  
+
+3. openvino  
+You can go to [openvino](./openvino) for details.  
 
 
 ## platform
+
 My platform is like this: 
+
 * ubuntu 18.04
 * nvidia Tesla T4 gpu, driver 450.51.05
 * cuda 10.2
@@ -45,10 +51,13 @@ My platform is like this:
 
 
 ## get start
+
 With a pretrained weight, you can run inference on an single image like this: 
+
 ```
 $ python tools/demo.py --config configs/bisenetv2_city.py --weight-path /path/to/your/weights.pth --img-path ./example.png
 ```
+
 This would run inference on the image and save the result image to `./res.jpg`.
 
 
@@ -95,7 +104,9 @@ Then you need to change the field of `im_root` and `train/val_im_anns` in the co
 
 
 ## train
+
 I used the following command to train the models:
+
 ```bash
 # bisenetv1 cityscapes
 export CUDA_VISIBLE_DEVICES=0,1
@@ -128,6 +139,7 @@ Note:
 
 
 ## finetune from trained model
+
 You can also load the trained model weights and finetune from it, like this:
 ```
 $ export CUDA_VISIBLE_DEVICES=0,1

ncnn/README.md

@@ -14,9 +14,9 @@ Though this demo runs on x86 platform, you can also use it on mobile platforms.
 ### Install ncnn
 
 #### 1. dependencies  
-  ```
-  $ python -m pip install onnx-simplifier
-  ```
+```
+$ python -m pip install onnx-simplifier
+```
 
 #### 2. build ncnn  
 Just follow the ncnn official tutoral of [build-for-linux](https://github.com/Tencent/ncnn/wiki/how-to-build#build-for-linux) to install ncnn:

openvino/CMakeLists.txt

@@ -0,0 +1,35 @@
+
+cmake_minimum_required (VERSION 3.10)
+
+cmake_policy(SET CMP0025 NEW)
+
+project(Samples)
+
+
+set (CMAKE_CXX_STANDARD 14)
+set(CMAKE_BUILD_TYPE "Release")
+set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -Wuninitialized -Winit-self")
+set (CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wall")
+
+
+find_package(OpenCV REQUIRED)
+find_package(InferenceEngine REQUIRED)
+find_package(ngraph REQUIRED)
+
+
+include_directories(
+    ${CMAKE_CURRENT_SOURCE_DIR}
+    ${CMAKE_CURRENT_BINARY_DIR}
+    ${OpenCV_INCLUDE_DIRS}
+)
+
+add_executable(segment main.cpp)
+target_link_libraries(
+    segment
+    ${InferenceEngine_LIBRARIES}
+    ${NGRAPH_LIBRARIES}
+    ${OpenCV_LIBS}
+)
+
+
+

openvino/README.md

@@ -0,0 +1,92 @@
+
+
+## A demo of using openvino to deploy
+
+Openvino is used to deploy model on intel cpus or "gpu inside cpu".  
+
+My cpu is Intel(R) Xeon(R) Gold 6240 CPU @ 2.60GHz.
+
+
+### preparation
+
+1.Train the model and export it to onnx  
+```
+$ cd BiSeNet/
+$ python tools/export_onnx.py --aux-mode eval --config configs/bisenetv2_city.py --weight-path /path/to/your/model.pth --outpath ./model_v2.onnx 
+```
+(Optional) 2.Install 'onnx-simplifier' to simplify the generated onnx model:
+```
+$ python -m pip install onnx-simplifier
+$ python -m onnxsim model_v2.onnx model_v2_sim.onnx
+```
+
+
+### Install and configure openvino
+
+1.pull docker image  
+```
+$ docker pull openvino/ubuntu18_dev
+```
+
+2.start a docker container and mount code into it  
+```
+$ docker run -itu root -v /path/to/BiSeNet:/BiSeNet openvino/ubuntu18_dev --device /dev/dri:/dev/dri bash
+
+```
+If your cpu does not have intel "gpu inside of cpu" or you do not want to use it, you can remove the option of `--device /dev/dri:/dev/dri`.  
+
+After running the above command, you will be in the container.  
+
+(optional) 3.install gpu dependencies  
+If you want to use gpu, you also need to install some dependencies inside the container:
+```
+# mkdir -p /tmp/opencl && cd /tmp/opencl
+# useradd -ms /bin/bash -G video,users openvino
+# chown openvino -R /home/openvino
+# apt update
+# apt install -y --no-install-recommends ocl-icd-libopencl1
+# curl -L "https://github.com/intel/compute-runtime/releases/download/19.41.14441/intel-gmmlib_19.3.2_amd64.deb" --output "intel-gmmlib_19.3.2_amd64.deb" 
+# curl -L "https://github.com/intel/compute-runtime/releases/download/19.41.14441/intel-igc-core_1.0.2597_amd64.deb" --output "intel-igc-core_1.0.2597_amd64.deb" 
+# curl -L "https://github.com/intel/compute-runtime/releases/download/19.41.14441/intel-igc-opencl_1.0.2597_amd64.deb" --output "intel-igc-opencl_1.0.2597_amd64.deb"
+# curl -L "https://github.com/intel/compute-runtime/releases/download/19.41.14441/intel-opencl_19.41.14441_amd64.deb" --output "intel-opencl_19.41.14441_amd64.deb" 
+# curl -L "https://github.com/intel/compute-runtime/releases/download/19.41.14441/intel-ocloc_19.41.14441_amd64.deb" --output "intel-ocloc_19.04.12237_amd64.deb" 
+# dpkg -i /tmp/opencl/*.deb
+# apt --fix-broken install
+# ldconfig
+```
+
+I got the above commands from the official docs but I did not test it since my cpu does not have integrated gpu.  
+
+
+4.configure environment  
+just run this script, and the environment would be ready:  
+```
+# source /opt/intel/openvino_2021.4.689/bin/setupvars.sh
+```
+
+
+### convert model and run demo
+
+1.convert onnx to openvino IR  
+In the docker container:  
+```
+# cd /opt/intel/openvino_2021.4.689/deployment_tools/model_optimizer
+# python3 mo.py --input_model /BiSeNet/model_v2.onnx --output_dir /BiSeNet/openvino/output_v2
+```
+
+2.compile and run the demo
+```
+# cd /BiSeNet/openvino
+# mkdir -p build && cd build
+# cmake .. && make
+# ./segment
+```
+After this, you will see a segmentation result image named `res.jpg` generated.
+
+
+
+### Tipes
+
+1.GPU support: openvino supports intel cpu and intel "gpu inside cpu". Until now(2021.11), other popular isolated gpus are not supported, such as nvidia/amd gpus. Also, other integrated gpus are not supported, such as aspeed graphics family.
+
+2.About low-precision: precision is optimized automatically, and the model will be run in one or several precision mode. We can also manually enforce to use bf16, as long as our cpu have `avx512_bf16` supports. If cpu does not support bf16, it will use simulation which would slow down the inference. If neither native bf16 nor simulation is supported, an error would occur.