Skip to content

Commit 9a442ab

Browse files
Li-HongdaLi-Hongda
authored
CodeCamp #106: Support SOLOv2 deployment with OpenVINO (#1521)
* support solov2 openvino * fix lint * Update benchmark.md * Update __init__.py * fix names and delete redundant codes * fix names and redundant codes * fix indent * fix lint * fix lint * delete 'ctx' arguments * [Fix]fix lint * 修改： mmdeploy/codebase/mmdet/models/dense_heads/solov2_head.py 修改： mmdeploy/codebase/mmdet/models/detectors/single_stage_instance_seg.py * Delete matrix_nms.py * Update __init__.py * Update linspace.py * Update solov2_head.py * Update solov2_head.py * update * fix num_groups * update docs * add end2endmodel for solo * add end2endmodel for solo * [Feature] Support End2EndModelSOLO * Update object_detection_model.py * add model_cfg to End2EndModel * add model_cfg to End2EndModel * Fix bugs for End2EndModel * fix bugs for End2EndModel * update docs. * update conmment * 更新 benchmark.md * update regression tests * fix errors in docs
1 parent 6bccf05 commit 9a442ab

File tree

14 files changed

+352
-44
lines changed

14 files changed

+352
-44
lines changed

‎docs/en/03-benchmark/benchmark.md‎

Lines changed: 21 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -613,6 +613,7 @@ Users can directly test the performance through [how_to_evaluate_a_model.md](../
613613
<td align="center">81.18</td>
614614
<td align="center">-</td>
615615
<td align="center">-</td>
616+
<td align="center">-</td>
616617
</tr>
617618
<tr>
618619
<td align="center">top-5</td>
@@ -623,6 +624,7 @@ Users can directly test the performance through [how_to_evaluate_a_model.md](../
623624
<td align="center">95.61</td>
624625
<td align="center">-</td>
625626
<td align="center">-</td>
627+
<td align="center">-</td>
626628
</tr>
627629
</tbody>
628630
</table>
@@ -907,20 +909,35 @@ Users can directly test the performance through [how_to_evaluate_a_model.md](../
907909
<td align="center">-</td>
908910
</tr>
909911
<tr>
910-
<td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo">SOLO</a></td>
911-
<td align="center" rowspan="2">Instance Segmentation</td>
912-
<td align="center" rowspan="2">COCO2017</td>
912+
<td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo/solo_r50_fpn_1x_coco.py">SOLO</a></td>
913+
<td align="center">Instance Segmentation</td>
914+
<td align="center">COCO2017</td>
913915
<td align="center">mask AP</td>
914916
<td align="center">33.1</td>
915917
<td align="center">-</td>
916-
<td align="center">-</td>
918+
<td align="center">32.7</td>
917919
<td align="center">-</td>
918920
<td align="center">-</td>
919921
<td align="center">-</td>
920922
<td align="center">-</td>
921923
<td align="center">-</td>
922924
<td align="center">32.7</td>
923925
</tr>
926+
<tr>
927+
<td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2/solov2_r50_fpn_1x_coco.py">SOLOv2</a></td>
928+
<td align="center">Instance Segmentation</td>
929+
<td align="center">COCO2017</td>
930+
<td align="center">mask AP</td>
931+
<td align="center">34.8</td>
932+
<td align="center">-</td>
933+
<td align="center">34.5</td>
934+
<td align="center">-</td>
935+
<td align="center">-</td>
936+
<td align="center">-</td>
937+
<td align="center">-</td>
938+
<td align="center">-</td>
939+
<td align="center">34.5</td>
940+
</tr>
924941
</tbody>
925942
</table>
926943
</div>

‎docs/en/03-benchmark/supported_models.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@ The table below lists the models that are guaranteed to be exportable to other b
2222
| [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints) | MMDetection | N | N | Y | N | ? | Y | N | N |
2323
| [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr) | MMDetection | N | Y | Y | N | ? | N | N | N |
2424
| [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet) | MMDetection | N | Y | Y | N | ? | N | N | N |
25-
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | MMDetection | N | N | N | N | N | Y | N | N |
25+
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | MMDetection | N | Y | N | N | N | Y | N | N |
26+
| [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2) | MMDetection | N | Y | N | N | N | Y | N | N |
2627
| [ResNet](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/resnet) | MMClassification | Y | Y | Y | Y | Y | Y | Y | Y |
2728
| [ResNeXt](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/resnext) | MMClassification | Y | Y | Y | Y | Y | Y | Y | Y |
2829
| [SE-ResNet](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/seresnet) | MMClassification | Y | Y | Y | Y | Y | Y | Y | Y |

‎docs/en/04-supported-codebases/mmdet.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -210,4 +210,5 @@ Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Inter
210210
| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | Instance Segmentation | Y | N | N | N | Y |
211211
| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn) | Instance Segmentation | Y | Y | N | N | Y |
212212
| [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin) | Instance Segmentation | Y | Y | N | N | N |
213-
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | Instance Segmentation | N | N | N | N | Y |
213+
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | Instance Segmentation | Y | N | N | N | Y |
214+
| [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2) | Instance Segmentation | Y | N | N | N | Y |

‎docs/zh_cn/03-benchmark/benchmark.md‎

Lines changed: 20 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -633,7 +633,7 @@ GPU: ncnn, TensorRT, PPLNN
633633
<th align="center" colspan="3">TensorRT</th>
634634
<th align="center">PPLNN</th>
635635
<th align="center">Ascend</th>
636-
<th algin="center">OpenVINO</th>
636+
<th align="center">OpenVINO</th>
637637
</tr>
638638
</thead>
639639
<tbody>
@@ -902,20 +902,35 @@ GPU: ncnn, TensorRT, PPLNN
902902
<td align="center">-</td>
903903
</tr>
904904
<tr>
905-
<td align="center" rowspan="2"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo">SOLO</a></td>
906-
<td align="center" rowspan="2">Instance Segmentation</td>
907-
<td align="center" rowspan="2">COCO2017</td>
905+
<td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo/solo_r50_fpn_1x_coco.py">SOLO</a></td>
906+
<td align="center">Instance Segmentation</td>
907+
<td align="center">COCO2017</td>
908908
<td align="center">mask AP</td>
909909
<td align="center">33.1</td>
910910
<td align="center">-</td>
911-
<td align="center">-</td>
911+
<td align="center">32.7</td>
912912
<td align="center">-</td>
913913
<td align="center">-</td>
914914
<td align="center">-</td>
915915
<td align="center">-</td>
916916
<td align="center">-</td>
917917
<td align="center">32.7</td>
918918
</tr>
919+
<tr>
920+
<td align="center"><a href="https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2/solov2_r50_fpn_1x_coco.py">SOLOv2</a></td>
921+
<td align="center">Instance Segmentation</td>
922+
<td align="center">COCO2017</td>
923+
<td align="center">mask AP</td>
924+
<td align="center">34.8</td>
925+
<td align="center">-</td>
926+
<td align="center">34.5</td>
927+
<td align="center">-</td>
928+
<td align="center">-</td>
929+
<td align="center">-</td>
930+
<td align="center">-</td>
931+
<td align="center">-</td>
932+
<td align="center">34.5</td>
933+
</tr>
919934
</tbody>
920935
</table>
921936
</div>

‎docs/zh_cn/03-benchmark/supported_models.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,8 @@
2222
| [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints) | MMDetection | N | N | Y | N | ? | Y | N | N |
2323
| [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr) | MMDetection | N | Y | Y | N | ? | N | N | N |
2424
| [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet) | MMDetection | N | Y | Y | N | ? | N | N | N |
25-
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | MMDetection | N | N | N | N | N | Y | N | N |
25+
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | MMDetection | N | Y | N | N | N | Y | N | N |
26+
| [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2) | MMDetection | N | Y | N | N | N | Y | N | N |
2627
| [ResNet](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/resnet) | MMClassification | Y | Y | Y | Y | Y | Y | Y | Y |
2728
| [ResNeXt](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/resnext) | MMClassification | Y | Y | Y | Y | Y | Y | Y | Y |
2829
| [SE-ResNet](https://github.com/open-mmlab/mmclassification/tree/1.x/configs/seresnet) | MMClassification | Y | Y | Y | Y | Y | Y | Y | Y |

‎docs/zh_cn/04-supported-codebases/mmdet.md‎

Lines changed: 23 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -192,25 +192,26 @@ cv2.imwrite('output_detection.png', img)
192192

193193
## 模型支持列表
194194

195-
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
196-
| :-------------------------------------------------------------------------------------------: | :-------------------: | :---------: | :------: | :--: | :---: | :------: |
197-
| [ATSS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/atss) | ObjectDetection | Y | Y | N | N | Y |
198-
| [FCOS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fcos) | ObjectDetection | Y | Y | Y | N | Y |
199-
| [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/foveabox) | ObjectDetection | Y | N | N | N | Y |
200-
| [FSAF](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fsaf) | ObjectDetection | Y | Y | Y | Y | Y |
201-
| [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/retinanet) | ObjectDetection | Y | Y | Y | Y | Y |
202-
| [SSD](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/ssd) | ObjectDetection | Y | Y | Y | N | Y |
203-
| [VFNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/vfnet) | ObjectDetection | N | N | N | N | Y |
204-
| [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolo) | ObjectDetection | Y | Y | Y | N | Y |
205-
| [YOLOX](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolox) | ObjectDetection | Y | Y | Y | N | Y |
206-
| [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | ObjectDetection | Y | Y | N | Y | Y |
207-
| [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn) | ObjectDetection | Y | Y | Y | Y | Y |
208-
| [Faster R-CNN + DCN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn) | ObjectDetection | Y | Y | Y | Y | Y |
209-
| [GFL](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/gfl) | ObjectDetection | Y | Y | N | ? | Y |
210-
| [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints) | ObjectDetection | N | Y | N | ? | Y |
211-
| [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr) | ObjectDetection | Y | Y | N | ? | Y |
212-
| [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet) | Object Detection | Y | Y | N | ? | ? |
213-
| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | InstanceSegmentation | Y | N | N | N | Y |
214-
| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn) | InstanceSegmentation | Y | Y | N | N | Y |
215-
| [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin) | InstanceSegmentation | Y | Y | N | N | N |
216-
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | Instance Segmentation | N | N | N | N | Y |
195+
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
196+
| :-------------------------------------------------------------------------------------------: | :------------------: | :---------: | :------: | :--: | :---: | :------: |
197+
| [ATSS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/atss) | ObjectDetection | Y | Y | N | N | Y |
198+
| [FCOS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fcos) | ObjectDetection | Y | Y | Y | N | Y |
199+
| [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/foveabox) | ObjectDetection | Y | N | N | N | Y |
200+
| [FSAF](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fsaf) | ObjectDetection | Y | Y | Y | Y | Y |
201+
| [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/retinanet) | ObjectDetection | Y | Y | Y | Y | Y |
202+
| [SSD](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/ssd) | ObjectDetection | Y | Y | Y | N | Y |
203+
| [VFNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/vfnet) | ObjectDetection | N | N | N | N | Y |
204+
| [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolo) | ObjectDetection | Y | Y | Y | N | Y |
205+
| [YOLOX](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolox) | ObjectDetection | Y | Y | Y | N | Y |
206+
| [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | ObjectDetection | Y | Y | N | Y | Y |
207+
| [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn) | ObjectDetection | Y | Y | Y | Y | Y |
208+
| [Faster R-CNN + DCN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn) | ObjectDetection | Y | Y | Y | Y | Y |
209+
| [GFL](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/gfl) | ObjectDetection | Y | Y | N | ? | Y |
210+
| [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints) | ObjectDetection | N | Y | N | ? | Y |
211+
| [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr) | ObjectDetection | Y | Y | N | ? | Y |
212+
| [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet) | Object Detection | Y | Y | N | ? | ? |
213+
| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | InstanceSegmentation | Y | N | N | N | Y |
214+
| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn) | InstanceSegmentation | Y | Y | N | N | Y |
215+
| [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin) | InstanceSegmentation | Y | Y | N | N | N |
216+
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | InstanceSegmentation | Y | N | N | N | Y |
217+
| [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2) | InstanceSegmentation | Y | N | N | N | Y |

‎mmdeploy/codebase/mmdet/deploy/object_detection_model.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,11 +44,13 @@ def __init__(self,
4444
backend_files: Sequence[str],
4545
device: str,
4646
deploy_cfg: Union[str, Config],
47+
model_cfg: Optional[Union[str, Config]] = None,
4748
data_preprocessor: Optional[Union[dict, nn.Module]] = None,
4849
**kwargs):
4950
super().__init__(
5051
deploy_cfg=deploy_cfg, data_preprocessor=data_preprocessor)
5152
self.deploy_cfg = deploy_cfg
53+
self.model_cfg = model_cfg
5254
self.device = device
5355
self._init_wrapper(
5456
backend=backend, backend_files=backend_files, device=device)
@@ -195,6 +197,8 @@ def forward(self,
195197
img_metas = [data_sample.metainfo for data_sample in data_samples]
196198
results = []
197199
rescale = kwargs.get('rescale', True)
200+
model_type = self.model_cfg.model.type if \
201+
self.model_cfg is not None else None
198202
for i in range(batch_size):
199203
dets, labels = batch_dets[i], batch_labels[i]
200204
result = InstanceData()
@@ -234,6 +238,8 @@ def forward(self,
234238

235239
result.scores = scores
236240
result.bboxes = bboxes
241+
if model_type in ['SOLO', 'SOLOv2']:
242+
result.bboxes = bboxes.new_zeros(bboxes.shape)
237243
result.labels = labels
238244

239245
if batch_masks is not None:

‎mmdeploy/codebase/mmdet/models/dense_heads/__init__.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,5 +9,6 @@
99
from . import rtmdet_head # noqa: F401,F403
1010
from . import rtmdet_ins_head # noqa: F401,F403
1111
from . import solo_head # noqa: F401,F403
12+
from . import solov2_head # noqa: F401,F403
1213
from . import yolo_head # noqa: F401,F403
1314
from . import yolox_head # noqa: F401,F403

‎mmdeploy/codebase/mmdet/models/dense_heads/solo_head.py‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,14 +12,15 @@
1212

1313

1414
@FUNCTION_REWRITER.register_rewriter(
15-
'mmdet.models.dense_heads.SOLOHead.predict_by_feat', backend='openvino')
16-
def solohead__predict_by_feat__openvino(self,
17-
mlvl_mask_preds: List[Tensor],
18-
mlvl_cls_scores: List[Tensor],
19-
batch_img_metas: List[Dict],
20-
cfg: OptConfigType = None,
21-
**kwargs):
22-
"""Rewrite `predict_by_feat` of `SOLOHead` for openvino backend."""
15+
func_name='mmdet.models.dense_heads.solo_head.'
16+
'SOLOHead.predict_by_feat')
17+
def solohead__predict_by_feat(self,
18+
mlvl_mask_preds: List[Tensor],
19+
mlvl_cls_scores: List[Tensor],
20+
batch_img_metas: List[Dict],
21+
cfg: OptConfigType = None,
22+
**kwargs):
23+
"""Rewrite `predict_by_feat` of `SOLOHead` for default backend."""
2324

2425
ctx = FUNCTION_REWRITER.get_context()
2526
batch_size = mlvl_cls_scores[0].size(0)

0 commit comments

Comments
 (0)