Skip to content

Commit b7024dd

Browse files
RunningLeondaigo0927
RunningLeon
and
daigo0927
authored
[Feature]: Support deployment of panoptic segmentation models (#2347)
* add semantic segmentation head Author: Daigo Hirooka <[email protected]> Date: Sun Jun 26 18:25:29 2022 +0900 * add panoptic detection model * add panoptic segmentation configs * support panoptic-fpn * remove interpolate * update * support panoptic-fpn mask2former maskformer * update * support dynamic * update * remove unused rewritings for mask2former * Revert "remove unused rewritings for mask2former" This reverts commit 2b6d24a. * update configs and regs * debug dynamic * fix for panoptic-fpn * update * remove rewritings for mask2former * update reg test config * fix * update docs * fix comments * fix --------- Co-authored-by: Daigo Hirooka <[email protected]>
1 parent 4dbba14 commit b7024dd

22 files changed

+652
-106
lines changed

‎.github/scripts/prepare_reg_test.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,12 @@ def prepare_codebases(codebases):
9898
f'{MMDEPLOY_DIR}/configs/mmyolo')
9999
shutil.copy(f'{target_dir}/tests/regression/mmyolo.yml',
100100
f'{MMDEPLOY_DIR}/tests/regression/mmyolo.yml')
101+
elif codebase == 'mmdet':
102+
# for panoptic
103+
run_cmd([
104+
'python -m pip install ',
105+
'git+https://github.com/cocodataset/panopticapi.git',
106+
])
101107

102108

103109
def install_torch(torch_version):

‎configs/mmdet/_base_/base_panoptic-seg_static.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
_base_ = ['../../_base_/onnx_config.py']
2+
3+
codebase_config = dict(
4+
type='mmdet',
5+
task='ObjectDetection',
6+
model_type='panoptic_end2end',
7+
post_processing=dict(
8+
export_postprocess_mask=False,
9+
score_threshold=0.0,
10+
iou_threshold=0.5,
11+
max_output_boxes_per_class=200,
12+
pre_top_k=5000,
13+
keep_top_k=100,
14+
background_label_id=-1,
15+
))

‎configs/mmdet/panoptic-seg/panoptic-seg_maskformer_onnxruntime_dynamic.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
_base_ = [
2+
'./panoptic-seg_maskformer_onnxruntime_static-800x1344.py',
3+
]
4+
onnx_config = dict(
5+
dynamic_axes={
6+
'input': {
7+
0: 'batch',
8+
2: 'height',
9+
3: 'width'
10+
},
11+
'cls_logits': {
12+
0: 'batch',
13+
},
14+
'mask_logits': {
15+
0: 'batch',
16+
2: 'h',
17+
3: 'w',
18+
},
19+
},
20+
input_shape=None)

‎configs/mmdet/panoptic-seg/panoptic-seg_maskformer_onnxruntime_static-800x1344.py‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
_base_ = [
2+
'../_base_/base_panoptic-seg_static.py',
3+
'../../_base_/backends/onnxruntime.py'
4+
]
5+
onnx_config = dict(
6+
opset_version=13,
7+
output_names=['cls_logits', 'mask_logits'],
8+
input_shape=[1344, 800])

‎configs/mmdet/panoptic-seg/panoptic-seg_maskformer_tensorrt_dynamic-320x512-1344x1344.py‎

Lines changed: 27 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
_base_ = ['./panoptic-seg_maskformer_tensorrt_static-800x1344.py']
2+
onnx_config = dict(
3+
dynamic_axes={
4+
'input': {
5+
0: 'batch',
6+
2: 'height',
7+
3: 'width'
8+
},
9+
'cls_logits': {
10+
0: 'batch',
11+
},
12+
'mask_logits': {
13+
0: 'batch',
14+
2: 'h',
15+
3: 'w',
16+
},
17+
},
18+
input_shape=None)
19+
20+
backend_config = dict(model_inputs=[
21+
dict(
22+
input_shapes=dict(
23+
input=dict(
24+
min_shape=[1, 3, 320, 512],
25+
opt_shape=[1, 3, 800, 1344],
26+
max_shape=[1, 3, 1344, 1344])))
27+
])

‎configs/mmdet/panoptic-seg/panoptic-seg_maskformer_tensorrt_static-800x1344.py‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
_base_ = [
2+
'../_base_/base_panoptic-seg_static.py',
3+
'../../_base_/backends/tensorrt.py'
4+
]
5+
onnx_config = dict(
6+
opset_version=13,
7+
output_names=['cls_logits', 'mask_logits'],
8+
input_shape=[1344, 800])
9+
10+
backend_config = dict(
11+
common_config=dict(max_workspace_size=1 << 30),
12+
model_inputs=[
13+
dict(
14+
input_shapes=dict(
15+
input=dict(
16+
min_shape=[1, 3, 800, 1344],
17+
opt_shape=[1, 3, 800, 1344],
18+
max_shape=[1, 3, 800, 1344])))
19+
])

‎configs/mmdet/panoptic-seg/panoptic-seg_panoptic-fpn_onnxruntime_dynamic.py‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
_base_ = [
2+
'../_base_/base_panoptic-seg_static.py',
3+
'../../_base_/backends/onnxruntime.py'
4+
]
5+
onnx_config = dict(
6+
input_shape=None,
7+
output_names=['dets', 'labels', 'masks', 'semseg'],
8+
dynamic_axes={
9+
'input': {
10+
0: 'batch',
11+
2: 'height',
12+
3: 'width'
13+
},
14+
'dets': {
15+
0: 'batch',
16+
1: 'num_dets',
17+
},
18+
'labels': {
19+
0: 'batch',
20+
1: 'num_dets',
21+
},
22+
'masks': {
23+
0: 'batch',
24+
1: 'num_dets',
25+
},
26+
'semseg': {
27+
0: 'batch',
28+
2: 'height',
29+
3: 'width'
30+
},
31+
},
32+
)

‎configs/mmdet/panoptic-seg/panoptic-seg_panoptic-fpn_tensorrt_dynamic-352x512-1344x1344.py‎

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
_base_ = [
2+
'../_base_/base_panoptic-seg_static.py',
3+
'../../_base_/backends/tensorrt.py'
4+
]
5+
onnx_config = dict(
6+
input_shape=None,
7+
output_names=['dets', 'labels', 'masks', 'semseg'],
8+
dynamic_axes={
9+
'input': {
10+
0: 'batch',
11+
2: 'height',
12+
3: 'width'
13+
},
14+
'dets': {
15+
0: 'batch',
16+
1: 'num_dets',
17+
},
18+
'labels': {
19+
0: 'batch',
20+
1: 'num_dets',
21+
},
22+
'masks': {
23+
0: 'batch',
24+
1: 'num_dets',
25+
},
26+
'semseg': {
27+
0: 'batch',
28+
2: 'height',
29+
3: 'width'
30+
},
31+
},
32+
)
33+
34+
backend_config = dict(
35+
common_config=dict(max_workspace_size=1 << 30),
36+
model_inputs=[
37+
dict(
38+
input_shapes=dict(
39+
input=dict(
40+
min_shape=[1, 3, 352, 512],
41+
opt_shape=[1, 3, 800, 1344],
42+
max_shape=[1, 3, 1344, 1344])))
43+
])

‎docs/en/04-supported-codebases/mmdet.md‎

Lines changed: 33 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
- [Backend model inference](#backend-model-inference)
1111
- [SDK model inference](#sdk-model-inference)
1212
- [Supported models](#supported-models)
13+
- [Reminder](#reminder)
1314

1415
______________________________________________________________________
1516

@@ -189,27 +190,35 @@ Besides python API, mmdeploy SDK also provides other FFI (Foreign Function Inter
189190

190191
## Supported models
191192

192-
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
193-
| :-------------------------------------------------------------------------------------------: | :-------------------: | :---------: | :------: | :--: | :---: | :------: |
194-
| [ATSS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/atss) | Object Detection | Y | Y | N | N | Y |
195-
| [FCOS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fcos) | Object Detection | Y | Y | Y | N | Y |
196-
| [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/foveabox) | Object Detection | Y | N | N | N | Y |
197-
| [FSAF](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fsaf) | Object Detection | Y | Y | Y | Y | Y |
198-
| [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/retinanet) | Object Detection | Y | Y | Y | Y | Y |
199-
| [SSD](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/ssd) | Object Detection | Y | Y | Y | N | Y |
200-
| [VFNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/vfnet) | Object Detection | N | N | N | N | Y |
201-
| [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolo) | Object Detection | Y | Y | Y | N | Y |
202-
| [YOLOX](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolox) | Object Detection | Y | Y | Y | N | Y |
203-
| [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | Object Detection | Y | Y | N | Y | Y |
204-
| [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn) | Object Detection | Y | Y | Y | Y | Y |
205-
| [Faster R-CNN + DCN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn) | Object Detection | Y | Y | Y | Y | Y |
206-
| [GFL](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/gfl) | Object Detection | Y | Y | N | ? | Y |
207-
| [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints) | Object Detection | N | Y | N | ? | Y |
208-
| [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr) | Object Detection | Y | Y | N | ? | Y |
209-
| [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet) | Object Detection | Y | Y | N | ? | Y |
210-
| [RTMDet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/rtmdet) | Object Detection | Y | Y | N | ? | Y |
211-
| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | Instance Segmentation | Y | Y | N | N | Y |
212-
| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn) | Instance Segmentation | Y | Y | N | N | Y |
213-
| [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin) | Instance Segmentation | Y | Y | N | N | Y |
214-
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | Instance Segmentation | Y | N | N | N | Y |
215-
| [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2) | Instance Segmentation | Y | N | N | N | Y |
193+
| Model | Task | OnnxRuntime | TensorRT | ncnn | PPLNN | OpenVINO |
194+
| :------------------------------------------------------------------------------------------------------: | :-------------------: | :---------: | :------: | :--: | :---: | :------: |
195+
| [ATSS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/atss) | Object Detection | Y | Y | N | N | Y |
196+
| [FCOS](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fcos) | Object Detection | Y | Y | Y | N | Y |
197+
| [FoveaBox](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/foveabox) | Object Detection | Y | N | N | N | Y |
198+
| [FSAF](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/fsaf) | Object Detection | Y | Y | Y | Y | Y |
199+
| [RetinaNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/retinanet) | Object Detection | Y | Y | Y | Y | Y |
200+
| [SSD](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/ssd) | Object Detection | Y | Y | Y | N | Y |
201+
| [VFNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/vfnet) | Object Detection | N | N | N | N | Y |
202+
| [YOLOv3](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolo) | Object Detection | Y | Y | Y | N | Y |
203+
| [YOLOX](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/yolox) | Object Detection | Y | Y | Y | N | Y |
204+
| [Cascade R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | Object Detection | Y | Y | N | Y | Y |
205+
| [Faster R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn) | Object Detection | Y | Y | Y | Y | Y |
206+
| [Faster R-CNN + DCN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/faster_rcnn) | Object Detection | Y | Y | Y | Y | Y |
207+
| [GFL](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/gfl) | Object Detection | Y | Y | N | ? | Y |
208+
| [RepPoints](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/reppoints) | Object Detection | N | Y | N | ? | Y |
209+
| [DETR](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/detr) | Object Detection | Y | Y | N | ? | Y |
210+
| [CenterNet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/centernet) | Object Detection | Y | Y | N | ? | Y |
211+
| [RTMDet](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/rtmdet) | Object Detection | Y | Y | N | ? | Y |
212+
| [Cascade Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/cascade_rcnn) | Instance Segmentation | Y | Y | N | N | Y |
213+
| [Mask R-CNN](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/mask_rcnn) | Instance Segmentation | Y | Y | N | N | Y |
214+
| [Swin Transformer](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/swin) | Instance Segmentation | Y | Y | N | N | Y |
215+
| [SOLO](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solo) | Instance Segmentation | Y | N | N | N | Y |
216+
| [SOLOv2](https://github.com/open-mmlab/mmdetection/tree/3.x/configs/solov2) | Instance Segmentation | Y | N | N | N | Y |
217+
| [Panoptic FPN](https://github.com/open-mmlab/mmdetection/tree/main/configs/panoptic_fpn) | Panoptic Segmentation | Y | Y | N | N | N |
218+
| [MaskFormer](https://github.com/open-mmlab/mmdetection/tree/main/configs/maskformer) | Panoptic Segmentation | Y | Y | N | N | N |
219+
| [Mask2Former](https://github.com/open-mmlab/mmdetection/tree/main/configs/mask2former)[\*](#mask2former) | Panoptic Segmentation | Y | Y | N | N | N |
220+
221+
## Reminder
222+
223+
- For transformer based models, strongly suggest use `TensorRT>=8.4`.
224+
- <i id="mask2former">Mask2Former</i> should use `TensorRT>=8.6.1` for dynamic shape inference.

0 commit comments

Comments
 (0)