Skip to content

Commit 5da2e9a

Browse files
author
Anna Grebneva
authored
Added MODNet models (#3373)
* Added MODNet models * Added demo support
1 parent 272d2d7 commit 5da2e9a

File tree

20 files changed

+519
-9
lines changed

20 files changed

+519
-9
lines changed

‎data/dataset_definitions.yml

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1493,3 +1493,14 @@ datasets:
14931493
annotation_file: object_detection/streams_1/high/annotations/instances_glb2bcls3.json
14941494
annotation: mscoco_detection_high_3cls.pickle
14951495
dataset_meta: mscoco_detection_high_3cls.json
1496+
1497+
- name: HumanMattingDataset
1498+
data_source: human_matting_dataset/clip_img/1803151818/clip_00000000
1499+
additional_data_source: human_matting_dataset/matting/1803151818/matting_00000000
1500+
annotation_conversion:
1501+
converter: background_matting
1502+
images_dir: human_matting_dataset/clip_img/1803151818/clip_00000000
1503+
masks_dir: human_matting_dataset/matting/1803151818/matting_00000000
1504+
image_postfix: '.jpg'
1505+
annotation: human_matting.pickle
1506+
dataset_meta: human_matting.json

‎demos/background_subtraction_demo/python/README.md

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -29,7 +29,10 @@ The demo application expects an instance segmentation or background matting mode
2929
* At least two outputs including:
3030
* `fgr` with normalized in [0, 1] range foreground
3131
* `pha` with normalized in [0, 1] range alpha
32-
4. for video background matting models based on RNN architecture:
32+
4. for image background matting models without trimap (background segmentation):
33+
* Single input for input image.
34+
* Single output with normalized in [0, 1] range alpha
35+
5. for video background matting models based on RNN architecture:
3336
* Five inputs:
3437
* `src` for input image
3538
* recurrent inputs: `r1`, `r2`, `r3`, `r4`
@@ -81,10 +84,12 @@ omz_converter --list models.lst
8184

8285
### Supported Models
8386

84-
* instance-segmentation-person-????
85-
* yolact-resnet50-fpn-pytorch
8687
* background-matting-mobilenetv2
88+
* instance-segmentation-person-????
89+
* modnet-photographic-portrait-matting
90+
* modnet-webcam-portrait-matting
8791
* robust-video-matting-mobilenetv3
92+
* yolact-resnet50-fpn-pytorch
8893

8994
> **NOTE**: Refer to the tables [Intel's Pre-Trained Models Device Support](../../../models/intel/device_support.md) and [Public Pre-Trained Models Device Support](../../../models/public/device_support.md) for the details on models inference support at different devices.
9095

‎demos/background_subtraction_demo/python/background_subtraction_demo.py

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,10 @@
2626

2727
sys.path.append(str(Path(__file__).resolve().parents[2] / 'common/python'))
2828

29-
from openvino.model_zoo.model_api.models import MaskRCNNModel, OutputTransform, RESIZE_TYPES, YolactModel, ImageMattingWithBackground, VideoBackgroundMatting
29+
from openvino.model_zoo.model_api.models import (
30+
MaskRCNNModel, OutputTransform, RESIZE_TYPES, YolactModel,
31+
ImageMattingWithBackground, VideoBackgroundMatting, PortraitBackgroundMatting
32+
)
3033
from openvino.model_zoo.model_api.models.utils import load_labels
3134
from openvino.model_zoo.model_api.performance_metrics import PerformanceMetrics
3235
from openvino.model_zoo.model_api.pipelines import get_user_config, AsyncPipeline
@@ -123,6 +126,9 @@ def get_model(model_adapter, configuration, args):
123126
model = ImageMattingWithBackground(model_adapter, configuration)
124127
need_bgr_input = True
125128
is_matting_model = True
129+
elif len(inputs) == 1 and len(outputs) == 1:
130+
model = PortraitBackgroundMatting(model_adapter, configuration)
131+
is_matting_model = True
126132
else:
127133
model = MaskRCNNModel(model_adapter, configuration)
128134
if not need_bgr_input and args.background is not None:

‎demos/background_subtraction_demo/python/models.lst

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# This file can be used with the --list option of the model downloader.
2-
instance-segmentation-person-????
3-
yolact-resnet50-fpn-pytorch
42
background-matting-mobilenetv2
3+
instance-segmentation-person-????
4+
modnet-photographic-portrait-matting
5+
modnet-webcam-portrait-matting
56
robust-video-matting-mobilenetv3
7+
yolact-resnet50-fpn-pytorch

‎demos/common/python/openvino/model_zoo/model_api/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ The following tasks can be solved with wrappers usage:
5959

6060
| Task type | Model API wrappers |
6161
|----------------------------|--------------------|
62-
| Background Matting | <ul><li>`VideoBackgroundMatting`</li><li>`ImageMattingWithBackground`</li></ul> |
62+
| Background Matting | <ul><li>`VideoBackgroundMatting`</li><li>`ImageMattingWithBackground`</li><li>`PortraitBackgroundMatting`</li></ul> |
6363
| Classification | <ul><li>`Classification`</li></ul> |
6464
| Deblurring | <ul><li>`Deblurring`</li></ul> |
6565
| Human Pose Estimation | <ul><li>`HpeAssociativeEmbedding`</li><li>`OpenPose`</li></ul> |

‎demos/common/python/openvino/model_zoo/model_api/models/__init__.py

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -16,7 +16,7 @@
1616

1717

1818
from .bert import BertEmbedding, BertNamedEntityRecognition, BertQuestionAnswering
19-
from .background_matting import ImageMattingWithBackground, VideoBackgroundMatting
19+
from .background_matting import ImageMattingWithBackground, VideoBackgroundMatting, PortraitBackgroundMatting
2020
from .centernet import CenterNet
2121
from .classification import Classification
2222
from .deblurring import Deblurring
@@ -58,6 +58,7 @@
5858
'MonoDepthModel',
5959
'OpenPose',
6060
'OutputTransform',
61+
'PortraitBackgroundMatting',
6162
'RESIZE_TYPES',
6263
'RetinaFace',
6364
'RetinaFacePyTorch',

‎demos/common/python/openvino/model_zoo/model_api/models/background_matting.py

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -150,3 +150,36 @@ def postprocess(self, outputs, meta):
150150
fgr = cv2.cvtColor(cv2.resize(fgr, (w, h)), cv2.COLOR_RGB2BGR)
151151
pha = np.expand_dims(cv2.resize(pha, (w, h)), axis=-1)
152152
return fgr, pha
153+
154+
155+
class PortraitBackgroundMatting(ImageModel):
156+
__model__ = 'Portrait-matting'
157+
158+
def __init__(self, model_adapter, configuration, preload=False):
159+
super().__init__(model_adapter, configuration, preload)
160+
self._check_io_number(1, 1)
161+
self.output_blob_name = self._get_outputs()
162+
163+
@classmethod
164+
def parameters(cls):
165+
return super().parameters()
166+
167+
def _get_outputs(self):
168+
output_blob_name = next(iter(self.outputs))
169+
output_size = self.outputs[output_blob_name].shape
170+
if len(output_size) != 4:
171+
self.raise_error("Unexpected output blob shape {}. Only 4D output blob is supported".format(output_size))
172+
173+
return output_blob_name
174+
175+
def preprocess(self, inputs):
176+
dict_inputs, meta = super().preprocess(inputs)
177+
meta.update({"original_image": inputs})
178+
return dict_inputs, meta
179+
180+
def postprocess(self, outputs, meta):
181+
output = outputs[self.output_blob_name][0].transpose(1, 2, 0)
182+
original_frame = meta['original_image'] / 255.0
183+
h, w = meta['original_shape'][:2]
184+
res_output = np.expand_dims(cv2.resize(output, (w, h)), -1)
185+
return original_frame, res_output

‎demos/tests/cases.py

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -757,7 +757,9 @@ def single_option_cases(key, *args):
757757
ModelArg('instance-segmentation-person-0007'),
758758
ModelArg('robust-video-matting-mobilenetv3'),
759759
ModelArg('background-matting-mobilenetv2'),
760-
ModelArg('yolact-resnet50-fpn-pytorch')),
760+
ModelArg('yolact-resnet50-fpn-pytorch'),
761+
ModelArg('modnet-photographic-portrait-matting'),
762+
ModelArg('modnet-webcam-portrait-matting')),
761763
)),
762764

763765
PythonDemo(name='bert_question_answering_demo', device_keys=['-d'], test_cases=combine_cases(

‎models/public/device_support.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,8 @@
7878
| mobilenet-v3-large-1.0-224-tf | YES | YES | YES |
7979
| mobilenet-v3-small-1.0-224-tf | YES | YES | YES |
8080
| mobilenet-yolo-v4-syg | YES | YES | |
81+
| modnet-photographic-portrait-matting | YES | YES | YES |
82+
| modnet-webcam-portrait-matting | YES | YES | YES |
8183
| mozilla-deepspeech-0.6.1 | YES | | |
8284
| mozilla-deepspeech-0.8.2 | YES | | |
8385
| mtcnn-o | YES | YES | |

‎models/public/index.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,8 @@
3333
:caption: Background Matting Models
3434
3535
omz_models_model_background_matting_mobilenetv2
36+
omz_models_model_modnet_photographic_portrait_matting
37+
omz_models_model_modnet_webcam_portrait_matting
3638
omz_models_model_robust_video_matting_mobilenetv3
3739
3840
.. toctree::
@@ -644,6 +646,8 @@ or mixed pixels. This distinguishes background matting from segmentation approac
644646
| Model Name | Implementation | OMZ Model Name | Accuracy | GFlops | mParams |
645647
| -------------- | -------------- | ------------------------------------------------------ | -------- | ------- | -------- |
646648
| background-matting-mobilenetv2 | PyTorch\* | [background-matting-mobilenetv2](./background-matting-mobilenetv2/README.md) | 4.32/1.0/2.48/2.7 | 6.7419 | 5.052 |
649+
| modnet-photographic-portrait-matting | PyTorch\* | [modnet-photographic-portrait-matting](./modnet-photographic-portrait-matting/README.md) | 5.21/727.95 | 31.1564 | 6.4597 |
650+
| modnet-webcam-portrait-matting | PyTorch\* | [modnet-webcam-portrait-matting](./modnet-webcam-portrait-matting/README.md) | 5.66/762.52 | 31.1564 | 6.4597 |
647651
| robust-video-matting-mobilenetv3 | PyTorch\* | [robust-video-matting-mobilenetv3](./robust-video-matting-mobilenetv3/README.md) | 20.8/15.1/4.42/4.05 | 9.3892 | 3.7363 |
648652

649653
## See Also

0 commit comments

Comments
 (0)