Skip to content

Commit 29ab2a8

Browse files
Dmitry SidneveaidovaAnna Grebneva
authored
models/public: new background matting models (#3054)
* Add description for new models * Minor fix * Fix name of argument Co-authored-by: Ekaterina Aidova <[email protected]> * Update index.md * Update device_support.md * Fix model.yml * Remove trailing space * Update task types in model_downloader * Fix link * Fix input shapes * Add AC configs and update documentation * Add support of background-matting-v2 model * Update * Add another background-matting model * Update * Update documentation in AC * Update index.md and name of model * Fix annotation converter * Update metrics * Fix scale * Fixes * Pylint errors fix * Add layout to inputs Co-authored-by: Ekaterina Aidova <[email protected]> * Add layout to inputs Co-authored-by: Ekaterina Aidova <[email protected]> * Fix year Co-authored-by: Ekaterina Aidova <[email protected]> * Update description in model.yml * Rename to robust-video-matting-mobilenetv3 * Minor fix Co-authored-by: Ekaterina Aidova <[email protected]> * Bug fix Co-authored-by: Ekaterina Aidova <[email protected]> * Add output verification * Pylint error fix * Fix dataset_definitions.yml * Fix converter * Add link to demo Co-authored-by: Anna Grebneva <[email protected]> * Add link to demo Co-authored-by: Anna Grebneva <[email protected]> * Add models to demo and tests * Fix framework Co-authored-by: Ekaterina Aidova <[email protected]> * Fix framework Co-authored-by: Ekaterina Aidova <[email protected]> * Fix model name in readme * Fix reference metric Co-authored-by: Anna Grebneva <[email protected]> * Fix reference metric Co-authored-by: Anna Grebneva <[email protected]> * Fix reference metric Co-authored-by: Anna Grebneva <[email protected]> * Fix reference metric Co-authored-by: Anna Grebneva <[email protected]> * Fix reference metric Co-authored-by: Anna Grebneva <[email protected]> * Fix reference metric Co-authored-by: Anna Grebneva <[email protected]> * Fix reference metric Co-authored-by: Anna Grebneva <[email protected]> * Remove redundant metric from readme Co-authored-by: Ekaterina Aidova <[email protected]> Co-authored-by: Anna Grebneva <[email protected]>
1 parent 8011247 commit 29ab2a8

File tree

25 files changed

+1122
-9
lines changed

25 files changed

+1122
-9
lines changed

‎data/dataset_definitions.yml

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,19 @@
11
datasets:
2+
- name: HumanMattingClips120
3+
annotation_conversion:
4+
converter: background_matting_sequential
5+
images_dir: HumanMattingClips120
6+
masks_dir: HumanMattingClips120
7+
backgrounds_dir: HumanMattingClips120
8+
image_prefix: com/
9+
mask_prefix: fgr/
10+
background_prefix: bgr/
11+
with_background: True
12+
with_alpha: True
13+
annotation: human_matting_120.pickle
14+
dataset_meta: human_matting_120_meta.json
15+
data_source: HumanMattingClips120
16+
217
- name: ms_coco_mask_rcnn
318
annotation_conversion:
419
converter: mscoco_mask_rcnn

‎demos/background_subtraction_demo/python/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -78,7 +78,7 @@ omz_converter --list models.lst
7878
* instance-segmentation-person-????
7979
* yolact-resnet50-fpn-pytorch
8080
* background-matting-mobilenetv2
81-
* robust-video-matting
81+
* robust-video-matting-mobilenetv3
8282

8383
> **NOTE**: Refer to the tables [Intel's Pre-Trained Models Device Support](../../../models/intel/device_support.md) and [Public Pre-Trained Models Device Support](../../../models/public/device_support.md) for the details on models inference support at different devices.
8484

‎demos/background_subtraction_demo/python/models.lst

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
# This file can be used with the --list option of the model downloader.
22
instance-segmentation-person-????
33
yolact-resnet50-fpn-pytorch
4-
# TODO: background-matting-mobilenetv2
5-
# TODO: robust-video-matting
4+
background-matting-mobilenetv2
5+
robust-video-matting-mobilenetv3

‎demos/tests/cases.py

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -717,8 +717,8 @@ def single_option_cases(key, *args):
717717
}),
718718
single_option_cases('-m',
719719
ModelArg('instance-segmentation-person-0007'),
720-
# ModelArg('robust-video-matting'),
721-
# ModelArg('background-matting-mobilenetv2'),
720+
ModelArg('robust-video-matting-mobilenetv3'),
721+
ModelArg('background-matting-mobilenetv2'),
722722
ModelArg('yolact-resnet50-fpn-pytorch')),
723723
)),
724724

‎models/public/background-matting-mobilenetv2/README.md

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
# background-matting-mobilenetv2
2+
3+
## Use Case and High-Level Description
4+
5+
The `background-matting-mobilenetv2` model is a high-resolution background replacement technique based on
6+
background matting (with MobileNetV2 backbone), where an additional frame of the background is
7+
captured and used in recovering the alpha matte and the foreground layer. This model is
8+
pre-trained in PyTorch\* framework and converted to ONNX\* format. More details provided in
9+
the [paper](https://arxiv.org/abs/2012.07810).
10+
For details see the [repository](https://github.com/PeterL1n/BackgroundMattingV2).
11+
For details regarding export to ONNX see [here](https://github.com/DmitriySidnev/BackgroundMattingV2/blob/master/export_onnx.py).
12+
13+
## Specification
14+
15+
| Metric | Value |
16+
|---------------------------------|-------------------------------------------|
17+
| Type | Background_matting |
18+
| GFlops | 6.7419 |
19+
| MParams | 5.052 |
20+
| Source framework | PyTorch\* |
21+
22+
## Accuracy
23+
24+
Accuracy measured on a dataset composed with foregrounds from the HumanMatting dataset and backgrounds from the OpenImagesV5 one with input resolution 1280x720.
25+
26+
| Metric | Original model | Converted model |
27+
| -------------- | -------------- | --------------- |
28+
| Alpha MAD | 4.32 | 4.35 |
29+
| Alpha MSE | 1.0 | 1.0 |
30+
| Alpha GRAD | 2.48 | 2.49 |
31+
| Foreground MSE | 2.7 | 2.69 |
32+
33+
* Alpha MAD - mean of absolute difference for alpha.
34+
* Alpha MSE - mean squared error for alpha.
35+
* Alpha GRAD - spatial-gradient metric for alpha.
36+
* Foreground MSE - mean squared error for foreground.
37+
38+
## Input
39+
40+
### Original Model
41+
42+
Image, name: `src`, shape: `1, 3, 720, 1280`, format: `B, C, H, W`, where:
43+
44+
- `B` - batch size
45+
- `C` - number of channels
46+
- `H` - image height
47+
- `W` - image width
48+
49+
Expected color order: `RGB`.
50+
scale factor: 255
51+
52+
Image, name: `bgr`, shape: `1, 3, 720, 1280`, format: `B, C, H, W`, where:
53+
54+
- `B` - batch size
55+
- `C` - number of channels
56+
- `H` - image height
57+
- `W` - image width
58+
59+
Expected color order: `RGB`.
60+
scale factor: 255
61+
62+
### Converted Model
63+
64+
Image, name: `src`, shape: `1, 3, 720, 1280`, format: `B, C, H, W`, where:
65+
66+
- `B` - batch size
67+
- `C` - number of channels
68+
- `H` - image height
69+
- `W` - image width
70+
71+
Expected color order: `BGR`.
72+
73+
Image, name: `bgr`, shape: `1, 3, 720, 1280`, format: `B, C, H, W`, where:
74+
75+
- `B` - batch size
76+
- `C` - number of channels
77+
- `H` - image height
78+
- `W` - image width
79+
80+
Expected color order: `BGR`.
81+
82+
## Output
83+
84+
### Original model
85+
86+
Alpha matte. Name: `pha` Shape: `1, 1, 720, 1280`, format: `B, C, H, W`, where:
87+
88+
- `B` - batch size
89+
- `C` - number of channels
90+
- `H` - image height
91+
- `W` - image width
92+
93+
Foreground. Name: `fgr` Shape: `1, 3, 720, 1280`, format: `B, C, H, W`, where:
94+
95+
- `B` - batch size
96+
- `C` - number of channels
97+
- `H` - image height
98+
- `W` - image width
99+
100+
### Converted model
101+
102+
Alpha matte. Name: `pha` Shape: `1, 1, 720, 1280`, format: `B, C, H, W`, where:
103+
104+
- `B` - batch size
105+
- `C` - number of channels
106+
- `H` - image height
107+
- `W` - image width
108+
109+
Foreground. Name: `fgr` Shape: `1, 3, 720, 1280`, format: `B, C, H, W`, where:
110+
111+
- `B` - batch size
112+
- `C` - number of channels
113+
- `H` - image height
114+
- `W` - image width
115+
116+
## Download a Model and Convert it into Inference Engine Format
117+
118+
You can download models and if necessary convert them into Inference Engine format using the [Model Downloader and other automation tools](../../../tools/model_tools/README.md) as shown in the examples below.
119+
120+
An example of using the Model Downloader:
121+
```
122+
omz_downloader --name <model_name>
123+
```
124+
125+
An example of using the Model Converter:
126+
```
127+
omz_converter --name <model_name>
128+
```
129+
130+
## Demo usage
131+
132+
The model can be used in the following demos provided by the Open Model Zoo to show its capabilities:
133+
134+
* [Background subtraction Python\* Demo](../../../demos/background_subtraction_demo/python/README.md)
135+
136+
## Legal Information
137+
138+
The original model is distributed under the
139+
[MIT License](https://github.com/DmitriySidnev/BackgroundMattingV2/blob/master/LICENSE).

‎models/public/background-matting-mobilenetv2/accuracy-check.yml

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,54 @@
1+
models:
2+
- name: background-matting-mobilenetv2
3+
4+
launchers:
5+
- framework: openvino
6+
adapter:
7+
type: background_matting_with_pha_and_fgr
8+
alpha_out: pha
9+
foreground_out: fgr
10+
inputs:
11+
- name: src
12+
type: INPUT
13+
value: com*
14+
- name: bgr
15+
type: INPUT
16+
value: bgr*
17+
18+
datasets:
19+
- name: HumanMattingClips120
20+
reader: pillow_imread
21+
22+
preprocessing:
23+
- type: rgb_to_bgr
24+
- type: resize
25+
dst_width: 1280
26+
dst_height: 720
27+
use_pillow: True
28+
29+
metrics:
30+
- name: alpha_MAD
31+
type: mad
32+
prediction_source: pha
33+
process_type: alpha
34+
reference: 4.35
35+
36+
- name: alpha_GRAD
37+
type: spatial_gradient
38+
prediction_source: pha
39+
process_type: alpha
40+
reference: 2.49
41+
42+
- name: alpha_MSE
43+
type: mse_with_mask
44+
prediction_source: pha
45+
process_type: alpha
46+
use_mask: False
47+
reference: 1.0
48+
49+
- name: foreground_MSE
50+
type: mse_with_mask
51+
prediction_source: fgr
52+
process_type: image
53+
use_mask: True
54+
reference: 2.69

‎models/public/background-matting-mobilenetv2/model.yml

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Copyright (c) 2022 Intel Corporation
2+
#
3+
# Licensed under the Apache License, Version 2.0 (the "License");
4+
# you may not use this file except in compliance with the License.
5+
# You may obtain a copy of the License at
6+
#
7+
# http://www.apache.org/licenses/LICENSE-2.0
8+
#
9+
# Unless required by applicable law or agreed to in writing, software
10+
# distributed under the License is distributed on an "AS IS" BASIS,
11+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
# See the License for the specific language governing permissions and
13+
# limitations under the License.
14+
15+
description: >-
16+
The "background-matting-mobilenetv2" model is a high-resolution background replacement
17+
technique based on background matting (with MobileNetV2 backbone), where an additional
18+
frame of the background is captured and used in recovering the alpha matte and the
19+
foreground layer. This model is pre-trained in PyTorch* framework and converted
20+
to ONNX* format. More details provided in the paper <https://arxiv.org/abs/2012.07810>.
21+
For details see the repository <https://github.com/PeterL1n/BackgroundMattingV2>.
22+
For details regarding export to ONNX see here <https://github.com/DmitriySidnev/BackgroundMattingV2/blob/master/export_onnx.py>.
23+
task_type: background_matting
24+
files:
25+
- name: bgm_mobilenetv2.onnx
26+
size: 20006715
27+
checksum: 3fc7bad659149e3e9e0feb4ef531c177ab158e5bbcfdabba9952c3832053f353d23560ff488aeae857245734ef639e6b
28+
source: https://github.com/DmitriySidnev/BackgroundMattingV2/raw/master/onnx/bgm_mobilenetv2.onnx
29+
model_optimizer_args:
30+
- --input_shape=[1,3,720,1280],[1,3,720,1280]
31+
- --input=src,bgr
32+
- --layout=src(NCHW),bgr(NCHW)
33+
- --output=pha,fgr
34+
- --scale=255
35+
- --reverse_input_channels
36+
- --input_model=$dl_dir/bgm_mobilenetv2.onnx
37+
framework: onnx
38+
license: https://github.com/DmitriySidnev/BackgroundMattingV2/blob/master/LICENSE

‎models/public/device_support.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
| aclnet-int8 | YES | YES | |
77
| alexnet | YES | YES | YES |
88
| anti-spoof-mn3 | YES | YES | YES |
9+
| background-matting-mobilenetv2 | YES | | |
910
| bert-base-ner | YES | YES | |
1011
| brain-tumor-segmentation-0001 | YES | | |
1112
| brain-tumor-segmentation-0002 | YES | | |
@@ -103,6 +104,7 @@
103104
| retinanet-tf | YES | YES | |
104105
| rexnet-v1-x1.0 | YES | YES | |
105106
| rfcn-resnet101-coco-tf | YES | YES | YES |
107+
| robust-video-matting-mobilenetv3 | YES | YES | |
106108
| se-inception | YES | YES | YES |
107109
| se-resnet-50 | YES | YES | YES |
108110
| se-resnext-50 | YES | YES | YES |

‎models/public/index.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -319,6 +319,17 @@ Named entity recognition (NER) is the task of tagging entities in text with thei
319319
| -------------- | -------------- | ------------------------------------------------------ | -------- | ------- | -------- |
320320
| vehicle-reid-0001 | PyTorch\* | [vehicle-reid-0001](./vehicle-reid-0001/README.md) | 96.31%/85.15 % | 2.643 | 2.183 |
321321

322+
## Background matting
323+
324+
Background matting is a method of separating a foreground from a background in an image or video,
325+
wherein some pixels may belong to foreground as well as background, such pixels are called partial
326+
or mixed pixels. This distinguishes background matting from segmentation approaches where the result is a binary mask.
327+
328+
| Model Name | Implementation | OMZ Model Name | Accuracy | GFlops | mParams |
329+
| -------------- | -------------- | ------------------------------------------------------ | -------- | ------- | -------- |
330+
| background-matting-mobilenetv2 | PyTorch\* | [background-matting-v2](./background-matting-mobilenetv2/README.md) | 4.32/1.0/2.48/2.7 | 6.7419 | 5.052 |
331+
| robust-video-matting-mobilenetv3 | PyTorch\* | [robust-video-matting-mobilenetv3](./robust-video-matting-mobilenetv3/README.md) | 20.8/15.1/4.42/4.05 | 9.3892 | 3.7363 |
332+
322333
## See Also
323334

324335
* [Open Model Zoo Demos](../../demos/README.md)

0 commit comments

Comments
 (0)