Skip to content

Commit 0aa5d7b

Browse files
gaotongxiaogaotongxiao
authored
[Model] Add SVTR framework and configs (#1621)
* [Model] Add SVTR framework and configs * update * update transform names * update base config * fix cfg * update cfgs * fix * update cfg * update decoder * fix encoder * fix encoder * fix * update cfg * update name
1 parent b0557c2 commit 0aa5d7b

File tree

15 files changed

+437
-14
lines changed

15 files changed

+437
-14
lines changed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,7 @@ Supported algorithms:
147147
- [x] [RobustScanner](configs/textrecog/robust_scanner/README.md) (ECCV'2020)
148148
- [x] [SAR](configs/textrecog/sar/README.md) (AAAI'2019)
149149
- [x] [SATRN](configs/textrecog/satrn/README.md) (CVPR'2020 Workshop on Text and Documents in the Deep Learning Era)
150+
- [x] [SVTR](configs/textrecog/svtr/README.md) (IJCAI'2022)
150151

151152
</details>
152153

‎README_zh-CN.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -147,6 +147,7 @@ pip3 install -e .
147147
- [x] [RobustScanner](configs/textrecog/robust_scanner/README.md) (ECCV'2020)
148148
- [x] [SAR](configs/textrecog/sar/README.md) (AAAI'2019)
149149
- [x] [SATRN](configs/textrecog/satrn/README.md) (CVPR'2020 Workshop on Text and Documents in the Deep Learning Era)
150+
- [x] [SVTR](configs/textrecog/svtr/README.md) (IJCAI'2022)
150151

151152
</details>
152153

‎configs/textrecog/svtr/README.md‎

Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# SVTR
2+
3+
> [SVTR: Scene Text Recognition with a Single Visual Model](https://arxiv.org/abs/2205.00159)
4+
5+
<!-- [ALGORITHM] -->
6+
7+
## Abstract
8+
9+
Dominant scene text recognition models commonly contain two building blocks, a visual model for feature extraction and a sequence model for text transcription. This hybrid architecture, although accurate, is complex and less efficient. In this study, we propose a Single Visual model for Scene Text recognition within the patch-wise image tokenization framework, which dispenses with the sequential modeling entirely. The method, termed SVTR, firstly decomposes an image text into small patches named character components. Afterward, hierarchical stages are recurrently carried out by component-level mixing, merging and/or combining. Global and local mixing blocks are devised to perceive the inter-character and intra-character patterns, leading to a multi-grained character component perception. Thus, characters are recognized by a simple linear prediction. Experimental results on both English and Chinese scene text recognition tasks demonstrate the effectiveness of SVTR. SVTR-L (Large) achieves highly competitive accuracy in English and outperforms existing methods by a large margin in Chinese, while running faster. In addition, SVTR-T (Tiny) is an effective and much smaller model, which shows appealing speed at inference.
10+
11+
<div align=center>
12+
<img src="https://user-images.githubusercontent.com/22607038/210541576-025df5d5-f4d2-4037-82e0-246cf8cd3c25.png"/>
13+
</div>
14+
15+
## Dataset
16+
17+
### Train Dataset
18+
19+
| trainset | instance_num | repeat_num | source |
20+
| :-------: | :----------: | :--------: | :----: |
21+
| SynthText | 7266686 | 1 | synth |
22+
| Syn90k | 8919273 | 1 | synth |
23+
24+
### Test Dataset
25+
26+
| testset | instance_num | type |
27+
| :-----: | :----------: | :-------: |
28+
| IIIT5K | 3000 | regular |
29+
| SVT | 647 | regular |
30+
| IC13 | 1015 | regular |
31+
| IC15 | 2077 | irregular |
32+
| SVTP | 645 | irregular |
33+
| CT80 | 288 | irregular |
34+
35+
## Results and Models
36+
37+
| Methods | | Regular Text | | | | Irregular Text | | download |
38+
| :-----------------------------------------------------------: | :----: | :----------: | :-------: | :-: | :-------: | :------------: | :----: | :------------------------------------------------------------------------------: |
39+
| | IIIT5K | SVT | IC13-1015 | | IC15-2077 | SVTP | CT80 | |
40+
| [SVTR-tiny](/configs/textrecog/svtr/svtr-tiny_20e_st_mj.py) | - | - | - | | - | - | - | [model](<>) \| [log](<>) |
41+
| [SVTR-small](/configs/textrecog/svtr/svtr-small_20e_st_mj.py) | 0.8553 | 0.9026 | 0.9448 | | 0.7496 | 0.8496 | 0.8854 | [model](https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-small_20e_st_mj/svtr-small_20e_st_mj-35d800d6.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-small_20e_st_mj/20230105_184454.log) |
42+
| [SVTR-base](/configs/textrecog/svtr/svtr-base_20e_st_mj.py) | 0.8570 | 0.9181 | 0.9438 | | 0.7448 | 0.8388 | 0.9028 | [model](https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-base_20e_st_mj/svtr-base_20e_st_mj-ea500101.pth) \| [log](https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-base_20e_st_mj/20221227_175415.log) |
43+
| [SVTR-large](/configs/textrecog/svtr/svtr-large_20e_st_mj.py) | - | - | - | | - | - | - | [model](<>) \| [log](<>) |
44+
45+
```{note}
46+
The implementation and configuration follow the original code and paper, but there is still a gap between the reproduced results and the official ones. We appreciate any suggestions to improve its performance.
47+
```
48+
49+
## Citation
50+
51+
```bibtex
52+
@inproceedings{ijcai2022p124,
53+
title = {SVTR: Scene Text Recognition with a Single Visual Model},
54+
author = {Du, Yongkun and Chen, Zhineng and Jia, Caiyan and Yin, Xiaoting and Zheng, Tianlun and Li, Chenxia and Du, Yuning and Jiang, Yu-Gang},
55+
booktitle = {Proceedings of the Thirty-First International Joint Conference on
56+
Artificial Intelligence, {IJCAI-22}},
57+
publisher = {International Joint Conferences on Artificial Intelligence Organization},
58+
editor = {Lud De Raedt},
59+
pages = {884--890},
60+
year = {2022},
61+
month = {7},
62+
note = {Main Track},
63+
doi = {10.24963/ijcai.2022/124},
64+
url = {https://doi.org/10.24963/ijcai.2022/124},
65+
}
66+
67+
```

‎configs/textrecog/svtr/_base_svtr-tiny.py‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
dictionary = dict(
2+
type='Dictionary',
3+
dict_file='{{ fileDirname }}/../../../dicts/lower_english_digits.txt',
4+
with_padding=True,
5+
with_unknown=True,
6+
)
7+
8+
model = dict(
9+
type='SVTR',
10+
preprocessor=dict(
11+
type='STN',
12+
in_channels=3,
13+
resized_image_size=(32, 64),
14+
output_image_size=(32, 100),
15+
num_control_points=20,
16+
margins=[0.05, 0.05]),
17+
encoder=dict(
18+
type='SVTREncoder',
19+
img_size=[32, 100],
20+
in_channels=3,
21+
out_channels=192,
22+
embed_dims=[64, 128, 256],
23+
depth=[3, 6, 3],
24+
num_heads=[2, 4, 8],
25+
mixer_types=['Local'] * 6 + ['Global'] * 6,
26+
window_size=[[7, 11], [7, 11], [7, 11]],
27+
merging_types='Conv',
28+
prenorm=False,
29+
max_seq_len=25),
30+
decoder=dict(
31+
type='SVTRDecoder',
32+
in_channels=192,
33+
module_loss=dict(
34+
type='CTCModuleLoss', letter_case='lower', zero_infinity=True),
35+
postprocessor=dict(type='CTCPostProcessor'),
36+
dictionary=dictionary),
37+
data_preprocessor=dict(
38+
type='TextRecogDataPreprocessor', mean=[127.5], std=[127.5]))

‎configs/textrecog/svtr/metafile.yml‎

Lines changed: 89 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
Collections:
2+
- Name: SVTR
3+
Metadata:
4+
Training Data: OCRDataset
5+
Training Techniques:
6+
- AdamW
7+
Training Resources: 4x Tesla A100
8+
Epochs: 20
9+
Batch Size: 2048
10+
Architecture:
11+
- STN
12+
- SVTREncoder
13+
- SVTRDecoder
14+
Paper:
15+
URL: https://arxiv.org/pdf/2205.00159.pdf
16+
Title: 'SVTR: Scene Text Recognition with a Single Visual Model'
17+
README: configs/textrecog/svtr/README.md
18+
19+
Models:
20+
- Name: svtr-small_20e_st_mj
21+
Alias: svtr-small
22+
In Collection: SVTR
23+
Config: configs/textrecog/svtr/svtr-small_20e_st_mj.py
24+
Metadata:
25+
Training Data:
26+
- SynthText
27+
- Syn90k
28+
Results:
29+
- Task: Text Recognition
30+
Dataset: IIIT5K
31+
Metrics:
32+
word_acc: 0.8553
33+
- Task: Text Recognition
34+
Dataset: SVT
35+
Metrics:
36+
word_acc: 0.9026
37+
- Task: Text Recognition
38+
Dataset: ICDAR2013
39+
Metrics:
40+
word_acc: 0.9448
41+
- Task: Text Recognition
42+
Dataset: ICDAR2015
43+
Metrics:
44+
word_acc: 0.7496
45+
- Task: Text Recognition
46+
Dataset: SVTP
47+
Metrics:
48+
word_acc: 0.8496
49+
- Task: Text Recognition
50+
Dataset: CT80
51+
Metrics:
52+
word_acc: 0.8854
53+
Weights: https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-small_20e_st_mj/svtr-small_20e_st_mj-35d800d6.pth
54+
55+
- Name: svtr-base_20e_st_mj
56+
Alias: svtr-base
57+
Batch Size: 1024
58+
In Collection: SVTR
59+
Config: configs/textrecog/svtr/svtr-base_20e_st_mj.py
60+
Metadata:
61+
Training Data:
62+
- SynthText
63+
- Syn90k
64+
Results:
65+
- Task: Text Recognition
66+
Dataset: IIIT5K
67+
Metrics:
68+
word_acc: 0.8570
69+
- Task: Text Recognition
70+
Dataset: SVT
71+
Metrics:
72+
word_acc: 0.9181
73+
- Task: Text Recognition
74+
Dataset: ICDAR2013
75+
Metrics:
76+
word_acc: 0.9438
77+
- Task: Text Recognition
78+
Dataset: ICDAR2015
79+
Metrics:
80+
word_acc: 0.7448
81+
- Task: Text Recognition
82+
Dataset: SVTP
83+
Metrics:
84+
word_acc: 0.8388
85+
- Task: Text Recognition
86+
Dataset: CT80
87+
Metrics:
88+
word_acc: 0.9028
89+
Weights: https://download.openmmlab.com/mmocr/textrecog/svtr/svtr-base_20e_st_mj/svtr-base_20e_st_mj-ea500101.pth

‎configs/textrecog/svtr/svtr-base_20e_st_mj.py‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
_base_ = [
2+
'svtr-tiny_20e_st_mj.py',
3+
]
4+
5+
model = dict(
6+
preprocessor=dict(output_image_size=(48, 160), ),
7+
encoder=dict(
8+
img_size=[48, 160],
9+
max_seq_len=40,
10+
out_channels=256,
11+
embed_dims=[128, 256, 384],
12+
depth=[3, 6, 9],
13+
num_heads=[4, 8, 12],
14+
mixer_types=['Local'] * 8 + ['Global'] * 10),
15+
decoder=dict(in_channels=256))
16+
17+
train_dataloader = dict(batch_size=256, )

‎configs/textrecog/svtr/svtr-large_20e_st_mj.py‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
_base_ = [
2+
'svtr-tiny_20e_st_mj.py',
3+
]
4+
5+
model = dict(
6+
preprocessor=dict(output_image_size=(48, 160), ),
7+
encoder=dict(
8+
img_size=[48, 160],
9+
max_seq_len=40,
10+
out_channels=384,
11+
embed_dims=[192, 256, 512],
12+
depth=[3, 9, 9],
13+
num_heads=[6, 8, 16],
14+
mixer_types=['Local'] * 10 + ['Global'] * 11),
15+
decoder=dict(in_channels=384))
16+
17+
train_dataloader = dict(batch_size=128, )
18+
19+
optim_wrapper = dict(optimizer=dict(lr=2.5 / (10**4)))

‎configs/textrecog/svtr/svtr-small_20e_st_mj.py‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
_base_ = [
2+
'svtr-tiny_20e_st_mj.py',
3+
]
4+
5+
model = dict(
6+
encoder=dict(
7+
embed_dims=[96, 192, 256],
8+
depth=[3, 6, 6],
9+
num_heads=[3, 6, 8],
10+
mixer_types=['Local'] * 8 + ['Global'] * 7))

0 commit comments

Comments
 (0)