Skip to content

Commit 3cc6572

Browse files
hcyanghcyang
authored
Trim model list and documentation for R13. (#6273)
1 parent d2d0dc0 commit 3cc6572

File tree

2 files changed

+6
-98
lines changed

2 files changed

+6
-98
lines changed

‎Orchestrator/docs/NLRModels.md‎

Lines changed: 5 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,11 @@ This is a high quality multilingual base model for intent detection. It's smalle
2121
It is a 6-layer pretrained pretrained [Transformer][7] model optimized for conversation.
2222
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box. The model supports in total 100 languages (full list can be found at [XLMR Supported Languages][8]). 8 languages (EN, ES, DE, FR, IT, JA, PT, and ZH) are fine-tuned with additional data (performance can be found [here](#multilingual-intent-detection-models-evaluation)).
2323

24+
### pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx (experimental)
25+
This is a high quality EN-only base model for entity extraction. It's smaller and faster than its 12-layer alternative.
26+
It is a 6-layer pretrained pretrained [Transformer][7] model optimized for conversation.
27+
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
28+
2429
## Alternate Models
2530

2631
### pretrained.20200924.microsoft.dte.00.03.en.onnx
@@ -42,26 +47,11 @@ Its architecture is pretrained for example-based use ([KNN][3]), thus it can be
4247

4348
## Experimental Models
4449

45-
### pretrained.20210205.microsoft.dte.00.12.bert_example_ner.en.onnx (experimental)
46-
This is a high quality EN-only base model for entity extraction.
47-
It is a 12-layer pretrained pretrained [Transformer][7] model optimized for conversation.
48-
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
49-
5050
### pretrained.20210218.microsoft.dte.00.12.bert_example_ner.en.onnx (experimental)
5151
This is a yet another high quality EN-only base model for entity extraction.
5252
It is a 12-layer pretrained pretrained [Transformer][7] model optimized for conversation.
5353
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
5454

55-
### pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx (experimental)
56-
This is a high quality EN-only base model for entity extraction. It's smaller and faster than its 12-layer alternative.
57-
It is a 6-layer pretrained pretrained [Transformer][7] model optimized for conversation.
58-
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
59-
60-
### pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx (experimental)
61-
This is a high quality EN-only base model for entity extraction. It's smaller and faster than its 12-layer alternative.
62-
It is a 6-layer pretrained pretrained [Transformer][7] model optimized for conversation.
63-
Its architecture is pretrained for example-based use ([KNN][3]), thus it can be used out of box.
64-
6555
## Models Evaluation
6656
For a more quantitative comparison analysis of the different models see the following performance characteristics.
6757

@@ -114,17 +104,13 @@ For a more quantitative comparison analysis of the different models see the foll
114104

115105
| Model | Base Model | Layers | Encoding time per query | Disk Allocation |
116106
| ------------------------------------------------------------ | ---------- | ------ | ----------------------- | --------------- |
117-
| pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx | BERT | 6 | ~ 23 ms | 259M |
118-
| pretrained.20210205.microsoft.dte.00.12.bert_example_ner.en.onnx | BERT | 12 | ~ 40 ms | 427M |
119107
| pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx | BERT | 6 | ~ 23 ms | 259M |
120108
| pretrained.20210218.microsoft.dte.00.12.bert_example_ner.en.onnx | BERT | 12 | ~ 40 ms | 425M |
121109

122110
- The following table shows how accurate is each model relative to provided training sample size using [Snips NLU][4] system, evaluated by **macro-average-F1**.
123111

124112
| Training samples per entity type | 10 | 20 | 50 | 100 | 200 |
125113
| ------------------------------------------------------------ | ----- | ----- | ----- | ----- | ----- |
126-
| pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx | 0.615 | 0.636 | 0.647 | 0.661 | 0.665 |
127-
| pretrained.20210205.microsoft.dte.00.12.bert_example_ner.en.onnx | 0.637 | 0.658 | 0.684 | 0.698 | 0.702 |
128114
| pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx | 0.637 | 0.658 | 0.673 | 0.686 | 0.684 |
129115
| pretrained.20210218.microsoft.dte.00.12.bert_example_ner.en.onnx | 0.661 | 0.664 | 0.670 | 0.685 | 0.681 |
130116

‎Orchestrator/v0.2/nlr_versions.json‎

Lines changed: 1 addition & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
"defaults": {
44
"en_intent": "pretrained.20200924.microsoft.dte.00.06.en.onnx",
55
"multilingual_intent": "pretrained.20210205.microsoft.dte.00.06.unicoder_multilingual.onnx",
6-
"en_entity": "pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx"
6+
"en_entity": "pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx"
77
},
88
"models": {
99
"pretrained.20200924.microsoft.dte.00.03.en.onnx": {
@@ -24,12 +24,6 @@
2424
"description": "Bot Framework SDK release 4.10 - English ONNX V1.4 12-layer per-token intent base model",
2525
"minSDKVersion": "4.10.0"
2626
},
27-
"pretrained.20210205.microsoft.dte.00.12.bert_example_ner.en.onnx": {
28-
"releaseDate": "02/05/2021",
29-
"modelUri": "https://aka.ms/pretrained.20210205.microsoft.dte.00.12.bert_example_ner.en.onnx",
30-
"description": "(experimental) Bot Framework SDK release 4.12 - English ONNX V1.4 12-layer per-token entity base model",
31-
"minSDKVersion": "4.12.0"
32-
},
3327
"pretrained.20210218.microsoft.dte.00.12.bert_example_ner.en.onnx": {
3428
"releaseDate": "02/18/2021",
3529
"modelUri": "https://aka.ms/pretrained.20210218.microsoft.dte.00.12.bert_example_ner.en.onnx",
@@ -42,12 +36,6 @@
4236
"description": "Bot Framework SDK release 4.12 - Multilingual ONNX V1.4 12-layer per-token intent base model",
4337
"minSDKVersion": "4.12.0"
4438
},
45-
"pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx": {
46-
"releaseDate": "02/05/2021",
47-
"modelUri": "https://aka.ms/pretrained.20210205.microsoft.dte.00.06.bert_example_ner.en.onnx",
48-
"description": "(experimental) Bot Framework SDK release 4.12 - English ONNX V1.4 6-layer per-token entity base model",
49-
"minSDKVersion": "4.12.0"
50-
},
5139
"pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx": {
5240
"releaseDate": "02/18/2021",
5341
"modelUri": "https://aka.ms/pretrained.20210218.microsoft.dte.00.06.bert_example_ner.en.onnx",
@@ -60,71 +48,5 @@
6048
"description": "Bot Framework SDK release 4.12 - Multilingual ONNX V1.4 6-layer per-token intent base model",
6149
"minSDKVersion": "4.12.0"
6250
},
63-
"pretrained.20210401.microsoft.dte.00.12.en.onnx": {
64-
"releaseDate": "04/01/2021",
65-
"modelUri": "https://aka.ms/pretrained.20210401.microsoft.dte.00.12.en.onnx",
66-
"description": "Bot Framework SDK release 4.13 - English ONNX V1.4 12-layer per-token intent base model",
67-
"minSDKVersion": "4.13.0"
68-
},
69-
"pretrained.20210320.microsoft.dte.00.12.bert_example_ner_addon_free.en.onnx": {
70-
"releaseDate": "03/20/2021",
71-
"modelUri": "https://aka.ms/onnx/pretrained.20210320.microsoft.dte.00.12.bert_example_ner_addon_free.en.onnx",
72-
"description": "(experimental) Bot Framework SDK release 4.13 - English ONNX V1.4 12-layer per-token entity base model",
73-
"minSDKVersion": "4.13.0"
74-
},
75-
"pretrained.20210105.microsoft.dte.00.12.bert_example_ner_multilingual.onnx": {
76-
"releaseDate": "01/05/2021",
77-
"modelUri": "https://aka.ms/pretrained.20210105.microsoft.dte.00.12.bert_example_ner_multilingual.onnx",
78-
"description": "(experimental) Bot Framework SDK release 4.13 - Multilingual ONNX V1.4 12-layer per-token entity base model",
79-
"minSDKVersion": "4.13.0"
80-
},
81-
"pretrained.20210105.microsoft.dte.00.12.tulr_example_ner_multilingual.onnx": {
82-
"releaseDate": "01/05/2021",
83-
"modelUri": "https://aka.ms/pretrained.20210105.microsoft.dte.00.12.tulr_example_ner_multilingual.onnx",
84-
"description": "(experimental) Bot Framework SDK release 4.13 - Multilingual ONNX V1.4 12-layer per-token entity base model",
85-
"minSDKVersion": "4.13.0"
86-
},
87-
"pretrained.20210326.microsoft.dte.00.12.tulr_example_ner_multilingual_addon_free.onnx": {
88-
"releaseDate": "03/26/2021",
89-
"modelUri": "https://aka.ms/pretrained.20210105.microsoft.dte.00.12.tulr_example_ner_multilingual_addon_free.onnx",
90-
"description": "(experimental) Bot Framework SDK release 4.13 - Multilingual ONNX V1.4 12-layer per-token entity base model",
91-
"minSDKVersion": "4.13.0"
92-
},
93-
"pretrained.20210326.microsoft.dte.00.06.bert_example_ner_addon_free.en.onnx": {
94-
"releaseDate": "03/26/2021",
95-
"modelUri": "https://aka.ms/pretrained.20210326.microsoft.dte.00.06.bert_example_ner_addon_free.en.onnx",
96-
"description": "(experimental) Bot Framework SDK release 4.13 - English ONNX V1.4 6-layer per-token entity base model",
97-
"minSDKVersion": "4.13.0"
98-
},
99-
"pretrained.20210401.microsoft.dte.00.06.bert_example_ner_addon_free.en.onnx": {
100-
"releaseDate": "04/01/2021",
101-
"modelUri": "https://aka.ms/pretrained.20210401.microsoft.dte.00.06.bert_example_ner_addon_free.en.onnx.zip",
102-
"description": "(experimental) Bot Framework SDK release 4.13 - English ONNX V1.4 6-layer per-token entity base model",
103-
"minSDKVersion": "4.13.0"
104-
},
105-
"pretrained.20210401.microsoft.dte.00.03.bert_example_ner_addon_free.en.onnx": {
106-
"releaseDate": "04/01/2021",
107-
"modelUri": "https://aka.ms/pretrained.20210401.microsoft.dte.00.03.bert_example_ner_addon_free.en.onnx",
108-
"description": "(experimental) Bot Framework SDK release 4.13 - English ONNX V1.4 3-layer per-token entity base model",
109-
"minSDKVersion": "4.13.0"
110-
},
111-
"pretrained.20210205.microsoft.dte.00.06.bert_example_ner_multilingual.onnx": {
112-
"releaseDate": "02/05/2021",
113-
"modelUri": "https://aka.ms/pretrained.20210205.microsoft.dte.00.06.bert_example_ner_multilingual.onnx",
114-
"description": "(experimental) Bot Framework SDK release 4.13 - Multilingual ONNX V1.4 6-layer per-token entity base model",
115-
"minSDKVersion": "4.13.0"
116-
},
117-
"pretrained.20210205.microsoft.dte.00.06.tulr_example_ner_multilingual.onnx": {
118-
"releaseDate": "02/05/2021",
119-
"modelUri": "https://aka.ms/pretrained.20210205.microsoft.dte.00.06.tulr_example_ner_multilingual.onnx",
120-
"description": "(experimental) Bot Framework SDK release 4.13 - Multilingual ONNX V1.4 6-layer per-token entity base model",
121-
"minSDKVersion": "4.10.0"
122-
},
123-
"pretrained.20210326.microsoft.dte.00.06.tulr_example_ner_multilingual_addon_free.onnx": {
124-
"releaseDate": "03/26/2021",
125-
"modelUri": "https://aka.ms/pretrained.20210205.microsoft.dte.00.06.tulr_example_ner_multilingual_addon_free.onnx",
126-
"description": "(experimental) Bot Framework SDK release 4.13 - Multilingual ONNX V1.4 6-layer per-token entity base model",
127-
"minSDKVersion": "4.13.0"
128-
}
12951
}
13052
}

0 commit comments

Comments
 (0)