You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
> ***Custom neural models** do not provide accuracy scores during training.
22
22
> * Confidence scores for tables, table rows and table cells are available starting with the **2024-02-29-preview** API version for **custom models**.
23
23
24
-
25
-
Custom template models generate an estimated accuracy score when trained. Documents analyzed with a custom model produce a confidence score for extracted fields. In this article, learn to interpret accuracy and confidence scores and best practices for using those scores to improve accuracy and confidence results.
24
+
Custom template models generate an estimated accuracy score when trained. Documents analyzed with a custom model produce a confidence score for extracted fields. A confidence score indicates probability by measuring the degree of statistical certainty that the extracted result is detected correctly. The estimated accuracy is calculated by running a few different combinations of the training data to predict the labeled values. In this article, learn to interpret accuracy and confidence scores and best practices for using those scores to improve accuracy and confidence results.
26
25
27
26
## Accuracy scores
28
27
29
-
The output of a `build` (v3.0) or `train` (v2.1) custom model operation includes the estimated accuracy score. This score represents the model's ability to accurately predict the labeled value on a visually similar document.
30
-
The accuracy value range is a percentage between 0% (low) and 100% (high). The estimated accuracy is calculated by running a few different combinations of the training data to predict the labeled values.
28
+
The output of a `build` (v3.0) or `train` (v2.1) custom model operation includes the estimated accuracy score. This score represents the model's ability to accurately predict the labeled value on a visually similar document. Accuracy is measured within a percentage value range from 0% (low) to 100% (high). It's best to target a score of 80% or higher. For more sensitive cases, like financial or medical records, we recommend a score of close to 100%. You can also require human review.
31
29
32
30
**Document Intelligence Studio** </br>
33
31
**Trained custom model (invoice)**
@@ -50,6 +48,20 @@ Field confidence indicates an estimated probability between 0 and 1 that the pre
50
48
51
49
:::image type="content" source="media/accuracy-confidence/confidence-scores.png" alt-text="confidence scores from Document Intelligence Studio":::
52
50
51
+
## Improve confidence scores
52
+
53
+
After an analysis operation, review the JSON output. Examine the `confidence` values for each key/value result under the `pageResults` node. You should also look at the confidence score in the `readResults` node, which corresponds to the text-read operation. The confidence of the read results doesn't affect the confidence of the key/value extraction results, so you should check both. Here are some tips:
54
+
55
+
* If the confidence score for the `readResults` object is low, improve the quality of your input documents.
56
+
57
+
* If the confidence score for the `pageResults` object is low, ensure that the documents you're analyzing are of the same type.
58
+
59
+
* Consider incorporating human review into your workflows.
60
+
61
+
* Use forms that have different values in each field.
62
+
63
+
* For custom models, use a larger set of training documents. Tagging more documents teaches your model to recognize fields with greater accuracy.
64
+
53
65
## Interpret accuracy and confidence scores for custom models
54
66
55
67
When interpreting the confidence score from a custom model, you should consider all the confidence scores returned from the model. Let's start with a list of all the confidence scores.
@@ -55,6 +55,16 @@ With the introduction of [**custom classification models**](./concept-custom-cla
55
55
> [!NOTE]
56
56
> With the addition of **_custom neural model_** , there are a few limits to the compatibility of models that can be composed together.
57
57
58
+
* With the model compose operation, you can assign up to 200 models to a single model ID. If the number of models that I want to compose exceeds the upper limit of a composed model, you can use one of these alternatives:
59
+
60
+
* Classify the documents before calling the custom model. You can use the [Read model](concept-read.md) and build a classification based on the extracted text from the documents and certain phrases by using sources like code, regular expressions, or search.
61
+
62
+
* If you want to extract the same fields from various structured, semi-structured, and unstructured documents, consider using the deep-learning [custom neural model](concept-custom-neural.md). Learn more about the [differences between the custom template model and the custom neural model](concept-custom.md#compare-model-features).
63
+
64
+
* Analyzing a document by using composed models is identical to analyzing a document by using a single model. The `Analyze Document` result returns a `docType` property that indicates which of the component models you selected for analyzing the document. There's no change in pricing for analyzing a document by using an individual custom model or a composed custom model.
65
+
66
+
* Model Compose is currently available only for custom models trained with labels.
67
+
58
68
### Composed model compatibility
59
69
60
70
|Custom model type|Models trained with v2.1 and v2.0 | Custom template models v3.0 |Custom neural models 3.0|Custom Neural models v3.1|
Copy file name to clipboardExpand all lines: articles/ai-services/document-intelligence/concept-custom-classifier.md
+5-5Lines changed: 5 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ author: vkurpad
6
6
manager: nitinme
7
7
ms.service: azure-ai-document-intelligence
8
8
ms.topic: conceptual
9
-
ms.date: 02/29/2024
9
+
ms.date: 07/09/2024
10
10
ms.author: lajanuar
11
11
ms.custom:
12
12
- references_regions
@@ -49,9 +49,9 @@ Custom classification models are deep-learning-model types that combine layout a
49
49
50
50
Custom classification models can analyze a single- or multi-file documents to identify if any of the trained document types are contained within an input file. Here are the currently supported scenarios:
51
51
52
-
* A single file containing one document. For instance, a loan application form.
52
+
* A single file containing one document type, such as a loan application form.
53
53
54
-
* A single file containing multiple documents. For instance, a loan application package containing a loan application form, payslip, and bank statement.
54
+
* A single file containing multiple document types. For instance, a loan application package that contains a loan application form, payslip, and bank statement.
55
55
56
56
* A single file containing multiple instances of the same document. For instance, a collection of scanned invoices.
57
57
@@ -98,7 +98,7 @@ Classification models can now be trained on documents of different languages. Se
98
98
99
99
Supported file formats:
100
100
101
-
|Model | PDF |Image:<br>jpeg/jpg, png, bmp, tiff, heif| Microsoft Office:<br> Word (docx), Excel (xlxs), PowerPoint (pptx)|
101
+
|Model | PDF |Image:<br>`jpeg/jpg`, `png`, `bmp`, `tiff`, `heif`| Microsoft Office:<br> Word (docx), Excel (xlxs), PowerPoint (pptx)|
When you have more than one document in a file, the classifier can identify the different document types contained within the input file. The classifier response contains the page ranges for each of the identified document types contained within a file. This response can include multiple instances of the same document type.
131
131
132
132
::: moniker range=">=doc-intel-4.0.0"
133
-
The analyze operation now includes a `splitMode` property that gives you granular control over the splitting behavior.
133
+
The `analyze` operation now includes a `splitMode` property that gives you granular control over the splitting behavior.
134
134
135
135
* To treat the entire input file as a single document for classification set the splitMode to `none`. When you do so, the service returns just one class for the entire input file.
136
136
* To classify each page of the input file, set the splitMode to `perPage`. The service attempts to classify each page as an individual document.
Document Intelligence uses advanced machine learning technology to identify documents, detect and extract information from forms and documents, and return the extracted data in a structured JSON output. With Document Intelligence, you can use document analysis models, pre-built/pre-trained, or your trained standalone custom models.
37
37
38
-
Custom models now include [custom classification models](./concept-custom-classifier.md) for scenarios where you need to identify the document type before invoking the extraction model. Classifier models are available starting with the ```2023-07-31 (GA)``` API. A classification model can be paired with a custom extraction model to analyze and extract fields from forms and documents specific to your business to create a document processing solution. Standalone custom extraction models can be combined to create [composed models](concept-composed-models.md).
38
+
Custom models now include [custom classification models](./concept-custom-classifier.md) for scenarios where you need to identify the document type before invoking the extraction model. Classifier models are available starting with the ```2023-07-31 (GA)``` API. A classification model can be paired with a custom extraction model to analyze and extract fields from forms and documents specific to your business. Standalone custom extraction models can be combined to create [composed models](concept-composed-models.md).
39
39
40
40
::: moniker range=">=doc-intel-3.0.0"
41
41
@@ -51,7 +51,7 @@ To create a custom extraction model, label a dataset of documents with the value
51
51
52
52
> [!IMPORTANT]
53
53
>
54
-
> Starting with version 4.0 — 2024-02-29-preview API, custom neural models now support **overlapping fields** and **table, row and cell level confidence**.
54
+
> Starting with version 4.0 (2024-02-29-preview) API, custom neural models now support **overlapping fields** and **table, row and cell level confidence**.
55
55
>
56
56
57
57
The custom neural (custom document) model uses deep learning models and base model trained on a large collection of documents. This model is then fine-tuned or adapted to your data when you train the model with a labeled dataset. Custom neural models support structured, semi-structured, and unstructured documents to extract fields. Custom neural models currently support English-language documents. When you're choosing between the two model types, start with a neural model to determine if it meets your functional needs. See [neural models](concept-custom-neural.md) to learn more about custom document models.
@@ -76,7 +76,7 @@ If the language of your documents and extraction scenarios supports custom neura
76
76
77
77
* Supported file formats:
78
78
79
-
|Model | PDF |Image: </br>jpeg/jpg, png, bmp, tiff, heif | Microsoft Office: </br> Word (docx), Excel (xlsx), PowerPoint (pptx)|
79
+
|Model | PDF |Image: </br>`jpeg/jpg`, `png`, `bmp`, `tiff`, `heif`| Microsoft Office: </br> Word (docx), Excel (xlsx), PowerPoint (pptx)|
@@ -103,9 +103,31 @@ If the language of your documents and extraction scenarios supports custom neura
103
103
104
104
* For custom classification model training, the total size of training data is `1GB` with a maximum of 10,000 pages.
105
105
106
+
### Optimal training data
107
+
108
+
Training input data is the foundation of any machine learning model. It determines the quality, accuracy, and performance of the model. Therefore, it's crucial to create the best training input data possible for your Document Intelligence project. When you use the Document Intelligence custom model, you provide your own training data. Here are a few tips to help train your models effectively:
109
+
110
+
* Use text-based instead of image-based PDFs when possible. One way to identify an image*based PDF is to try selecting specific text in the document. If you can select only the entire image of the text, the document is image based, not text based.
111
+
112
+
* Organize your training documents by using a subfolder for each format (JPEG/JPG, PNG, BMP, PDF, or TIFF).
113
+
114
+
* Use forms that have all of the available fields completed.
115
+
116
+
* Use forms with differing values in each field.
117
+
118
+
* Use a larger dataset (more than five training documents) if your images are low quality.
119
+
120
+
* Determine if you need to use a single model or multiple models composed into a single model.
121
+
122
+
* Consider segmenting your dataset into folders, where each folder is a unique template. Train one model per folder, and compose the resulting models into a single endpoint. Model accuracy can decrease when you have different formats analyzed with a single model.
123
+
124
+
* Consider segmenting your dataset to train multiple models if your form has variations with formats and page breaks. Custom forms rely on a consistent visual template.
125
+
126
+
* Ensure that you have a balanced dataset by accounting for formats, document types, and structure.
127
+
106
128
### Build mode
107
129
108
-
The build custom model operation adds support for the *template* and *neural* custom models. Previous versions of the REST API and client libraries only supported a single build mode that is now known as the *template* mode.
130
+
The `build custom model` operation adds support for the *template* and *neural* custom models. Previous versions of the REST API and client libraries only supported a single build mode that is now known as the *template* mode.
109
131
110
132
* Template models only accept documents that have the same basic page structure—a uniform visual appearance—or the same relative positioning of elements within the document.
111
133
@@ -149,6 +171,10 @@ Document Intelligence v3.1 and later models support the following tools, applica
The life cycle of a custom model depends on the API version that is used to train it. If the API version is a general availability (GA) version, the custom model has the same life cycle as that version. The custom model isn't available for inference when the API version is deprecated. If the API version is a preview version, the custom model has the same life cycle as the preview version of the API.
0 commit comments