Skip to content

Commit b0a58f9

Browse files
authored
Update concept-accuracy-confidence.md
1 parent fdd71b1 commit b0a58f9

File tree

1 file changed

+12
-8
lines changed

1 file changed

+12
-8
lines changed

articles/ai-services/document-intelligence/concept-accuracy-confidence.md

Lines changed: 12 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -18,11 +18,11 @@ ms.author: lajanuar
1818

1919
> [!NOTE]
2020
>
21-
> * **Custom neural models do not provide accuracy scores during training**.
22-
> * Confidence scores for structured fields such as tables are currently unavailable.
21+
> * **Custom neural models** do not provide accuracy scores during training.
22+
> * Confidence scores for tables, table rows and table cells are available starting with the **2024-02-29-preview** API version for **custom models**.
2323
2424

25-
Custom models generate an estimated accuracy score when trained. Documents analyzed with a custom model produce a confidence score for extracted fields. In this article, learn to interpret accuracy and confidence scores and best practices for using those scores to improve accuracy and confidence results.
25+
Custom template models generate an estimated accuracy score when trained. Documents analyzed with a custom model produce a confidence score for extracted fields. In this article, learn to interpret accuracy and confidence scores and best practices for using those scores to improve accuracy and confidence results.
2626

2727
## Accuracy scores
2828

@@ -38,21 +38,25 @@ The accuracy value range is a percentage between 0% (low) and 100% (high). The e
3838

3939
> [!NOTE]
4040
>
41-
> * **Table cell confidence scores are now included with the 2024-02-29-preview API version**.
41+
> * **Table, row and cell confidence scores are now included with the 2024-02-29-preview API version**.
4242
> * Confidence scores for table cells from custom models is added to the API starting with the 2024-02-29-preview API.
4343
4444
Document Intelligence analysis results return an estimated confidence for predicted words, key-value pairs, selection marks, regions, and signatures. Currently, not all document fields return a confidence score.
4545

4646
Field confidence indicates an estimated probability between 0 and 1 that the prediction is correct. For example, a confidence value of 0.95 (95%) indicates that the prediction is likely correct 19 out of 20 times. For scenarios where accuracy is critical, confidence can be used to determine whether to automatically accept the prediction or flag it for human review.
4747

48-
Confidence scores have two data points: the field level confidence score and the text extraction confidence score. In addition to the field confidence of position and span, the text extraction confidence in the ```pages``` section of the response is the model's confidence in the text extraction (OCR) process. The two confidence scores should be combined to generate one overall confidence score.
49-
5048
**Document Intelligence Studio** </br>
5149
**Analyzed invoice prebuilt-invoice model**
5250

5351
:::image type="content" source="media/accuracy-confidence/confidence-scores.png" alt-text="confidence scores from Document Intelligence Studio":::
5452

55-
## Interpret accuracy and confidence scores
53+
## Interpret accuracy and confidence scores for custom models
54+
55+
When interpreting the confidence score from a custom model, you should consider all the confidence scores returned from the model. Let's start with a list of all the confidence scores.
56+
1. **Document type confidence score**: The document type confidence is an indicator of closely the analyzed document resembleds documents in the training dataset. When the document type confidence is low, this is indicative of template or structural variations in the analyzed document. To improve the document type confidence, label a document with that specific variation and add it to your training dataset. Once the model is re-trained, it should be better equipped to handl that class of variations.
57+
2. **Field level confidence**: Each labled field extracted has an associated confidence score. This score reflects the model's confidence on the position of the value extracted. While evaluating the confidence you should also look at the underlying extraction confidence to generate a comprehensive confidence for the extracted result. Evaluate the OCR results for text extraction or selection marks depending on the field type to generate a composite confidence score for the field.
58+
3. **Word confidence score** Each word extracted within the document has an associated confidence score. The score represents the confidence of the transcription. The pages array contains an array of words, each word has an associated span and confidence. Spans from the custom field extracted values will match the spans of the extracted words.
59+
4. **Selection mark confidence score**: The pages array also contains an array of selection marks, each selection mark has a confidence score representing the confidence of the seletion mark and selection state detection. When a labeled field is a selection mark, the custom field selection confidence combined with the selection mark confidence is an accurate representation of the overall confidence that the field was extracted correctly.
5660

5761
The following table demonstrates how to interpret both the accuracy and confidence scores to measure your custom model's performance.
5862

@@ -65,7 +69,7 @@ The following table demonstrates how to interpret both the accuracy and confiden
6569

6670
## Table, row, and cell confidence
6771

68-
With the addition of table, row and cell confidence with the ```2024-02-29-preview``` API, here are some common questions that should help with interpreting the scores:
72+
With the addition of table, row and cell confidence with the ```2024-02-29-preview``` API, here are some common questions that should help with interpreting the table, row and cell scores:
6973

7074
**Q:** Is it possible to see a high confidence score for cells, but a low confidence score for the row?<br>
7175

0 commit comments

Comments
 (0)