confidence and accuracy

Author: laujan

articles/ai-services/content-understanding/concepts/accuracy-confidence.md

@@ -1,61 +1,59 @@
 ---
-title: Understanding Confidence Scores in Azure AI Content Understanding.
+title: Understand and improve confidence scores in Azure AI Content Understanding.
 titleSuffix: Azure AI services
-description: Best practices to interpret and improve Azure AI Content Understanding accuracy and confidence scores.
+description: Tips for interpreting and improving Azure AI Content Understanding accuracy and confidence scores.
 author: laujan
 ms.author: admaheshwari
 manager: nitinme
 ms.service: azure-ai-content-understanding
 ms.topic: overview
-ms.date: 02/20/2025
+ms.date: 02/28/2025
 ---
 
-# Interpret and improve accuracy and confidence scores
+# Interpret and improve confidence and accuracy scores
 
-A confidence score indicates probability by measuring the degree of statistical certainty that the extracted result is detected correctly. The estimated accuracy is calculated by running a few different combinations of the training data to predict the labeled values. In this article, we share how to interpret accuracy and confidence scores and best practices for using those scores to improve accuracy and confidence results.
+> [!NOTE]
+>
+> Azure AI Content Understanding is currently available in preview. 
+>
+> While the service is in active development, confidence scores are only available for the document modality.
 
+Confidence scores quantify the probability that a result is accurately detected, by gauging the degree of statistical certainty. The estimated accuracy is derived from evaluating various combinations of the training data to forecast the labeled values. In this article, we share how to interpret accuracy and confidence scores and best practices for using those scores to improve both accuracy and confidence results.
+
+Confidence scores are essential as they indicate the model's level of certainty in its predictions. These scores enable users to assess the reliability of extracted data, guiding whether a human review is necessary. Additionally, confidence scores are instrumental in streamlining workflows and enhancing efficiency by minimizing the need for manual validation.
 
-Understanding Confidence Scores
-What are confidence scores?
-Confidence scores represent the probability that the extracted result is correct. For example, a confidence score of 0.95 (95%) suggests that the prediction is likely correct 19 out of 20 times. These scores are derived from various factors, including the quality of the input document, the similarity between the training data and the document being analyzed, and the model's ability to recognize patterns and features in the document.
-Why are confidence scores important?
-Confidence scores are important because they provide a measure of the model's certainty in its predictions. They help users make informed decisions about the reliability of the extracted data and determine whether human review is necessary. Confidence scores also play a crucial role in automating workflows and improving efficiency by reducing the need for manual validation.
-Supported fields
 Confidence scores are supported for extractive fields, including text, tables for documents and speech transcription. The specific fields supported may vary depending on the model and the use case.
 
-JSON output for documents
-"fields": {
-                    "ClientProjectManager": {
-                        "type": "string",
-                        "valueString": "Nestor Wilke",
-                        "spans": [
-                            {
-                                "offset": 4345,
-                                "length": 12
-                            }
-                        ],
-                        "confidence": 0.964,
-                        "source": "D(2,3.5486,8.3139,4.2943,8.3139,4.2943,8.4479,3.5486,8.4479)"
-                    },
-What are thresholds for confidence scores?
-Thresholds for confidence scores are predefined values that determine whether a prediction is considered reliable or requires further review. These thresholds can be set across different modalities to ensure consistent and accurate results. Setting appropriate thresholds is important because it helps balance the trade-off between automation and accuracy. By setting the right thresholds, users can ensure that only high-confidence predictions are automated, while low-confidence predictions are flagged for human review. This helps improve the overall accuracy and reliability of the predictions
-
-Improving Confidence Scores
-What are some common challenges with confidence scores?
-Common challenges with confidence scores include low-quality input documents, variability in document types, complexity of the documents, and limitations of the model in recognizing certain types of content or features. 
-Human in the Loop (HITL)
-What is Human in the Loop (HITL)?
-Human in the Loop (HITL) is a process that involves human intervention in the model's predictions to validate and correct the results. HITL helps improve the accuracy and reliability of the predictions by incorporating human expertise and judgment. HITL helps identify and correct errors, improve the model's performance, and enhance the overall quality of the predictions by human experts intervening only when the confidence scores are below a certain threshold.
-It can improved accuracy and reliability of the predictions, reduced errors, and enhanced overall quality of the results. 
-
-How can customers access confidence score in CU?
-For every field extraction, confidence score is listed as part of the field extraction output. You can also check confidence score as part of your JSON output under "confidence"
+## Supported fields 
+
+Confidence scores are supported for extractive various fields, including text, tables, and images. The specific fields supported may vary depending on the model and the use case. 
+
+## Confidence scores
+
+Confidence scores are listed for every field as part of the field extraction output:
+
+  :::image type="content" source="../media/confidence-accuracy/field-extraction-score.png" alt-text="Screenshot of field extraction scores from Azure AI Foundry.":::
+
+Confidence scores are also part of extraction output JSON file:
+
+  :::image type="content" source="../media/confidence-accuracy/json-output.png" alt-text="Screenshot of field extraction JSON output.":::
 
- Tips to improve confidence score
-1.    Correcting an expected output so that the model can understand the definition better.  Example: Here we can see the confidence score is 12%, to improve confidence score, we can go to label data, select auto label which will give us predicted field labels. Now we can correct our definition and it will show corrected field label. Test the analyzer again for better confidence score. Here it jumped to 98%. Confidence improvement will vary as per the complexity and nature of document. 
-
-2.    Adding more samples and label them for different variation and templates the model may expect. 
-3.    Add documents that contains various input values for the schema you want to extract. 
-4.    Improve the quality of your input documents. 
-5.    Incorporate human in the loop for lower confidence results. 
-Note: Confidence score is only available for document modality in the preview. For other modalities it will be added soon. 
+## Improving accuracy results
+
+Common challenges with confidence scores include the quality of input documents, diversity in document types, complexity of the documents, and limitations of the model to recognize certain types of content or features. These limitations underscore the need for continuous improvements and adaptations in the modeling process to enhance reliability and accuracy. Here are some tips
+
+* **Establish appropriate thresholds**. Setting thresholds can enhance the accuracy and reliability of predictions. These thresholds are predefined values that determine whether a prediction is considered reliable or requires further review. Establishing the right thresholds ensures that only high-confidence predictions are automated, while low-confidence predictions are flagged for human review. This approach helps increases the overall accuracy and reliability of predictions.
+
+* **Incorporate human review into workflows**. Human in the Loop (`HITL`) is a process where human intervention is introduced to validate and correct the  model's predictions. Utilizing human expertise and judgment enhances the accuracy and reliability of predictions. `HITL` allows for the identification and correction of errors, improves the model's performance, and elevates the overall quality of predictions by involving human experts only when confidence scores fall below a specified threshold.
+
+* **Include diverse input values for the schema you aim to extract**. To enrich the dataset and account for different variations and templates the model might encounter, use forms with unique values in each field and add labeled samples.
+
+* **Improve the quality of your input documents**. Clear, well-structured forms with consistent formatting typically result in higher confidence scores.
+
+## Related content
+
+* [Best practices for Content Understanding](best-practices.md)
+
+* [Document Intelligence accuracy and confidence scores](../../document-intelligence/concept/accuracy-confidence.md)
+
+