Merge branch 'MicrosoftDocs:main' into master

Author: rathishrms

articles/ai-services/document-intelligence/concept-custom-generative.md

@@ -1,87 +1,116 @@
 ---
-title: Azure AI Document Intelligence (formerly Form Recognizer) custom generative field extraction
+title: Azure AI Document Intelligence (formerly Form Recognizer) custom generative document field extraction
 titleSuffix: Azure AI services
 description: Custom generative AI model extracts user-specified fields from documents across a wide variety of visual templates.
 author: laujan
 manager: nitinme
 ms.service: azure-ai-document-intelligence
-ms.custom:
-  - ignite-2023
 ms.topic: overview
-ms.date: 08/07/2024
+ms.date: 08/09/2024
 ms.author: lajanuar
 monikerRange: '>=doc-intel-4.0.0'
 ---
 
-# Document Intelligence custom generative model
+# Document Field extraction - custom generative AI model
 
-[!INCLUDE [preview-version-notice](includes/preview-notice.md)]
+> [!IMPORTANT]
+>
+> * Document Intelligence public preview releases provide early access to features that are in active development. Features, approaches, and processes may change, prior to General Availability (GA), based on user feedback.
+> * The public preview version of Document Intelligence client libraries default to REST API version [**2024-07-31-preview**](/rest/api/aiservices/operation-groups?view=rest-aiservices-2024-07-31-preview&preserve-view=true) and is currently only available in the following Azure regions.
+>   * **East US**
+>   * **West US2**
+>   * **West Europe**
+>   * **North Central US**
+>
+> * **The new custom generative model in AI Studio is only available in the North Central US region**:
 
-The custom generative model combines the power of document understanding with Large Language Models (LLMs) and the rigor and schema from custom extraction capabilities. Custom generative extraction enables you to easily automate data extraction workflows for any type of document, with minimal labeling and greater accuracy and speed.
+The document field extraction (custom generative AI) model utilizes generative AI to extract user-specified fields from documents across a wide variety of visual templates. The custom generative AI model combines the power of document understanding with Large Language Models (LLMs) and the rigor and schema from custom extraction capabilities to create a model with high accuracy in minutes. With this generative model type, you can start with a single document and go through the schema addition and model creation process with minimal labeling. The custom generative model allows developers and enterprises to easily automate data extraction workflows with greater accuracy and speed for any type of document. The custom generative AI model excels in extracting simple fields from documents without labeled samples. However, providing a few labeled samples improves the extraction accuracy for complex fields and user-defined fields like tables. You can use the [REST API](/rest/api/aiservices/operation-groups?view=rest-aiservices-2024-07-31-preview&preserve-view=true) or client libraries to submit a document for analysis with a model build and use the custom generative process.
 
-## Custom generative model key features
+## Custom generative AI model benefits
 
 * **Automatic labeling**. Utilize large language models (LLM) and extract user-specified fields for various document types and visual templates.
+
 * **Improved Generalization**. Extract data from unstructured data and varying document templates with higher accuracy.
-* **Grounded results**. Localize the data extracted in documents and ensure the response is generated from the content and enables human review workflows.
-* **High confidence scores**. Use confidence scores and quickly filter high quality extracted data for downstream processing and lower manual review time.
+
+* **Grounded results**. Localize the data extracted in the documents. Custom generative models ground the results where applicable, ensuring the response is generated from the content and enable human review workflows.
+
+* **Confidence scores**. Use confidence scores for each extracted field to, filter high quality extracted data, maximize straight through processing of documents and minimize human review costs.
 
 ### Common use cases
 
-* **Contract Lifecycle Management**. Build a generative model and extract the fields, clauses, and obligations from a wide array of contract types.  
-* **Loan & Mortgage Applications**. Automation of loan and mortgage application process enables banks, lenders, and government entities to quickly process loan and mortgage application.  
-* **Financial Services**. Analyze complex documents like financial reports and asset management reports.
-* **Expense management**. The custom generative model can extract expenses, receipts, and invoices with varying formats and templates.  
+* **Contract Lifecycle Management**. Build a generative model and extract the fields, clauses, and obligations from a wide array of contract types.
+
+* **Loan & Mortgage Applications**. Automation of loan and mortgage application process enables banks, lenders, and government entities to quickly process loan and mortgage application.
+
+* **Financial Services**. With the custom generative AI model, analyze complex documents like financial reports and asset management reports.
+
+* **Expense management**. Receipts and invoices from various retailers and businesses need to be parsed to validate the expenses. The custom generative AI model can extract expenses across different formats and documents with varying templates.
+
+### Managing the training dataset
+
+With our other custom models, you need to maintain the dataset, add new samples, and train the model for accuracy improvements. With the custom generative AI model, the labeled documents are transformed, encrypted, and stored as part of the model. This process ensures that the model can continually use the labeled samples to improve the extraction quality. As with other custom models, models are stored in Microsoft storage, and you can delete them anytime.
+
+The Document Intelligence service does manage your datasets, but your documents are stored encrypted and only used to improve the model results for your specific model. A service-manged key can be used to encrypt your data or it can be optionally encrypted with a customer managed key. The change in management and lifecycle of the dataset only applies to custom generative models.
 
 ## Model capabilities  
 
-The Custom generative model currently supports dynamic table with the `2024-07-31-preview` and the following fields:
+Field extraction custom generative model currently supports dynamic table with the `2024-07-31-preview` and the following fields:
 
 | Form fields | Selection marks | Tabular fields | Signature | Region labeling | Overlapping fields |
 |:--:|:--:|:--:|:--:|:--:|:--:|
 |Supported| Supported |Supported| Unsupported |Unsupported |Supported|
 
 ## Build mode  
 
-The `build custom model` operation supports custom _template_, _neural_ and _generative_ models, _see_ [Custom model build mode](concept-custom.md#build-mode):
+The `build custom model` operation supports custom **template**, **neural**, and **generative** models, _see_ [Custom model build mode](concept-custom.md#build-mode). Here are the differences in the model types:
+
+* **Custom generative AI models** can process complex documents with various formats, varied templates, and unstructured data.
+
+* **Custom neural models** support complex document processing and also support more variance in pages for structured and semi-structured documents.
 
-* **Custom generative models** can process complex documents in various formats, templates, and unstructured data.
-* **Custom neural models** support complex document processing and also support more variance in page for structured and semi-structured documents.
 * **Custom template models** rely on consistent visual templates, such as questionnaires or applications, to extract the labeled data.
 
 ## Languages and locale support
 
-The custom generative model `2024-07-31-preview` version supports the **en-us** locale. For more information on language support, *see* [Language support - custom models](language-support-custom.md).
+Field extraction custom generative model `2024-07-31-preview` version supports the **en-us** locale. For more information on language support, _see_ [Language support - custom models](language-support-custom.md).
 
 ## Region support
 
-The custom generative model `2024-07-31-preview` version is only available in `North Central US`.  
+Field extraction custom generative model `2024-07-31-preview` version is only available in `North Central US`.  
 
-## Input requirements 
+## Input requirements
 
 [!INCLUDE [input requirements](./includes/input-requirements.md)]
 
 ## Best practices  
 
 * **Representative data**. Use representative documents that target actual data distribution, and train a high-quality custom generative model. For example, if the target document includes partially filled tabular fields, add training documents that consist of partially filled tables. Or if field is named date, values for this field should be a date as random strings can affect model performance.
+
 * **Field naming**. Choose a precise field name that represents the field values. For example, for a field value containing the Transaction Date, consider naming the field _TransactionDate_ instead of `Date1`.
-* **Field Description**. Provide more contextual information in description to help clarify the field that needs to be extracted. Examples include location in the document, potential field labels it may be associated with, ways to differentiate with other terms that could be ambiguous.  
-* **Variation**. Custom generative models can generalize across different document templates of the same document type. As a best practice, create a single model for all variations of a document type. Ideally, include a visual template for each type, especially for ones that 
+
+* **Field Description**. Provide more contextual information in description to help clarify the field that needs to be extracted. Examples include location in the document, potential field labels it can be associated with, and ways to differentiate with other terms that could be ambiguous.
+
+* **Variation**. Custom generative models can generalize across different document templates of the same document type. As a best practice, create a single model for all variations of a document type. Ideally, include a visual template for each type, especially for ones that
 
 ## Service guidance
 
 * The Custom Generative preview model doesn't currently support fixed table and signature extraction.
+
 * Inference on the same document could yield slightly different results across calls and is a known limitation of current `GPT` models.
+
 * Confidence scores for each field might vary. We recommend testing with your representative data to establish the confidence thresholds for your scenario.
-* Grounding, especially for tabular fields, is challenging and might not be perfect in some cases.  
+
+* Grounding, especially for tabular fields, is challenging and might not be perfect in some cases.
+
 * Latency for large documents is high and a known limitation in preview.
+
 * Composed models don't support custom generative extraction.
 
 ## Training a model  
 
 Custom generative models are available with the `2024-07-31-preview` version and later models.
 
-The `build operation` to train model supports the ```buildMode``` property, to train a custom generative model, set the ```buildMode``` to ```generative```.
+The `build operation` to train model supports the `buildMode` property, to train a custom generative model, set the `buildMode` to `generative`.
 
 ```bash
 

articles/ai-services/document-intelligence/includes/preview-notice.md

@@ -13,8 +13,7 @@ ms.date: 07/31/2024
 
 > [!IMPORTANT]
 >
-> * Document Intelligence public preview releases provide early access to features that are in active development.
-> * Features, approaches, and processes may change, prior to General Availability (GA), based on user feedback.
+> * Document Intelligence public preview releases provide early access to features that are in active development. Features, approaches, and processes may change, prior to General Availability (GA), based on user feedback.
 > * The public preview version of Document Intelligence client libraries default to REST API version [**2024-07-31-preview**](/rest/api/aiservices/operation-groups?view=rest-aiservices-2024-07-31-preview&preserve-view=true).
 > * Public preview version [**2024-07-31-preview**](/rest/api/aiservices/operation-groups?view=rest-aiservices-2024-07-31-preview&preserve-view=true) is currently only available in the following Azure regions. Note that the custom generative (document field extraction) model in AI Studio is only available in North Central US region:
 >   * **East US**

articles/azure-monitor/agents/azure-monitor-agent-manage.md

@@ -373,8 +373,7 @@ The AgentSettings DCR currently supports configuring the following parameters:
 
 | Parameter | Description | Valid values |
 | --------- | ----------- | ----------- |
-
-| `DiscQuotaInMb` | Defines the amount of disk space used by the Azure Monitor Agent log files and cache. | 1,000-50,000 (or 1-50 GB) |
+| `MaxDiskQuotaInMB` | Defines the amount of disk space used by the Azure Monitor Agent log files and cache. | 1000-50000 (in MB) |
 | `TimeReceivedForForwardedEvents` | Changes WEF column in the Sentinel WEF table to use TimeReceived instead of TimeGenerated data | 0 or 1 |
 
 ### Setting up AgentSettings DCR
@@ -397,9 +396,9 @@ N/A
 
     [Install AMA](#installation-options) on your VM.
 
-1. **Create a DCR via template deployment:**
+1. **Create a DCR:**
 
-    The following example changes the maximum amount of disk space used by AMA cache to 5 GB.
+    This example sets the maximum amount of disk space used by AMA cache to 5000 MB.
 
     ```json
     {
@@ -431,24 +430,70 @@ N/A
     }
     ```
     
-    > [!NOTE]
-    > You can use the Get DataCollectionRule API to get the DCR payload you created with this template.
-    
-1. **Associate DCR with your machine:**
+1. **Associate the DCR with your machine:**
 
-    This can be done with a template or by using the [Create API](/rest/api/monitor/data-collection-rule-associations/create) with the following details:
-  
-    * **AssociationName:** agentSettings
-    * **ResourceUri:** Full ARM ID of the VM
-    * **api-version:** 2023-03-11 (Old API version is also fine)
-    * **Body:**
-        ```json
-            {
-            "properties": {
-                "dataCollectionRuleId": “Full ARM ID for agent setting DCR”
-                }
-            }
-        ```
+   Use these ARM template and parameter files:
+
+    **ARM template file**
+    
+    ```json
+    {
+      "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
+      "contentVersion": "1.0.0.0",
+      "parameters": {
+        "vmName": {
+          "type": "string",
+          "metadata": {
+            "description": "The name of the virtual machine."
+          }
+        },
+        "associationName": {
+          "type": "string",
+          "metadata": {
+            "description": "The name of the association."
+          }
+        },
+        "dataCollectionRuleId": {
+          "type": "string",
+          "metadata": {
+            "description": "The resource ID of the data collection rule."
+          }
+        }
+      },
+      "resources": [
+        {
+          "type": "Microsoft.Insights/dataCollectionRuleAssociations",
+          "apiVersion": "2021-09-01-preview",
+          "scope": "[format('Microsoft.Compute/virtualMachines/{0}', parameters('vmName'))]",
+          "name": "[parameters('associationName')]",
+          "properties": {
+            "description": "Association of data collection rule. Deleting this association will break the data collection for this virtual machine.",
+            "dataCollectionRuleId": "[parameters('dataCollectionRuleId')]"
+          }
+        }
+      ]
+    }
+    ```
+    
+    **Parameter file**
+    
+    ```json
+    {
+      "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
+      "contentVersion": "1.0.0.0",
+      "parameters": {
+        "vmName": {
+          "value": "my-azure-vm"
+        },
+        "associationName": {
+          "value": "my-windows-vm-my-dcr"
+        },
+        "dataCollectionRuleId": {
+          "value": "/subscriptions/00000000-0000-0000-0000-000000000000/resourcegroups/my-resource-group/providers/microsoft.insights/datacollectionrules/my-dcr"
+        }
+       }
+    }
+    ```
 
 1. **Activate the settings:**
 

articles/azure-monitor/alerts/resource-manager-alerts-metric.md

@@ -4911,22 +4911,23 @@ resource metricAlert 'Microsoft.Insights/metricAlerts@2018-03-01' = {
   "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
   "contentVersion": "1.0.0.0",
   "metadata": {
-  "parameters": {
-    "appName": {
-      "type": "string"
-    },
-    "pingURL": {
-      "type": "string"
-    },
-    "pingText": {
-      "type": "string",
-      "defaultValue": ""
-    },
-    "actionGroupId": {
-      "type": "string"
-    },
-    "location": {
-      "type": "string"
+    "parameters": {
+      "appName": {
+        "type": "string"
+      },
+      "pingURL": {
+        "type": "string"
+      },
+      "pingText": {
+        "type": "string",
+        "defaultValue": ""
+      },
+      "actionGroupId": {
+        "type": "string"
+      },
+      "location": {
+        "type": "string"
+      }
     }
   },
   "variables": {

articles/azure-monitor/app/opentelemetry-configuration.md

@@ -15,16 +15,14 @@ This article covers configuration settings for the Azure Monitor OpenTelemetry d
 
 ## Connection string
 
-A connection string in Application Insights defines the target location for sending telemetry data, ensuring it reaches the appropriate resource for monitoring and analysis.
-
+A connection string in Application Insights defines the target location for sending telemetry data.
 ### [ASP.NET Core](#tab/aspnetcore)
 
 Use one of the following three ways to configure the connection string:
 
-- Add `UseAzureMonitor()` to your application startup, in your `program.cs` class.
+- Add `UseAzureMonitor()` to your `program.cs` file:
 
     ```csharp
-    // Create a new ASP.NET Core web application builder.    
     var builder = WebApplication.CreateBuilder(args);
 
     // Add the OpenTelemetry telemetry service to the application.
@@ -33,10 +31,8 @@ Use one of the following three ways to configure the connection string:
         options.ConnectionString = "<Your Connection String>";
     });
 
-    // Build the ASP.NET Core web application.
     var app = builder.Build();
 
-    // Start the ASP.NET Core web application.    
     app.Run();
     ```
 

articles/azure-monitor/best-practices-logs.md

@@ -14,7 +14,11 @@ This article provides architectural best practices for Azure Monitor Logs. The g
 
 
 ## Reliability
-[Reliability](/azure/well-architected/resiliency/overview) refers to the ability of a system to recover from failures and continue to function. Instead of trying to prevent failures altogether in the cloud, the goal is to minimize the effects of a single failing component. Use the following information to minimize failure of your Log Analytics workspaces and to protect the data they collect.
+[Reliability](/azure/well-architected/resiliency/overview) refers to the ability of a system to recover from failures and continue to function. The goal is to minimize the effects of a single failing component. Use the following information to minimize failure of your Log Analytics workspaces and to protect the data they collect.
+
+This video provides an overview of reliability and resilience options available for Log Analytics workspaces:
+
+> [!VIDEO https://www.youtube.com/embed/CYspm1Yevx8?cc_load_policy=1&cc_lang_pref=auto]
 
 [!INCLUDE [waf-logs-reliability](includes/waf-logs-reliability.md)]