Merge pull request #192038 from aahill/baher-orchestration

screenshots, quickstart updates, evaluation metrics

Author: v-regandowner

articles/cognitive-services/language-service/orchestration-workflow/concepts/evaluation-metrics.md

@@ -0,0 +1,42 @@
+---
+title: Orchestration workflow model evaluation metrics
+titleSuffix: Azure Cognitive Services
+description: Learn about evaluation metrics in orchestration workflow
+services: cognitive-services
+author: aahill
+manager: nitinme
+ms.service: cognitive-services
+ms.subservice: language-service
+ms.topic: overview
+ms.date: 03/17/2022
+ms.author: aahi
+ms.custom: language-service-orchestration
+---
+
+# Evaluation metrics for orchestration workflow models
+
+Model evaluation in orchestration workflow uses the following metrics:
+
+|Metric |Description  |Calculation  |
+|---------|---------|---------|
+|Precision     |  The ratio of successful recognitions to all attempted recognitions. This shows how many times the model's entity recognition is truly a good recognition.       | `Precision = #True_Positive / (#True_Positive + #False_Positive)`        |
+|Recall     | The ratio of successful recognitions to the actual number of entities present.        | `Recall = #True_Positive / (#True_Positive + #False_Negatives)`        |
+|F1 score    |  The combination of precision and recall.       |  `F1 Score = 2 * Precision * Recall / (Precision + Recall)`       |
+
+## Confusion matrix
+
+A Confusion matrix is an N x N matrix used for model performance evaluation, where N is the number of intents.
+The matrix compares the actual tags with the tags predicted by the model.
+This gives a holistic view of how well the model is performing and what kinds of errors it is making.
+
+You can use the Confusion matrix to identify intents that are too close to each other and often get mistaken (ambiguity). In this case consider merging these intents  together. If that isn't possible, consider adding more tagged examples of both intents to help the model differentiate between them.
+
+You can calculate the model-level evaluation metrics from the confusion matrix:
+
+* The *true positive* of the model is the sum of *true Positives* for all intents.
+* The *false positive* of the model is the sum of *false positives* for all intents.
+* The *false Negative* of the model is the sum of *false negatives* for all intents.
+
+## Next steps
+
+[Train a model in Language Studio](../how-to/train-model.md)

articles/cognitive-services/language-service/orchestration-workflow/faq.md

@@ -16,22 +16,22 @@ ms.custom: ignite-fall-2021, mode-other
 # Frequently asked questions for orchestration workflows
 
 Use this article to quickly get the answers to common questions about orchestration workflows
-<!--
+
 ## How do I create a project?
 
 See the [quickstart](./quickstart.md) to quickly create your first project, or the [how-to article](./how-to/create-project.md) for more details. 
--->
+
 ## How do I connect other service applications in orchestration workflow projects?
 
 See [How to create projects and build schemas](./how-to/create-project.md) for information on connecting another project as an intent.
 
 ## Which LUIS applications can I connect to in orchestration workflow projects?
 
-LUIS applications that use the Language resource as their authoring resource will be available for connection. You can only connect to LUIS applications that are owned by the same resource. This option will only be available for resources in West Europe, as it's the only common available region between LUIS and CLU. <!--See [region limits](./service-limits.md#region-limits) for more information. -->
+LUIS applications that use the Language resource as their authoring resource will be available for connection. You can only connect to LUIS applications that are owned by the same resource. This option will only be available for resources in West Europe, as it's the only common available region between LUIS and CLU.
 
 ## Training is taking a long time, is this expected?
 
-For conversation projects, long training times are expected. Based on the number of examples you have your training times may vary from 5 minutes to 1 hour or more. 
+For orchestration projects, long training times are expected. Based on the number of examples you have your training times may vary from 5 minutes to 1 hour or more. 
 
 ## Can I add entities to orchestration workflow projects?
 
@@ -64,4 +64,4 @@ Yes, all the APIs [are available](https://aka.ms/clu-apis).
 
 ## Next steps
 
-[Conversational language understanding overview](overview.md)
+[Orchestration workflow overview](overview.md)

articles/cognitive-services/language-service/orchestration-workflow/how-to/deploy-query-model.md

@@ -14,25 +14,31 @@ ms.devlang: csharp, python
 ms.custom: language-service-orchestration
 ---
 
-# Deploy and test an orchestration workflow model
+# Deploy and test model
 
 After you have [trained a model](./train-model.md) on your dataset, you're ready to deploy it. After deploying your model, you'll be able to query it for predictions. 
 
-## Deploy model
+> [!Tip]
+> Before deploying a model, make sure to view the model details to make sure that the model is performing as expected.
+> You can only have ten deployment names.
 
-Deploying a model is to host it and make it available for predictions through an endpoint. You can only have 1 deployed model per project, deploying another one will overwrite the previously deployed model.
+## Orchestration workflow model deployments
+
+Deploying a model hosts and makes it available for predictions through an endpoint.
 
 When a model is deployed, you will be able to test the model directly in the portal or by calling the API associated to it.
 
-When you're deploying an orchestration workflow project, A small window will show up for you to confirm your deployment, and configure parameters for connected services.
+1. From the left side, click on **Deploy model**.
+
+2.	Click on **Add deployment** to submit a new deployment job.
+
+    In the window that appears, you can create a new deployment name by giving the deployment a name or override an existing deployment name. Then, you can add a trained model to this deployment name and press next.
 
-If you're connecting one or more LUIS applications, specify the deployment name, and whether you're using *slot* or *version* type deployment.       
-* The *slot* deployment type requires you to pick between a production and staging slot.
-* The *version* deployment type requires you to specify the version you have published.
+3. If you're connecting one or more LUIS applications or conversational language understanding projects, you have to specify the deployment name.
 
-No configurations are required for custom question answering and conversational language understanding connections, or unlinked intents.
+    No configurations are required for custom question answering or unlinked intents.
 
-LUIS projects **must be published** to the slot configured during the orchestration workflow deployment, and custom question answering KBs must also be published to their production slots.
+    LUIS projects **must be published** to the slot configured during the Orchestration deployment, and custom question answering KBs must also be published to their Production slots.
 
 1.	Click on **Add deployment** to submit a new deployment job.
 
@@ -48,7 +54,6 @@ LUIS projects **must be published** to the slot configured during the orchestrat
 
     :::image type="content" source="../media/deploy-connected-services.png" alt-text="A screenshot showing the deployment screen for orchestration workflow projects." lightbox="../media/deploy-connected-services.png":::
 
-
 ## Send a request to your model
 
 Once your model is deployed, you can begin using the deployed model for predictions. Outside of the test model page, you can begin calling your deployed model via API requests to your provided custom endpoint. This endpoint request obtains the intent and entity predictions defined within the model.
@@ -58,17 +63,16 @@ You can get the full URL for your endpoint by going to the **Deploy model** page
 :::image type="content" source="../media/prediction-url.png" alt-text="Screenshot showing the prediction request and URL" lightbox="../media/prediction-url.png":::
 
 Add your key to the `Ocp-Apim-Subscription-Key` header value, and replace the query and language parameters.
-<!--
+ 
 > [!TIP]
 > As you construct your requests, see the [quickstart](../quickstart.md?pivots=rest-api#query-model) and REST API [reference documentation](https://aka.ms/clu-apis) for more information.
--->
 
 ### Use the client libraries (Azure SDK)
 
 You can also use the client libraries provided by the Azure SDK to send requests to your model. 
 
 > [!NOTE]
-> The client library for conversational language understanding is only available for:
+> The client library for Orchestration workflow is only available for:
 > * .NET
 > * Python
 

articles/cognitive-services/language-service/orchestration-workflow/how-to/train-model.md

@@ -1,7 +1,7 @@
 ---
 title: How to train and evaluate models in orchestration workflow projects
 titleSuffix: Azure Cognitive Services
-description: Use this article to train an orchestration workflow model and view its evaluation details to make improvements.
+description: Use this article to train an orchestration model and view its evaluation details to make improvements.
 services: cognitive-services
 author: aahill
 manager: nitinme
@@ -15,41 +15,39 @@ ms.custom: language-service-orchestration
 
 # Train and evaluate orchestration workflow models
 
-After you have completed tagging your utterances, you can train your model. Training is the act of converting the current state of your project's training data to build a model that can be used for predictions. Every time you train, you have to name your training instance. 
+After you have completed [tagging your utterances](./tag-utterances.md), you can train your model. Training is the act of converting the current state of your project's training data to build a model that can be used for predictions. Every time you train, you have to name your training instance. 
 
 You can create and train multiple models within the same project. However, if you re-train a specific model it overwrites the last state.
 
-The training times can be anywhere from a few seconds, up to a couple of hours when you reach the maximum limit of utterances. Before training, you will have the option to enable evaluation, which lets you view how your model performs. 
+The training times can be anywhere from a few seconds, up to a couple of hours when you reach high numbers of utterances.
 
 ## Train model
 
 Select **Train model** on the left of the screen. Select **Start a training job** from the top menu.
 
 Enter a new model name or select an existing model from the **Model Name** dropdown. 
 
-Select whether you want to evaluate your model by changing the **Run evaluation with training** toggle. If enabled, your tagged utterances will be spilt into 2 parts; 80% for training, 20% for testing. Afterwards, you'll be able to see the model's evaluation results.
-
 :::image type="content" source="../media/train-model.png" alt-text="A screenshot showing the Train model page for Conversational Language Understanding projects." lightbox="../media/train-model.png":::
 
-Click the **Train** button and wait for training to complete. You will see the training status of your model in the view model details page. Only successfully completed tasks will generate models.
+Click the **Train** button and wait for training to complete. You will see the training status of your model in the view model details page. Only successfully completed jobs will generate models.
 
 ## Evaluate model
 
-After model training is completed, you can view your model details and see how well it performs against the test set if you enabled evaluation in the training step. Observing how well your model performed is called evaluation. The test set is composed of 20% of your utterances, and this split is done at random before training. The test set consists of data that was not introduced to the model during the training process. For the evaluation process to complete there must be at least 10 utterances in your training set.
+After model training is completed, you can view your model details and see how well it performs against the test set in the training step. Observing how well your model performed is called evaluation. The test set is composed of 20% of your utterances, and this split is done at random before training. The test set consists of data that was not introduced to the model during the training process. For the evaluation process to complete there must be at least 10 utterances in your training set.
 
 In the **view model details** page, you'll be able to see all your models, with their current training status, and the date they were last trained.
 
 :::image type="content" source="../media/model-page-1.png" alt-text="A screenshot showing the model details page for Conversational Language Understanding projects." lightbox="../media/model-page-1.png":::
 
 * Click on the model name for more details. A model name is only clickable if you've enabled evaluation before hand. 
-* In the **Overview** section you can find the macro precision, recall and F1 score for the collective intents or entities, based on which option you select. 
-* Under the **Intents** and **Entities** tabs you can find the micro precision, recall and F1 score for each intent or entity separately.
+* In the **Overview** section you can find the macro precision, recall and F1 score for the collective intents. 
+* Under the **Intents** tab you can find the micro precision, recall and F1 score for each intent separately.
 
 > [!NOTE]
-> If you don't see any of the intents or entities you have in your model displayed here, it is because they weren't in any of the utterances that were used for the test set.
+> If you don't see any of the intents you have in your model displayed here, it is because they weren't in any of the utterances that were used for the test set.
 
-You can view the confusion matrix for intents and entities by clicking on the **Test set confusion matrix** tab at the top fo the screen. 
+You can view the [confusion matrix](../concepts/evaluation-metrics.md) for intents by clicking on the **Test set confusion matrix** tab at the top fo the screen. 
 
 ## Next steps
-<!--* [Model evaluation metrics](../concepts/evaluation-metrics.md)-->
+* [Model evaluation metrics](../concepts/evaluation-metrics.md)
 * [Deploy and query the model](./deploy-query-model.md)

articles/cognitive-services/language-service/orchestration-workflow/includes/quickstarts/language-studio.md

@@ -30,15 +30,10 @@ Once you have a Language resource associated with your account, create an orches
 
 1. In [Language Studio](https://aka.ms/languageStudio), find the section labeled **Understand questions and conversational language** and select **Orchestration Workflow**.  
 
-   :::image type="content" source="../../../conversational-language-understanding/media/select-custom-clu.png" alt-text="A screenshot showing the location of Custom Language Understanding in the Language Studio landing page." lightbox="../../../conversational-language-understanding/media/select-custom-clu.png"::: 
+   :::image type="content" source="../../media/select-orchestration.png" alt-text="A screenshot showing the location of the orchestration workflow section in the Language Studio landing page." lightbox="../../media/select-orchestration.png"::: 
 
 
-2. This will bring you to the **Orchestration workflow project** page. Select **Create new project**.
-
-    :::image type="content" source="../../../conversational-language-understanding/media/projects-page.png" alt-text="A screenshot showing the conversation project page in Language Studio." lightbox="../../../conversational-language-understanding/media/projects-page.png":::
-
-
-You then need to provide the following details:
+2. This will bring you to the **Orchestration workflow project** page. Select **+ Create new project**. To create a project, you will need to provide the following details:
 
 |Value  | Description  |
 |---------|---------|
@@ -53,20 +48,21 @@ Once you're done, select **Next** and review the details. Select **create projec
 1. Click on *+Add* button to add your intent.
 2. Give your intent a name and choose to connect the intent to an existing project.
 3. Click on *Add intent* button. 
-4. Another option is to select *no* when creating an intent and don't connect it to an existing project.
+4. Enter **Greeting** as an intent. For this quickstart, select **No, I don't want to connect to a project**.
 
-When you select the intent, you will see the [tag utterances](../../how-to/tag-utterances.md) page, where you can add examples for intents.
 
+    :::image type="content" source="../../media/quickstart-intent.png" alt-text="A screenshot showing the schema page in Language studio." lightbox="../../media/quickstart-intent.png":::
+
+When you select the intent, you will see the [tag utterances](../../how-to/tag-utterances.md) page, where you can add examples for intents.
 
-:::image type="content" source="../../../conversational-language-understanding/media/quickstart-intents.png" alt-text="A screenshot showing the schema page in Language studio." lightbox="../../../conversational-language-understanding/media/quickstart-intents.png":::
 
 ## Tag utterances
 
 In the tag utterances page, let's add a few examples to the intents. Select the **Greeting** intent from the drop-down box that says **Select Intent**.
 
-In the text box that says **Write your example utterance and press enter**, write the sentence "*Good evening*" and press enter to add it as an example utterance.
+In the text box that says **Write your example utterance and press enter**, write the sentence "*Good evening*" and press enter to add it as an example utterance. 
 
-Add the rest of these utterances to the **Greeting** intent to the Training set.
+Add the rest of these utterances to the **Greeting** intent to the Training set. 
 
 |Utterance|
 |--|
@@ -77,7 +73,7 @@ Add the rest of these utterances to the **Greeting** intent to the Training set.
 
 When you're done, select **Save Changes** to save the utterances and labels to the project. The icon next to the button will turn green when the changes are saved. Next, go to the **Train Model** page.
 
-:::image type="content" source="../../../conversational-language-understanding/media/quickstart-utterances.png" alt-text="A screenshot showing the intents tagging screen in Language Studio." lightbox="../../../conversational-language-understanding/media/quickstart-utterances.png":::
+:::image type="content" source="../../media/tagged-utterances.png" alt-text="A screenshot showing the intents tagging screen in Language Studio." lightbox="../../media/tagged-utterances.png":::
 
 ## Train your model and view its details
 
@@ -95,4 +91,4 @@ Once you're done, select **Next** and review the details. For the connected proj
 
 Select **Test model** on the left of the screen, and select the deployment name from the drop-down menu. Add your test, for example *Good morning* in the text field and click on **Run the test**. 
 
-You now see the top intent as **Greeting**.
+You now see the top intent is **Greeting**.