Skip to content

Commit ba4a0b3

Browse files
JamesJBarnettJamesJBarnett
authored
Merge pull request #248346 from aahill/clu-best-practices
CLU best practices
2 parents 461a565 + b40a9aa commit ba4a0b3

File tree

7 files changed

+268
-5
lines changed

7 files changed

+268
-5
lines changed

‎articles/ai-services/language-service/conversational-language-understanding/concepts/app-architecture.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
---
2+
title: When to choose conversational language understanding or orchestration workflow
3+
titleSuffix: Azure AI services
4+
description: Learn when to choose conversational language understanding or orchestration workflow
5+
services: cognitive-services
6+
author: aahill
7+
manager: nitinme
8+
ms.service: cognitive-services
9+
ms.subservice: language-service
10+
ms.topic: best-practice
11+
ms.date: 08/15/2023
12+
ms.author: aahi
13+
ms.custom: language-service-clu
14+
---
15+
16+
# When to use conversational language understanding or orchestration workflow apps
17+
18+
When you create large applications, you should consider whether your use-case would be best served by a single conversational app (flat architecture), or multiple apps that are orchestrated.
19+
20+
21+
## Orchestration overview
22+
23+
Orchestration workflow is a feature that allows you to connect different projects from [LUIS](../../../LUIS/what-is-luis.md) [conversational language understanding](../overview.md), and [custom question answering](../../question-answering/overview.md) in one project. You can then use this project for predictions using one endpoint. The orchestration project makes a prediction on which child project should be called, automatically routes the request, and returns with its response.
24+
25+
The key point is that orchestration involves two steps:
26+
27+
1. Predicting which child project to call. <!--The model that performs this classification can be trained either with a standard or an advanced recipe. (Please see footnotes on instructions for training with advanced recipe).-->
28+
2. Routing the utterance to the destination child app, and returning the child app's response.
29+
30+
### Advantages
31+
32+
* Clear decomposition and faster development:
33+
* If your overall schema has a substantial number of domains, the orchestration approach can help decompose your application into several child apps (each serving a specific domain). For example, an automotive conversational app might have a *navigation domain*, a *media domain*, and so on.
34+
* Developing each domain app in parallel is easier. People and teams with specific domain expertise can work on individual apps collaboratively and in parallel.
35+
* Since each domain app is smaller, the development cycle becomes faster. Smaller sized domain apps take much less time to train than a single large app.
36+
* More flexible [confidence score thresholds](/legal/cognitive-services/clu/clu-characteristics-and-limitations?context=/azure/ai-services/language-service/context/context#understand-confidence-scores):
37+
* Since there are separate child apps serving each domain, it's easy to set separate thresholds for different child apps.
38+
* AI quality improvements where appropriate:
39+
* Some applications require that certain entities are domain restricted. Orchestration makes this easy to achieve. Once the orchestration project has predicted which child app should be called, the other child apps won't be called.
40+
41+
For example, if your app contains a `Person.Name` prebuilt entity, consider the utterance *"How do I use a jack?"*, in the context of a vehicle question. In this context, *jack* is an automotive tool, and shouldn’t be recognized as a person's name. Using orchestration, this utterance can be redirected to a child app created to answer such questions, which doesn’t have a `Person.Name` entity.
42+
43+
### Disadvantages
44+
45+
* Redundant entities in child apps:
46+
* If you need a particular prebuilt entity being returned in all utterances irrespective of the domain, for example `Quantity.Number` or `Geography.Location`, there is no way of adding an entity to the Orchestration app (it is an intent-only model). You would need to add it to all individual child apps.
47+
* Efficiency:
48+
* Orchestration apps take two model inferences. One for predicting which child app to call, another for the prediction in the child app. Inference times will typically be slower than single apps with a flat architecture.
49+
* Train/test split for orchestrator:
50+
* Training an orchestration app does not allow you to granularly split data between the testing and training sets. For example, you cannot train a 90-10 split for child app A, and then an 80-20 split for child app B. This may be a minor point, but worth keeping in mind.
51+
52+
## Flat architecture overview
53+
54+
Flat architecture is the other method of developing conversational apps. Instead of using an orchestration app to send utterances to one of multiple child apps, you develop a singular (or flat) app to handle utterances.
55+
56+
### Advantages
57+
58+
* Simplicity:
59+
* For small sized apps or domains, the orchestrator approach can be overly complex.
60+
* Since all intents and entities are at the same app level, it might be easier to make changes to all of them together.
61+
* It's easier to add entities that should always be returned:
62+
* If you want certain prebuilt or list entities to be returned for all utterances, you only need to add it alongside other entities in a single app. If you use orchestration, as mentioned above, you would need to add it to every child app.
63+
64+
### Disadvantages
65+
66+
* Unwieldy for large apps:
67+
* For large apps (say > 50 intents or entities) it can become difficult to keep track of evolving schemas and datasets. This is particularly evident in cases where the app has to serve several domains. For example an automotive conversational app might have a *navigation domain*, a *media domain*, and so on.
68+
* Limited control over entity matches:
69+
* In a flat architecture, there is no way to restrict entities to be returned only in certain cases. You can accomplish this using orchestration by assigning those specific entities to particular child apps.
70+
71+
## Next steps
72+
* [Orchestration workflow overview](../../orchestration-workflow/overview.md)
73+
* [Conversational language understanding overview](../overview.md)

‎articles/ai-services/language-service/conversational-language-understanding/concepts/best-practices.md

Lines changed: 130 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ manager: nitinme
88
ms.service: cognitive-services
99
ms.subservice: language-service
1010
ms.topic: best-practice
11-
ms.date: 10/11/2022
11+
ms.date: 08/30/2023
1212
ms.author: aahi
1313
ms.custom: language-service-clu
1414
---
@@ -26,22 +26,149 @@ Schema is the definition of your intents and entities. There are different appro
2626

2727
You can typically think of actions and queries as _intents_, while the information required to fulfill those queries as _entities_.
2828

29-
For example, assume you want your customers to cancel subscriptions for various products that you offer through your chatbot. You can create a _Cancel_ intent with various examples like _"Cancel the Contoso service"_, or _"stop charging me for the Fabrikam subscription"_. The user's intent here is to _cancel_, the _Contoso service_ or _Fabrikam subscription_ are the subscriptions they would like to cancel. Therefore, you can create an entity for _subscriptions_. You can then model your entire project to capture actions as intents and use entities to fill in those actions. This allows you to cancel anything you define as an entity, such as other products. You can then have intents for signing up, renewing, upgrading, etc. that all make use of the _subscriptions_ and other entities.
29+
For example, assume you want your customers to cancel subscriptions for various products that you offer through your chatbot. You can create a _Cancel_ intent with various examples like _"Cancel the Contoso service,"_ or _"stop charging me for the Fabrikam subscription."_ The user's intent here is to _cancel,_ the _Contoso service_ or _Fabrikam subscription_ are the subscriptions they would like to cancel. Therefore, you can create an entity for _subscriptions_. You can then model your entire project to capture actions as intents and use entities to fill in those actions. This allows you to cancel anything you define as an entity, such as other products. You can then have intents for signing up, renewing, upgrading, etc. that all make use of the _subscriptions_ and other entities.
3030

3131
The above schema design makes it easy for you to extend existing capabilities (canceling, upgrading, signing up) to new targets by creating a new entity.
3232

33-
Another approach is to model the _information_ as intents and _actions_ as entities. Let's take the same example, allowing your customers to cancel subscriptions through your chatbot. You can create an intent for each subscription available, such as _Contoso_ with utterances like _"cancel Contoso"_, _"stop charging me for contoso services"_, _"Cancel the Contoso subscription"_. You would then create an entity to capture the action, _cancel_. You can define different entities for each action or consolidate actions as one entity with a list component to differentiate between actions with different keys.
33+
Another approach is to model the _information_ as intents and _actions_ as entities. Let's take the same example, allowing your customers to cancel subscriptions through your chatbot. You can create an intent for each subscription available, such as _Contoso_ with utterances like _"cancel Contoso,"_ _"stop charging me for contoso services,"_ _"Cancel the Contoso subscription."_ You would then create an entity to capture the action, _cancel._ You can define different entities for each action or consolidate actions as one entity with a list component to differentiate between actions with different keys.
3434

3535
This schema design makes it easy for you to extend new actions to existing targets by adding new action entities or entity components.
3636

3737
Make sure to avoid trying to funnel all the concepts into just intents, for example don't try to create a _Cancel Contoso_ intent that only has the purpose of that one specific action. Intents and entities should work together to capture all the required information from the customer.
3838

3939
You also want to avoid mixing different schema designs. Do not build half of your application with actions as intents and the other half with information as intents. Ensure it is consistent to get the possible results.
4040

41+
[!INCLUDE [Balance training data](../includes/balance-training-data.md)]
4142

43+
[!INCLUDE [Label data](../includes/label-data-best-practices.md)]
4244

45+
## Use standard training before advanced training
4346

47+
[Standard training](../how-to/train-model.md#training-modes) is free and faster than Advanced training, making it useful to quickly understand the effect of changing your training set or schema while building the model. Once you are satisfied with the schema, consider using advanced training to get the best AIQ out of your model.
4448

49+
## Use the evaluation feature
50+
51+
When you build an app, it's often helpful to catch errors early. It’s usually a good practice to add a test set when building the app, as training and evaluation results are very useful in identifying errors or issues in your schema.
4552

53+
## Machine-learning components and composition
4654

55+
See [Component types](./entity-components.md#component-types).
4756

57+
## Using the "none" score Threshold
58+
59+
If you see too many false positives, such as out-of-context utterances being marked as valid intents, See [confidence threshold](./none-intent.md) for information on how it affects inference.
60+
61+
* Non machine-learned entity components like lists and regex are by definition not contextual. If you see list or regex entities in unintended places, try labeling the list synonyms as the machine-learned component.
62+
63+
* For entities, you can use learned component as the ‘Required’ component, to restrict when a composed entity should fire.
64+
65+
For example, suppose you have an entity called "*ticket quantity*" that attempts to extract the number of tickets you want to reserve for booking flights, for utterances such as "*Book two tickets tomorrow to Cairo.*"
66+
67+
68+
Typically, you would add a prebuilt component for `Quantity.Number` that already extracts all numbers in utterances. However if your entity was only defined with the prebuilt component, it would also extract other numbers as part of the *ticket quantity* entity, such as "*Book two tickets tomorrow to Cairo at 3 PM.*"
69+
70+
To resolve this, you would label a learned component in your training data for all the numbers that are meant to be a *ticket quantity*. The entity now has two components:
71+
* The prebuilt component that can interpret all numbers, and
72+
* The learned component that predicts where the *ticket quantity* is in a sentence.
73+
74+
If you require the learned component, make sure that *ticket quantity* is only returned when the learned component predicts it in the right context. If you also require the prebuilt component, you can then guarantee that the returned *ticket quantity* entity is both a number and in the correct position.
75+
76+
77+
## Addressing casing inconsistencies
78+
79+
If you have poor AI quality and determine the casing used in your training data is dissimilar to the testing data, you can use the `normalizeCasing` project setting. This normalizes the casing of utterances when training and testing the model. If you've migrated from LUIS, you might recognize that LUIS did this by default.
80+
81+
```json
82+
{
83+
"projectFileVersion": "2022-10-01-preview",
84+
...
85+
"settings": {
86+
"confidenceThreshold": 0.5,
87+
"normalizeCasing": true
88+
}
89+
...
90+
```
91+
92+
## Addressing model overconfidence
93+
94+
Customers can use the LoraNorm recipe version in case the model is being incorrectly overconfident. An example of this can be like the below (note that the model predicts the incorrect intent with 100% confidence). This makes the confidence threshold project setting unusable.
95+
96+
| Text | Predicted intent | Confidence score |
97+
|----|----|----|
98+
| "*Who built the Eiffel Tower?*" | `Sports` | 1.00 |
99+
| "*Do I look good to you today?*" | `QueryWeather` | 1.00 |
100+
| "*I hope you have a good evening.*" | `Alarm` | 1.00 |
101+
102+
To address this, use the `2023-04-15` configuration version that normalizes confidence scores. The confidence threshold project setting can then be adjusted to achieve the desired result.
103+
104+
```console
105+
curl --location 'https://<your-resource>.cognitiveservices.azure.com/language/authoring/analyze-conversations/projects/<your-project>/:train?api-version=2022-10-01-preview' \
106+
--header 'Ocp-Apim-Subscription-Key: <your subscription key>' \
107+
--header 'Content-Type: application/json' \
108+
--data '{
109+
      "modelLabel": "<modelLabel>",
110+
      "trainingMode": "advanced",
111+
      "trainingConfigVersion": "2023-04-15",
112+
      "evaluationOptions": {
113+
            "kind": "percentage",
114+
            "testingSplitPercentage": 0,
115+
            "trainingSplitPercentage": 100
116+
      }
117+
}
118+
```
119+
120+
Once the request is sent, you can track the progress of the training job in Language Studio as usual.
121+
122+
> [!NOTE]
123+
> You have to retrain your model after updating the `confidenceThreshold` project setting. Afterwards, you'll need to republish the app for the new threshold to take effect.
124+
125+
## Debugging composed entities
126+
127+
Entities are functions that emit spans in your input with an associated type. The function is defined by one or more components. You can mark components as needed, and you can decide whether to enable the *combine components* setting. When you combine components, all spans that overlap will be merged into a single span. If the setting isn't used, each individual component span will be emitted.
128+
129+
To better understand how individual components are performing, you can disable the setting and set each component to "not required". This lets you inspect the individual spans that are emitted, and experiment with removing components so that only problematic components are generated.
130+
131+
## Evaluate a model using multiple test sets
132+
133+
Data in a conversational language understanding project can have two data sets. A "testing" set, and a "training" set. If you want to use multiple test sets to evaluate your model, you can:
134+
135+
* Give your test sets different names (for example, "test1" and "test2").
136+
* Export your project to get a JSON file with its parameters and configuration.
137+
* Use the JSON to import a new project, and rename your second desired test set to "test".
138+
* Train the model to run the evaluation using your second test set.
139+
140+
## Custom parameters for target apps and child apps
141+
142+
If you are using [orchestrated apps](./app-architecture.md), you may want to send custom parameter overrides for various child apps. The `targetProjectParameters` field allows users to send a dictionary representing the parameters for each target project. For example, consider an orchestrator app named `Orchestrator` orchestrating between a conversational language understanding app named `CLU1` and a custom question answering app named `CQA1`. If you want to send a parameter named "top" to the question answering app, you can use the above parameter.
143+
144+
```console
145+
curl --request POST \
146+
--url 'https://<your-language-resource>.cognitiveservices.azure.com/language/:analyze-conversations?api-version=2022-10-01-preview' \
147+
--header 'ocp-apim-subscription-key: <your subscription key>' \
148+
--data '{
149+
"kind": "Conversation",
150+
"analysisInput": {
151+
"conversationItem": {
152+
"id": "1",
153+
"text": "Turn down the volume",
154+
"modality": "text",
155+
"language": "en-us",
156+
"participantId": "1"
157+
}
158+
},
159+
"parameters": {
160+
"projectName": "Orchestrator",
161+
"verbose": true,
162+
"deploymentName": "std",
163+
"stringIndexType": "TextElement_V8",
164+
"targetProjectParameters": {
165+
"CQA1": {
166+
"targetProjectKind": "QuestionAnswering",
167+
"callingOptions": {
168+
"top": 1
169+
}
170+
}
171+
}
172+
}
173+
}'
174+
```

‎articles/ai-services/language-service/conversational-language-understanding/how-to/tag-utterances.md

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ manager: nitinme
88
ms.service: cognitive-services
99
ms.subservice: language-service
1010
ms.topic: how-to
11-
ms.date: 06/03/2022
11+
ms.date: 08/25/2023
1212
ms.author: aahi
1313
ms.custom: language-service-clu, ignite-fall-2021
1414
---
@@ -41,6 +41,8 @@ As you add utterances and label them, keep in mind:
4141
* **Label consistently**: The same entity should have the same label across all the utterances.
4242
* **Label completely**: Provide varied utterances for every intent. Label all the instances of the entity in all your utterances.
4343

44+
[!INCLUDE [Label data best practices](../includes/label-data-best-practices.md)]
45+
4446
* For [Multilingual projects](../language-support.md#multi-lingual-option), adding utterances in other languages increases the model's performance in these languages, but avoid duplicating your data across all the languages you would like to support. For example, to improve a calender bot's performance with users, a developer might add examples mostly in English, and a few in Spanish or French as well. They might add utterances such as:
4547

4648
* "_Set a meeting with **Matt** and **Kevin** **tomorrow** at **12 PM**._" (English)

‎articles/ai-services/language-service/conversational-language-understanding/how-to/train-model.md

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ manager: nitinme
88
ms.service: cognitive-services
99
ms.subservice: language-service
1010
ms.topic: how-to
11-
ms.date: 05/12/2022
11+
ms.date: 08/25/2023
1212
ms.author: aahi
1313
ms.custom: language-service-clu, ignite-fall-2021
1414
---
@@ -30,6 +30,10 @@ Model evaluation is triggered automatically after training is completed successf
3030

3131
<!--See the [project development lifecycle](../overview.md#project-development-lifecycle) for more information.-->
3232

33+
[!INCLUDE [Balance training data](../includes/balance-training-data.md)]
34+
35+
36+
3337
## Data splitting
3438

3539
Before you start the training process, labeled utterances in your project are divided into a training set and a testing set. Each one of them serves a different function.

0 commit comments

Comments
 (0)