Skip to content

Commit 07f18c5

Browse files
authored
Update best-practices.md
Include the new OOS recipe
1 parent a7fd4d8 commit 07f18c5

File tree

1 file changed

+35
-0
lines changed
  • articles/ai-services/language-service/conversational-language-understanding/concepts

1 file changed

+35
-0
lines changed

articles/ai-services/language-service/conversational-language-understanding/concepts/best-practices.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,3 +216,38 @@ curl --request POST \
216216
"targetResourceRegion": "<target-region>"
217217
}'
218218
```
219+
220+
221+
## Addressing Out of Domain utterances
222+
223+
Customers can use the new recipe version '2024-06-01-preview' in case the model has poor AIQ on out of domain utterances. An example of this with the default recipe can be like the below where the model has 3 intents Sports, QueryWeather and Alarm. The test utterances are out of domain utterances and the model classifies them as InDomain with a relatively high confidence score.
224+
225+
| Text | Predicted intent | Confidence score |
226+
|----|----|----|
227+
| "*Who built the Eiffel Tower?*" | `Sports` | 0.90 |
228+
| "*Do I look good to you today?*" | `QueryWeather` | 1.00 |
229+
| "*I hope you have a good evening.*" | `Alarm` | 0.80 |
230+
231+
To address this, use the `2024-06-01-preview` configuration version that is built specifically to address this issue while also maintaining reasonably good quality on In Domain utterances.
232+
233+
```console
234+
curl --location 'https://<your-resource>.cognitiveservices.azure.com/language/authoring/analyze-conversations/projects/<your-project>/:train?api-version=2022-10-01-preview' \
235+
--header 'Ocp-Apim-Subscription-Key: <your subscription key>' \
236+
--header 'Content-Type: application/json' \
237+
--data '{
238+
      "modelLabel": "<modelLabel>",
239+
      "trainingMode": "advanced",
240+
      "trainingConfigVersion": "2024-06-01-preview",
241+
      "evaluationOptions": {
242+
            "kind": "percentage",
243+
            "testingSplitPercentage": 0,
244+
            "trainingSplitPercentage": 100
245+
      }
246+
}
247+
```
248+
249+
Once the request is sent, you can track the progress of the training job in Language Studio as usual.
250+
251+
Caveats:
252+
- The None Score threshold for the app (confidence threshold below which the topIntent is marked as None) when using this recipe should be set to 0. This is because this new recipe attributes a certain portion of the in domain probabiliities to out of domain so that the model is not incorrectly overconfident about in domain utterances. As a result, users may see slightly reduced confidence scores for in domain utterances as compared to the prod recipe.
253+
- This recipe is not recommended for apps with just 2 intents, eg: IntentA and None.

0 commit comments

Comments
 (0)