You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/cognitive-services/QnAMaker/Concepts/content-types.md
+1-1Lines changed: 1 addition & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -16,7 +16,7 @@ The table below summarizes the types of content and file formats that are suppor
16
16
|Source Type|Content Type| Examples|
17
17
|--|--|--|
18
18
|URL|FAQs<br> (Flat, with sections or with a topics homepage)<br>Support pages <br> (Single page how-to articles, troubleshooting articles etc.)|[Plain FAQ](https://docs.microsoft.com/azure/cognitive-services/qnamaker/faqs), <br>[FAQ with links](https://www.microsoft.com/en-us/software-download/faq),<br> [FAQ with topics homepage](https://www.microsoft.com/Licensing/servicecenter/Help/Faq.aspx)<br>[Support article](https://docs.microsoft.com/azure/cognitive-services/qnamaker/concepts/best-practices)|
description: Improve the quality of your knowledge base with active learning. Review, accept or reject, add without removing or changing existing questions.
5
5
author: diberry
6
-
manager: nitinme
6
+
manager: nitinme
7
7
services: cognitive-services
8
8
ms.service: cognitive-services
9
9
ms.subservice: qna-maker
10
10
ms.topic: conceptual
11
-
ms.date: 10/14/2019
11
+
ms.date: 01/28/2020
12
12
ms.author: diberry
13
13
---
14
14
15
15
# Use active learning to improve your knowledge base
16
16
17
-
Active learning allows you to improve the quality of your knowledge base by suggesting alternative questions, based on user-submissions, to your question and answer pair. You review those suggestions, either adding them to existing questions or rejecting them.
17
+
Active learning allows you to improve the quality of your knowledge base by suggesting alternative questions, based on user-submissions, to your question and answer pair. You review those suggestions, either adding them to existing questions or rejecting them.
18
18
19
19
Your knowledge base doesn't change automatically. In order for any change to take effect, you must accept the suggestions. These suggestions add questions but don't change or remove existing questions.
20
20
21
21
## What is active learning?
22
22
23
23
QnA Maker learns new question variations with implicit and explicit feedback.
24
-
24
+
25
25
*[Implicit feedback](#how-qna-makers-implicit-feedback-works) – The ranker understands when a user question has multiple answers with scores that are very close and considers this as feedback. You don't need to do anything for this to happen.
26
-
*[Explicit feedback](#how-you-give-explicit-feedback-with-the-train-api) – When multiple answers with little variation in scores are returned from the knowledge base, the client application asks the user which question is the correct question. The user's explicit feedback is sent to QnA Maker with the [Train API](#train-api).
26
+
*[Explicit feedback](#how-you-give-explicit-feedback-with-the-train-api) – When multiple answers with little variation in scores are returned from the knowledge base, the client application asks the user which question is the correct question. The user's explicit feedback is sent to QnA Maker with the [Train API](#train-api).
27
27
28
28
Both methods provide the ranker with similar queries that are clustered.
29
29
@@ -39,53 +39,54 @@ Once questions are suggested in the QnA Maker portal, you need to review and acc
39
39
40
40
QnA Maker's implicit feedback uses an algorithm to determine score proximity then make active learning suggestions. The algorithm to determine proximity is not a simple calculation. The ranges in the following example are not meant to be fixed but should be used as a guide to understand the impact of the algorithm only.
41
41
42
-
When a question's score is highly confident, such as 80%, the range of scores that are considered for active learning are wide, approximately within 10%. As the confidence score decreases, such as 40%, the range of scores decreases as well, approximately within 4%.
42
+
When a question's score is highly confident, such as 80%, the range of scores that are considered for active learning are wide, approximately within 10%. As the confidence score decreases, such as 40%, the range of scores decreases as well, approximately within 4%.
43
43
44
44
## How you give explicit feedback with the Train API
45
45
46
46
It is important that QnA Maker gets explicit feedback about which of the answers was the best answer. How the best answer is determined is up to you and can include:
47
47
48
48
* User feedback, selecting one of the answers.
49
-
* Business logic, such as determining an acceptable score range.
49
+
* Business logic, such as determining an acceptable score range.
50
50
* A combination of both user feedback and business logic.
51
51
52
52
## Upgrade your runtime version to use active learning
53
53
54
-
Active Learning is supported in runtime version 4.4.0 and above. If your knowledge base was created on an earlier version, [upgrade your runtime](set-up-qnamaker-service-azure.md#get-the-latest-runtime-updates) to use this feature.
54
+
Active Learning is supported in runtime version 4.4.0 and above. If your knowledge base was created on an earlier version, [upgrade your runtime](set-up-qnamaker-service-azure.md#get-the-latest-runtime-updates) to use this feature.
55
55
56
56
## Turn on active learning to see suggestions
57
57
58
58
Active learning is off by default. Turn it on to see suggested questions. After you turn on active learning, you need to send information from the client app to QnA Maker. For more information, see [Architectural flow for using GenerateAnswer and Train APIs from a bot](#architectural-flow-for-using-generateanswer-and-train-apis-from-a-bot).
59
59
60
60
1. Select **Publish** to publish the knowledge base. Active learning queries are collected from the GenerateAnswer API prediction endpoint only. The queries to the Test pane in the QnA Maker portal do not impact active learning.
61
61
62
-
1. To turn active learning on in the QnA Maker portal, go to the top-right corner, select your **Name**, go to [**Service Settings**](https://www.qnamaker.ai/UserSettings).
62
+
1. To turn active learning on in the QnA Maker portal, go to the top-right corner, select your **Name**, go to [**Service Settings**](https://www.qnamaker.ai/UserSettings).
63
63
64
64
65
65
66
66
67
-
1. Find the QnA Maker service then toggle **Active Learning**.
67
+
1. Find the QnA Maker service then toggle **Active Learning**.
68
68
69
-
[](../media/improve-knowledge-base/turn-active-learning-on-at-service-setting.png#lightbox)
69
+
> [!div class="mx-imgBorder"]
70
+
> [](../media/improve-knowledge-base/turn-active-learning-on-at-service-setting.png#lightbox)
70
71
71
72
> [!Note]
72
-
> The exact version on the preceding image is shown as an example only. Your version may be different.
73
+
> The exact version on the preceding image is shown as an example only. Your version may be different.
73
74
74
75
Once **Active Learning** is enabled, the knowledge base suggests new questions at regular intervals based on user-submitted questions. You can disable **Active Learning** by toggling the setting again.
75
76
76
77
## Accept an active learning suggestion in the knowledge base
77
78
78
79
Active Learning alters the Knowledge Base or Search Service after you approve the suggestion, then save and train. If you approve the suggestion it will be added as an alternate question.
79
80
80
-
1. In order to see the suggested questions, on the **Edit** knowledge base page, select **View Options**, then select **Show active learning suggestions**.
81
+
1. In order to see the suggested questions, on the **Edit** knowledge base page, select **View Options**, then select **Show active learning suggestions**.
81
82
82
83
[](../media/improve-knowledge-base/show-suggestions-button.png#lightbox)
83
84
84
85
1. Filter the knowledge base with question and answer pairs to show only suggestions by selecting **Filter by Suggestions**.
85
86
86
87
[](../media/improve-knowledge-base/filter-by-suggestions.png#lightbox)
87
88
88
-
1. Each QnA pair suggests the new question alternatives with a check mark, `✔` , to accept the question or an `x` to reject the suggestions. Select the check mark to add the question.
89
+
1. Each QnA pair suggests the new question alternatives with a check mark, `✔` , to accept the question or an `x` to reject the suggestions. Select the check mark to add the question.
89
90
90
91
[](../media/improve-knowledge-base/accept-active-learning-suggestions.png#lightbox)
91
92
@@ -113,7 +114,7 @@ A bot or other client application should use the following architectural flow to
113
114
114
115
### Use the top property in the GenerateAnswer request to get several matching answers
115
116
116
-
When submitting a question to QnA Maker for an answer, the `top` property of the JSON body sets the number of answers to return.
117
+
When submitting a question to QnA Maker for an answer, the `top` property of the JSON body sets the number of answers to return.
117
118
118
119
```json
119
120
{
@@ -125,7 +126,7 @@ When submitting a question to QnA Maker for an answer, the `top` property of the
125
126
126
127
### Use the score property along with business logic to get list of answers to show user
127
128
128
-
When the client application (such as a chat bot) receives the response, the top 3 questions are returned. Use the `score` property to analyze the proximity between scores. This proximity range is determined by your own business logic.
129
+
When the client application (such as a chat bot) receives the response, the top 3 questions are returned. Use the `score` property to analyze the proximity between scores. This proximity range is determined by your own business logic.
129
130
130
131
```json
131
132
{
@@ -166,9 +167,9 @@ When the client application (such as a chat bot) receives the response, the top
166
167
167
168
## Client application follow-up when questions have similar scores
168
169
169
-
Your client application displays the questions with an option for the user to select _the single question_ that most represents their intention.
170
+
Your client application displays the questions with an option for the user to select _the single question_ that most represents their intention.
170
171
171
-
Once the user selects one of the existing questions, the client application sends the user's choice as feedback using QnA Maker's Train API. This feedback completes the active learning feedback loop.
172
+
Once the user selects one of the existing questions, the client application sends the user's choice as feedback using QnA Maker's Train API. This feedback completes the active learning feedback loop.
172
173
173
174
## Train API
174
175
@@ -212,11 +213,11 @@ An example JSON body looks like:
212
213
}
213
214
```
214
215
215
-
A successful response returns a status of 204 and no JSON response body.
216
+
A successful response returns a status of 204 and no JSON response body.
216
217
217
218
### Batch many feedback records into a single call
218
219
219
-
In the client-side application, such as a bot, you can store the data, then send many records in a single JSON body in the `feedbackRecords` array.
220
+
In the client-side application, such as a bot, you can store the data, then send many records in a single JSON body in the `feedbackRecords` array.
220
221
221
222
An example JSON body looks like:
222
223
@@ -253,7 +254,7 @@ Your bot framework code needs to call the Train API, if the user's query should
253
254
* Determine if query should be used for active learning
254
255
* Send query back to QnA Maker's Train API for active learning
255
256
256
-
In the [Azure Bot sample](https://aka.ms/activelearningsamplebot), both of these activities have been programmed.
257
+
In the [Azure Bot sample](https://aka.ms/activelearningsamplebot), both of these activities have been programmed.
257
258
258
259
### Example C# code for Train API with Bot Framework 4.x
### Example Node.js code for Train API with Bot Framework 4.x
322
+
### Example Node.js code for Train API with Bot Framework 4.x
322
323
323
324
The following code illustrates how to send information back to QnA Maker with the Train API. This [complete code sample](https://github.com/microsoft/BotBuilder-Samples/blob/master/experimental/qnamaker-activelearning/javascript_nodejs) is available on GitHub.
## Active learning is saved in the exported knowledge base
365
366
366
-
When your app has active learning enabled, and you export the app, the `SuggestedQuestions` column in the tsv file retains the active learning data.
367
+
When your app has active learning enabled, and you export the app, the `SuggestedQuestions` column in the tsv file retains the active learning data.
367
368
368
369
The `SuggestedQuestions` column is a JSON object of information of implicit, `autosuggested`, and explicit, `usersuggested` feedback. An example of this JSON object for a single user-submitted question of `help` is:
369
370
@@ -389,7 +390,7 @@ You can also use the download alterations API to review these alterations, using
> [](../media/qnamaker-how-to-key-management/Endpoint-keys1.png#lightbox)
95
96
96
97
>[!NOTE]
97
98
>Refresh your keys if you think they've been compromised. This may require corresponding changes to your client application or bot code.
@@ -130,7 +131,7 @@ QnA Maker's **App Service** resource uses the Cognitive Search resource. In orde
130
131
1. Select **Settings**, then **Configuration**. This will display all existing settings for the QnA Maker's App Service.
131
132
132
133
> [!div class="mx-imgBorder"]
133
-
> 
134
+
> [](../media/qnamaker-how-to-upgrade-qnamaker/change-search-service-app-service-configuration.png#lightbox)
134
135
135
136
1. Change the values for the following keys:
136
137
@@ -140,7 +141,7 @@ QnA Maker's **App Service** resource uses the Cognitive Search resource. In orde
140
141
1. To use the new settings, you need to restart the App service. Select **Overview**, then select **Restart**.
141
142
142
143
> [!div class="mx-imgBorder"]
143
-
> 
144
+
> [](../media/qnamaker-how-to-upgrade-qnamaker/screenshot-azure-portal-restart-app-service.png)
144
145
145
146
If you create a QnA service through Azure Resource Manager templates, you can create all resources and control the App Service creation to use an existing Search service.
0 commit comments