Skip to content

Commit 258b9d8

Browse files
tsuwandytsuwandy
authored
Orchestrator samples reorganization (#6289)
* Orchestrator reorganization 1. Moved ModelTuning from botbuilder-samples repo 2. Delete OrchestratorDispatch Composer sample (moved to botbuilder-samples repo) * Update FAQ.md
1 parent de28d1e commit 258b9d8

File tree

70 files changed

+372
-16405
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

70 files changed

+372
-16405
lines changed

‎Orchestrator/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -135,7 +135,7 @@ C# .NET Core 3.1
135135
- [NodeJS samples][13]
136136
- [BF Orchestrator Command Usage][23]
137137

138-
See the [School skill navigator](https://github.com/microsoft/BotBuilder-Samples/tree/main/experimental/orchestrator/Composer/01.school-skill-navigator#school-skill-navigator-bot) for an example of using Orchestrator commandlets to improve the quality of a .lu training set and using Composer to build a bot from examples in .lu format.
138+
See the [School skill navigator](https://github.com/microsoft/BotBuilder-Samples/tree/main/composer-samples/csharp_dotnetcore/projects/OrchestratorSchoolNavigator) for an example of using Orchestrator commandlets to improve the quality of a .lu training set and using Composer to build a bot from examples in .lu format.
139139

140140
[1]:https://luis.ai
141141
[2]:https://qnamaker.ai

‎Orchestrator/Samples/CLI/ModelTuning/README.md‎

Lines changed: 159 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,159 @@
1+
# Improve Language Model with BF CLI
2+
3+
The following sample illustrates how to evaluate and improve a simple language model using the Orchestrator report. You may adopt this process simply by editing the [demo.cmd](./demo.cmd) script to use your language model files.
4+
5+
## Prerequisites
6+
7+
* [Bot Framework CLI][5]
8+
* [Bot Framework CLI Orchestrator plugin][1]
9+
* An understanding of [Orchestrator][6] feature usage.
10+
11+
## Walkthrough
12+
13+
The following files make up this illustration:
14+
15+
```
16+
demo.cmd: Script run evaluation and produces report
17+
common.lu: Bot Language Model LU
18+
common.test.lu: A test set containing examples not present in original LU
19+
common.fixed.lu: A corrected Language Model based on evaluation run
20+
```
21+
22+
Assume a bot with a simple language model in [common.lu](./common.lu) file as follows:
23+
24+
```
25+
# Help
26+
- help
27+
- i need help
28+
- please help
29+
- can you please help
30+
- what do you do
31+
- what is this bot for
32+
33+
34+
# BuySurface
35+
- How can I buy {ProductType=Surface PRO}
36+
- I want to buy {ProductType=Surface PRO}
37+
- I want to buy {ProductType=Surface laptop}
38+
- Can I buy {ProductType=Surface PRO} online
39+
40+
# USConstitution
41+
- Tell me about the US Constitution
42+
- US Constitution
43+
- Info on the Constitution
44+
- tell me about the Constitution
45+
- constitution knowledgebase
46+
- constitution faq
47+
- what do you know about the constitution
48+
- The united states constitution
49+
50+
# Restart
51+
- restart
52+
- reset
53+
- start over
54+
- menu
55+
- top menu
56+
57+
# Finance
58+
- pay my bills
59+
- transfer funds
60+
- what's my balance
61+
- get $500 cash
62+
63+
# LUIS
64+
- luis
65+
66+
# Orchestrator
67+
- Orchestrator
68+
- Orch
69+
70+
```
71+
72+
This evaluation examines how the Orchestrator engine can generalize language understanding from a few examples to include phrases never seen before. Hence, for proper language evaluation, the test file [common.test.lu](./common.test.lu) should contain utterances not present in the original "training" set:
73+
74+
```
75+
# Help
76+
- help
77+
- do you have help
78+
- any assistance in this bot?
79+
80+
# BuySurface
81+
- Looking for a computer made by MSFT
82+
- what kind of MS products do you have?
83+
84+
# USConstitution
85+
- Where can I read our founding fathers famous document?
86+
- Is there an analysis of the bill of rights?
87+
- What is that proclamation thing?
88+
89+
# Restart
90+
- reset
91+
- go to the beginning
92+
93+
# Finance
94+
- can i pay with credit card?
95+
- do you use Zelle?
96+
- what is my bank borrowing limit?
97+
98+
# LUIS
99+
- Language Understanding Inteligent Service
100+
101+
# Orchestrator
102+
- Conductor
103+
104+
```
105+
106+
Next, run the [demo.cmd](./demo.cmd) script.
107+
108+
In first run you need to download the Orchestrator basemodel so execute ```demo.cmd getmodel``` which will download the model and run the test. Consequent runs can reuse the same basemodel (hint: see ```bf orchestrator:basemodel:list``` for alternate models). Also, if you wish to compare to [LUIS](https://luis.ai) results, edit the script with your LUIS application info.
109+
110+
This will produce a report in report folder such as follows:
111+
112+
![Before Correction](./reportbefore.png)
113+
114+
Notice how 5 utterances were misclassified. For example ```do you use Zelle?``` was classified as *Help* (with low score of 0.2998) instead of *Finance*. Also, note that what led to this misclassification is that the nearest example scored 0.5669 under the *Help* label.
115+
116+
The observation shows that the concepts for those misclassified utterances are not present in the original common.lu language model. To correct, we'll add representative examples in the original common.lu file. Note how we don't add the exact utterances, only representative examples:
117+
118+
![Compared LU](./comparedLU.png)
119+
120+
The corrected file is available for reference as [common.fixed.lu](./common.fixed.lu). You may copy it over the common.lu.
121+
122+
Now, re-run the test ```demo.cmd``` and view the resulting report:
123+
124+
125+
126+
![Report After](./reportafter.png)
127+
128+
## Summary
129+
130+
This walkthrough showed how to improve your bot's language model before ever deploying it. It uses BF CLI to test the language model, and use the resulting report to correct the language model. One should construct a language model with as many examples representing expected user utterances. However, it also illustrates how one does not need to account for all utterance permutations, rather only to present concepts within those utterances. The Orchestrator engine generalizes and can accurately detect similar utterances.
131+
132+
## Additional Reading
133+
134+
- [Orchestrator Documentation][6]
135+
- [BF CLI Orchestrator Command Usage][4]
136+
- [Report Interpretation][3]
137+
- [LU File Format][2]
138+
139+
140+
141+
[1]: https://aka.ms/bforchestratorcli "Orchestrator Plugin"
142+
[2]: https://docs.microsoft.com/en-us/azure/bot-service/file-format/bot-builder-lu-file-format?view=azure-bot-service-4.0 "LU file format"
143+
[3]: https://aka.ms/bforchestratorreport "report interpretation"
144+
[4]: https://github.com/microsoft/botframework-sdk/blob/main/Orchestrator/docs/BFOrchestratorUsage.md "BF Orchestrator usage"
145+
[5]: https://github.com/microsoft/botframework-cli
146+
[6]: https://aka.ms/bf-orchestrator
147+
148+
149+
150+
151+
152+
153+
154+
155+
156+
157+
158+
159+

‎Orchestrator/Samples/CLI/ModelTuning/common.fixed.lu‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
# Help
2+
- help
3+
- i need help
4+
- please help
5+
- can you please help
6+
- what do you do
7+
- what is this bot for
8+
9+
10+
# BuySurface
11+
- How can I buy {ProductType=Surface PRO}
12+
- I want to buy {ProductType=Surface PRO}
13+
- I want to buy {ProductType=Surface laptop}
14+
- Can I buy {ProductType=Surface PRO} online
15+
- Types of MS products?
16+
17+
# USConstitution
18+
- Tell me about the US Constitution
19+
- US Constitution
20+
- Info on the Constitution
21+
- tell me about the Constitution
22+
- constitution knowledgebase
23+
- constitution faq
24+
- what do you know about the constitution
25+
- The united states constitution
26+
27+
# Restart
28+
- restart
29+
- reset
30+
- start over
31+
- menu
32+
- top menu
33+
34+
# Finance
35+
- pay my bills
36+
- transfer funds
37+
- what's my balance
38+
- get $500 cash
39+
- use Zelle
40+
- borrowing limits?
41+
42+
# LUIS
43+
- luis
44+
- Language Understanding
45+
46+
# Orchestrator
47+
- Orchestrator
48+
- Orch
49+
- Conductor organizer
50+

‎Orchestrator/Samples/CLI/ModelTuning/common.lu‎

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Help
2+
- help
3+
- i need help
4+
- please help
5+
- can you please help
6+
- what do you do
7+
- what is this bot for
8+
9+
10+
# BuySurface
11+
- How can I buy {ProductType=Surface PRO}
12+
- I want to buy {ProductType=Surface PRO}
13+
- I want to buy {ProductType=Surface laptop}
14+
- Can I buy {ProductType=Surface PRO} online
15+
16+
# USConstitution
17+
- Tell me about the US Constitution
18+
- US Constitution
19+
- Info on the Constitution
20+
- tell me about the Constitution
21+
- constitution knowledgebase
22+
- constitution faq
23+
- what do you know about the constitution
24+
- The united states constitution
25+
26+
# Restart
27+
- restart
28+
- reset
29+
- start over
30+
- menu
31+
- top menu
32+
33+
# Finance
34+
- pay my bills
35+
- transfer funds
36+
- what's my balance
37+
- get $500 cash
38+
39+
# LUIS
40+
- luis
41+
42+
# Orchestrator
43+
- Orchestrator
44+
- Orch
45+

‎Orchestrator/Samples/CLI/ModelTuning/common.test.lu‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Help
2+
- help
3+
- do you have help
4+
- any assistance in this bot?
5+
6+
7+
# BuySurface
8+
- Looking for a computer made by MSFT
9+
- what kind of MS products do you have?
10+
11+
# USConstitution
12+
- Where can I read our founding fathers famous document?
13+
- Is there an analysis of the bill of rights?
14+
- What is that proclamation thing?
15+
16+
# Restart
17+
- reset
18+
- go to the beginning
19+
20+
# Finance
21+
- can i pay with credit card?
22+
- do you use Zelle?
23+
- what is my bank borrowing limit?
24+
25+
26+
# LUIS
27+
- Language Understanding Inteligent Service
28+
29+
# Orchestrator
30+
- Conductor
31+

‎Orchestrator/Samples/CLI/ModelTuning/comparedLU.png‎

212 KB
Loading

‎Orchestrator/Samples/CLI/ModelTuning/demo.cmd‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
@echo off
2+
@echo ORCHESTRATOR EVALUATION DEMO
3+
4+
@rem set SEED for different test sets
5+
set SEED=common
6+
set BLU=generated\%SEED%.blu
7+
set LUFILE=%SEED%.lu
8+
@rem test file contains sample utterances that are not in main LU file.
9+
set TESTFILE=%SEED%.test.lu
10+
@rem proper test
11+
@rem set TESTFILE=%SEED%.test.lu
12+
13+
14+
set LUISKEY=
15+
set LUISAPP=
16+
set LUISHOST=
17+
18+
set SKIPLUIS=0
19+
set QUERYRUN=0
20+
21+
if "%LUISKEY%" == "" (
22+
@echo Skipping comparison with LUIS. Fill in LUIS info to compare results.
23+
set SKIPLUIS=1
24+
)
25+
26+
@rem set QUERY to run a single utterance test
27+
set QUERY="what is the american declaration of independence?"
28+
29+
if "%1" == "qonly" (
30+
set QUERYRUN=1
31+
goto QUERYONLY
32+
)
33+
34+
@rem model folder needs to be downloaded only once.
35+
if "%1" == "getmodel" (
36+
if EXIST model rd /s /q model
37+
)
38+
39+
@echo cleaning folders
40+
if EXIST report (
41+
rd /s /q report && md report
42+
)
43+
if EXIST generated (
44+
rd /s /q generated && md generated
45+
)
46+
47+
48+
@rem Only need to retrieve model once
49+
IF NOT EXIST .\model (
50+
@rem see bf orchestrator:basemodel:get --help to get the non-default model
51+
@rem see available models via bf orchestrator:basemodel:list
52+
@echo getting base model
53+
md model
54+
call bf orchestrator:basemodel:get --out model
55+
)
56+
@echo Create orchestrator snapshot .blu file
57+
call bf orchestrator:create --model model --out %BLU% --in %LUFILE%
58+
59+
@echo running orchestrator test to generate a report (see report folder)
60+
call bf orchestrator:test --in %BLU% --model ./model --out report --test %TESTFILE%
61+
62+
if "%SKIPLUIS%" == "0" (
63+
@echo running LUIS test...
64+
call bf luis:test --subscriptionKey %LUISKEY% --endpoint %LUISHOST% --appId %LUISAPP% --in %TESTFILE% --out report/luisresult.txt
65+
)
66+
67+
:QUERYONLY
68+
@rem Illustrates how to query for only a single utterance. Edit %QUERY% above.
69+
if "%QUERYRUN%" == "1" (
70+
71+
echo Orchestrator single utterance query:
72+
echo bf orchestrator:query --in %BLU% --model model --query %QUERY%
73+
call bf orchestrator:query --in %BLU% --model model --query %QUERY%
74+
75+
76+
if "%SKIPLUIS%" == "0" (
77+
echo LUIS single utterance query:
78+
echo bf luis:application:query --appId LUISAPP --endpoint LUISHOST --subscriptionKey LUISKEY --query %QUERY%
79+
call bf luis:application:query --appId %LUISAPP% --endpoint %LUISHOST% --subscriptionKey %LUISKEY% --query %QUERY%
80+
)
81+
)
82+
83+
:DONE
84+

‎Orchestrator/Samples/CLI/ModelTuning/reportafter.png‎

15.2 KB
Loading

‎Orchestrator/Samples/CLI/ModelTuning/reportbefore.png‎

39.1 KB
Loading

‎Orchestrator/Samples/Composer/OrchestratorDispatch/.gitignore‎

Lines changed: 0 additions & 2 deletions
This file was deleted.

0 commit comments

Comments
 (0)