Skip to content

Commit d16ca7a

Browse files
authored
Merge pull request #105 from ks6088ts-labs/feature/issue-69_promptflow-demo
fork chat-math-variant sample
2 parents ae906b0 + 6bcd9d4 commit d16ca7a

File tree

12 files changed

+338
-1
lines changed

12 files changed

+338
-1
lines changed

.pre-commit-config.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,4 +13,4 @@ repos:
1313
rev: 24.2.0
1414
hooks:
1515
- id: black
16-
exclude: 'generated/.*|artifacts/.*|.jsonl|.csv'
16+
exclude: 'generated/.*|artifacts/.*|.jsonl|.csv|.json'

apps/11_promptflow/README.md

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -203,6 +203,32 @@ $ pf run create \
203203
$ pf run show-details --name $RUN_NAME
204204
```
205205

206+
### chat-math-variant
207+
208+
Tuning prompts using `variants` is a powerful feature in Prompt flow. It allows you to test different prompts and see which one works best for your use case.
209+
210+
Prompt flow repository provides an example of a chat flow with math variants at [examples/flows/chat/chat-math-variant](https://github.com/microsoft/promptflow/tree/main/examples/flows/chat/chat-math-variant).
211+
212+
To understand how to use variants, you can refer to the [How-to Guides > Tune prompts using variants](https://microsoft.github.io/promptflow/how-to-guides/tune-prompts-with-variants.html) document.
213+
214+
```shell
215+
cd apps/11_promptflow/chat-math-variant
216+
217+
# Create run with multiple lines data with variant
218+
$ RUN_NAME=chat-math-variant-$(date +%s)
219+
$ VARIANT='${chat.variant_0}'
220+
$ pf run create \
221+
--name $RUN_NAME \
222+
--flow . \
223+
--data ./data.jsonl \
224+
--column-mapping question='${data.question}' \
225+
--variant $VARIANT \
226+
--stream
227+
228+
# Show run details
229+
$ pf run show-details --name $RUN_NAME
230+
```
231+
206232
## References
207233

208234
- [Prompt flow > repos](https://github.com/microsoft/promptflow)
Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
.env
2+
__pycache__/
3+
.promptflow/*
4+
!.promptflow/flow.tools.json
5+
.runs/
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
{
2+
"package": {},
3+
"code": {
4+
"chat.jinja2": {
5+
"type": "llm",
6+
"inputs": {
7+
"chat_history": {
8+
"type": [
9+
"string"
10+
]
11+
},
12+
"question": {
13+
"type": [
14+
"string"
15+
]
16+
}
17+
},
18+
"source": "chat.jinja2"
19+
},
20+
"chat_variant_1.jinja2": {
21+
"type": "llm",
22+
"inputs": {
23+
"chat_history": {
24+
"type": [
25+
"string"
26+
]
27+
},
28+
"question": {
29+
"type": [
30+
"string"
31+
]
32+
}
33+
},
34+
"source": "chat_variant_1.jinja2"
35+
},
36+
"chat_variant_2.jinja2": {
37+
"type": "llm",
38+
"inputs": {
39+
"chat_history": {
40+
"type": [
41+
"string"
42+
]
43+
},
44+
"question": {
45+
"type": [
46+
"string"
47+
]
48+
}
49+
},
50+
"source": "chat_variant_2.jinja2"
51+
},
52+
"extract_result.py": {
53+
"type": "python",
54+
"inputs": {
55+
"input1": {
56+
"type": [
57+
"string"
58+
]
59+
}
60+
},
61+
"source": "extract_result.py",
62+
"function": "my_python_tool"
63+
}
64+
}
65+
}
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Test your prompt variants for chat with math
2+
This is a prompt tuning case with 3 prompt variants for math question answering.
3+
4+
By utilizing this flow, in conjunction with the `evaluation/eval-chat-math` flow, you can quickly grasp the advantages of prompt tuning and experimentation with prompt flow. Here we provide a [video](https://www.youtube.com/watch?v=gcIe6nk2gA4) and a [tutorial]((../../../tutorials/flow-fine-tuning-evaluation/promptflow-quality-improvement.md)) for you to get started.
5+
6+
Tools used in this flow:
7+
- `llm` tool
8+
- custom `python` Tool
9+
10+
## Prerequisites
11+
12+
Install promptflow sdk and other dependencies in this folder:
13+
```bash
14+
pip install -r requirements.txt
15+
```
16+
17+
## Getting started
18+
19+
### 1 Create connection for LLM tool to use
20+
Go to "Prompt flow" "Connections" tab. Click on "Create" button, select one of LLM tool supported connection types and fill in the configurations.
21+
22+
Currently, there are two connection types supported by LLM tool: "AzureOpenAI" and "OpenAI". If you want to use "AzureOpenAI" connection type, you need to create an Azure OpenAI service first. Please refer to [Azure OpenAI Service](https://azure.microsoft.com/en-us/products/cognitive-services/openai-service/) for more details. If you want to use "OpenAI" connection type, you need to create an OpenAI account first. Please refer to [OpenAI](https://platform.openai.com/) for more details.
23+
24+
```bash
25+
# Override keys with --set to avoid yaml file changes
26+
pf connection create --file ../../../connections/azure_openai.yml --set api_key=<your_api_key> api_base=<your_api_base> --name open_ai_connection
27+
```
28+
29+
Note in [flow.dag.yaml](flow.dag.yaml) we are using connection named `open_ai_connection`.
30+
```bash
31+
# show registered connection
32+
pf connection show --name open_ai_connection
33+
```
34+
35+
### 2 Start chatting
36+
37+
```bash
38+
# run chat flow with default question in flow.dag.yaml
39+
pf flow test --flow .
40+
41+
# run chat flow with new question
42+
pf flow test --flow . --inputs question="2+5=?"
43+
44+
# start a interactive chat session in CLI
45+
pf flow test --flow . --interactive
46+
47+
# start a interactive chat session in CLI with verbose info
48+
pf flow test --flow . --interactive --verbose
Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
# system:
2+
You are an assistant to calculate the answer to the provided math problems.
3+
Please return the final numerical answer only, without any accompanying reasoning or explanation.
4+
5+
{% for item in chat_history %}
6+
# user:
7+
{{item.inputs.question}}
8+
# assistant:
9+
{{item.outputs.answer}}
10+
{% endfor %}
11+
12+
# user:
13+
{{question}}
Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# system:
2+
You are an assistant to calculate the answer to the provided math problems.
3+
Please think step by step.
4+
Return the final numerical answer only and any accompanying reasoning or explanation seperately as json format.
5+
6+
# user:
7+
A jar contains two red marbles, three green marbles, ten white marbles and no other marbles. Two marbles are randomly drawn from this jar without replacement. What is the probability that these two marbles drawn will both be red? Express your answer as a common fraction.
8+
# assistant:
9+
{Chain of thought: "The total number of marbles is $2+3+10=15$. The probability that the first marble drawn will be red is $2/15$. Then, there will be one red left, out of 14. Therefore, the probability of drawing out two red marbles will be: $$\\frac{2}{15}\\cdot\\frac{1}{14}=\\boxed{\\frac{1}{105}}$$.", "answer": "1/105"}
10+
# user:
11+
Find the greatest common divisor of $7!$ and $(5!)^2.$
12+
# assistant:
13+
{"Chain of thought": "$$ \\begin{array} 7! &=& 7 \\cdot 6 \\cdot 5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1 &=& 2^4 \\cdot 3^2 \\cdot 5^1 \\cdot 7^1 \\\\ (5!)^2 &=& (5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1)^2 &=& 2^6 \\cdot 3^2 \\cdot 5^2 \\\\ \\text{gcd}(7!, (5!)^2) &=& 2^4 \\cdot 3^2 \\cdot 5^1 &=& \\boxed{720} \\end{array} $$.", "answer": "720"}
14+
{% for item in chat_history %}
15+
16+
# user:
17+
{{item.inputs.question}}
18+
# assistant:
19+
{{item.outputs.answer}}
20+
{% endfor %}
21+
22+
# user:
23+
{{question}}
Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
# system:
2+
You are an assistant to calculate the answer to the provided math problems.
3+
Please think step by step.
4+
Return the final numerical answer only and any accompanying reasoning or explanation seperately as json format.
5+
6+
# user:
7+
A jar contains two red marbles, three green marbles, ten white marbles and no other marbles. Two marbles are randomly drawn from this jar without replacement. What is the probability that these two marbles drawn will both be red? Express your answer as a common fraction.
8+
# assistant:
9+
{Chain of thought: "The total number of marbles is $2+3+10=15$. The probability that the first marble drawn will be red is $2/15$. Then, there will be one red left, out of 14. Therefore, the probability of drawing out two red marbles will be: $$\\frac{2}{15}\\cdot\\frac{1}{14}=\\boxed{\\frac{1}{105}}$$.", "answer": "1/105"}
10+
# user:
11+
Find the greatest common divisor of $7!$ and $(5!)^2.$
12+
# assistant:
13+
{"Chain of thought": "$$ \\begin{array} 7! &=& 7 \\cdot 6 \\cdot 5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1 &=& 2^4 \\cdot 3^2 \\cdot 5^1 \\cdot 7^1 \\\\ (5!)^2 &=& (5 \\cdot 4 \\cdot 3 \\cdot 2 \\cdot 1)^2 &=& 2^6 \\cdot 3^2 \\cdot 5^2 \\\\ \\text{gcd}(7!, (5!)^2) &=& 2^4 \\cdot 3^2 \\cdot 5^1 &=& \\boxed{720} \\end{array} $$.", "answer": "720"}
14+
# user:
15+
A club has 10 members, 5 boys and 5 girls. Two of the members are chosen at random. What is the probability that they are both girls?
16+
# assistant:
17+
{"Chain of thought": "There are $\\binomial{10}{2} = 45$ ways to choose two members of the group, and there are $\\binomial{5}{2} = 10$ ways to choose two girls. Therefore, the probability that two members chosen at random are girls is $\\dfrac{10}{45} = \\boxed{\\dfrac{2}{9}}$.", "answer": "2/9"}
18+
# user:
19+
Allison, Brian and Noah each have a 6-sided cube. All of the faces on Allison's cube have a 5. The faces on Brian's cube are numbered 1, 2, 3, 4, 5 and 6. Three of the faces on Noah's cube have a 2 and three of the faces have a 6. All three cubes are rolled. What is the probability that Allison's roll is greater than each of Brian's and Noah's? Express your answer as a common fraction.
20+
# assistant:
21+
{"Chain of thought": "Since Allison will always roll a 5, we must calculate the probability that both Brian and Noah roll a 4 or lower. The probability of Brian rolling a 4 or lower is $\\frac{4}{6} = \\frac{2}{3}$ since Brian has a standard die. Noah, however, has a $\\frac{3}{6} = \\frac{1}{2}$ probability of rolling a 4 or lower, since the only way he can do so is by rolling one of his 3 sides that have a 2. So, the probability of both of these independent events occurring is $\\frac{2}{3} \\cdot \\frac{1}{2} = \\boxed{\\frac{1}{3}}$.", "answer": "1/3"}
22+
# user:
23+
Compute $\\density binomial{50}{2}$.
24+
# assistant:
25+
{"Chain of thought": "$\\density binomial{50}{2} = \\dfrac{50!}{2!48!}=\\dfrac{50\\times 49}{2\\times 1}=\\boxed{1225}.$", "answer": "1225"}
26+
# user:
27+
The set $S = \\{1, 2, 3, \\ldots , 49, 50\\}$ contains the first $50$ positive integers. After the multiples of 2 and the multiples of 3 are removed, how many integers remain in the set $S$?
28+
# assistant:
29+
{"Chain of thought": "The set $S$ contains $25$ multiples of 2 (that is, even numbers). When these are removed, the set $S$ is left with only the odd integers from 1 to 49. At this point, there are $50-25=25$ integers in $S$. We still need to remove the multiples of 3 from $S$.\n\nSince $S$ only contains odd integers after the multiples of 2 are removed, we must remove the odd multiples of 3 between 1 and 49. These are 3, 9, 15, 21, 27, 33, 39, 45, of which there are 8. Therefore, the number of integers remaining in the set $S$ is $25 - 8 = \\boxed{17}$.", "answer": "17"}
30+
{% for item in chat_history %}
31+
32+
# user:
33+
{{item.inputs.question}}
34+
# assistant:
35+
{{item.outputs.answer}}
36+
{% endfor %}
37+
38+
# user:
39+
{{question}}

0 commit comments

Comments
 (0)