From 23c25d4f78b09873862a582daab0017b6bca97a8 Mon Sep 17 00:00:00 2001 From: Claudio Spiess Date: Thu, 10 Apr 2025 10:13:46 -0700 Subject: [PATCH 1/4] Initial prompt library Signed-off-by: Claudio Spiess --- contrib/prompt_library/CoT.pdl | 124 ++++++++++ contrib/prompt_library/PoT.pdl | 227 ++++++++++++++++++ contrib/prompt_library/RAG.pdl | 26 ++ contrib/prompt_library/ReAct.pdl | 217 +++++++++++++++++ contrib/prompt_library/ReWoo.pdl | 163 +++++++++++++ .../examples/evalplus/general.pdl | 144 +++++++++++ .../prompt_library/examples/fever/general.pdl | 84 +++++++ .../prompt_library/examples/gsm8k/general.pdl | 78 ++++++ contrib/prompt_library/tools.pdl | 135 +++++++++++ 9 files changed, 1198 insertions(+) create mode 100644 contrib/prompt_library/CoT.pdl create mode 100644 contrib/prompt_library/PoT.pdl create mode 100644 contrib/prompt_library/RAG.pdl create mode 100644 contrib/prompt_library/ReAct.pdl create mode 100644 contrib/prompt_library/ReWoo.pdl create mode 100644 contrib/prompt_library/examples/evalplus/general.pdl create mode 100644 contrib/prompt_library/examples/fever/general.pdl create mode 100644 contrib/prompt_library/examples/gsm8k/general.pdl create mode 100644 contrib/prompt_library/tools.pdl diff --git a/contrib/prompt_library/CoT.pdl b/contrib/prompt_library/CoT.pdl new file mode 100644 index 000000000..639cba07a --- /dev/null +++ b/contrib/prompt_library/CoT.pdl @@ -0,0 +1,124 @@ +description: CoT pattern introduced by Wei et al. (2022) +defs: + # Chain of Thought + cot_block: + function: + question: str + reasoning: str + answer: str + return: |- + Question: ${ question } + Answer: Let's think step by step. ${ reasoning } + The answer is ${ answer } + + # Auto Chain of Thought Zhang et al. (2022) + # The idea is to use a _model_ to generate a reasoning path, even if not very accurate. + # It is best combined with some fewshot examples + auto_chain_of_thought: + function: + question: str + model: str + answer: str + return: + text: + - |- + Question: ${ question } + - "Answer: Let's think step by step. " + - model: ${ model } + parameters: + # decoding_method: "greedy" + temperature: 0 + stop: + - "The answer is" + include_stop_sequence: false + - "The answer is ${ answer }" + + fewshot_cot: + function: + examples: + { list: { obj: { question: str, reasoning: str, answer: str } } } + return: + text: + - for: + example: ${ examples } + repeat: + text: + - call: ${ cot_block } + args: + question: ${ example.question } + reasoning: ${ example.reasoning } + answer: ${ example.answer } + - "\n\n" + join: + with: "" + + chain_of_thought: + function: + question: str + model: str + examples: + { list: { obj: { question: str, reasoning: str, answer: str } } } + return: + text: + - call: ${ fewshot_cot } + args: + examples: ${ examples } + - "Question: ${ question }\n" + - "Answer: Let's think step by step. " + - model: ${ model } + parameters: + temperature: 0 + stop: + - "<|endoftext|>" + include_stop_sequence: false + + claim_cot: + function: + question: str + reasoning: str + answer: str + return: |- + Task: On June 2017, the following claim was made: ${ question } + Q: Was this claim true or false? + Thought: Let's think step by step. ${ reasoning|trim } + Answer: The claim is ${ answer } + + fewshot_cot_claim: + function: + examples: + { list: { obj: { question: str, reasoning: str, answer: str } } } + return: + text: + - for: + example: ${ examples } + repeat: + text: + - call: ${ claim_cot } + args: + question: ${ example.question } + reasoning: ${ example.reasoning } + answer: ${ example.answer } + - "\n\n" + join: + with: "" + + chain_of_thought_claim: + function: + question: str + model: str + examples: + { list: { obj: { question: str, reasoning: str, answer: str } } } + return: + text: + - call: ${ fewshot_cot_claim } + args: + examples: ${ examples } + - "${ question }\n" + - "Thought: Let's think step by step. " + - model: ${ model } + parameters: + temperature: 0 + stop: + - "<|endoftext|>" + max_tokens: 1024 + include_stop_sequence: false \ No newline at end of file diff --git a/contrib/prompt_library/PoT.pdl b/contrib/prompt_library/PoT.pdl new file mode 100644 index 000000000..cf5f4bcf4 --- /dev/null +++ b/contrib/prompt_library/PoT.pdl @@ -0,0 +1,227 @@ +description: Program of Thoughts pattern Chen (2022), TMLR +defs: + program_of_thought: + function: + question: str + model: str + return: + - | + Question: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? + # Python code, return ans + total_eggs = 16 + eaten_eggs = 3 + baked_eggs = 4 + sold_eggs = total_eggs - eaten_eggs - baked_eggs + dollars_per_egg = 2 + result = sold_eggs * dollars_per_egg + + Question: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? + # Python code, return ans + bolts_of_blue_fiber = 2 + bolts_of_white_fiber = num_of_blue_fiber / 2 + result = bolts_of_blue_fiber + bolts_of_white_fiber + + Question: Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make? + # Python code, return ans + cost_of_original_house = 80000 + increase_rate = 150 / 100 + value_of_house = (1 + increase_rate) * cost_of_original_house + cost_of_repair = 50000 + result = value_of_house - cost_of_repair - cost_of_original_house + + Question: Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she need to give her chickens in the final meal of the day if the size of Wendi's flock is 20 chickens? + # Python code, return ans + numb_of_chickens = 20 + cups_for_each_chicken = 3 + cups_for_all_chicken = num_of_chickens * cups_for_each_chicken + cups_in_the_morning = 15 + cups_in_the_afternoon = 25 + result = cups_for_all_chicken - cups_in_the_morning - cups_in_the_afternoon + + Question: Kylar went to the store to buy glasses for his new apartment. One glass costs $5, but every second glass costs only 60% of the price. Kylar wants to buy 16 glasses. How much does he need to pay for them? + # Python code, return ans + num_glasses = 16 + first_glass_cost = 5 + second_glass_cost = 5 * 0.6 + result = 0 + for i in range(num_glasses): + if i % 2 == 0: + result += first_glass_cost + else: + result += second_glass_cost + + Question: Marissa is hiking a 12-mile trail. She took 1 hour to walk the first 4 miles, then another hour to walk the next two miles. If she wants her average speed to be 4 miles per hour, what speed (in miles per hour) does she need to walk the remaining distance? + # Python code, return ans + average_mile_per_hour = 4 + total_trail_miles = 12 + remaining_miles = total_trail_miles - 4 - 2 + total_hours = total_trail_miles / average_mile_per_hour + remaining_hours = total_hours - 2 + result = remaining_miles / remaining_hours + + Question: Carlos is planting a lemon tree. The tree will cost $90 to plant. Each year it will grow 7 lemons, which he can sell for $1.5 each. It costs $3 a year to water and feed the tree. How many years will it tak + e before he starts earning money on the lemon tree? + # Python code, return ans + total_cost = 90 + cost_of_watering_and_feeding = 3 + cost_of_each_lemon = 1.5 + num_of_lemon_per_year = 7 + result = 0 + while total_cost > 0: + total_cost += cost_of_watering_and_feeding + total_cost -= num_of_lemon_per_year * cost_of_each_lemon + result += 1 + + Question: When Freda cooks canned tomatoes into sauce, they lose half their volume. Each 16 ounce can of tomatoes that she uses contains three tomatoes. Freda's last batch of tomato sauce made 32 ounces of sauce. How many tomatoes did Freda use? + # Python code, return ans + lose_rate = 0.5 + num_tomato_contained_in_per_ounce_sauce = 3 / 16 + ounce_sauce_in_last_batch = 32 + num_tomato_in_last_batch = ounce_sauce_in_last_batch * num_tomato_contained_in_per_ounce_sauce + result = num_tomato_in_last_batch / (1 - lose_rate) + + Question: Jordan wanted to surprise her mom with a homemade birthday cake. From reading the instructions, she knew it would take 20 minutes to make the cake batter and 30 minutes to bake the cake. The cake would require 2 hours to cool and an additional 10 minutes to frost the cake. If she plans to make the cake all on the same day, what is the latest time of day that Jordan can start making the cake to be ready to serve it at 5:00 pm? + # Python code, return ans + minutes_to_make_batter = 20 + minutes_to_bake_cake = 30 + minutes_to_cool_cake = 2 * 60 + minutes_to_frost_cake = 10 + total_minutes = minutes_to_make_batter + minutes_to_bake_cake + minutes_to_cool_cake + minutes_to_frost_cake + total_hours = total_minutes / 60 + result = 5 - total_hours + + Question: ${ question } + # Python code, return ans + + - def: PROGRAM + model: ${ model } + parameters: + stop: ["\nAnswer: "] + include_stop_sequence: false + - def: ANSWER + lang: python + code: ${ PROGRAM } + - get: ANSWER + + program_of_thought_backtick: + function: + question: str + model: str + return: + - | + Question: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? + # Python code, return ans + ```python + total_eggs = 16 + eaten_eggs = 3 + baked_eggs = 4 + sold_eggs = total_eggs - eaten_eggs - baked_eggs + dollars_per_egg = 2 + result = sold_eggs * dollars_per_egg + ``` + + Question: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? + # Python code, return ans + ```python + bolts_of_blue_fiber = 2 + bolts_of_white_fiber = num_of_blue_fiber / 2 + result = bolts_of_blue_fiber + bolts_of_white_fiber + ``` + + Question: Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make? + # Python code, return ans + ```python + cost_of_original_house = 80000 + increase_rate = 150 / 100 + value_of_house = (1 + increase_rate) * cost_of_original_house + cost_of_repair = 50000 + result = value_of_house - cost_of_repair - cost_of_original_house + ``` + + Question: Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she need to give her chickens in the final meal of the day if the size of Wendi's flock is 20 chickens? + # Python code, return ans + ```python + numb_of_chickens = 20 + cups_for_each_chicken = 3 + cups_for_all_chicken = num_of_chickens * cups_for_each_chicken + cups_in_the_morning = 15 + cups_in_the_afternoon = 25 + result = cups_for_all_chicken - cups_in_the_morning - cups_in_the_afternoon + ``` + + Question: Kylar went to the store to buy glasses for his new apartment. One glass costs $5, but every second glass costs only 60% of the price. Kylar wants to buy 16 glasses. How much does he need to pay for them? + # Python code, return ans + ```python + num_glasses = 16 + first_glass_cost = 5 + second_glass_cost = 5 * 0.6 + result = 0 + for i in range(num_glasses): + if i % 2 == 0: + result += first_glass_cost + else: + result += second_glass_cost + ``` + + Question: Marissa is hiking a 12-mile trail. She took 1 hour to walk the first 4 miles, then another hour to walk the next two miles. If she wants her average speed to be 4 miles per hour, what speed (in miles per hour) does she need to walk the remaining distance? + # Python code, return ans + ```python + average_mile_per_hour = 4 + total_trail_miles = 12 + remaining_miles = total_trail_miles - 4 - 2 + total_hours = total_trail_miles / average_mile_per_hour + remaining_hours = total_hours - 2 + result = remaining_miles / remaining_hours + ``` + + Question: Carlos is planting a lemon tree. The tree will cost $90 to plant. Each year it will grow 7 lemons, which he can sell for $1.5 each. It costs $3 a year to water and feed the tree. How many years will it tak + e before he starts earning money on the lemon tree? + # Python code, return ans + ```python + total_cost = 90 + cost_of_watering_and_feeding = 3 + cost_of_each_lemon = 1.5 + num_of_lemon_per_year = 7 + result = 0 + while total_cost > 0: + total_cost += cost_of_watering_and_feeding + total_cost -= num_of_lemon_per_year * cost_of_each_lemon + result += 1 + ``` + + Question: When Freda cooks canned tomatoes into sauce, they lose half their volume. Each 16 ounce can of tomatoes that she uses contains three tomatoes. Freda's last batch of tomato sauce made 32 ounces of sauce. How many tomatoes did Freda use? + # Python code, return ans + ```python + lose_rate = 0.5 + num_tomato_contained_in_per_ounce_sauce = 3 / 16 + ounce_sauce_in_last_batch = 32 + num_tomato_in_last_batch = ounce_sauce_in_last_batch * num_tomato_contained_in_per_ounce_sauce + result = num_tomato_in_last_batch / (1 - lose_rate) + ``` + + Question: Jordan wanted to surprise her mom with a homemade birthday cake. From reading the instructions, she knew it would take 20 minutes to make the cake batter and 30 minutes to bake the cake. The cake would require 2 hours to cool and an additional 10 minutes to frost the cake. If she plans to make the cake all on the same day, what is the latest time of day that Jordan can start making the cake to be ready to serve it at 5:00 pm? + # Python code, return ans + ```python + minutes_to_make_batter = 20 + minutes_to_bake_cake = 30 + minutes_to_cool_cake = 2 * 60 + minutes_to_frost_cake = 10 + total_minutes = minutes_to_make_batter + minutes_to_bake_cake + minutes_to_cool_cake + minutes_to_frost_cake + total_hours = total_minutes / 60 + result = 5 - total_hours + ``` + + Question: ${ question } + # Python code, return ans + - def: PROGRAM + model: ${ model } + parser: + regex: '```.*\n((?:.|\n|$)*?)$\n\s*```' # extracts code from backtick blocks + mode: findall + parameters: + stop: ["\nAnswer: "] + include_stop_sequence: false + - def: ANSWER + lang: python + code: ${ PROGRAM|join('\n') } + - get: ANSWER \ No newline at end of file diff --git a/contrib/prompt_library/RAG.pdl b/contrib/prompt_library/RAG.pdl new file mode 100644 index 000000000..6c0a64355 --- /dev/null +++ b/contrib/prompt_library/RAG.pdl @@ -0,0 +1,26 @@ +description: Retrieval-Augmented Generation (RAG) following Lewis et al. +defs: + # Corpus: Store the retrieval object in the PDL session + corpus: + function: + corpus: {list: str} + return: + - lang: python + code: | + from rank_bm25 import BM25Okapi + PDL_SESSION.corpus = corpus + PDL_SESSION.tokenized_corpus = [doc.split(" ") for doc in corpus] + PDL_SESSION.bm25_corpus = BM25Okapi(PDL_SESSION.tokenized_corpus) + result = None + # Retrieve from corpus in PDL session + retrieve: + function: + query: str + num_examples: int + spec: {list: str} + return: + - lang: python + code: | + from rank_bm25 import BM25Okapi + tokenized_query = query.split(" ") + result = PDL_SESSION.bm25_corpus.get_top_n(tokenized_query, PDL_SESSION.corpus, n=num_examples) \ No newline at end of file diff --git a/contrib/prompt_library/ReAct.pdl b/contrib/prompt_library/ReAct.pdl new file mode 100644 index 000000000..b1bd23fb5 --- /dev/null +++ b/contrib/prompt_library/ReAct.pdl @@ -0,0 +1,217 @@ +description: ReAct pattern from Yao et al., [ICLR 2023](https://openreview.net/forum?id=WE_vluYUL-X) +# See alternative implementation here: https://smith.langchain.com/hub/hwchase17/react-chat +defs: + react_block: + function: + trajectory: { list: obj } + return: + text: + - for: + trajectory: ${ trajectory } + repeat: + text: + - def: type + text: ${ trajectory.keys()|first } + contribute: [] + - if: ${ type == 'question'} + then: | + Question: ${ trajectory[type]|trim } + - if: ${ type == 'task'} + then: | + Task: ${ trajectory[type]|trim } + - if: ${ type == 'thought'} + then: | + Tho: ${ trajectory[type]|trim } + - if: ${ type == 'action'} + then: | + Act: ${ trajectory[type]|trim } + - if: ${ type == 'observation'} + then: | + Obs: ${ trajectory[type]|trim } + - if: ${ type not in ['question', 'task', 'thought', 'action', 'observation'] } + then: "${ type }: ${ trajectory[type]|trim }" + - "\n" + + react_code_block: + function: + trajectory: { list: obj } + return: + text: + - for: + trajectory: ${ trajectory } + repeat: + text: + - def: type + text: ${ trajectory.keys()|first } + contribute: [] + - if: ${ type == 'task'} + then: | + Task: + ${ trajectory[type]|trim } + - if: ${ type == 'thought'} + then: + text: + - "\n" + - | + Assistant: + ${ trajectory[type]|trim } + - if: ${ type == 'action'} + then: | + + ${ trajectory[type]|trim } + + - if: ${ type == 'observation'} + then: + text: + - "\n" + - | + Observation: + ${ trajectory[type]|trim } + - if: ${ type == 'solution'} + then: |- + + ${ trajectory[type]|trim } + + - if: ${ type not in ['question', 'task', 'thought', 'action', 'observation', 'solution'] } + then: "${ type }: ${ trajectory[type]|trim }" + - "\n" + + finish_action: + data: + display_name: Finish + pdl_function: + description: Respond with the Answer + parameters: + - name: answer + type: string + description: The answer + examples: [] + + demonstrate_tools: + function: + tools: { list: obj } + return: + for: + tool: ${ tools } + repeat: + for: + example: ${ tool.examples } + repeat: + call: ${ react_block } + args: + trajectory: ${ example } + + react: + function: + task: str + model: str + tools: { list: obj } + trajectories: { list: list } + return: + lastOf: + - if: ${ system_prompt == "llama3"} + then: + text: + - role: system + contribute: [context] + text: + - "Cutting Knowledge Date: December 2023\nToday Date: " + - lang: python + code: | + from datetime import datetime + result = datetime.today().strftime('%d %B %Y\n\n') + - role: system + text: You are a helpful assistant with tool calling capabilities. When you receive a tool call response, use the output to format an answer to the orginal use question. + contribute: [context] + - role: user + text: | + Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt. + + Respond in the format {"name": function name, "arguments": dictionary of argument name and its value}. Do not use variables. + + ${ tools } + contribute: [context] + - if: ${ system_prompt == "granite_llama" } + then: + text: + - role: system + text: | + You are a helpful assistant with access to the following function calls. Your task is to produce a sequence of function calls necessary to generate response to the user utterance. Use the following function calls as required. + + Respond in the format {"name": function name, "arguments": dictionary of argument name and its value}. Do not use variables. + + ${ tools } + contribute: [context] + - if: ${ system_prompt == "granite_tools" } + then: + text: + - role: system + contribute: [context] + text: + - "Knowledge Cutoff Date: April 2024.\nToday's Date: " + - lang: python + code: | + from datetime import datetime + result = datetime.today().strftime('%B %d, %Y.\n') + - | + You are Granite, developed by IBM. You are a helpful AI assistant with access to the following tools. When a tool is required to answer the user's query, respond with a JSON object of the tool to use. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request. + + ${ tools } + - "\n" + - for: + traj: ${ trajectories } + repeat: + text: + call: ${ react_block } + args: + trajectory: ${ traj } + - ${ task } + - def: prev_action + contribute: [] + data: none + - def: exit + contribute: [] + data: False + - repeat: + text: + - "\nTho: " + - def: thought + model: "${ model }" + parameters: + stop: + - "Act:" + max_tokens: 256 + include_stop_sequence: false + - "\nAct: " + - def: action + model: "${ model }" + parser: json + parameters: + temperature: 0 + stop: ["\n", "Obs:", "<|eom_id|>"] + include_stop_sequence: false + spec: { name: str, arguments: obj } + - if: ${ action != prev_action } + then: + def: observation + if: ${ action.name.lower() != 'finish' } + then: + text: + - "\nObs: " + - if: ${ action.name.lower() in pdl_tools } + then: + call: ${ pdl_tools[action.name.lower()] } + args: + arguments: ${ action.arguments } + else: "Invalid action. Valid actions are ${ (pdl_tools.keys()|list)[:-1]|join(', ') }, and ${ (pdl_tools.keys()|list)[-1] }." + - "\n" + else: + def: exit + contribute: [] + data: True + - def: prev_action + contribute: [] + data: ${ action } + until: ${ action.name.lower() == "finish" or exit } + - data: + answer: ${ (action.arguments.answer|default("No answer found."))|trim } # Factor 5: answer argument name. some models really like putting answer in. diff --git a/contrib/prompt_library/ReWoo.pdl b/contrib/prompt_library/ReWoo.pdl new file mode 100644 index 000000000..67e57df70 --- /dev/null +++ b/contrib/prompt_library/ReWoo.pdl @@ -0,0 +1,163 @@ +description: ReWOO (Reasoning without observation) pattern from Xu et al., (http://arxiv.org/abs/2305.18323) +# Compared to ReAct, reduced token consumption (and thus execution time), +# by generating full chain of tools in a single pass +# see: https://github.com/langchain-ai/langgraph/blob/main/examples/rewoo/rewoo.ipynb +defs: + rewoo_block: + function: + trajectory: { list: obj } + return: + text: + - defs: + i: + data: 1 + - for: + trajectory: ${ trajectory } + repeat: + text: + - defs: + type: + text: ${ trajectory.keys()|first } + content: + text: ${ trajectory.values()|first } + - if: ${ type in ['task', 'question'] } + then: |- + Task: ${ content|trim } + - if: ${ type == 'thought'} + then: |- + + Plan: ${ content|trim } + - if: ${ type == 'action'} + then: + text: + - " #E${ i } = ${ content|trim }" + - defs: + i: + data: ${ i+1 } + - if: ${ type == 'observation'} + then: "" + - if: ${ type not in ['question', 'task', 'thought', 'action', 'observation'] } + then: "${ type }: ${ content|trim }\n" + - "\n" + + rewoo: + function: + task: str + model: str + tools: { list: obj } + trajectories: { list: list } + show_plans: bool + return: + lastOf: + - | + For the following task, make plans that can solve the problem step by step. For each plan, indicate which external tool together with tool input to retrieve evidence. You can store the evidence into a variable #E that can be called by later tools. (Plan, #E1, Plan, #E2, Plan, ...) + + Tools can be one of the following: + ${ tools } + - for: + traj: ${ trajectories } + repeat: + text: + - call: ${ rewoo_block } + args: + trajectory: ${ traj } + - "\n" + - | + Begin! + Describe your plans with rich details. Each Plan should be followed by only one #E. + + ${ task } + - def: PLANS + model: ${ model } + parser: # plan, step_name, tool, tool_input + regex: 'Plan:\s*(?P(?:.|\n)*?)\s*(?P#E\d+)\s*=\s*(?P\{.+\})' + mode: findall + parameters: + temperature: 0 + stop: + - "<|endoftext|>" + - "\n\n" + - "Task:" + include_stop_sequence: false + max_tokens: 1024 + - if: ${ show_plans } + then: + text: + - "\n\n--- Raw plans ---\n" + - ${ PLANS } + - "\n\n--- Extracted Blueprint ---\n" + - for: + plan: ${ PLANS } + repeat: + text: + - "Plan: ${ plan[0] }\n" + - "${ plan[1] } = ${ plan[2] }\n" + - "----------------------------\n\n" + - defs: + output: + data: {} + plans: + for: + plan: ${ PLANS } + repeat: + lastOf: + - defs: + PLAN: ${ plan[0] } + ID: ${ plan[1] } + ACTION_RAW: ${ plan[2] } + ACTION: + parser: json + lang: python + code: |- + for k,v in output.items(): + if k in ACTION_RAW: + ACTION_RAW = ACTION_RAW.replace(k, v) + result = ACTION_RAW + tool_output: + if: ${ ACTION.name.lower() in pdl_tools } + then: + call: ${ pdl_tools[ACTION.name.lower()] } + args: + arguments: ${ ACTION.arguments } + else: "Invalid action. Valid actions are ${ pdl_tools.keys() }" + - def: output + lang: python + contribute: [] + code: | + output[ID] = str(tool_output) + result = output + - | + Plan: ${ PLAN } + Evidence: ${ tool_output } + - if: ${ plans is not none and plans|length > 0 } + then: + lastOf: + - def: solution_input + text: |- + Solve the following task or problem. To solve the problem, we have made step-by-step Plan and retrieved corresponding Evidence to each Plan. Use them with caution since long evidence might contain irrelevant information. + + ${ plans|join } + Now solve the question or task according to provided Evidence above. Respond with the answer directly with no extra words. + + ${ task } + Response: + - def: SOLUTION + model: ${ model } + parameters: + timeout: 30 + time_limit: 30000 + temperature: 0 + stop: + - "<|endoftext|>" + include_stop_sequence: false + max_tokens: 1024 + input: + text: ${ solution_input } + - data: + answer: ${ SOLUTION } + else: + lastOf: + - def: SOLUTION + text: "No plans found." + - data: + answer: ${ SOLUTION } diff --git a/contrib/prompt_library/examples/evalplus/general.pdl b/contrib/prompt_library/examples/evalplus/general.pdl new file mode 100644 index 000000000..f7c9acd51 --- /dev/null +++ b/contrib/prompt_library/examples/evalplus/general.pdl @@ -0,0 +1,144 @@ +description: EvalPlus simple +defs: + react_demos: + data: + - - task: |- + Write a function to find the similar elements from the given two tuple lists. + assert similar_elements((3, 4, 5, 6),(5, 7, 4, 10)) == (4, 5) + - thought: |- + The similar elements here are just elements that are in both lists. I should run a solution on the test case before proposing a solution. + - action: |- + def similar_elements(test_tup1, test_tup2): + res = tuple(set(test_tup1) | set(test_tup2)) + return res + res = similar_elements((3, 4, 5, 6), (5, 7, 4, 10)) + assert res == (4, 5), "Expected (4, 5) but got {}".format(res) + - observation: |- + --------------------------------------------------------------------------- + AssertionError Traceback (most recent call last) + Cell In[4], line 5 + 3 return res + 4 res = similar_elements((3, 4, 5, 6), (5, 7, 4, 10)) + ----> 5 assert res == (4, 5), "Expected (4, 5) but got {}".format(res) + + AssertionError: Expected (4, 5) but got (3, 4, 5, 6, 7, 10) + - thought: |- + To find the similar elements, I should find the intersection of the two sets. I should use the & operator instead of the | operator. + - action: |- + def similar_elements(test_tup1, test_tup2): + res = tuple(set(test_tup1) & set(test_tup2)) + return res + res = similar_elements((3, 4, 5, 6), (5, 7, 4, 10)) + assert res == (4, 5), "Expected (4, 5) but got {}".format(res) + - observation: "[Executed Successfully with No Output]" + - thought: There is no more AssertionError. I can now submit the solution. + - solution: |- + def similar_elements(test_tup1, test_tup2): + res = tuple(set(test_tup1) & set(test_tup2)) + return res +text: + # CoT + - if: ${ prompt_pattern == 'cot' } + then: + text: + - for: + example: ${ demonstrations } + repeat: + text: |+ + Task: ${ example.question } + Solution: + ${ example.answer } + + - | + Task: ${ prompt } + Solution: + - model: ${ model } + parameters: + stop: + - "<|endoftext|>" + max_tokens: 512 + include_stop_sequence: false + temperature: 0 + + # ReAct + - if: ${ prompt_pattern == 'react' } + then: + text: + # see https://github.com/xingyaoww/mint-bench/blob/main/mint/tasks/in_context_examples/mbpp/with_tool.txt for prompt + - include: ../../ReAct.pdl + - for: + traj: ${ demonstrations } + repeat: + text: + - call: ${ react_code_block } + args: + trajectory: ${ traj } + - "\n" + - for: + traj: ${ react_demos } + repeat: + text: + - call: ${ react_code_block } + args: + trajectory: ${ traj } + - |- + + Task: + ${ prompt|trim } + + Assistant: + + - defs: + temperature: 0.0 + SOLUTION: "" + iterations: 0 + max_iterations: 5 + EXECUTE: "" + - repeat: + text: + - repeat: + text: + - defs: + iterations: ${ iterations+1 } + - def: THOUGHT + text: + - model: ${ model } + def: THOUGHT_inner + parameters: + temperature: ${ temperature } + stop: + - + - + - <|endoftext|> + max_tokens: 1024 + include_stop_sequences: true + - def: thought_trim + contribute: [] + text: ${ THOUGHT_inner|trim|trim('\n') } + until: ${ thought_trim.endswith('') or (thought_trim).endswith('') or iterations>max_iterations } + - if: ${ (thought_trim).endswith('') } + then: + text: + - "\n\nObservation:\n" + - def: observation + contribute: [] + lang: ipython + code: | + ${ thought_trim.split('')[-1].strip('\n\n').lstrip('```python').strip('`') or thought_trim } + - "${ (observation|trim|trim('\n')) if observation is not none else '[Executed Successfully with No Output]' }" + - |- + + + Assistant: + + - if: ${ thought_trim.endswith('') } + then: + defs: + SOLUTION: ${ thought_trim.split('')[-1].strip('') or thought_trim } + until: ${ SOLUTION != '' or iterations>max_iterations } + - if: ${ iterations>max_iterations } + then: "[ Failed to find a solution within ${ max_iterations } iterations. ]" + - |- + + Solution: + ${ SOLUTION|trim } \ No newline at end of file diff --git a/contrib/prompt_library/examples/fever/general.pdl b/contrib/prompt_library/examples/fever/general.pdl new file mode 100644 index 000000000..a29245577 --- /dev/null +++ b/contrib/prompt_library/examples/fever/general.pdl @@ -0,0 +1,84 @@ +description: Demo of ReAct template fever +defs: + search_tools_react: + data: + - name: Search ReAct + display_name: Search + pdl_function: Search + description: Search Wikipedia for a summary + parameters: + - name: topic + type: string + description: The topic of interest + examples: [] + + search_tools_rewoo: + data: + - name: Search ReWOO + display_name: Search + pdl_function: Search + description: Search Wikipedia for a summary + parameters: + - name: topic + type: string + description: The topic of interest + examples: [] + + search_tools: + data: + - type: function + function: + name: Search + description: Search Wikipedia for a summary + parameters: + type: object + properties: + topic: + type: string + description: The topic of interest + required: + - topic + + task: |- + Task: On June 2017, the following claim was made: ${ claim } + Q: Was this claim true or false? +text: + - include: ../../tools.pdl + + # CoT + - if: ${ prompt_pattern == 'cot' } + then: + text: + - include: ../../CoT.pdl + - call: ${ chain_of_thought_claim } + args: + examples: ${ demonstrations } + question: ${ task } + model: ${ model } + + # ReAct + - if: ${ prompt_pattern == 'react' } + then: + text: + - include: ../../ReAct.pdl + - call: ${ react } + def: ANSWER + args: + task: ${ task } + model: ${ model } + tools: ${ search_tools } + trajectories: ${ demonstrations } + + # ReWOO + - if: ${ prompt_pattern == 'rewoo' } + then: + text: + - include: ../../ReWoo.pdl + - call: ${ rewoo } + def: ANSWER + args: + task: ${ task } + model: ${ model } + tools: ${ search_tools } + trajectories: ${ demonstrations } + show_plans: false \ No newline at end of file diff --git a/contrib/prompt_library/examples/gsm8k/general.pdl b/contrib/prompt_library/examples/gsm8k/general.pdl new file mode 100644 index 000000000..babcc9df5 --- /dev/null +++ b/contrib/prompt_library/examples/gsm8k/general.pdl @@ -0,0 +1,78 @@ +description: Demo of template +defs: + cot: + import: ../../CoT + react: + import: ../../ReAct + rewoo: + import: ../../ReWoo + tools: + import: ../../tools + model: watsonx/ibm/granite-3-8b-instruct + prompt_pattern: cot + system_prompt: granite_llama + question: "Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day?" + demonstrations: + data: + - question: |- + Noah charges $60 for a large painting and $30 for a small painting. + Last month he sold eight large paintings and four small paintings. + If he sold twice as much this month, how much is his sales for this month? + reasoning: |- + He sold 8 large paintings and 4 small paintings last month. + He sold twice as many this month. + 8 large paintings x $60 = << 8*60= 480 >> 480 + 4 small paintings x $30 = << 4*30= 120 >> 120 + So he sold << 480+120= 600 >> 600 paintings last month. + Therefore he sold << 600*2= 1200 >> this month. + answer: $1200 + - question: |- + Noah charges $30 for a large vases and $10 for a small vases. + Last month he sold five large vases and three small vases. + If he sold three times as much this month, how much is his sales for this month? + reasoning: |- + He sold 5 large vases and 3 small vases last month. + He sold three times as many this month. + 5 large vases x $30 = << 5*30= 150 >> 150 + 3 small vases x $10 = << 3*10= 30 >> 30 + So he sold << 150+30= 180 >> 180 vases last month. + Therefore he sold << 180*3= 540 >> this month. + answer: $540 +text: + # CoT + - if: "${ prompt_pattern == 'cot' }" + then: + text: + - "Answer the questions to the best of your abilities.\n\n" + - call: ${ cot.chain_of_thought } + args: + examples: "${ demonstrations }" + question: "${ question|trim }" + model: "${ model }" + + # ReAct + - if: "${ prompt_pattern == 'react' }" + then: + text: + - call: ${ react.react } + def: ANSWER + args: + task: "Question: ${ question|trim }" + model: "${ model }" + tools: ${ tools.tools } + trajectories: ${ demonstrations } + - "\nThe answer is ${ ANSWER.answer|trim }" + + # ReWOO + - if: "${ prompt_pattern == 'rewoo' }" + then: + text: + - call: ${ rewoo.rewoo } + def: ANSWER + args: + task: "Task: ${ question }" + model: "${ model }" + tools: ${ tools.tools } + trajectories: ${ demonstrations } + show_plans: false + - "\nThe answer is ${ ANSWER.answer|trim }" diff --git a/contrib/prompt_library/tools.pdl b/contrib/prompt_library/tools.pdl new file mode 100644 index 000000000..db77827c4 --- /dev/null +++ b/contrib/prompt_library/tools.pdl @@ -0,0 +1,135 @@ +description: Toolbox of PDL functions for agents +defs: + # Note: Although PDL functions can be properly typed, + # the input to a function via the LLM is fundamentally a string. + # Therefore, parsing the input is the responsibility of the + # function, not the caller. In the future, one could + # imagine the use of constrained decoding to force + # LLM to produce a type-compliant JSON as input. + + wrap_document: + data: true + + filter_tools_by_name: + function: + tools: { list: obj } + tool_names: { list: str } + return: + data: ${ tools|selectattr('display_name', 'in', tool_names)|list } + + list_tools: + function: + tools: { list: obj } + return: + lastOf: + - defs: + signatures: + for: + tool: ${ tools } + repeat: "${ tool.display_name }[<${ tool.parameters|join('>, <', attribute='name') }>]" + join: + as: array + typed_signatures: + for: + tool: ${ tools } + repeat: + lastOf: + - defs: + params: + for: + param: ${ tool.parameters } + repeat: "${ param.name}${ (': ' ~ param.type) if param.type else '' }" + - "${ tool.display_name }(${ params|join(', ') })" + join: + as: array + display_names: ${ tools|map(attribute='display_name')|list } + pdl_functions: ${ tools|map(attribute='pdl_function')|list } + name_map: + lang: python + code: | + result = dict(zip(display_names, pdl_functions, strict=True)) + - data: + names: ${ tools|map(attribute='display_name')|list } + display_names: ${ tools|map(attribute='display_name')|list } + pdl_functions: ${ tools|map(attribute='pdl_function')|list } + name_map: ${ name_map } + signatures: ${ signatures } + typed_signatures: ${ typed_signatures } + descriptions: ${ tools|map(attribute='description')|list } + + tools: + data: + - type: function + function: + name: Calculator + description: Calculator function + parameters: + type: object + properties: + expr: + type: string + description: Arithmetic expression to calculate + required: + - expr + + - type: function + function: + name: finish + description: Respond with the answer + parameters: + type: object + properties: + answer: + type: str + description: The answer + required: + - answer + + pdl_tools: + object: + calculator: + function: + arguments: obj + return: + lang: python + code: | + from sympy.parsing.sympy_parser import parse_expr + result = None + + def main(expr: str, *args, **kwargs) -> str: + try: + exp = "${ arguments.expr|replace(",", "")|replace("$","")|replace("^","**")|replace("%","*0.01") }" + + result = float(parse_expr(exp).evalf()) + result = round(result, 2) + if result.is_integer(): + result = int(result) + except Exception as e: + result = f"There was an error calculating: {e}. Please try again." + + return result + + result = main(**arguments) + search: + function: + arguments: obj + return: + lastOf: + - def: result + lang: python + code: | + import warnings, wikipedia + warnings.simplefilter("ignore") + + def main(topic: str, *args, **kwargs) -> str: + try: + return wikipedia.summary(topic) + except wikipedia.DisambiguationError as d: + return f"\"{topic}\" may refer to one of {d.args[1]}. Please retry the search with a more specific subject." + except wikipedia.WikipediaException as e: + return str(e) + + result = main(**arguments) + - if: ${ wrap_document } + then: "[Document]\n${ result|trim }\n[End]" + else: ${ result } \ No newline at end of file From f29cc4a8702e75d76a528bf2bb3c423f9275135a Mon Sep 17 00:00:00 2001 From: Claudio Spiess Date: Mon, 14 Apr 2025 08:58:03 -0700 Subject: [PATCH 2/4] Documentation and fixes Signed-off-by: Claudio Spiess --- contrib/prompt_library/CoT.pdl | 3 + contrib/prompt_library/PoT.pdl | 227 ------------------ contrib/prompt_library/RAG.pdl | 26 -- contrib/prompt_library/ReAct.pdl | 1 + .../examples/evalplus/general.pdl | 2 +- .../prompt_library/examples/fever/general.pdl | 1 + .../prompt_library/examples/gsm8k/example.pdl | 79 ++++++ .../prompt_library/examples/gsm8k/general.pdl | 61 ++--- docs/tutorial.md | 178 +++++++++----- examples/prompt_library/gsm8k_cot.pdl | 41 ++++ 10 files changed, 274 insertions(+), 345 deletions(-) delete mode 100644 contrib/prompt_library/PoT.pdl delete mode 100644 contrib/prompt_library/RAG.pdl create mode 100644 contrib/prompt_library/examples/gsm8k/example.pdl create mode 100644 examples/prompt_library/gsm8k_cot.pdl diff --git a/contrib/prompt_library/CoT.pdl b/contrib/prompt_library/CoT.pdl index 639cba07a..a325d0286 100644 --- a/contrib/prompt_library/CoT.pdl +++ b/contrib/prompt_library/CoT.pdl @@ -27,6 +27,7 @@ defs: - model: ${ model } parameters: # decoding_method: "greedy" + max_tokens: 1024 temperature: 0 stop: - "The answer is" @@ -67,9 +68,11 @@ defs: - "Answer: Let's think step by step. " - model: ${ model } parameters: + max_tokens: 1024 temperature: 0 stop: - "<|endoftext|>" + - "Question:" include_stop_sequence: false claim_cot: diff --git a/contrib/prompt_library/PoT.pdl b/contrib/prompt_library/PoT.pdl deleted file mode 100644 index cf5f4bcf4..000000000 --- a/contrib/prompt_library/PoT.pdl +++ /dev/null @@ -1,227 +0,0 @@ -description: Program of Thoughts pattern Chen (2022), TMLR -defs: - program_of_thought: - function: - question: str - model: str - return: - - | - Question: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? - # Python code, return ans - total_eggs = 16 - eaten_eggs = 3 - baked_eggs = 4 - sold_eggs = total_eggs - eaten_eggs - baked_eggs - dollars_per_egg = 2 - result = sold_eggs * dollars_per_egg - - Question: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? - # Python code, return ans - bolts_of_blue_fiber = 2 - bolts_of_white_fiber = num_of_blue_fiber / 2 - result = bolts_of_blue_fiber + bolts_of_white_fiber - - Question: Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make? - # Python code, return ans - cost_of_original_house = 80000 - increase_rate = 150 / 100 - value_of_house = (1 + increase_rate) * cost_of_original_house - cost_of_repair = 50000 - result = value_of_house - cost_of_repair - cost_of_original_house - - Question: Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she need to give her chickens in the final meal of the day if the size of Wendi's flock is 20 chickens? - # Python code, return ans - numb_of_chickens = 20 - cups_for_each_chicken = 3 - cups_for_all_chicken = num_of_chickens * cups_for_each_chicken - cups_in_the_morning = 15 - cups_in_the_afternoon = 25 - result = cups_for_all_chicken - cups_in_the_morning - cups_in_the_afternoon - - Question: Kylar went to the store to buy glasses for his new apartment. One glass costs $5, but every second glass costs only 60% of the price. Kylar wants to buy 16 glasses. How much does he need to pay for them? - # Python code, return ans - num_glasses = 16 - first_glass_cost = 5 - second_glass_cost = 5 * 0.6 - result = 0 - for i in range(num_glasses): - if i % 2 == 0: - result += first_glass_cost - else: - result += second_glass_cost - - Question: Marissa is hiking a 12-mile trail. She took 1 hour to walk the first 4 miles, then another hour to walk the next two miles. If she wants her average speed to be 4 miles per hour, what speed (in miles per hour) does she need to walk the remaining distance? - # Python code, return ans - average_mile_per_hour = 4 - total_trail_miles = 12 - remaining_miles = total_trail_miles - 4 - 2 - total_hours = total_trail_miles / average_mile_per_hour - remaining_hours = total_hours - 2 - result = remaining_miles / remaining_hours - - Question: Carlos is planting a lemon tree. The tree will cost $90 to plant. Each year it will grow 7 lemons, which he can sell for $1.5 each. It costs $3 a year to water and feed the tree. How many years will it tak - e before he starts earning money on the lemon tree? - # Python code, return ans - total_cost = 90 - cost_of_watering_and_feeding = 3 - cost_of_each_lemon = 1.5 - num_of_lemon_per_year = 7 - result = 0 - while total_cost > 0: - total_cost += cost_of_watering_and_feeding - total_cost -= num_of_lemon_per_year * cost_of_each_lemon - result += 1 - - Question: When Freda cooks canned tomatoes into sauce, they lose half their volume. Each 16 ounce can of tomatoes that she uses contains three tomatoes. Freda's last batch of tomato sauce made 32 ounces of sauce. How many tomatoes did Freda use? - # Python code, return ans - lose_rate = 0.5 - num_tomato_contained_in_per_ounce_sauce = 3 / 16 - ounce_sauce_in_last_batch = 32 - num_tomato_in_last_batch = ounce_sauce_in_last_batch * num_tomato_contained_in_per_ounce_sauce - result = num_tomato_in_last_batch / (1 - lose_rate) - - Question: Jordan wanted to surprise her mom with a homemade birthday cake. From reading the instructions, she knew it would take 20 minutes to make the cake batter and 30 minutes to bake the cake. The cake would require 2 hours to cool and an additional 10 minutes to frost the cake. If she plans to make the cake all on the same day, what is the latest time of day that Jordan can start making the cake to be ready to serve it at 5:00 pm? - # Python code, return ans - minutes_to_make_batter = 20 - minutes_to_bake_cake = 30 - minutes_to_cool_cake = 2 * 60 - minutes_to_frost_cake = 10 - total_minutes = minutes_to_make_batter + minutes_to_bake_cake + minutes_to_cool_cake + minutes_to_frost_cake - total_hours = total_minutes / 60 - result = 5 - total_hours - - Question: ${ question } - # Python code, return ans - - - def: PROGRAM - model: ${ model } - parameters: - stop: ["\nAnswer: "] - include_stop_sequence: false - - def: ANSWER - lang: python - code: ${ PROGRAM } - - get: ANSWER - - program_of_thought_backtick: - function: - question: str - model: str - return: - - | - Question: Janet's ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers' market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers' market? - # Python code, return ans - ```python - total_eggs = 16 - eaten_eggs = 3 - baked_eggs = 4 - sold_eggs = total_eggs - eaten_eggs - baked_eggs - dollars_per_egg = 2 - result = sold_eggs * dollars_per_egg - ``` - - Question: A robe takes 2 bolts of blue fiber and half that much white fiber. How many bolts in total does it take? - # Python code, return ans - ```python - bolts_of_blue_fiber = 2 - bolts_of_white_fiber = num_of_blue_fiber / 2 - result = bolts_of_blue_fiber + bolts_of_white_fiber - ``` - - Question: Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make? - # Python code, return ans - ```python - cost_of_original_house = 80000 - increase_rate = 150 / 100 - value_of_house = (1 + increase_rate) * cost_of_original_house - cost_of_repair = 50000 - result = value_of_house - cost_of_repair - cost_of_original_house - ``` - - Question: Every day, Wendi feeds each of her chickens three cups of mixed chicken feed, containing seeds, mealworms and vegetables to help keep them healthy. She gives the chickens their feed in three separate meals. In the morning, she gives her flock of chickens 15 cups of feed. In the afternoon, she gives her chickens another 25 cups of feed. How many cups of feed does she need to give her chickens in the final meal of the day if the size of Wendi's flock is 20 chickens? - # Python code, return ans - ```python - numb_of_chickens = 20 - cups_for_each_chicken = 3 - cups_for_all_chicken = num_of_chickens * cups_for_each_chicken - cups_in_the_morning = 15 - cups_in_the_afternoon = 25 - result = cups_for_all_chicken - cups_in_the_morning - cups_in_the_afternoon - ``` - - Question: Kylar went to the store to buy glasses for his new apartment. One glass costs $5, but every second glass costs only 60% of the price. Kylar wants to buy 16 glasses. How much does he need to pay for them? - # Python code, return ans - ```python - num_glasses = 16 - first_glass_cost = 5 - second_glass_cost = 5 * 0.6 - result = 0 - for i in range(num_glasses): - if i % 2 == 0: - result += first_glass_cost - else: - result += second_glass_cost - ``` - - Question: Marissa is hiking a 12-mile trail. She took 1 hour to walk the first 4 miles, then another hour to walk the next two miles. If she wants her average speed to be 4 miles per hour, what speed (in miles per hour) does she need to walk the remaining distance? - # Python code, return ans - ```python - average_mile_per_hour = 4 - total_trail_miles = 12 - remaining_miles = total_trail_miles - 4 - 2 - total_hours = total_trail_miles / average_mile_per_hour - remaining_hours = total_hours - 2 - result = remaining_miles / remaining_hours - ``` - - Question: Carlos is planting a lemon tree. The tree will cost $90 to plant. Each year it will grow 7 lemons, which he can sell for $1.5 each. It costs $3 a year to water and feed the tree. How many years will it tak - e before he starts earning money on the lemon tree? - # Python code, return ans - ```python - total_cost = 90 - cost_of_watering_and_feeding = 3 - cost_of_each_lemon = 1.5 - num_of_lemon_per_year = 7 - result = 0 - while total_cost > 0: - total_cost += cost_of_watering_and_feeding - total_cost -= num_of_lemon_per_year * cost_of_each_lemon - result += 1 - ``` - - Question: When Freda cooks canned tomatoes into sauce, they lose half their volume. Each 16 ounce can of tomatoes that she uses contains three tomatoes. Freda's last batch of tomato sauce made 32 ounces of sauce. How many tomatoes did Freda use? - # Python code, return ans - ```python - lose_rate = 0.5 - num_tomato_contained_in_per_ounce_sauce = 3 / 16 - ounce_sauce_in_last_batch = 32 - num_tomato_in_last_batch = ounce_sauce_in_last_batch * num_tomato_contained_in_per_ounce_sauce - result = num_tomato_in_last_batch / (1 - lose_rate) - ``` - - Question: Jordan wanted to surprise her mom with a homemade birthday cake. From reading the instructions, she knew it would take 20 minutes to make the cake batter and 30 minutes to bake the cake. The cake would require 2 hours to cool and an additional 10 minutes to frost the cake. If she plans to make the cake all on the same day, what is the latest time of day that Jordan can start making the cake to be ready to serve it at 5:00 pm? - # Python code, return ans - ```python - minutes_to_make_batter = 20 - minutes_to_bake_cake = 30 - minutes_to_cool_cake = 2 * 60 - minutes_to_frost_cake = 10 - total_minutes = minutes_to_make_batter + minutes_to_bake_cake + minutes_to_cool_cake + minutes_to_frost_cake - total_hours = total_minutes / 60 - result = 5 - total_hours - ``` - - Question: ${ question } - # Python code, return ans - - def: PROGRAM - model: ${ model } - parser: - regex: '```.*\n((?:.|\n|$)*?)$\n\s*```' # extracts code from backtick blocks - mode: findall - parameters: - stop: ["\nAnswer: "] - include_stop_sequence: false - - def: ANSWER - lang: python - code: ${ PROGRAM|join('\n') } - - get: ANSWER \ No newline at end of file diff --git a/contrib/prompt_library/RAG.pdl b/contrib/prompt_library/RAG.pdl deleted file mode 100644 index 6c0a64355..000000000 --- a/contrib/prompt_library/RAG.pdl +++ /dev/null @@ -1,26 +0,0 @@ -description: Retrieval-Augmented Generation (RAG) following Lewis et al. -defs: - # Corpus: Store the retrieval object in the PDL session - corpus: - function: - corpus: {list: str} - return: - - lang: python - code: | - from rank_bm25 import BM25Okapi - PDL_SESSION.corpus = corpus - PDL_SESSION.tokenized_corpus = [doc.split(" ") for doc in corpus] - PDL_SESSION.bm25_corpus = BM25Okapi(PDL_SESSION.tokenized_corpus) - result = None - # Retrieve from corpus in PDL session - retrieve: - function: - query: str - num_examples: int - spec: {list: str} - return: - - lang: python - code: | - from rank_bm25 import BM25Okapi - tokenized_query = query.split(" ") - result = PDL_SESSION.bm25_corpus.get_top_n(tokenized_query, PDL_SESSION.corpus, n=num_examples) \ No newline at end of file diff --git a/contrib/prompt_library/ReAct.pdl b/contrib/prompt_library/ReAct.pdl index b1bd23fb5..a2ae932b0 100644 --- a/contrib/prompt_library/ReAct.pdl +++ b/contrib/prompt_library/ReAct.pdl @@ -107,6 +107,7 @@ defs: model: str tools: { list: obj } trajectories: { list: list } + system_prompt: str return: lastOf: - if: ${ system_prompt == "llama3"} diff --git a/contrib/prompt_library/examples/evalplus/general.pdl b/contrib/prompt_library/examples/evalplus/general.pdl index f7c9acd51..9eb7cc447 100644 --- a/contrib/prompt_library/examples/evalplus/general.pdl +++ b/contrib/prompt_library/examples/evalplus/general.pdl @@ -89,7 +89,7 @@ text: Assistant: - defs: - temperature: 0.0 + temperature: 0 SOLUTION: "" iterations: 0 max_iterations: 5 diff --git a/contrib/prompt_library/examples/fever/general.pdl b/contrib/prompt_library/examples/fever/general.pdl index a29245577..90b9dbe1c 100644 --- a/contrib/prompt_library/examples/fever/general.pdl +++ b/contrib/prompt_library/examples/fever/general.pdl @@ -68,6 +68,7 @@ text: model: ${ model } tools: ${ search_tools } trajectories: ${ demonstrations } + system_prompt: ${ system_prompt } # ReWOO - if: ${ prompt_pattern == 'rewoo' } diff --git a/contrib/prompt_library/examples/gsm8k/example.pdl b/contrib/prompt_library/examples/gsm8k/example.pdl new file mode 100644 index 000000000..06217e31f --- /dev/null +++ b/contrib/prompt_library/examples/gsm8k/example.pdl @@ -0,0 +1,79 @@ +description: Demo of template +defs: + cot: + import: ../../CoT + react: + import: ../../ReAct + rewoo: + import: ../../ReWoo + tools: + import: ../../tools + model: watsonx_text/ibm/granite-3-8b-instruct + prompt_pattern: cot + system_prompt: granite_llama + question: "Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day?" + demonstrations: + data: + - question: |- + Noah charges $60 for a large painting and $30 for a small painting. + Last month he sold eight large paintings and four small paintings. + If he sold twice as much this month, how much is his sales for this month? + reasoning: |- + He sold 8 large paintings and 4 small paintings last month. + He sold twice as many this month. + 8 large paintings x $60 = << 8*60= 480 >> 480 + 4 small paintings x $30 = << 4*30= 120 >> 120 + So he sold << 480+120= 600 >> 600 paintings last month. + Therefore he sold << 600*2= 1200 >> this month. + answer: $1200 + - question: |- + Noah charges $30 for a large vases and $10 for a small vases. + Last month he sold five large vases and three small vases. + If he sold three times as much this month, how much is his sales for this month? + reasoning: |- + He sold 5 large vases and 3 small vases last month. + He sold three times as many this month. + 5 large vases x $30 = << 5*30= 150 >> 150 + 3 small vases x $10 = << 3*10= 30 >> 30 + So he sold << 150+30= 180 >> 180 vases last month. + Therefore he sold << 180*3= 540 >> this month. + answer: $540 +text: + # CoT + - if: "${ prompt_pattern == 'cot' }" + then: + text: + - "Answer the questions to the best of your abilities.\n\n" + - call: ${ cot.chain_of_thought } + args: + examples: "${ demonstrations }" + question: "${ question|trim }" + model: "${ model }" + + # ReAct + - if: "${ prompt_pattern == 'react' }" + then: + text: + - call: ${ react.react } + def: ANSWER + args: + task: "Question: ${ question|trim }" + model: "${ model }" + tools: ${ tools.tools } + trajectories: ${ demonstrations } + system_prompt: ${ system_prompt } + - "\nThe answer is ${ ANSWER.answer|trim }" + + # ReWOO + - if: "${ prompt_pattern == 'rewoo' }" + then: + text: + - call: ${ rewoo.rewoo } + def: ANSWER + args: + task: "Task: ${ question }" + model: "${ model }" + tools: ${ tools.tools } + trajectories: ${ demonstrations } + show_plans: false + - "\nThe answer is ${ ANSWER.answer|trim }" diff --git a/contrib/prompt_library/examples/gsm8k/general.pdl b/contrib/prompt_library/examples/gsm8k/general.pdl index babcc9df5..f58d53cc7 100644 --- a/contrib/prompt_library/examples/gsm8k/general.pdl +++ b/contrib/prompt_library/examples/gsm8k/general.pdl @@ -8,36 +8,36 @@ defs: import: ../../ReWoo tools: import: ../../tools - model: watsonx/ibm/granite-3-8b-instruct - prompt_pattern: cot - system_prompt: granite_llama - question: "Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day?" - demonstrations: - data: - - question: |- - Noah charges $60 for a large painting and $30 for a small painting. - Last month he sold eight large paintings and four small paintings. - If he sold twice as much this month, how much is his sales for this month? - reasoning: |- - He sold 8 large paintings and 4 small paintings last month. - He sold twice as many this month. - 8 large paintings x $60 = << 8*60= 480 >> 480 - 4 small paintings x $30 = << 4*30= 120 >> 120 - So he sold << 480+120= 600 >> 600 paintings last month. - Therefore he sold << 600*2= 1200 >> this month. - answer: $1200 - - question: |- - Noah charges $30 for a large vases and $10 for a small vases. - Last month he sold five large vases and three small vases. - If he sold three times as much this month, how much is his sales for this month? - reasoning: |- - He sold 5 large vases and 3 small vases last month. - He sold three times as many this month. - 5 large vases x $30 = << 5*30= 150 >> 150 - 3 small vases x $10 = << 3*10= 30 >> 30 - So he sold << 150+30= 180 >> 180 vases last month. - Therefore he sold << 180*3= 540 >> this month. - answer: $540 + # model: watsonx/ibm/granite-3-8b-instruct + # prompt_pattern: cot + # system_prompt: granite_llama + # question: "Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day?" + # demonstrations: + # data: + # - question: |- + # Noah charges $60 for a large painting and $30 for a small painting. + # Last month he sold eight large paintings and four small paintings. + # If he sold twice as much this month, how much is his sales for this month? + # reasoning: |- + # He sold 8 large paintings and 4 small paintings last month. + # He sold twice as many this month. + # 8 large paintings x $60 = << 8*60= 480 >> 480 + # 4 small paintings x $30 = << 4*30= 120 >> 120 + # So he sold << 480+120= 600 >> 600 paintings last month. + # Therefore he sold << 600*2= 1200 >> this month. + # answer: $1200 + # - question: |- + # Noah charges $30 for a large vases and $10 for a small vases. + # Last month he sold five large vases and three small vases. + # If he sold three times as much this month, how much is his sales for this month? + # reasoning: |- + # He sold 5 large vases and 3 small vases last month. + # He sold three times as many this month. + # 5 large vases x $30 = << 5*30= 150 >> 150 + # 3 small vases x $10 = << 3*10= 30 >> 30 + # So he sold << 150+30= 180 >> 180 vases last month. + # Therefore he sold << 180*3= 540 >> this month. + # answer: $540 text: # CoT - if: "${ prompt_pattern == 'cot' }" @@ -61,6 +61,7 @@ text: model: "${ model }" tools: ${ tools.tools } trajectories: ${ demonstrations } + system_prompt: ${ system_prompt } - "\nThe answer is ${ ANSWER.answer|trim }" # ReWOO diff --git a/docs/tutorial.md b/docs/tutorial.md index 2f8c2984b..4c7921b19 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -1,12 +1,10 @@ - # PDL Language Tutorial The following sections give a step-by-step overview of PDL language features. All the examples in this tutorial can be found in `examples/tutorial`. - -## Simple text +## Simple text The simplest PDL program is one that generates a small text ([file](https://github.com/IBM/prompt-declaration-language/blob/main/examples/tutorial/simple_program.pdl)): @@ -28,9 +26,7 @@ This results in the following output: Hello, world! ``` - - -## Calling an LLM +## Calling an LLM ```yaml --8<-- "./examples/tutorial/calling_llm.pdl" @@ -54,6 +50,7 @@ Hello where the second `Hello` has been generated by Granite. Here's another example of model call that includes an `input` field ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/calling_llm_with_input.pdl)): + ```yaml --8<-- "./examples/tutorial/calling_llm_with_input.pdl" ``` @@ -64,10 +61,11 @@ In this case, the input passed to the model is the sentence: `Translate the word Hello Bonjour (pronounced bon-zhoor) is the translation for "Hello" in French. It's an informal greeting used during the day, similar to how we use "Hi" or "Hello." For a more formal context, you might say "Bonjour," which means "Good day." ``` -where the second line is generated by the model. +where the second line is generated by the model. Using the `input` field, we can also give a directly an array of messages (`role`/`content`) to the model ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/calling_llm_with_input_messages.pdl)): + ```yaml --8<-- "./examples/tutorial/calling_llm_with_input_messages.pdl" ``` @@ -78,22 +76,20 @@ This has the same output as the previous program. An alternative way of writing When using Granite models, we use the following defaults for model parameters: - - `temperature`: 0 - - `max_new_tokens`: 1024 - - `min_new_tokens`: 1 - - `repetition_penalty`: 1.05 +- `temperature`: 0 +- `max_new_tokens`: 1024 +- `min_new_tokens`: 1 +- `repetition_penalty`: 1.05 Also if the `decoding_method` is `sample` (`watsonx_text text completion endpoint), then the following defaults are used: - - `temperature`: 0.7 - - `top_p`: 0.85 - - `top_k`: 50 +- `temperature`: 0.7 +- `top_p`: 0.85 +- `top_k`: 50 The user can override these defaults by explicitly including them in the model call. - - -## Variable Definition and Use +## Variable Definition and Use Any block can define a variable using a `def: ` field. This means that the output of that block is assigned to the variable ``, which may be reused at a later point in the document. @@ -107,13 +103,14 @@ Here we assign the output of the model to variable `GEN` using the `def` field. are also used to specify conditions for loops and conditionals. See for example this [file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/programs/chatbot.pdl). When we execute this program, we obtain: + ``` Hello Hello GEN is equal to: Hello ``` -## Model Chaining +## Model Chaining In PDL, we can declaratively chain models together as in the following example ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/calling_llm_chaining.pdl)): @@ -132,14 +129,11 @@ Did you just say Hello? Yes, I did. That's how I greet people in this conversation. It's a common way to start a dialogue. How can I assist you today? ``` - - ## Function Definition PDL also supports function definitions to make it easier to reuse code. Suppose we want to define a translation function that takes a string and calls a Granite model for the translation. This would be written in PDL as follows ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/function_definition.pdl)): - ```yaml --8<-- "./examples/tutorial/function_definition.pdl" ``` @@ -165,7 +159,7 @@ The context inherited by a function can be reset at the call site (see [example] Functions can be declared with optional parameters (see [example](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/function_optional_params.pdl)). -## Grouping Variable Definitions in Defs +## Grouping Variable Definitions in Defs In PDL, the above program can be written more neatly by grouping certain variable definitions into a `defs` section, as follows ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/defs.pdl)): @@ -173,7 +167,6 @@ In PDL, the above program can be written more neatly by grouping certain variabl --8<-- "./examples/tutorial/defs.pdl" ``` - This program has the same output has the one from the previous section. Any block can have a `defs` field defining variables used in that block. Notice it's different than the `def` field which stores the @@ -181,7 +174,7 @@ result of the block after execution. For another example, see [file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/defs-hello.pdl). -## Muting Block Output with contribute +## Muting Block Output with contribute By default, when a PDL block is executed it produces a result that is contributed to the overall result, and it also contributes to the background context. It is possible to mute both contributions by setting `contribute` to `[]` for any block. This feature allows the computation of intermediate values that are not necessarily output as a result. The value of the variable specified in `def` is still set to the result of the block. @@ -191,7 +184,6 @@ Consider the similar example as above, but with `contribute` set to `[]` ([file] --8<-- "./examples/tutorial/muting_block_output.pdl" ``` - The call to the translator with French as language does not produce an output. However, we save the result in variable `FRENCH` and use it in the last sentence of the document. When we execute this program, we obtain: ``` @@ -200,6 +192,7 @@ The french sentence was: 'J'aime Paris !' In general, `contribute` can be used to set how the result of the block contribute to the final result and the background context. Here are its possible values: + - `[]`: no contribution to either the final result or the background context - `[result]`: contribute to the final result but not the background context @@ -223,6 +216,7 @@ array: ``` This results in the following output: + ``` ["apple", "orange", "banana"] ``` @@ -238,6 +232,7 @@ object: ``` This results in the following output: + ``` {"name": "Bob", "job": "manager"} ``` @@ -254,17 +249,17 @@ lastOf: ``` This results in the following output: + ``` 3 ``` Each list item can contain any PDL block (strings are shown here), and the result of the whole list is that of the last block. -Notice that block types that require lists (`repeat`, `for`, `if-then-else`) need to specify the shape of data in their bodies, for +Notice that block types that require lists (`repeat`, `for`, `if-then-else`) need to specify the shape of data in their bodies, for example `text` or `array`. It is a syntax error to omit it. For more detailed discussion on this see [this section](#conditionals-and-loops). - -## Input from File or Stdin +## Input from File or Stdin PDL can accept textual input from a file or stdin. In the following example ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/input_file.pdl)), the contents of this [file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/data.txt) are read by PDL and incorporated in the document. The result is also assigned to a variable `HELLO`. @@ -272,14 +267,12 @@ PDL can accept textual input from a file or stdin. In the following example ([fi --8<-- "./examples/tutorial/input_file.pdl" ``` - In the next example, prompts are obtained from stdin ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/input_stdin.pdl)). This is indicated by assigning the value `null` to the `read` field. ```yaml --8<-- "./examples/tutorial/input_stdin.pdl" ``` - If the `message` field is omitted then one is provided for you. The following example shows a multiline stdin input ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/input_stdin_multiline.pdl)). When executing this code and to exit from the multiline input simply press control D (on macOS). @@ -288,7 +281,6 @@ The following example shows a multiline stdin input ([file](https://github.com/I --8<-- "./examples/tutorial/input_stdin_multiline.pdl" ``` - Finally, the following example shows reading content in JSON format. Consider the JSON content in this [file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/input.json): @@ -304,47 +296,50 @@ The following PDL program reads this content and assigns it to variable `PERSON` ``` When we execute this program, we obtain: + ``` Bob lives at the following address: 87 Smith Road in the town of Armonk, NY ``` -## Parsing the output of a block +## Parsing the output of a block As we saw in the previous section, it is possible to use the `parser: json` setting to parse the result of a block as a JSON. Other possible values for `parser` are `yaml`, `jsonl`, or `regex`. The following example extracts using a regular expression parser the code between triple backtick generated by a model: + ```yaml --8<-- "./examples/tutorial/parser_regex_code.pdl" ``` Here is another example using a regular expression: + ```yaml --8<-- "./examples/tutorial/parser_regex.pdl" ``` - We support the following operations with the`regex` parser (indicated with the `mode` field): - `fullmatch` (default) - `search` - + - `match` - + - `split` - + - `findall` Here is an example using the `findall` mode that returns the list `['1', '2', '3', '4']`: + ```yaml --8<-- "./examples/tutorial/parser_findall.pdl" ``` See [here](https://docs.python.org/3/library/re.html) for more information on how to write regular expressions. -## Calling code +## Calling code The following script shows how to execute python code ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/code_python.pdl)). The python code is executed locally (or in a containerized way if using `pdl --sandbox`). In principle, PDL is agnostic of any specific programming language, but we currently only support Python, Jinja, and shell commands. Variables defined in PDL are copied into the global scope of the Python code, so those variables can be used directly in the code. However, mutating variables in Python has no effect on the variables in the PDL program. The result of the code must be assigned to the variable `result` internally to be propagated to the result of the block. A variable `def` on the code block will then be set to this result. @@ -357,13 +352,14 @@ See for example: ([file](https://github.com/IBM/prompt-declaration-language//blo ``` This results in the following output (for example): + ``` Hello, r! ``` -PDL also supports Jinja code blocks, shell commands, as well as PDL code blocks for meta-cycle programming. For more examples, see -([Jinja code](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/code_jinja.pdl)), -([shell command](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/code_command.pdl)), +PDL also supports Jinja code blocks, shell commands, as well as PDL code blocks for meta-cycle programming. For more examples, see +([Jinja code](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/code_jinja.pdl)), +([shell command](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/code_command.pdl)), ([PDL code](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/code_pdl.pdl)). ## Calling REST APIs @@ -376,8 +372,7 @@ PDL programs can contain calls to REST APIs with Python code. Consider a simple In this program, we first define a query about the weather in some location (assigned to variable `QUERY`). The next block is a call to a Granite model with few-shot examples to extract the location, which we assign to variable `LOCATION`. The next block makes an API call with Python (mocked in this example). Here the `LOCATION` is appended to the `url`. The result is a JSON object, which may be hard to interpret for a human user. So we make a final call to an LLM to interpret the JSON in terms of weather. Notice that many blocks have `contribute` set to `[]` to hide intermediate results. - -## Data Block +## Data Block PDL offers the ability to create JSON data as illustrated by the following example (described in detail in the [Overview](https://ibm.github.io/prompt-declaration-language/#overview) section). The `data` block can gather previously defined variables into a JSON structure. This feature is useful for data generation. Programs such as this one can be generalized to read jsonl files to generate data en masse by piping into another jsonl file ([file](https://github.com/IBM/prompt-declaration-language/blob/main/examples/tutorial/programs/code-json.pdl)). @@ -395,7 +390,6 @@ The `raw` field can be used to turn off the interpreter inside a `data` block. F --8<-- "./examples/tutorial/data_block_raw.pdl" ``` - The result of this program is the JSON object: ``` @@ -418,14 +412,57 @@ following [example](https://github.com/IBM/prompt-declaration-language//blob/mai which imports the following [file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/import_lib.pdl): - The `import` block means that the PDL code at that file is executed and its scope is assigned to the variable defined in that block. So all the `defs` in the imported file are made available via that variable. This feature allows reuse of common templates and patterns and to build libraries. Notice that relative paths are relative to the containing file. +## Prompt Library -## Conditionals and Loops +A prompt library is included in the `contrib/` directory. These modules define some common patterns such Chain-of-Thought, ReAct, and ReWOO. -PDL supports conditionals and loops as illustrated in the following example ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/programs/chatbot.pdl)), which implements a chatbot. +In the `GSM8K` example below, we import `CoT` and define our model, demonstrations (a list of dictionaries), and the question. + +``` yaml +--8<-- "./examples/prompt_library/gsm8k_cot.pdl" +``` + +Executing this example results in the following output (model response highlighted): + +``` text linenums="1" hl_lines="27-31" +Answer the questions to the best of your abilities. +Question: Noah charges $60 for a large painting and $30 for a small painting. +Last month he sold eight large paintings and four small paintings. +If he sold twice as much this month, how much is his sales for this month? +Answer: Let's think step by step. He sold 8 large paintings and 4 small paintings last month. +He sold twice as many this month. +8 large paintings x $60 = << 8*60= 480 >> 480 +4 small paintings x $30 = << 4*30= 120 >> 120 +So he sold << 480+120= 600 >> 600 paintings last month. +Therefore he sold << 600*2= 1200 >> this month. +The answer is $1200 + +Question: Noah charges $30 for a large vases and $10 for a small vases. +Last month he sold five large vases and three small vases. +If he sold three times as much this month, how much is his sales for this month? +Answer: Let's think step by step. He sold 5 large vases and 3 small vases last month. +He sold three times as many this month. +5 large vases x $30 = << 5*30= 150 >> 150 +3 small vases x $10 = << 3*10= 30 >> 30 +So he sold << 150+30= 180 >> 180 vases last month. +Therefore he sold << 180*3= 540 >> this month. +The answer is $540 + +Question: Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day? +Answer: Let's think step by step. +1. Jacob earns $6 per hour. +2. Jake earns thrice what Jacob does, so Jake earns 3 * $6 = $18 per hour. +3. Jake works 8 hours a day, so he earns $18 * 8 = $144 per day. +4. Jake works 5 days, so he earns $144 * 5 = $720 in 5 days. +The answer is $720. +``` + +## Conditionals and Loops + +PDL supports conditionals and loops as illustrated in the following example ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/programs/chatbot.pdl)), which implements a chatbot. ```yaml --8<-- "./examples/tutorial/programs/chatbot.pdl" @@ -437,7 +474,7 @@ true. The field `repeat` can contain a string, or a block, or a list. If it cont `array` or `lastOf` (which means that all the blocks in the list are executed and the result of the body is that of the last block). The example also shows the use of an `if-then-else` block. The `if` field contains a condition, the `then` field -can also contain either a string, or a block, or a list (and similarly for `else`). +can also contain either a string, or a block, or a list (and similarly for `else`). The chatbot keeps looping by making a call to a model, asking the user if the generated text is a good answer, and asking `why not?` if the answer (stored in variable `eval`) is `no`. The loop ends when `eval` becomes `yes`. This is specified with a Jinja expression on line 18. @@ -459,11 +496,13 @@ The following [example](https://github.com/IBM/prompt-declaration-language//blob ``` This program outputs: + ``` 1234 ``` To output a number of each line, we can specify which string to use to join the results. + ```yaml --8<-- "./examples/tutorial/for_with.pdl" ``` @@ -475,28 +514,32 @@ To output a number of each line, we can specify which string to use to join the 4 ``` - To create an array as a result of iteration, we would write: + ```yaml --8<-- "./examples/tutorial/for_array.pdl" ``` which outputs the following list: + ``` [1, 2, 3, 4] ``` To retain only the result of the last iteration of the loop, we would write: + ```yaml --8<-- "./examples/tutorial/for_lastOf.pdl" ``` which outputs: + ``` 4 ``` When `join` is not specified, the collation defaults to + ``` join: as: text @@ -508,7 +551,6 @@ meaning that result of each iteration is stringified and concatenated with that Note that `join` can be added to any looping construct (`repeat`) not just `for` loops. - The `for` loop construct also allows iterating over 2 or more lists of the same length simultaneously: ```yaml @@ -525,6 +567,7 @@ Ernest's number is 4 ``` The loop constructs also allow to build an object: + ```yaml --8<-- "./examples/tutorial/for_object.pdl" ``` @@ -538,6 +581,7 @@ This results in the following output: ### While Loop The following example shows a while loop in PDL: + ```yaml --8<-- "./examples/tutorial/while.pdl" ``` @@ -546,13 +590,14 @@ The `while` field indicates the looping condition and `repeat` contains the body Notice that `for`, `while`, `until`, and `maxIterations` can all be combined in the same `repeat` block. The loop exits as soon as one of the exit conditions is satisfied: + ```yaml --8<-- "./examples/tutorial/repeat.pdl" ``` ### Match block -PDL provides a match block for convenience. +PDL provides a match block for convenience. Consider the [example](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/programs/demo-hallucination.pdl). This shows retrieved RAG documents that are then submitted with a query to a RAG Granite model. The output contains an answer to the query together with hallucination @@ -562,6 +607,7 @@ To obtain and install the Granite model locally follow these [instructions](https://github.com/IBM/prompt-declaration-language//blob/main/examples/intrinsics/README.md). The end of this program contains a match block: + ```yaml ... The answer is: ${ out[0].sentence } @@ -579,13 +625,11 @@ The end of this program contains a match block: The `match` field indicates an expression to match on. The cases follow the `with` field. Additional conditions can be indicated as shown in the second case. - ## Roles and Chat Templates Consider again the chatbot example ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/programs/chatbot.pdl)). By default blocks have role `user`, except for model call blocks, which have role `assistant`. If we write roles explicitly for the chatbot, we obtain: - ```yaml description: chatbot text: @@ -617,6 +661,7 @@ need to be specified explicitly). PDL takes care of applying appropriate chat templates (done either in LiteLLM or at the server side). The prompt that is actually submitted to the first model call (with query `What is APR?`) is the following: + ``` <|start_of_role|>user<|end_of_role|>What is APR?<|end_of_text|> <|start_of_role|>assistant<|end_of_role|> @@ -657,6 +702,7 @@ Similarly, the output of the model call is parsed as YAML, and the `spec` indica 2 fields: `name` of type string, and `age` of type integer. When we run this program, we obtain the output: + ``` {'name': 'John', 'age': '30'} type_checking.pdl:9 - Type errors during spec checking: @@ -700,20 +746,21 @@ The following [program](https://github.com/IBM/prompt-declaration-language//blob ``` produces the output: + ``` What is the color of the sky? {'color': 'blue'} ``` -## Python SDK +## Python SDK See examples of PDL being called programmatically in Python [here](https://github.com/IBM/prompt-declaration-language/blob/main/examples/sdk). For a more sophisticated example, see [here](https://github.com/IBM/prompt-declaration-language/blob/main/examples/callback). -## Debugging PDL Programs +## Debugging PDL Programs We highly recommend to edit PDL programs using an editor that support YAML with JSON Schema validation. For example, you can use VSCode with the [YAML extension](https://marketplace.visualstudio.com/items?itemName=redhat.vscode-yaml) and configure it to use the [PDL schema](https://github.com/IBM/prompt-declaration-language/blob/main/src/pdl/pdl-schema.json). The PDL repository has been configured so that every `*.pdl` file is associated with the PDL grammar JSONSchema (see [settings](https://github.com/IBM/prompt-declaration-language/blob/main/.vscode/settings.json)). This enables the editor to display error messages when the yaml deviates from the PDL syntax and grammar. It also provides code completion. The PDL interpreter also provides similar error messages. To make sure that the schema is associated with your PDL files, be sure that `PDL Schemas` appear at the bottom right of your VSCode window, or on top of the editor window. @@ -725,8 +772,9 @@ may also kick the tires with a web version of the GUI [here](https://pdl.s3-web.us-east.cloud-object-storage.appdomain.cloud/). To generate a trace for use in the GUI: + ```bash -pdl --trace +pdl --trace ``` This is similar to a spreadsheet for tabular data, where data is in the forefront and the user can inspect the formula that generates the data in each cell. In the Live Document, cells are not uniform but can take arbitrary extents. Clicking on them similarly reveals the part of the code that produced them. @@ -736,16 +784,12 @@ be used for debugging or performance analysis, and to see the shape of prompts s For more information see [here](https://github.com/IBM/prompt-declaration-language/blob/main/docs/telemetry.md). - - - ## Using Ollama models 1. Install Ollama e.g., `brew install --cask ollama` 2. Run a model e.g., `ollama run granite-code:8b`. See [the Ollama library for more models](https://ollama.com/library/granite-code/tags) 3. An OpenAI style server is running locally at [http://localhost:11434/](http://localhost:11434/), see [the Ollama blog](https://ollama.com/blog/openai-compatibility) for more details. - Example: ``` @@ -776,9 +820,10 @@ Multiline strings are commonly used when writing PDL programs. There are two typ ### Block Scalars -**Block Style Indicator**: The block style indicates how newlines inside the block should behave. If you would like them to be **kept** as newlines, use the literal style, indicated by a pipe `|`. Note that without a chomping indicator, described next, only the _last_ newline is kept. +**Block Style Indicator**: The block style indicates how newlines inside the block should behave. If you would like them to be **kept** as newlines, use the literal style, indicated by a pipe `|`. Note that without a chomping indicator, described next, only the *last* newline is kept. PDL: + ``` text: - | @@ -796,6 +841,7 @@ text: ``` Output: + ``` Several lines of text, with some "quotes" of various 'types', @@ -808,9 +854,10 @@ plus another line at the end. End. ``` -If instead you want them to be replaced by spaces, use the folded style, indicated by a right angle bracket `>`. To get a newline using the folded style, leave a blank line by putting _two_ newlines in. Lines with extra indentation are also not folded. +If instead you want them to be replaced by spaces, use the folded style, indicated by a right angle bracket `>`. To get a newline using the folded style, leave a blank line by putting *two* newlines in. Lines with extra indentation are also not folded. PDL: + ``` text: - > @@ -828,6 +875,7 @@ text: ``` Output: + ``` Several lines of text, with some "quotes" of various 'types', and also a blank line: and some text with @@ -839,6 +887,7 @@ End. **Block Chomping Indicator**: The chomping indicator controls what should happen with newlines at the end of the string. The default, clip, puts a single newline at the end of the string. To remove all newlines, strip them by putting a minus sign `-` after the style indicator. Both clip and strip ignore how many newlines are actually at the end of the block; to keep them all put a plus sign `+` after the style indicator. PDL: + ``` text: - |- @@ -856,6 +905,7 @@ text: ``` Output: + ``` Several lines of text, with some "quotes" of various 'types', @@ -868,6 +918,7 @@ plus another line at the end.End. ``` PDL: + ``` text: - |+ @@ -885,6 +936,7 @@ text: ``` Output: + ``` Several lines of text, with some "quotes" of various 'types', @@ -900,6 +952,7 @@ End. ``` If you don't have enough newline characters using the above methods, you can always add more like so: + ``` text: - |- @@ -918,6 +971,7 @@ text: ``` Output: + ``` Several lines of text, with some "quotes" of various 'types', @@ -936,6 +990,7 @@ End. **Indentation Indicator**: Ordinarily, the number of spaces you're using to indent a block will be automatically guessed from its first line. You may need a block indentation indicator if the first line of the block starts with extra spaces. In this case, simply put the number of spaces used for indentation (between 1 and 9) at the end of the header. PDL: + ``` text: - |1 @@ -949,6 +1004,7 @@ text: ``` Output: + ``` Several lines of text, with some "quotes" of various 'types', diff --git a/examples/prompt_library/gsm8k_cot.pdl b/examples/prompt_library/gsm8k_cot.pdl new file mode 100644 index 000000000..f8c3fc69c --- /dev/null +++ b/examples/prompt_library/gsm8k_cot.pdl @@ -0,0 +1,41 @@ +description: Demo of template +defs: + cot: + import: ../../contrib/prompt_library/CoT + model: watsonx_text/ibm/granite-3-8b-instruct + demonstrations: + data: + - question: |- + Noah charges $60 for a large painting and $30 for a small painting. + Last month he sold eight large paintings and four small paintings. + If he sold twice as much this month, how much is his sales for this month? + reasoning: |- + He sold 8 large paintings and 4 small paintings last month. + He sold twice as many this month. + 8 large paintings x $60 = << 8*60= 480 >> 480 + 4 small paintings x $30 = << 4*30= 120 >> 120 + So he sold << 480+120= 600 >> 600 paintings last month. + Therefore he sold << 600*2= 1200 >> this month. + answer: $1200 + - question: |- + Noah charges $30 for a large vases and $10 for a small vases. + Last month he sold five large vases and three small vases. + If he sold three times as much this month, how much is his sales for this month? + reasoning: |- + He sold 5 large vases and 3 small vases last month. + He sold three times as many this month. + 5 large vases x $30 = << 5*30= 150 >> 150 + 3 small vases x $10 = << 3*10= 30 >> 30 + So he sold << 150+30= 180 >> 180 vases last month. + Therefore he sold << 180*3= 540 >> this month. + answer: $540 + question: "Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day?" + +text: + # CoT + - "Answer the questions to the best of your abilities.\n\n" + - call: ${ cot.chain_of_thought } + args: + examples: "${ demonstrations }" + question: "${ question }" + model: "${ model }" \ No newline at end of file From ae96a00d199e460d862704d28e9aeddefd2dad35 Mon Sep 17 00:00:00 2001 From: Claudio Spiess Date: Mon, 21 Apr 2025 10:54:46 -0700 Subject: [PATCH 3/4] Finish prompt lib docs Signed-off-by: Claudio Spiess --- contrib/prompt_library/CoT.pdl | 129 +++------- contrib/prompt_library/ReAct.pdl | 148 ++--------- contrib/prompt_library/ReWoo.pdl | 203 ++++++++------- .../examples/evalplus/general.pdl | 144 ----------- .../prompt_library/examples/fever/general.pdl | 85 ------- .../prompt_library/examples/gsm8k/example.pdl | 79 ------ .../prompt_library/examples/gsm8k/general.pdl | 79 ------ contrib/prompt_library/tools.pdl | 109 ++++---- docs/tutorial.md | 234 +++++++++++++++++- examples/prompt_library/gsm8k_cot.pdl | 7 +- examples/prompt_library/gsm8k_react.pdl | 41 +++ examples/prompt_library/gsm8k_rewoo.pdl | 34 +++ 12 files changed, 506 insertions(+), 786 deletions(-) delete mode 100644 contrib/prompt_library/examples/evalplus/general.pdl delete mode 100644 contrib/prompt_library/examples/fever/general.pdl delete mode 100644 contrib/prompt_library/examples/gsm8k/example.pdl delete mode 100644 contrib/prompt_library/examples/gsm8k/general.pdl create mode 100644 examples/prompt_library/gsm8k_react.pdl create mode 100644 examples/prompt_library/gsm8k_rewoo.pdl diff --git a/contrib/prompt_library/CoT.pdl b/contrib/prompt_library/CoT.pdl index a325d0286..3d52a4972 100644 --- a/contrib/prompt_library/CoT.pdl +++ b/contrib/prompt_library/CoT.pdl @@ -11,47 +11,24 @@ defs: Answer: Let's think step by step. ${ reasoning } The answer is ${ answer } - # Auto Chain of Thought Zhang et al. (2022) - # The idea is to use a _model_ to generate a reasoning path, even if not very accurate. - # It is best combined with some fewshot examples - auto_chain_of_thought: - function: - question: str - model: str - answer: str - return: - text: - - |- - Question: ${ question } - - "Answer: Let's think step by step. " - - model: ${ model } - parameters: - # decoding_method: "greedy" - max_tokens: 1024 - temperature: 0 - stop: - - "The answer is" - include_stop_sequence: false - - "The answer is ${ answer }" - fewshot_cot: function: examples: { list: { obj: { question: str, reasoning: str, answer: str } } } return: text: - - for: - example: ${ examples } - repeat: - text: - - call: ${ cot_block } - args: - question: ${ example.question } - reasoning: ${ example.reasoning } - answer: ${ example.answer } - - "\n\n" - join: - with: "" + - for: + example: ${ examples } + repeat: + text: + - call: ${ cot_block } + args: + question: ${ example.question } + reasoning: ${ example.reasoning } + answer: ${ example.answer } + - "\n\n" + join: + with: "" chain_of_thought: function: @@ -60,68 +37,20 @@ defs: examples: { list: { obj: { question: str, reasoning: str, answer: str } } } return: - text: - - call: ${ fewshot_cot } - args: - examples: ${ examples } - - "Question: ${ question }\n" - - "Answer: Let's think step by step. " - - model: ${ model } - parameters: - max_tokens: 1024 - temperature: 0 - stop: - - "<|endoftext|>" - - "Question:" - include_stop_sequence: false - - claim_cot: - function: - question: str - reasoning: str - answer: str - return: |- - Task: On June 2017, the following claim was made: ${ question } - Q: Was this claim true or false? - Thought: Let's think step by step. ${ reasoning|trim } - Answer: The claim is ${ answer } - - fewshot_cot_claim: - function: - examples: - { list: { obj: { question: str, reasoning: str, answer: str } } } - return: - text: - - for: - example: ${ examples } - repeat: - text: - - call: ${ claim_cot } - args: - question: ${ example.question } - reasoning: ${ example.reasoning } - answer: ${ example.answer } - - "\n\n" - join: - with: "" - - chain_of_thought_claim: - function: - question: str - model: str - examples: - { list: { obj: { question: str, reasoning: str, answer: str } } } - return: - text: - - call: ${ fewshot_cot_claim } - args: - examples: ${ examples } - - "${ question }\n" - - "Thought: Let's think step by step. " - - model: ${ model } - parameters: - temperature: 0 - stop: - - "<|endoftext|>" - max_tokens: 1024 - include_stop_sequence: false \ No newline at end of file + lastOf: + - call: ${ fewshot_cot } + args: + examples: ${ examples } + - "Question: ${ question }\n" + - "Answer: Let's think step by step. " + - model: ${ model } + def: answer + parameters: + max_tokens: 1024 + temperature: 0 + stop: + - "<|endoftext|>" + - "Question:" + include_stop_sequence: false + - data: + answer: ${ answer|trim } diff --git a/contrib/prompt_library/ReAct.pdl b/contrib/prompt_library/ReAct.pdl index a2ae932b0..d47136b1a 100644 --- a/contrib/prompt_library/ReAct.pdl +++ b/contrib/prompt_library/ReAct.pdl @@ -32,132 +32,29 @@ defs: then: "${ type }: ${ trajectory[type]|trim }" - "\n" - react_code_block: - function: - trajectory: { list: obj } - return: - text: - - for: - trajectory: ${ trajectory } - repeat: - text: - - def: type - text: ${ trajectory.keys()|first } - contribute: [] - - if: ${ type == 'task'} - then: | - Task: - ${ trajectory[type]|trim } - - if: ${ type == 'thought'} - then: - text: - - "\n" - - | - Assistant: - ${ trajectory[type]|trim } - - if: ${ type == 'action'} - then: | - - ${ trajectory[type]|trim } - - - if: ${ type == 'observation'} - then: - text: - - "\n" - - | - Observation: - ${ trajectory[type]|trim } - - if: ${ type == 'solution'} - then: |- - - ${ trajectory[type]|trim } - - - if: ${ type not in ['question', 'task', 'thought', 'action', 'observation', 'solution'] } - then: "${ type }: ${ trajectory[type]|trim }" - - "\n" - - finish_action: - data: - display_name: Finish - pdl_function: - description: Respond with the Answer - parameters: - - name: answer - type: string - description: The answer - examples: [] - - demonstrate_tools: - function: - tools: { list: obj } - return: - for: - tool: ${ tools } - repeat: - for: - example: ${ tool.examples } - repeat: - call: ${ react_block } - args: - trajectory: ${ example } - react: function: task: str model: str - tools: { list: obj } + tool_schema: { list: obj } + tools: obj trajectories: { list: list } - system_prompt: str return: lastOf: - - if: ${ system_prompt == "llama3"} - then: - text: - - role: system - contribute: [context] - text: - - "Cutting Knowledge Date: December 2023\nToday Date: " - - lang: python - code: | - from datetime import datetime - result = datetime.today().strftime('%d %B %Y\n\n') - - role: system - text: You are a helpful assistant with tool calling capabilities. When you receive a tool call response, use the output to format an answer to the orginal use question. - contribute: [context] - - role: user - text: | - Given the following functions, please respond with a JSON for a function call with its proper arguments that best answers the given prompt. - - Respond in the format {"name": function name, "arguments": dictionary of argument name and its value}. Do not use variables. - - ${ tools } - contribute: [context] - - if: ${ system_prompt == "granite_llama" } - then: - text: - - role: system - text: | - You are a helpful assistant with access to the following function calls. Your task is to produce a sequence of function calls necessary to generate response to the user utterance. Use the following function calls as required. + - role: system + text: + - "Today's Date: " + - lang: python + code: | + from datetime import datetime + result = datetime.today().strftime('%B %d, %Y.\n') + - | + You are a helpful assistant with access to the following function calls. Your task is to produce a sequence of function calls necessary to generate response to the user utterance. Use the following function calls as required. - Respond in the format {"name": function name, "arguments": dictionary of argument name and its value}. Do not use variables. + Respond in the format {"name": function name, "arguments": dictionary of argument name and its value}. Do not use variables. - ${ tools } - contribute: [context] - - if: ${ system_prompt == "granite_tools" } - then: - text: - - role: system - contribute: [context] - text: - - "Knowledge Cutoff Date: April 2024.\nToday's Date: " - - lang: python - code: | - from datetime import datetime - result = datetime.today().strftime('%B %d, %Y.\n') - - | - You are Granite, developed by IBM. You are a helpful AI assistant with access to the following tools. When a tool is required to answer the user's query, respond with a JSON object of the tool to use. If a tool does not exist in the provided list of tools, notify the user that you do not have the ability to fulfill the request. - - ${ tools } + ${ tool_schema } + contribute: [context] - "\n" - for: traj: ${ trajectories } @@ -173,16 +70,21 @@ defs: - def: exit contribute: [] data: False + - def: tool_names + contribute: [] + text: ${ tool_schema|map(attribute='function.name')|list } - repeat: text: - "\nTho: " - def: thought model: "${ model }" + contribute: [] parameters: stop: - "Act:" max_tokens: 256 include_stop_sequence: false + - "${ thought|trim }" - "\nAct: " - def: action model: "${ model }" @@ -195,17 +97,17 @@ defs: - if: ${ action != prev_action } then: def: observation - if: ${ action.name.lower() != 'finish' } + if: ${ action.name.lower() != "finish" } then: text: - "\nObs: " - - if: ${ action.name.lower() in pdl_tools } + - if: ${ action.name in tools } then: - call: ${ pdl_tools[action.name.lower()] } + call: ${ tools[action.name] } args: arguments: ${ action.arguments } - else: "Invalid action. Valid actions are ${ (pdl_tools.keys()|list)[:-1]|join(', ') }, and ${ (pdl_tools.keys()|list)[-1] }." - - "\n" + else: "Invalid action. Valid actions are ${ tool_names[:-1]|join(', ') }, and ${ tool_names[-1] }." + # - "\n" else: def: exit contribute: [] @@ -215,4 +117,4 @@ defs: data: ${ action } until: ${ action.name.lower() == "finish" or exit } - data: - answer: ${ (action.arguments.answer|default("No answer found."))|trim } # Factor 5: answer argument name. some models really like putting answer in. + answer: ${ (action.arguments.answer|default("No answer found."))|trim } diff --git a/contrib/prompt_library/ReWoo.pdl b/contrib/prompt_library/ReWoo.pdl index 67e57df70..5879e8818 100644 --- a/contrib/prompt_library/ReWoo.pdl +++ b/contrib/prompt_library/ReWoo.pdl @@ -8,43 +8,44 @@ defs: trajectory: { list: obj } return: text: - - defs: - i: - data: 1 - - for: - trajectory: ${ trajectory } - repeat: - text: - - defs: - type: - text: ${ trajectory.keys()|first } - content: - text: ${ trajectory.values()|first } - - if: ${ type in ['task', 'question'] } - then: |- - Task: ${ content|trim } - - if: ${ type == 'thought'} - then: |- - - Plan: ${ content|trim } - - if: ${ type == 'action'} - then: - text: - - " #E${ i } = ${ content|trim }" + - defs: + i: + data: 1 + - for: + trajectory: ${ trajectory } + repeat: + text: - defs: - i: - data: ${ i+1 } - - if: ${ type == 'observation'} - then: "" - - if: ${ type not in ['question', 'task', 'thought', 'action', 'observation'] } - then: "${ type }: ${ content|trim }\n" - - "\n" + type: + text: ${ trajectory.keys()|first } + content: + text: ${ trajectory.values()|first } + - if: ${ type in ['task', 'question'] } + then: |- + Task: ${ content|trim } + - if: ${ type == 'thought'} + then: |- + + Plan: ${ content|trim } + - if: ${ type == 'action'} + then: + text: + - " #E${ i } = ${ content|trim }" + - defs: + i: + data: ${ i+1 } + - if: ${ type == 'observation'} + then: "" + - if: ${ type not in ['question', 'task', 'thought', 'action', 'observation'] } + then: "${ type }: ${ content|trim }\n" + - "\n" rewoo: function: task: str model: str - tools: { list: obj } + tool_schema: { list: obj } + tools: obj trajectories: { list: list } show_plans: bool return: @@ -53,15 +54,16 @@ defs: For the following task, make plans that can solve the problem step by step. For each plan, indicate which external tool together with tool input to retrieve evidence. You can store the evidence into a variable #E that can be called by later tools. (Plan, #E1, Plan, #E2, Plan, ...) Tools can be one of the following: - ${ tools } + ${ tool_schema } + - "\n" - for: traj: ${ trajectories } repeat: text: - - call: ${ rewoo_block } - args: - trajectory: ${ traj } - - "\n" + - call: ${ rewoo_block } + args: + trajectory: ${ traj } + - "\n" - | Begin! Describe your plans with rich details. Each Plan should be followed by only one #E. @@ -69,6 +71,7 @@ defs: ${ task } - def: PLANS model: ${ model } + contribute: [] parser: # plan, step_name, tool, tool_input regex: 'Plan:\s*(?P(?:.|\n)*?)\s*(?P#E\d+)\s*=\s*(?P\{.+\})' mode: findall @@ -81,19 +84,22 @@ defs: include_stop_sequence: false max_tokens: 1024 - if: ${ show_plans } + contribute: [result, context] then: text: - - "\n\n--- Raw plans ---\n" - - ${ PLANS } - - "\n\n--- Extracted Blueprint ---\n" - - for: - plan: ${ PLANS } - repeat: - text: - - "Plan: ${ plan[0] }\n" - - "${ plan[1] } = ${ plan[2] }\n" - - "----------------------------\n\n" + - "\n\n--- Raw plans ---\n" + - ${ PLANS } + - "\n\n--- Extracted Blueprint ---\n" + - for: + plan: ${ PLANS } + repeat: + text: + - "Plan: ${ plan[0] }\n" + - "${ plan[1] } = ${ plan[2] }\n" + - "----------------------------\n\n" - defs: + SOLUTION: + text: "No plans found." output: data: {} plans: @@ -101,63 +107,56 @@ defs: plan: ${ PLANS } repeat: lastOf: - - defs: - PLAN: ${ plan[0] } - ID: ${ plan[1] } - ACTION_RAW: ${ plan[2] } - ACTION: - parser: json - lang: python - code: |- - for k,v in output.items(): - if k in ACTION_RAW: - ACTION_RAW = ACTION_RAW.replace(k, v) - result = ACTION_RAW - tool_output: - if: ${ ACTION.name.lower() in pdl_tools } - then: - call: ${ pdl_tools[ACTION.name.lower()] } - args: - arguments: ${ ACTION.arguments } - else: "Invalid action. Valid actions are ${ pdl_tools.keys() }" - - def: output - lang: python - contribute: [] - code: | - output[ID] = str(tool_output) - result = output - - | - Plan: ${ PLAN } - Evidence: ${ tool_output } + - defs: + PLAN: ${ plan[0] } + ID: ${ plan[1] } + ACTION_RAW: ${ plan[2] } + ACTION: + parser: json + lang: python + code: |- + for k,v in output.items(): + if k in ACTION_RAW: + ACTION_RAW = ACTION_RAW.replace(k, v) + result = ACTION_RAW + tool_output: + if: ${ ACTION.name in tools } + then: + call: ${ tools[ACTION.name] } + args: + arguments: ${ ACTION.arguments } + else: "Invalid action. Valid actions are ${ tools.keys() }" + - def: output + lang: python + contribute: [] + code: | + output[ID] = str(tool_output) + result = output + - | + Plan: ${ PLAN } + Evidence: ${ tool_output } - if: ${ plans is not none and plans|length > 0 } then: - lastOf: - - def: solution_input - text: |- - Solve the following task or problem. To solve the problem, we have made step-by-step Plan and retrieved corresponding Evidence to each Plan. Use them with caution since long evidence might contain irrelevant information. + text: + - "\n\n" + - def: solution_input + text: |- + Solve the following task or problem. To solve the problem, we have made step-by-step Plan and retrieved corresponding Evidence to each Plan. Use them with caution since long evidence might contain irrelevant information. - ${ plans|join } - Now solve the question or task according to provided Evidence above. Respond with the answer directly with no extra words. + ${ plans|join } + Now solve the question or task according to provided Evidence above. Respond with the answer directly with no extra words. - ${ task } - Response: - - def: SOLUTION - model: ${ model } - parameters: - timeout: 30 - time_limit: 30000 - temperature: 0 - stop: - - "<|endoftext|>" - include_stop_sequence: false - max_tokens: 1024 - input: - text: ${ solution_input } - - data: - answer: ${ SOLUTION } - else: - lastOf: - - def: SOLUTION - text: "No plans found." - - data: - answer: ${ SOLUTION } + ${ task } + Response: + - def: SOLUTION + model: ${ model } + parameters: + temperature: 0 + stop: + - "<|endoftext|>" + include_stop_sequence: false + max_tokens: 1024 + input: + text: ${ solution_input } + - data: + answer: ${ SOLUTION|trim } diff --git a/contrib/prompt_library/examples/evalplus/general.pdl b/contrib/prompt_library/examples/evalplus/general.pdl deleted file mode 100644 index 9eb7cc447..000000000 --- a/contrib/prompt_library/examples/evalplus/general.pdl +++ /dev/null @@ -1,144 +0,0 @@ -description: EvalPlus simple -defs: - react_demos: - data: - - - task: |- - Write a function to find the similar elements from the given two tuple lists. - assert similar_elements((3, 4, 5, 6),(5, 7, 4, 10)) == (4, 5) - - thought: |- - The similar elements here are just elements that are in both lists. I should run a solution on the test case before proposing a solution. - - action: |- - def similar_elements(test_tup1, test_tup2): - res = tuple(set(test_tup1) | set(test_tup2)) - return res - res = similar_elements((3, 4, 5, 6), (5, 7, 4, 10)) - assert res == (4, 5), "Expected (4, 5) but got {}".format(res) - - observation: |- - --------------------------------------------------------------------------- - AssertionError Traceback (most recent call last) - Cell In[4], line 5 - 3 return res - 4 res = similar_elements((3, 4, 5, 6), (5, 7, 4, 10)) - ----> 5 assert res == (4, 5), "Expected (4, 5) but got {}".format(res) - - AssertionError: Expected (4, 5) but got (3, 4, 5, 6, 7, 10) - - thought: |- - To find the similar elements, I should find the intersection of the two sets. I should use the & operator instead of the | operator. - - action: |- - def similar_elements(test_tup1, test_tup2): - res = tuple(set(test_tup1) & set(test_tup2)) - return res - res = similar_elements((3, 4, 5, 6), (5, 7, 4, 10)) - assert res == (4, 5), "Expected (4, 5) but got {}".format(res) - - observation: "[Executed Successfully with No Output]" - - thought: There is no more AssertionError. I can now submit the solution. - - solution: |- - def similar_elements(test_tup1, test_tup2): - res = tuple(set(test_tup1) & set(test_tup2)) - return res -text: - # CoT - - if: ${ prompt_pattern == 'cot' } - then: - text: - - for: - example: ${ demonstrations } - repeat: - text: |+ - Task: ${ example.question } - Solution: - ${ example.answer } - - - | - Task: ${ prompt } - Solution: - - model: ${ model } - parameters: - stop: - - "<|endoftext|>" - max_tokens: 512 - include_stop_sequence: false - temperature: 0 - - # ReAct - - if: ${ prompt_pattern == 'react' } - then: - text: - # see https://github.com/xingyaoww/mint-bench/blob/main/mint/tasks/in_context_examples/mbpp/with_tool.txt for prompt - - include: ../../ReAct.pdl - - for: - traj: ${ demonstrations } - repeat: - text: - - call: ${ react_code_block } - args: - trajectory: ${ traj } - - "\n" - - for: - traj: ${ react_demos } - repeat: - text: - - call: ${ react_code_block } - args: - trajectory: ${ traj } - - |- - - Task: - ${ prompt|trim } - - Assistant: - - - defs: - temperature: 0 - SOLUTION: "" - iterations: 0 - max_iterations: 5 - EXECUTE: "" - - repeat: - text: - - repeat: - text: - - defs: - iterations: ${ iterations+1 } - - def: THOUGHT - text: - - model: ${ model } - def: THOUGHT_inner - parameters: - temperature: ${ temperature } - stop: - - - - - - <|endoftext|> - max_tokens: 1024 - include_stop_sequences: true - - def: thought_trim - contribute: [] - text: ${ THOUGHT_inner|trim|trim('\n') } - until: ${ thought_trim.endswith('') or (thought_trim).endswith('') or iterations>max_iterations } - - if: ${ (thought_trim).endswith('') } - then: - text: - - "\n\nObservation:\n" - - def: observation - contribute: [] - lang: ipython - code: | - ${ thought_trim.split('')[-1].strip('\n\n').lstrip('```python').strip('`') or thought_trim } - - "${ (observation|trim|trim('\n')) if observation is not none else '[Executed Successfully with No Output]' }" - - |- - - - Assistant: - - - if: ${ thought_trim.endswith('') } - then: - defs: - SOLUTION: ${ thought_trim.split('')[-1].strip('') or thought_trim } - until: ${ SOLUTION != '' or iterations>max_iterations } - - if: ${ iterations>max_iterations } - then: "[ Failed to find a solution within ${ max_iterations } iterations. ]" - - |- - - Solution: - ${ SOLUTION|trim } \ No newline at end of file diff --git a/contrib/prompt_library/examples/fever/general.pdl b/contrib/prompt_library/examples/fever/general.pdl deleted file mode 100644 index 90b9dbe1c..000000000 --- a/contrib/prompt_library/examples/fever/general.pdl +++ /dev/null @@ -1,85 +0,0 @@ -description: Demo of ReAct template fever -defs: - search_tools_react: - data: - - name: Search ReAct - display_name: Search - pdl_function: Search - description: Search Wikipedia for a summary - parameters: - - name: topic - type: string - description: The topic of interest - examples: [] - - search_tools_rewoo: - data: - - name: Search ReWOO - display_name: Search - pdl_function: Search - description: Search Wikipedia for a summary - parameters: - - name: topic - type: string - description: The topic of interest - examples: [] - - search_tools: - data: - - type: function - function: - name: Search - description: Search Wikipedia for a summary - parameters: - type: object - properties: - topic: - type: string - description: The topic of interest - required: - - topic - - task: |- - Task: On June 2017, the following claim was made: ${ claim } - Q: Was this claim true or false? -text: - - include: ../../tools.pdl - - # CoT - - if: ${ prompt_pattern == 'cot' } - then: - text: - - include: ../../CoT.pdl - - call: ${ chain_of_thought_claim } - args: - examples: ${ demonstrations } - question: ${ task } - model: ${ model } - - # ReAct - - if: ${ prompt_pattern == 'react' } - then: - text: - - include: ../../ReAct.pdl - - call: ${ react } - def: ANSWER - args: - task: ${ task } - model: ${ model } - tools: ${ search_tools } - trajectories: ${ demonstrations } - system_prompt: ${ system_prompt } - - # ReWOO - - if: ${ prompt_pattern == 'rewoo' } - then: - text: - - include: ../../ReWoo.pdl - - call: ${ rewoo } - def: ANSWER - args: - task: ${ task } - model: ${ model } - tools: ${ search_tools } - trajectories: ${ demonstrations } - show_plans: false \ No newline at end of file diff --git a/contrib/prompt_library/examples/gsm8k/example.pdl b/contrib/prompt_library/examples/gsm8k/example.pdl deleted file mode 100644 index 06217e31f..000000000 --- a/contrib/prompt_library/examples/gsm8k/example.pdl +++ /dev/null @@ -1,79 +0,0 @@ -description: Demo of template -defs: - cot: - import: ../../CoT - react: - import: ../../ReAct - rewoo: - import: ../../ReWoo - tools: - import: ../../tools - model: watsonx_text/ibm/granite-3-8b-instruct - prompt_pattern: cot - system_prompt: granite_llama - question: "Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day?" - demonstrations: - data: - - question: |- - Noah charges $60 for a large painting and $30 for a small painting. - Last month he sold eight large paintings and four small paintings. - If he sold twice as much this month, how much is his sales for this month? - reasoning: |- - He sold 8 large paintings and 4 small paintings last month. - He sold twice as many this month. - 8 large paintings x $60 = << 8*60= 480 >> 480 - 4 small paintings x $30 = << 4*30= 120 >> 120 - So he sold << 480+120= 600 >> 600 paintings last month. - Therefore he sold << 600*2= 1200 >> this month. - answer: $1200 - - question: |- - Noah charges $30 for a large vases and $10 for a small vases. - Last month he sold five large vases and three small vases. - If he sold three times as much this month, how much is his sales for this month? - reasoning: |- - He sold 5 large vases and 3 small vases last month. - He sold three times as many this month. - 5 large vases x $30 = << 5*30= 150 >> 150 - 3 small vases x $10 = << 3*10= 30 >> 30 - So he sold << 150+30= 180 >> 180 vases last month. - Therefore he sold << 180*3= 540 >> this month. - answer: $540 -text: - # CoT - - if: "${ prompt_pattern == 'cot' }" - then: - text: - - "Answer the questions to the best of your abilities.\n\n" - - call: ${ cot.chain_of_thought } - args: - examples: "${ demonstrations }" - question: "${ question|trim }" - model: "${ model }" - - # ReAct - - if: "${ prompt_pattern == 'react' }" - then: - text: - - call: ${ react.react } - def: ANSWER - args: - task: "Question: ${ question|trim }" - model: "${ model }" - tools: ${ tools.tools } - trajectories: ${ demonstrations } - system_prompt: ${ system_prompt } - - "\nThe answer is ${ ANSWER.answer|trim }" - - # ReWOO - - if: "${ prompt_pattern == 'rewoo' }" - then: - text: - - call: ${ rewoo.rewoo } - def: ANSWER - args: - task: "Task: ${ question }" - model: "${ model }" - tools: ${ tools.tools } - trajectories: ${ demonstrations } - show_plans: false - - "\nThe answer is ${ ANSWER.answer|trim }" diff --git a/contrib/prompt_library/examples/gsm8k/general.pdl b/contrib/prompt_library/examples/gsm8k/general.pdl deleted file mode 100644 index f58d53cc7..000000000 --- a/contrib/prompt_library/examples/gsm8k/general.pdl +++ /dev/null @@ -1,79 +0,0 @@ -description: Demo of template -defs: - cot: - import: ../../CoT - react: - import: ../../ReAct - rewoo: - import: ../../ReWoo - tools: - import: ../../tools - # model: watsonx/ibm/granite-3-8b-instruct - # prompt_pattern: cot - # system_prompt: granite_llama - # question: "Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day?" - # demonstrations: - # data: - # - question: |- - # Noah charges $60 for a large painting and $30 for a small painting. - # Last month he sold eight large paintings and four small paintings. - # If he sold twice as much this month, how much is his sales for this month? - # reasoning: |- - # He sold 8 large paintings and 4 small paintings last month. - # He sold twice as many this month. - # 8 large paintings x $60 = << 8*60= 480 >> 480 - # 4 small paintings x $30 = << 4*30= 120 >> 120 - # So he sold << 480+120= 600 >> 600 paintings last month. - # Therefore he sold << 600*2= 1200 >> this month. - # answer: $1200 - # - question: |- - # Noah charges $30 for a large vases and $10 for a small vases. - # Last month he sold five large vases and three small vases. - # If he sold three times as much this month, how much is his sales for this month? - # reasoning: |- - # He sold 5 large vases and 3 small vases last month. - # He sold three times as many this month. - # 5 large vases x $30 = << 5*30= 150 >> 150 - # 3 small vases x $10 = << 3*10= 30 >> 30 - # So he sold << 150+30= 180 >> 180 vases last month. - # Therefore he sold << 180*3= 540 >> this month. - # answer: $540 -text: - # CoT - - if: "${ prompt_pattern == 'cot' }" - then: - text: - - "Answer the questions to the best of your abilities.\n\n" - - call: ${ cot.chain_of_thought } - args: - examples: "${ demonstrations }" - question: "${ question|trim }" - model: "${ model }" - - # ReAct - - if: "${ prompt_pattern == 'react' }" - then: - text: - - call: ${ react.react } - def: ANSWER - args: - task: "Question: ${ question|trim }" - model: "${ model }" - tools: ${ tools.tools } - trajectories: ${ demonstrations } - system_prompt: ${ system_prompt } - - "\nThe answer is ${ ANSWER.answer|trim }" - - # ReWOO - - if: "${ prompt_pattern == 'rewoo' }" - then: - text: - - call: ${ rewoo.rewoo } - def: ANSWER - args: - task: "Task: ${ question }" - model: "${ model }" - tools: ${ tools.tools } - trajectories: ${ demonstrations } - show_plans: false - - "\nThe answer is ${ ANSWER.answer|trim }" diff --git a/contrib/prompt_library/tools.pdl b/contrib/prompt_library/tools.pdl index db77827c4..0893142ab 100644 --- a/contrib/prompt_library/tools.pdl +++ b/contrib/prompt_library/tools.pdl @@ -15,53 +15,28 @@ defs: tools: { list: obj } tool_names: { list: str } return: - data: ${ tools|selectattr('display_name', 'in', tool_names)|list } + data: ${ tools|selectattr('function.name', 'in', tool_names)|list } - list_tools: - function: - tools: { list: obj } - return: - lastOf: - - defs: - signatures: - for: - tool: ${ tools } - repeat: "${ tool.display_name }[<${ tool.parameters|join('>, <', attribute='name') }>]" - join: - as: array - typed_signatures: - for: - tool: ${ tools } - repeat: - lastOf: - - defs: - params: - for: - param: ${ tool.parameters } - repeat: "${ param.name}${ (': ' ~ param.type) if param.type else '' }" - - "${ tool.display_name }(${ params|join(', ') })" - join: - as: array - display_names: ${ tools|map(attribute='display_name')|list } - pdl_functions: ${ tools|map(attribute='pdl_function')|list } - name_map: - lang: python - code: | - result = dict(zip(display_names, pdl_functions, strict=True)) - - data: - names: ${ tools|map(attribute='display_name')|list } - display_names: ${ tools|map(attribute='display_name')|list } - pdl_functions: ${ tools|map(attribute='pdl_function')|list } - name_map: ${ name_map } - signatures: ${ signatures } - typed_signatures: ${ typed_signatures } - descriptions: ${ tools|map(attribute='description')|list } + finish_action: + data: + type: function + function: + name: finish + description: Respond with the answer + parameters: + type: object + properties: + answer: + type: str + description: The answer + required: + - answer - tools: + tool_schema: data: - type: function function: - name: Calculator + name: calculator description: Calculator function parameters: type: object @@ -71,21 +46,21 @@ defs: description: Arithmetic expression to calculate required: - expr - - type: function function: - name: finish - description: Respond with the answer + name: Search + description: Search Wikipedia for a summary parameters: type: object properties: - answer: - type: str - description: The answer - required: - - answer + topic: + type: string + description: The topic of interest + required: + - topic + - ${ finish_action } - pdl_tools: + tools: object: calculator: function: @@ -115,21 +90,21 @@ defs: arguments: obj return: lastOf: - - def: result - lang: python - code: | - import warnings, wikipedia - warnings.simplefilter("ignore") + - def: result + lang: python + code: | + import warnings, wikipedia + warnings.simplefilter("ignore") - def main(topic: str, *args, **kwargs) -> str: - try: - return wikipedia.summary(topic) - except wikipedia.DisambiguationError as d: - return f"\"{topic}\" may refer to one of {d.args[1]}. Please retry the search with a more specific subject." - except wikipedia.WikipediaException as e: - return str(e) + def main(topic: str, *args, **kwargs) -> str: + try: + return wikipedia.summary(topic) + except wikipedia.DisambiguationError as d: + return f"\"{topic}\" may refer to one of {d.args[1]}. Please retry the search with a more specific subject." + except wikipedia.WikipediaException as e: + return str(e) - result = main(**arguments) - - if: ${ wrap_document } - then: "[Document]\n${ result|trim }\n[End]" - else: ${ result } \ No newline at end of file + result = main(**arguments) + - if: ${ wrap_document } + then: "[Document]\n${ result|trim }\n[End]" + else: ${ result } diff --git a/docs/tutorial.md b/docs/tutorial.md index 4c7921b19..efce6e5d2 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -416,17 +416,19 @@ The `import` block means that the PDL code at that file is executed and its scop ## Prompt Library -A prompt library is included in the `contrib/` directory. These modules define some common patterns such Chain-of-Thought, ReAct, and ReWOO. +A prompt library is included in the `contrib/` directory. These modules define some common patterns such Chain-of-Thought, ReAct, and ReWOO. Example usage can be found in `examples/prompt_library`. Note that `import` blocks resolve file paths relative to the PDL file being executed. -In the `GSM8K` example below, we import `CoT` and define our model, demonstrations (a list of dictionaries), and the question. +### Chain-of-Thought -``` yaml +In the `GSM8K` example below, we import `CoT` [(Wei et al., 2022)](https://arxiv.org/abs/2201.11903) and define our model, demonstrations, and the question. Demonstrations are a list of dictionaries with `question`, `reasoning`, and `answer` keys. The `cot.chain_of_thought` function returns a dictionary with an `answer` key. + +``` yaml title="examples/prompt_library/gsm8k_cot.pdl" --8<-- "./examples/prompt_library/gsm8k_cot.pdl" ``` -Executing this example results in the following output (model response highlighted): +Executing this example produces the following background context (model response highlighted): -``` text linenums="1" hl_lines="27-31" +``` text hl_lines="27-31" Answer the questions to the best of your abilities. Question: Noah charges $60 for a large painting and $30 for a small painting. @@ -460,6 +462,228 @@ Answer: Let's think step by step. The answer is $720. ``` +The final result is: + +``` text +Answer the questions to the best of your abilities. + +Result: {'answer': '1. Jacob earns $6 per hour.\n2. Jake earns thrice what Jacob does, so Jake earns 3 * $6 = $18 per hour.\n3. Jake works 8 hours a day, so he earns $18 * 8 = $144 per day.\n4. Jake works 5 days, so he earns $144 * 5 = $720 in 5 days.\nThe answer is $720.'} +``` + +### ReAct + +In the ReAct [(Yao et al., 2022)](https://arxiv.org/abs/2210.03629) example below, we import the ReAct library and provide tools from `tools.pdl`. Demonstrations consist of agent trajectories represented as a list of lists with key names `question` or `task`, `thought`, `action`, and `observation`. Actions follow this pattern: `{"name": "tool_name", "arguments": {"arg1": "..."}}`. The function returns a dictionary with a `answer` key, containing the answer. + +``` yaml title="examples/prompt_library/gsm8k_react.pdl" +--8<-- "./examples/prompt_library/gsm8k_react.pdl" +``` + +Produces background context (model responses highlighted): + +``` text hl_lines="22 23 25 26 28 29 31 32" +Question: Noah charges $60 for a large painting and $30 for a small painting. Last month he sold eight large paintings and four small paintings. If he sold twice as much this month, how much is his sales for this month? +Tho: He sold 8 large paintings and 4 small paintings last month. He sold twice as many this month. I need to calculate (8 large paintings x $60 + 4 small paintings x $30) +Act: {"name": "calculator", "arguments": {"expr": "8*60+4*30"}} +Obs: 600 +Tho: He sold twice as many paintings this month, therefore I need to calculate 600*2. +Act: {"name": "calculator", "arguments": {"expr": "600*2"}} +Obs: 1200 +Tho: He sold $1200 this month. +Act: {"name": "finish", "arguments": {"answer": "$1200"}} + +Question: Teresa is 59 and her husband Morio is 71 years old. Their daughter, Michiko was born when Morio was 38. How old was Teresa when she gave birth to Michiko? +Tho: I need to calculate the difference in age between Teresa and Morio. +Act: {"name": "calculator", "arguments": {"expr": "71-59"}} +Obs: 12 +Tho: I need to calculate how old Teresa is when their daughter is born. +Act: {"name": "calculator", "arguments": {"expr": "38-12"}} +Obs: 26 +Tho: Teresa was 26 when she gave birth to Michiko. +Act: {"name": "finish", "arguments": {"answer": "26"}} + +Question: Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day? +Tho: Jacob earns $6 per hour. Jake earns thrice as much. +Act: {"name": "calculator", "arguments": {"expr": "6*3"}} +Obs: 18 +Tho: Jake earns $18 per hour. +Act: {"name": "calculator", "arguments": {"expr": "18*8"}} +Obs: 144 +Tho: Jake earns $144 in a day. +Act: {"name": "calculator", "arguments": {"expr": "144*5"}} +Obs: 720 +Tho: Jake earns $720 in 5 days. +Act: {"name": "finish", "arguments": {"answer": "$720"}} +``` + +The final result is: + +``` text +Result: {'answer': '$720'} +``` + +#### Tools + +Tools allow the agentic patterns to call PDL functions. The tools are defined in two parts. The `tools` object is a dictionary of tool names to PDL functions. The `tool_schema` is a JSON schema, represented as a `data` block consisting of list of dictionaries, that describes the tools to the LLM. The `tools` library contains a calculator and a Wikipedia search tool by default. There are a few requirements to consider when using tools. + +1. You must have a `finish` tool in the _schema_. You do not need a PDL function for this, but the LLM needs to be aware of the `finish` tool so the `ReAct` loop can end. +2. A PDL tool function must accept only `arguments: obj` as parameters. + +Here we define our tools in PDL, with an example using a `python` block. Note the use of `arguments: obj`. + +``` yaml +tools: + object: + hello: + function: + arguments: obj + return: + lang: python + code: | + result = None + def main(name: str, *args, **kwargs) -> str: + return f"hello {name}" + result = main(**arguments) + + another_function: ... +``` + +Once we have defined our PDL functions, we describe them in the `tool_schema`. + +``` yaml +tool_schema: + data: + - type: function + function: + name: hello + description: Hello function, returns "hello " + parameters: + type: object + properties: + name: + type: string + description: Name to greet + required: + - name + + - type: function + function: + name: finish + description: Respond with the answer + parameters: + type: object + properties: + answer: + type: str + description: The answer + required: + - answer +``` + +Another useful thing to remember is that data blocks are templated in PDL. For example, this is valid: + +``` yaml +finish_action: + data: + type: function + function: + name: finish + description: Respond with the answer + parameters: + type: object + properties: + answer: + type: str + description: The answer + required: + - answer + +tool_schema: + data: + - type: function + function: + name: calculator + description: Calculator function + parameters: + type: object + properties: + expr: + type: string + description: Arithmetic expression to calculate + required: + - expr + - ${ finish_action } +``` + +In fact, you can reuse the `finish_action` from `tools` in your own tool schemas: + +``` yaml +defs: + tools: + import: ../../contrib/prompt_library/tools + my_tool_schema: + data: + - ${ tools.finish_action } + - ... +``` + +Additionally, there is a helper function `filter_tools_by_name` in `tools`, that given a JSON tools schema and a list of tool names, returns a schema with only those tools in it. This is useful if you have defined many tools, but don't need all of them for a particular task. + +``` yaml +call: ${ tools.filter_tools_by_name } +args: + tools: ${ tools.tool_schema } + tool_names: ['calculator', 'finish'] +``` + +### ReWOO + +The tools can be reused for ReWOO [(Xu et al., 2023)](https://arxiv.org/abs/2305.18323). Demonstrations follow a similar pattern as for ReAct, except that ReWOO does not use a `finish` tool/action, nor does it include `observation`, and thus should be omitted from the demonstrations. The argument `show_plans` is used for displaying the extracted ReWOO plans in the background context for debugging purposes i.e. `--stream context`. The function returns a dictionary with a `answer` key, containing the answer. + +``` yaml title="examples/prompt_library/gsm8k_rewoo.pdl" +--8<-- "./examples/prompt_library/gsm8k_rewoo.pdl" +``` + +This results in the following background context: + +``` text hl_lines="18 28-30" +For the following task, make plans that can solve the problem step by step. For each plan, indicate which external tool together with tool input to retrieve evidence. You can store the evidence into a variable #E that can be called by later tools. (Plan, #E1, Plan, #E2, Plan, ...) + +Tools can be one of the following: +[{'type': 'function', 'function': {'name': 'calculator', 'description': 'Calculator function', 'parameters': {'type': 'object', 'properties': {'expr': {'type': 'string', 'description': 'Arithmetic expression to calculate'}}, 'required': ['expr']}}}] + +Task: Noah charges $60 for a large painting and $30 for a small painting. Last month he sold eight large paintings and four small paintings. If he sold twice as much this month, how much is his sales for this month? +Plan: He sold 8 large paintings and 4 small paintings last month. He sold twice as many this month. I need to calculate (8 large paintings x $60 + 4 small paintings x $30) #E1 = {"name": "calculator", "arguments": {"expr": "8*60+4*30"}} +Plan: He sold twice as many paintings this month, therefore I need to calculate 600*2. #E2 = {"name": "calculator", "arguments": {"expr": "600*2"}} + +Task: Teresa is 59 and her husband Morio is 71 years old. Their daughter, Michiko was born when Morio was 38. How old was Teresa when she gave birth to Michiko? +Plan: I need to calculate the difference in age between Teresa and Morio. #E1 = {"name": "calculator", "arguments": {"expr": "71-59"}} +Plan: I need to calculate how old Teresa is when their daughter is born. #E2 = {"name": "calculator", "arguments": {"expr": "38-12"}} + +Begin! +Describe your plans with rich details. Each Plan should be followed by only one #E. + +Question: Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day? +[["First, I need to calculate Jake's hourly wage. Since Jake earns thrice what Jacob does, and Jacob earns $6 per hour, Jake earns 3 * $6 = $18 per hour.", "#E1", "{\"name\": \"calculator\", \"arguments\": {\"expr\": \"6*3\"}}"]] + +Solve the following task or problem. To solve the problem, we have made step-by-step Plan and retrieved corresponding Evidence to each Plan. Use them with caution since long evidence might contain irrelevant information. + +Plan: First, I need to calculate Jake's hourly wage. Since Jake earns thrice what Jacob does, and Jacob earns $6 per hour, Jake earns 3 * $6 = $18 per hour. +Evidence: 18 + +Now solve the question or task according to provided Evidence above. Respond with the answer directly with no extra words. + +Question: Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day? +Response: Jake earns $18 per hour. In 5 days, working 8 hours a day, Jake works 5 * 8 = 40 hours. Therefore, Jake earns 40 * $18 = $720 in 5 days. + +Answer: $720 +``` + +The final result of this example is + +``` text +Result: {"answer": "Jake earns $18 per hour. In 5 days, working 8 hours a day, Jake works 5 * 8 = 40 hours. Therefore, Jake earns 40 * $18 = $720 in 5 days.\n\nAnswer: $720"} +``` + ## Conditionals and Loops PDL supports conditionals and loops as illustrated in the following example ([file](https://github.com/IBM/prompt-declaration-language//blob/main/examples/tutorial/programs/chatbot.pdl)), which implements a chatbot. diff --git a/examples/prompt_library/gsm8k_cot.pdl b/examples/prompt_library/gsm8k_cot.pdl index f8c3fc69c..834e5266a 100644 --- a/examples/prompt_library/gsm8k_cot.pdl +++ b/examples/prompt_library/gsm8k_cot.pdl @@ -1,4 +1,4 @@ -description: Demo of template +description: Demo of CoT pattern defs: cot: import: ../../contrib/prompt_library/CoT @@ -35,7 +35,10 @@ text: # CoT - "Answer the questions to the best of your abilities.\n\n" - call: ${ cot.chain_of_thought } + def: cot_result + contribute: [] args: examples: "${ demonstrations }" question: "${ question }" - model: "${ model }" \ No newline at end of file + model: "${ model }" + - "Result: ${ cot_result }" \ No newline at end of file diff --git a/examples/prompt_library/gsm8k_react.pdl b/examples/prompt_library/gsm8k_react.pdl new file mode 100644 index 000000000..90648dca5 --- /dev/null +++ b/examples/prompt_library/gsm8k_react.pdl @@ -0,0 +1,41 @@ +description: Demo of ReAct pattern +defs: + react: + import: ../../contrib/prompt_library/ReAct + tools: + import: ../../contrib/prompt_library/tools + model: watsonx_text/ibm/granite-3-8b-instruct + demonstrations: + data: + - - question: Noah charges $60 for a large painting and $30 for a small painting. Last month he sold eight large paintings and four small paintings. If he sold twice as much this month, how much is his sales for this month? + - thought: He sold 8 large paintings and 4 small paintings last month. He sold twice as many this month. I need to calculate (8 large paintings x $60 + 4 small paintings x $30) + - action: '{"name": "calculator", "arguments": {"expr": "8*60+4*30"}}' + - observation: 600 + - thought: He sold twice as many paintings this month, therefore I need to calculate 600*2. + - action: '{"name": "calculator", "arguments": {"expr": "600*2"}}' + - observation: 1200 + - thought: He sold $1200 this month. + - action: '{"name": "finish", "arguments": {"answer": "$1200"}}' + - - question: Teresa is 59 and her husband Morio is 71 years old. Their daughter, Michiko was born when Morio was 38. How old was Teresa when she gave birth to Michiko? + - thought: I need to calculate the difference in age between Teresa and Morio. + - action: '{"name": "calculator", "arguments": {"expr": "71-59"}}' + - observation: 12 + - thought: I need to calculate how old Teresa is when their daughter is born. + - action: '{"name": "calculator", "arguments": {"expr": "38-12"}}' + - observation: 26 + - thought: Teresa was 26 when she gave birth to Michiko. + - action: '{"name": "finish", "arguments": {"answer": "26"}}' + question: "Question: Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day?" + +text: + # ReAct + - call: ${ react.react } + def: react_result + contribute: [] + args: + task: ${ question } + model: ${ model } + tool_schema: ${ tools.tool_schema } + tools: ${ tools.tools } + trajectories: ${ demonstrations } + - "Result: ${ react_result }" \ No newline at end of file diff --git a/examples/prompt_library/gsm8k_rewoo.pdl b/examples/prompt_library/gsm8k_rewoo.pdl new file mode 100644 index 000000000..38fbf5433 --- /dev/null +++ b/examples/prompt_library/gsm8k_rewoo.pdl @@ -0,0 +1,34 @@ +description: Demo of ReWOO pattern +defs: + rewoo: + import: ../../contrib/prompt_library/ReWoo + tools: + import: ../../contrib/prompt_library/tools + model: watsonx_text/ibm/granite-3-8b-instruct + demonstrations: + data: + - - question: Noah charges $60 for a large painting and $30 for a small painting. Last month he sold eight large paintings and four small paintings. If he sold twice as much this month, how much is his sales for this month? + - thought: He sold 8 large paintings and 4 small paintings last month. He sold twice as many this month. I need to calculate (8 large paintings x $60 + 4 small paintings x $30) + - action: '{"name": "calculator", "arguments": {"expr": "8*60+4*30"}}' + - thought: He sold twice as many paintings this month, therefore I need to calculate 600*2. + - action: '{"name": "calculator", "arguments": {"expr": "600*2"}}' + - - question: Teresa is 59 and her husband Morio is 71 years old. Their daughter, Michiko was born when Morio was 38. How old was Teresa when she gave birth to Michiko? + - thought: I need to calculate the difference in age between Teresa and Morio. + - action: '{"name": "calculator", "arguments": {"expr": "71-59"}}' + - thought: I need to calculate how old Teresa is when their daughter is born. + - action: '{"name": "calculator", "arguments": {"expr": "38-12"}}' + question: "Question: Jake earns thrice what Jacob does. If Jacob earns $6 per hour, how much does Jake earn in 5 days working 8 hours a day?" + +text: + # ReWoo + - call: ${ rewoo.rewoo } + def: rewoo_result + contribute: [] + args: + task: ${ question } + model: ${ model } + tool_schema: ${ tools.tool_schema } + tools: ${ tools.tools } + trajectories: ${ demonstrations } + show_plans: false + - "Result: ${ rewoo_result }" \ No newline at end of file From 0691677405423e5bb4bd4c58c4c19cd854d8f4a3 Mon Sep 17 00:00:00 2001 From: Claudio Spiess Date: Tue, 22 Apr 2025 18:32:30 -0600 Subject: [PATCH 4/4] Address feedback Signed-off-by: Claudio Spiess --- contrib/prompt_library/CoT.pdl | 14 +++--- contrib/prompt_library/ReAct.pdl | 2 +- contrib/prompt_library/tools.pdl | 68 ++++++++++++-------------- docs/tutorial.md | 84 +++++++++++++++----------------- 4 files changed, 76 insertions(+), 92 deletions(-) diff --git a/contrib/prompt_library/CoT.pdl b/contrib/prompt_library/CoT.pdl index 3d52a4972..4d559ac34 100644 --- a/contrib/prompt_library/CoT.pdl +++ b/contrib/prompt_library/CoT.pdl @@ -20,15 +20,13 @@ defs: - for: example: ${ examples } repeat: - text: - - call: ${ cot_block } - args: - question: ${ example.question } - reasoning: ${ example.reasoning } - answer: ${ example.answer } - - "\n\n" + call: ${ cot_block } + args: + question: ${ example.question } + reasoning: ${ example.reasoning } + answer: ${ example.answer } join: - with: "" + with: "\n\n" chain_of_thought: function: diff --git a/contrib/prompt_library/ReAct.pdl b/contrib/prompt_library/ReAct.pdl index d47136b1a..2f44fdf98 100644 --- a/contrib/prompt_library/ReAct.pdl +++ b/contrib/prompt_library/ReAct.pdl @@ -72,7 +72,7 @@ defs: data: False - def: tool_names contribute: [] - text: ${ tool_schema|map(attribute='function.name')|list } + text: ${ tool_schema|map(attribute='name')|list } - repeat: text: - "\nTho: " diff --git a/contrib/prompt_library/tools.pdl b/contrib/prompt_library/tools.pdl index 0893142ab..182fff4e5 100644 --- a/contrib/prompt_library/tools.pdl +++ b/contrib/prompt_library/tools.pdl @@ -15,49 +15,43 @@ defs: tools: { list: obj } tool_names: { list: str } return: - data: ${ tools|selectattr('function.name', 'in', tool_names)|list } + data: ${ tools|selectattr('name', 'in', tool_names)|list } finish_action: data: - type: function - function: - name: finish - description: Respond with the answer - parameters: - type: object - properties: - answer: - type: str - description: The answer - required: - - answer + name: finish + description: Respond with the answer + parameters: + type: object + properties: + answer: + type: str + description: The answer + required: + - answer tool_schema: data: - - type: function - function: - name: calculator - description: Calculator function - parameters: - type: object - properties: - expr: - type: string - description: Arithmetic expression to calculate - required: - - expr - - type: function - function: - name: Search - description: Search Wikipedia for a summary - parameters: - type: object - properties: - topic: - type: string - description: The topic of interest - required: - - topic + - name: calculator + description: Calculator function + parameters: + type: object + properties: + expr: + type: string + description: Arithmetic expression to calculate + required: + - expr + - name: Search + description: Search Wikipedia for a summary + parameters: + type: object + properties: + topic: + type: string + description: The topic of interest + required: + - topic - ${ finish_action } tools: diff --git a/docs/tutorial.md b/docs/tutorial.md index efce6e5d2..f8c5380eb 100644 --- a/docs/tutorial.md +++ b/docs/tutorial.md @@ -552,41 +552,18 @@ Once we have defined our PDL functions, we describe them in the `tool_schema`. ``` yaml tool_schema: data: - - type: function - function: - name: hello - description: Hello function, returns "hello " - parameters: - type: object - properties: - name: - type: string - description: Name to greet - required: - - name - - - type: function - function: - name: finish - description: Respond with the answer - parameters: - type: object - properties: - answer: - type: str - description: The answer - required: - - answer -``` - -Another useful thing to remember is that data blocks are templated in PDL. For example, this is valid: + - name: hello + description: Hello function, returns "hello " + parameters: + type: object + properties: + name: + type: string + description: Name to greet + required: + - name -``` yaml -finish_action: - data: - type: function - function: - name: finish + - name: finish description: Respond with the answer parameters: type: object @@ -596,21 +573,36 @@ finish_action: description: The answer required: - answer +``` + +Another useful thing to remember is that data blocks are templated in PDL. For example, this is valid: + +``` yaml +finish_action: + data: + name: finish + description: Respond with the answer + parameters: + type: object + properties: + answer: + type: str + description: The answer + required: + - answer tool_schema: data: - - type: function - function: - name: calculator - description: Calculator function - parameters: - type: object - properties: - expr: - type: string - description: Arithmetic expression to calculate - required: - - expr + - name: calculator + description: Calculator function + parameters: + type: object + properties: + expr: + type: string + description: Arithmetic expression to calculate + required: + - expr - ${ finish_action } ```