Skip to content

Commit b280f6a

Browse files
vazirimvazirim
authored
granite-io notebook demo (#592)
Signed-off-by: Mandana Vaziri <[email protected]>
1 parent 9081b13 commit b280f6a

File tree

1 file changed

+304
-0
lines changed

1 file changed

+304
-0
lines changed

‎examples/notebooks/granite_io_demo.ipynb‎

Lines changed: 304 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,304 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "50f4ce4a",
6+
"metadata": {},
7+
"source": [
8+
"# PDL - Granite IO Processor Demo\n",
9+
"\n",
10+
"The Prompt Declaration Language (PDL) is a YAML-based declarative approach to prompt programming, where prompts are at the forefront. PDL facilitates model chaining and tool use, abstracting away the plumbing necessary for such compositions, enables type checking of the input and output of models, and is based on LiteLLM to support a variety of model providers. PDL has been used with RAG, CoT, ReAct, and an agent for solving SWE-bench. PDL is [open-source](https://github.com/IBM/prompt-declaration-language) and works well with watsonx.ai and Granite models.\n",
11+
"\n",
12+
"You can use PDL stand-alone or from a Python SDK or, as shown here, in a notebook via a notebook extension. In the cell output, model-generated text is rendered in green font, and tool-generated text is rendered in purple font.\n",
13+
"\n",
14+
"In this notebook, we demonstrate how PDL is integrated with the [Granite IO Processor](https://github.com/ibm-granite/granite-io) framework, which enables a developer to transform how a user calls an IBM Granite model and how the output from the model is returned to the user. PDL uses granite-io as an alternative backend to LiteLLM. The following examples show how to call an Ollama Granite model via PDL and granite-io, how to extract hallucination scores and citations, and how to toggle the thinking control, which turns on reasoning.\n",
15+
"\n",
16+
"First make sure you have Ollama installed and ollama serve is running, and that you have pulled the `granite3.2:2b` model."
17+
]
18+
},
19+
{
20+
"cell_type": "code",
21+
"execution_count": null,
22+
"id": "bfc303da",
23+
"metadata": {},
24+
"outputs": [],
25+
"source": [
26+
"! pip install 'prompt-declaration-language[examples]'"
27+
]
28+
},
29+
{
30+
"cell_type": "code",
31+
"execution_count": 2,
32+
"id": "e25a6874-54d9-4167-82ed-ab2f4fdc0a6f",
33+
"metadata": {},
34+
"outputs": [],
35+
"source": [
36+
"%load_ext pdl.pdl_notebook_ext"
37+
]
38+
},
39+
{
40+
"cell_type": "markdown",
41+
"id": "b2234ce9",
42+
"metadata": {},
43+
"source": [
44+
"## Model call with granite-io\n",
45+
"\n",
46+
"In PDL, the user specifies step-by-step the shape of data they want to generate. In the following, the `text` construct indicates a text block containing a prompt and a model call. Implicitly, PDL builds a background conversational context (list of role/content) which is used to make model calls. Each model call uses the context built so far as its input prompt.\n",
47+
"\n",
48+
"In this example, we infer using the `granite3.2:2b` model on Ollama via `granite-io`. Note that the `platform` field can be omited."
49+
]
50+
},
51+
{
52+
"cell_type": "code",
53+
"execution_count": 6,
54+
"id": "f3c62df1-0347-4711-acd7-3892cfd5df30",
55+
"metadata": {},
56+
"outputs": [
57+
{
58+
"name": "stdout",
59+
"output_type": "stream",
60+
"text": [
61+
"Hello!\n",
62+
"\u001b[32mHello! It seems like there's no question or context provided for me to respond to yet. How can I assist you today? Maybe you have some questions on a wide range of topics, from general knowledge, science, history, literature, technology, and more. Feel free to share what's on your mind. For instance, here are a few areas I'm prepared to cover:\n",
63+
"\n",
64+
"1. **Science**: Explain concepts in physics, biology, chemistry or even space exploration.\n",
65+
"2. **History**: Provide information or context about historical events.\n",
66+
"3. **Literature and Arts**: Discuss various literary works, authors, artists, their periods, movements, and styles.\n",
67+
"4. **Technology**: Talk about current trends, explain technical terms or concepts, even describe how certain tech products might work.\n",
68+
"5. **Current Events**: Give a summary of recent news across the globe or discuss topics like climate change, politics, or popular culture.\n",
69+
"\n",
70+
"Please share what you're interested in learning more about today, and I'll do my best to provide an informative and engaging response.\u001b[0m"
71+
]
72+
}
73+
],
74+
"source": [
75+
"%%pdl --reset-context\n",
76+
"text:\n",
77+
"- \"Hello!\\n\"\n",
78+
"- model: \"granite3.2:2b\"\n",
79+
" platform: granite-io\n",
80+
" backend: openai"
81+
]
82+
},
83+
{
84+
"cell_type": "markdown",
85+
"id": "152180fe-2c69-4760-9989-8c52ec60b341",
86+
"metadata": {},
87+
"source": [
88+
"## Model call with thinking flag\n",
89+
"\n",
90+
"In the following example, we pass the `thinking` flag to the model, which causes it to reason. This flag is passed to the Ollama model via the `granite-io` library, which shapes the prompt appropriately given the `thinking` flag."
91+
]
92+
},
93+
{
94+
"cell_type": "code",
95+
"execution_count": 5,
96+
"id": "bb01f89d-afaa-409c-ad48-10cc50c3fbc5",
97+
"metadata": {},
98+
"outputs": [
99+
{
100+
"name": "stdout",
101+
"output_type": "stream",
102+
"text": [
103+
"Find the fastest way for a seller to visit all the cities in their region\n",
104+
">> Response:\n",
105+
"\u001b[32m\n",
106+
"\n",
107+
"To find the fastest way—in terms of minimizing cities visited and ensuring every city is covered without repetition—for a seller (let's assume they operate as a last-mile courier within a defined region) to visit all cities along a circular path from any one starting point, your best bet would be to employ specialized algorithms designed to solve variants of the Traveling Salesman Problem. One such tool is Concorde TSP Solver, an open-source exact solver that's renowned for its performance on real-world data sets and industrial scale instances due to sophisticated optimizations, primarily:\n",
108+
"\n",
109+
"1. **Branch-and-Cut Method**: This technique cuts away non-integral solutions iteratively while retaining feasible paths to maintain a balance between improving the objective (i.e., cities visited) and adhering to constraints, thus finding near-optimal results efficiently.\n",
110+
"\n",
111+
"2. **Preprocessing**: Concorde uses advanced preprocessing steps like the Lin–Kernighan algorithm followed by linear programming refinement to further improve its running time and solution quality as large instances scale.\n",
112+
"\n",
113+
"3. **Integration and Use** (Pseudo-Python Implementation Using Python Concorde Library):\n",
114+
"\n",
115+
" ```python\n",
116+
" import concorde\n",
117+
"\n",
118+
" # List of cities, use 0 for home base\n",
119+
" cities = [1, 2, 3, 4, 5, 6, 7, 8, 9, 1] # Example: Cities in order from 0 to nine (e.g., [1-based index)]\n",
120+
"\n",
121+
" # Set starting city (index)\n",
122+
" start = 0\n",
123+
" \n",
124+
" # Run Concorde as an exact TSP solver with a specified timeout (optional; you can set it for more time if needed)\n",
125+
" solver = concorde.TspSolver()\n",
126+
" results = solver.solve(cities_from=list(range(len(cities))), start=start, max_seconds=3600) # Adjust 'max_seconds' based on your tolerance\n",
127+
"\n",
128+
" print(\"Optimal Route with Minimum Cities Visited:\", results.best_path())\n",
129+
" ```\n",
130+
"\n",
131+
">> Thoughts:will be a sequence of indices representing the optimal cities to visit that minimizes total visits while covering all predefined locations exactly once, which in the city context signifies the most efficient path for the seller’s courier rounds without revisiting any city till every stop is made. Note that Concorde's results are typically very close to optimal with real-world applications—the primary benefit being scalability and execution speed across diverse dataset sizes. While not an exact polynomial-time algorithm, it outperforms most other methods for large TSP instances due to its sophisticated heuristics and optimization techniques.\u001b[0m\n",
132+
"\n",
133+
"\n",
134+
"1. **Understand the Problem**: The goal is to find the quickest route that allows a seller, who operates in a specific geographical region (let's assume this for this exercise as a circular map of cities), to visit each city exactly once and return to their starting point without revisiting any city until all visits are complete.\n",
135+
"\n",
136+
"2. **Type of Problem**: This problem isn't about distance traveled between two points but rather the number of cities crossed while minimizing repetitions until no more routes are possible. It's similar to the Traveling Salesman Problem (TSP), a well-known NP-hard combinatorial optimization problem, with an added constraint: we know some starting and ending (home) point, which simplifies the task somewhat.\n",
137+
"\n",
138+
"3. **Strategy**: To find the fastest route in this context, I'll focus on minimizing total cities visited while ensuring every city is covered—akin to a 1-sequence that visits each city exactly once. This type of solution aligns with what you might consider when planning the quickest delivery schedule for multiple stops from one starting point.\n",
139+
"\n",
140+
"4. **Algorithm Choice**: For such an optimization problem, efficient heuristics and approximation algorithms are often used because exact solutions become impractical for large sets of cities due to their computational complexity. One such popular, widely-used algorithm is Concorde TSP Solver, which employs a branch-and-cut method and can solve very large TSP problems quickly.\n",
141+
"\n",
142+
"5. **Tool Utilization**: Since Concorde is an offline exact solver, I would suggest using it to find the optimal or near-optimal solution. It's freely available for non-commercial use with open source licensing for programming implementations in various languages including C++, Java, and Python.\n",
143+
"\n",
144+
"6. **Implementation Notes (pseudo-code)**:\n",
145+
" - Initialize the software with a list of cities.\n",
146+
" - Set the starting city as prescribed, i.e., their home base.\n",
147+
" - Run the algorithm (Concorde):\n",
148+
" ```python\n",
149+
" # In pseudo-Python:\n",
150+
" solver = ConcordeTsp(cities, method=\"exact\", timeout=3600) # Adjust time limit per your need\n",
151+
" results = solver.solve()\n",
152+
" ```\n",
153+
" - The output `results` will contain the optimal path that visits each city exactly once with the shortest possible total distance (or cost if weights are used), minus some tolerance for computational reasons, which should be negligible in this context since it's an exact solver.\n",
154+
"\n",
155+
"\n"
156+
]
157+
}
158+
],
159+
"source": [
160+
"%%pdl --reset-context\n",
161+
"text:\n",
162+
"- |\n",
163+
" Find the fastest way for a seller to visit all the cities in their region\n",
164+
" >> Response:\n",
165+
"- model: \"granite3.2:2b\"\n",
166+
" backend: openai\n",
167+
" parameters: \n",
168+
" thinking: true\n",
169+
" modelResponse: outputs\n",
170+
"- |\n",
171+
" >> Thoughts:\n",
172+
" ${ outputs.reasoning_content }\n"
173+
]
174+
},
175+
{
176+
"cell_type": "markdown",
177+
"id": "c9d405f8",
178+
"metadata": {},
179+
"source": [
180+
"## Hallucination Score and Citations\n",
181+
"\n",
182+
"In the following example, we pass the hallucination and citations controls to the model call."
183+
]
184+
},
185+
{
186+
"cell_type": "code",
187+
"execution_count": 9,
188+
"id": "d7149b3f",
189+
"metadata": {},
190+
"outputs": [
191+
{
192+
"name": "stdout",
193+
"output_type": "stream",
194+
"text": [
195+
"Did Faith Hill take a break from recording after releasing her second album, It Matters to Me?\u001b[32mYes, after the release of her sophomore album, It Matters to Me (1995), Faith Hill indeed took a three-year break from recording <co>1</co>. This pause in music allowed her to prepare for motherhood, as she started a family with fellow country artist Tim McGraw at that time. During this period, which began in 1996 after HIll's engagement with producer Scott Hendricks turned into an affair and subsequent marriage, she collaborated on the hit single \"It's Your Love\" with her future husband (released post-separation).\n",
196+
"\n",
197+
"<co>1</co> The documents suggest that Faith Hill, after becoming successful with her albums Take Me as I Am (1993) and It Matters to Me (1995), decided to step back from the studio for approximately three years. This decision was partly driven by her desire to start a family with Tim McGraw and provide more time to their young kids, who included Gracie Katherine (born 1997), Maggie Elizabeth (born 1998), and Audrey Caroline (born 2001) at the time of this break.\n",
198+
"\n",
199+
"# Hallucinations:\n",
200+
"1. Risk low: Yes, after the release of her sophomore album, It Matters to Me (1995), Faith Hill indeed took a three-year break from recording <co>1</co>.\n",
201+
"2. Risk low: This pause in music allowed her to prepare for motherhood, as she started a family with fellow country artist Tim McGraw at that time.\n",
202+
"3. Risk high: During this period, which began in 1996 after HIll's engagement with producer Scott Hendricks turned into an affair and subsequent marriage, she collaborated on the hit single \"It's Your Love\" with her future husband (released post-separation).\n",
203+
"4. Risk low: <co>1</co> The documents suggest that Faith Hill, after becoming successful with her albums Take Me as I Am (1993) and It Matters to Me (1995), decided to step back from the studio for approximately three years.\n",
204+
"5. Risk low: This decision was partly driven by her desire to start a family with Tim McGraw and provide more time to their young kids, who included Gracie Katherine (born 1997), Maggie Elizabeth (born 1998), and Audrey Caroline (born 2001) at the time of this break.\u001b[0m"
205+
]
206+
}
207+
],
208+
"source": [
209+
"%%pdl\n",
210+
"defs:\n",
211+
" doc:\n",
212+
" data:\n",
213+
" text: |\n",
214+
" Audrey Faith McGraw (born September 21, 1967) is an American singer \n",
215+
" and record producer. She is one of the most successful country artists \n",
216+
" of all time, having sold more than 40 million albums worldwide. Hill is \n",
217+
" married to American singer Tim McGraw, with whom she has recorded several duets. \n",
218+
" Hill's first two albums, Take Me as I Am (1993) and It Matters to Me (1995), \n",
219+
" were major successes and placed a combined three number ones on Billboard's \n",
220+
" country charts. Hill's debut album was Take Me as I Am (1993); sales were strong, \n",
221+
" buoyed by the chart success of \"Wild One\". Hill became the first female country \n",
222+
" singer in 30 years to hold Billboard's number one position for four consecutive \n",
223+
" weeks when \"Wild One\" managed the feat in 1994. Her version of \"Piece of My Heart\", \n",
224+
" also went to the top of the country charts in 1994. The album sold a total of \n",
225+
" 3 million copies. Other singles from the album include \"Take Me as I Am\". The recording \n",
226+
" of Faith's second album was delayed by surgery to repair a ruptured blood vessel on \n",
227+
" her vocal cords. It Matters to Me finally appeared in 1995 and was another \n",
228+
" success, with the title track becoming her third number-one country single. \n",
229+
" Several other top 10 singles followed, and more than 3 million copies of the \n",
230+
" album were sold. The fifth single from the album, \"I Can't Do That Anymore\", \n",
231+
" was written by country music artist Alan Jackson. Other singles from the album \n",
232+
" include \"You Can't Lose Me\", \"Someone Else's Dream\", and \"Let's Go to Vegas\". \n",
233+
" During this period, Hill appeared on the acclaimed PBS music program Austin City Limits. \n",
234+
" In spring 1996, Hill began the Spontaneous Combustion Tour with country singer Tim McGraw. \n",
235+
" At that time, Hill had recently become engaged to her former producer, Scott Hendricks, \n",
236+
" and McGraw had recently broken an engagement. McGraw and Hill were quickly \n",
237+
" attracted to each other and began an affair. After discovering that Hill was \n",
238+
" pregnant with their first child, the couple married on October 6, 1996. The \n",
239+
" couple have three daughters together: Gracie Katherine (born 1997), Maggie Elizabeth (born 1998) \n",
240+
" and Audrey Caroline (born 2001). Since their marriage, Hill and McGraw have endeavored \n",
241+
" never to be apart for more than three consecutive days. After the release of It Matters to Me, \n",
242+
" Hill took a three-year break from recording to give herself a rest from four years of touring\n",
243+
" and to begin a family with McGraw. During her break, she joined forces with her husband \n",
244+
" for their first duet, \"It's Your Love\". The song stayed at number one for six weeks, \n",
245+
" and won awards from both the Academy of Country Music and the Country Music Association. \n",
246+
" Hill has remarked that sometimes when they perform the song together, \n",
247+
" \"it [doesn't] feel like anybody else was really watching.\"\n",
248+
"\n",
249+
"text:\n",
250+
"- Did Faith Hill take a break from recording after releasing her second album, It Matters to Me?\n",
251+
"- model: \"granite3.2:2b\"\n",
252+
" backend: openai\n",
253+
" parameters:\n",
254+
" documents:\n",
255+
" - ${ doc }\n",
256+
" controls:\n",
257+
" hallucinations: true\n",
258+
" citations: true\n",
259+
" "
260+
]
261+
},
262+
{
263+
"cell_type": "markdown",
264+
"id": "61c40266",
265+
"metadata": {},
266+
"source": [
267+
"## Conclusion\n",
268+
"\n",
269+
"Since prompts are at the forefront, PDL makes users more productive in their trial-and-error with LLMs. With the `granite-io` platform, PDL users can take advantage of controls such as thinking, hallucination scores and citations. Try it!\n",
270+
"\n",
271+
"https://github.com/IBM/prompt-declaration-language"
272+
]
273+
},
274+
{
275+
"cell_type": "code",
276+
"execution_count": null,
277+
"id": "35899e04-c75f-40ed-be5e-34e031c22573",
278+
"metadata": {},
279+
"outputs": [],
280+
"source": []
281+
}
282+
],
283+
"metadata": {
284+
"kernelspec": {
285+
"display_name": "Python 3 (ipykernel)",
286+
"language": "python",
287+
"name": "python3"
288+
},
289+
"language_info": {
290+
"codemirror_mode": {
291+
"name": "ipython",
292+
"version": 3
293+
},
294+
"file_extension": ".py",
295+
"mimetype": "text/x-python",
296+
"name": "python",
297+
"nbconvert_exporter": "python",
298+
"pygments_lexer": "ipython3",
299+
"version": "3.12.5"
300+
}
301+
},
302+
"nbformat": 4,
303+
"nbformat_minor": 5
304+
}

0 commit comments

Comments
 (0)