Skip to content

Commit 562e852

Browse files
willhath-openaiwillhath-openai
committed
wip
1 parent df31b9f commit 562e852

File tree

1 file changed

+303
-0
lines changed

1 file changed

+303
-0
lines changed

‎examples/evaluation/use-cases/responses-evaluation.ipynb‎

Lines changed: 303 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,303 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Evaluating a new model on existing responses"
8+
]
9+
},
10+
{
11+
"cell_type": "markdown",
12+
"metadata": {},
13+
"source": [
14+
"In the following eval, we are going to compare how a new model (gpt-4.1-mini) compares to our old model (gpt-4o-mini) by evaluating it on some stored responses. The benefit of this is for most developers, they won't have to spend any time putting together a whole eval -- all of their data will already be stored in their [logs page](https://platform.openai.com/logs)."
15+
]
16+
},
17+
{
18+
"cell_type": "code",
19+
"execution_count": 68,
20+
"metadata": {},
21+
"outputs": [],
22+
"source": [
23+
"import openai\n",
24+
"import os\n",
25+
"\n",
26+
"client = openai.OpenAI()"
27+
]
28+
},
29+
{
30+
"cell_type": "markdown",
31+
"metadata": {},
32+
"source": [
33+
"We want to see how gpt-4.1 compares to gpt-4o on explaining a code base. Since can only use the responses datasource if you already have user traffic, we're going to generate some example traffic using 4o, and then compare how it does to gpt-4.1. \n",
34+
"\n",
35+
"We're going to get some example code files from the OpenAI SDK, and ask gpt-4o to explain them to us."
36+
]
37+
},
38+
{
39+
"cell_type": "code",
40+
"execution_count": null,
41+
"metadata": {},
42+
"outputs": [],
43+
"source": [
44+
"openai_sdk_file_path = os.path.dirname(openai.__file__)\n",
45+
"\n",
46+
"# Get some example code files from the OpenAI SDK\n",
47+
"file_paths = [\n",
48+
" os.path.join(openai_sdk_file_path, \"resources\", \"evals\", \"evals.py\"),\n",
49+
" os.path.join(openai_sdk_file_path, \"resources\", \"responses\", \"responses.py\"),\n",
50+
" os.path.join(openai_sdk_file_path, \"resources\", \"images.py\"),\n",
51+
" os.path.join(openai_sdk_file_path, \"resources\", \"embeddings.py\"),\n",
52+
" os.path.join(openai_sdk_file_path, \"resources\", \"files.py\"),\n",
53+
"]\n",
54+
"\n",
55+
"print(file_paths[0])\n"
56+
]
57+
},
58+
{
59+
"cell_type": "markdown",
60+
"metadata": {},
61+
"source": [
62+
"Now, lets generate some responses"
63+
]
64+
},
65+
{
66+
"cell_type": "code",
67+
"execution_count": 66,
68+
"metadata": {},
69+
"outputs": [],
70+
"source": [
71+
"for file_path in file_paths:\n",
72+
" response = client.responses.create(\n",
73+
" input=[\n",
74+
" {\"role\": \"user\",\n",
75+
" \"content\": [\n",
76+
" {\n",
77+
" \"type\": \"input_text\",\n",
78+
" \"text\": \"What does this file do?\"\n",
79+
" },\n",
80+
" {\n",
81+
" \"type\": \"input_text\",\n",
82+
" \"text\": open(file_path, \"r\").read(),\n",
83+
" },\n",
84+
" ]},\n",
85+
" ],\n",
86+
" model=\"gpt-4o-mini\",\n",
87+
" )\n",
88+
" break\n",
89+
" print(response.id)"
90+
]
91+
},
92+
{
93+
"cell_type": "code",
94+
"execution_count": 49,
95+
"metadata": {},
96+
"outputs": [],
97+
"source": [
98+
"grader_system_prompt = \"\"\"\n",
99+
"We've created a consumer-facing Evals product to help AI integrators quickly and clearly understand their models' real-world performance. Your role is to serve as a Universal Evaluator, automatically grading responses to measure how well each model output addresses user needs and expectations.\n",
100+
"\n",
101+
"Given the conversation messages, assign a quality score in the `result` key of the response in the inclusive range between 1.0 (poor) and 7.0 (excellent). Customers will analyze your collective scores and reasoning to gain actionable insights into their models' performance.\n",
102+
"\n",
103+
"---\n",
104+
"\n",
105+
"## Things to Consider\n",
106+
"\n",
107+
"- Evaluate the overall value provided to the user\n",
108+
"- Verify all claims and do not take the AI's statements at face value! Errors might be very hard to find and well hidden.\n",
109+
"- Differentiate between minor errors (slight utility reduction) and major errors (significant trust or safety impact).\n",
110+
"- Reward answers that closely follow user instructions.\n",
111+
"- Reserve the highest and lowest reward scores for cases where you have complete certainty about correctness and utility.\n",
112+
"\n",
113+
"\n",
114+
"---\n",
115+
"\n",
116+
"## Secondary Labels to Support Final Utility Score Prediction\n",
117+
"\n",
118+
"To help you assign an accurate final utility score, first analyze and predict several important aspects of the AI response. Crucially, these intermediate evaluations should precede your final utility score prediction.\n",
119+
"\n",
120+
"Your structured output must match the provided schema:\n",
121+
"\n",
122+
"- `steps`: A JSON array of objects, each containing:\n",
123+
" - `description`: A detailed explanation of your reasoning for each step.\n",
124+
" - `result`: The float score reached based on the reasoning in this step.\n",
125+
"\n",
126+
"### Steps to Predict (in order):\n",
127+
"\n",
128+
"1. **major_errors**\n",
129+
" - *description*: Identify and explain any significant errors.\n",
130+
" - *conclusion*: List major errors found, or indicate \"None\".\n",
131+
"\n",
132+
"2. **minor_errors**\n",
133+
" - *description*: Identify and explain any minor inaccuracies.\n",
134+
" - *conclusion*: List minor errors found, or indicate \"None\".\n",
135+
"\n",
136+
"3. **potential_improvements**\n",
137+
" - *description*: Suggest enhancements that would improve the response.\n",
138+
" - *conclusion*: List suggested improvements, or indicate \"None\".\n",
139+
"\n",
140+
"---\n",
141+
"\n",
142+
"## JSON Response Structure\n",
143+
"\n",
144+
"Once you predicted all the above fields you need to assign a float between 1 and 7 to indicate the response's utility compared to the alternative responses. Use your best judgment for the meaning of `final_score`. Your response should be a JSON that can be loaded with json.loads in Python and contains:\n",
145+
"- steps: An array of objects representing your reasoning steps. Each step includes:\n",
146+
" - description (string): Detailed reasoning for this step.\n",
147+
" - result (string): The float score derived from this reasoning.\n",
148+
"- result (float): A numeric quality score as a string, in the inclusive range [1,7].\n",
149+
"\n",
150+
"---\n",
151+
"\n",
152+
"## Notes\n",
153+
"\n",
154+
"- Be meticulous in identifying errors, especially subtle or high-impact ones.\n",
155+
"- Avoid being too kind by giving overly high scores easily, it's important to often keep a gap at the top to continue having signal for improvement. Only use [6.5, 7) if the answer is truly mind blowing and you don't see how it could have been improved.\n",
156+
"- Never take the AI's responses at face value—verify everything thoroughly.\n",
157+
"\"\"\"\n",
158+
"user_input_message = \"\"\"**User input**\n",
159+
"\n",
160+
"{{item.input}}\n",
161+
"\n",
162+
"**Response to evaluate**\n",
163+
"\n",
164+
"{{sample.output_text}\n",
165+
"\"\"\""
166+
]
167+
},
168+
{
169+
"cell_type": "code",
170+
"execution_count": 71,
171+
"metadata": {},
172+
"outputs": [],
173+
"source": [
174+
"logs_eval = client.evals.create(\n",
175+
" name=\"Code QA Eval\",\n",
176+
" data_source_config={\n",
177+
" \"type\": \"logs\",\n",
178+
" },\n",
179+
" testing_criteria=[\n",
180+
" {\n",
181+
"\t\t\t\"type\": \"score_model\",\n",
182+
" \"name\": \"General Evaluator\",\n",
183+
" \"model\": \"o3\",\n",
184+
" \"input\": [{\n",
185+
" \"role\": \"system\",\n",
186+
" \"content\": grader_system_prompt,\n",
187+
" }, {\n",
188+
" \"role\": \"user\",\n",
189+
" \"content\": user_input_message,\n",
190+
" },\n",
191+
" ],\n",
192+
" \"pass_threshold\": 0.5\n",
193+
" }\n",
194+
" ]\n",
195+
")"
196+
]
197+
},
198+
{
199+
"cell_type": "markdown",
200+
"metadata": {},
201+
"source": [
202+
"First, lets kick off a run to evaluate how good the original responses were. To do this, we just set the filters for what responses we want to evaluate on"
203+
]
204+
},
205+
{
206+
"cell_type": "code",
207+
"execution_count": 72,
208+
"metadata": {},
209+
"outputs": [],
210+
"source": [
211+
"gpt_4o_run = client.evals.runs.create(\n",
212+
" eval_id=logs_eval.id,\n",
213+
" data_source={\n",
214+
" \"type\": \"responses\",\n",
215+
" \"source\": {\"type\": \"responses\", \"limit\": len(file_paths)},\n",
216+
" },\n",
217+
")"
218+
]
219+
},
220+
{
221+
"cell_type": "markdown",
222+
"metadata": {},
223+
"source": [
224+
"Now, let's see how 4.1-mini does!"
225+
]
226+
},
227+
{
228+
"cell_type": "code",
229+
"execution_count": 73,
230+
"metadata": {},
231+
"outputs": [],
232+
"source": [
233+
"gpt_41_run = client.evals.runs.create(\n",
234+
" eval_id=logs_eval.id,\n",
235+
" data_source={\n",
236+
" \"type\": \"responses\",\n",
237+
" \"source\": {\"type\": \"responses\", \"limit\": len(file_paths)},\n",
238+
" \"input_messages\": {\n",
239+
" \"type\": \"item_reference\",\n",
240+
" \"item_reference\": \"item.input\",\n",
241+
" },\n",
242+
" \"model\": \"gpt-4.1-mini\",\n",
243+
" }\n",
244+
")"
245+
]
246+
},
247+
{
248+
"cell_type": "markdown",
249+
"metadata": {},
250+
"source": [
251+
"Now, lets go to the dashboard to see how we did!"
252+
]
253+
},
254+
{
255+
"cell_type": "code",
256+
"execution_count": 74,
257+
"metadata": {},
258+
"outputs": [
259+
{
260+
"data": {
261+
"text/plain": [
262+
"'https://platform.openai.com/evaluations/eval_6822dd1c4d308190855601dc7fe982bd?project_id=proj_YFvXZA3VxHbZx5aSJv6SAM5B&run_id=evalrun_6822dd1f89b88190a3287c706fd599f4'"
263+
]
264+
},
265+
"execution_count": 74,
266+
"metadata": {},
267+
"output_type": "execute_result"
268+
}
269+
],
270+
"source": [
271+
"gpt_41_run.report_url"
272+
]
273+
},
274+
{
275+
"cell_type": "code",
276+
"execution_count": null,
277+
"metadata": {},
278+
"outputs": [],
279+
"source": []
280+
}
281+
],
282+
"metadata": {
283+
"kernelspec": {
284+
"display_name": "openai",
285+
"language": "python",
286+
"name": "python3"
287+
},
288+
"language_info": {
289+
"codemirror_mode": {
290+
"name": "ipython",
291+
"version": 3
292+
},
293+
"file_extension": ".py",
294+
"mimetype": "text/x-python",
295+
"name": "python",
296+
"nbconvert_exporter": "python",
297+
"pygments_lexer": "ipython3",
298+
"version": "3.11.8"
299+
}
300+
},
301+
"nbformat": 4,
302+
"nbformat_minor": 2
303+
}

0 commit comments

Comments
 (0)