wip #1841
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,303 @@ | ||
| { | ||
| "cells": [ | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "# Evaluating a new model on existing responses" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "In the following eval, we are going to compare how a new model (gpt-4.1-mini) compares to our old model (gpt-4o-mini) by evaluating it on some stored responses. The benefit of this is for most developers, they won't have to spend any time putting together a whole eval -- all of their data will already be stored in their [logs page](https://platform.openai.com/logs)." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 68, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "import openai\n", | ||
| "import os\n", | ||
| "\n", | ||
| "client = openai.OpenAI()" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "We want to see how gpt-4.1 compares to gpt-4o on explaining a code base. Since can only use the responses datasource if you already have user traffic, we're going to generate some example traffic using 4o, and then compare how it does to gpt-4.1. \n", | ||
| "\n", | ||
| "We're going to get some example code files from the OpenAI SDK, and ask gpt-4o to explain them to us." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "openai_sdk_file_path = os.path.dirname(openai.__file__)\n", | ||
| "\n", | ||
| "# Get some example code files from the OpenAI SDK\n", | ||
| "file_paths = [\n", | ||
| " os.path.join(openai_sdk_file_path, \"resources\", \"evals\", \"evals.py\"),\n", | ||
| " os.path.join(openai_sdk_file_path, \"resources\", \"responses\", \"responses.py\"),\n", | ||
| " os.path.join(openai_sdk_file_path, \"resources\", \"images.py\"),\n", | ||
| " os.path.join(openai_sdk_file_path, \"resources\", \"embeddings.py\"),\n", | ||
| " os.path.join(openai_sdk_file_path, \"resources\", \"files.py\"),\n", | ||
| "]\n", | ||
| "\n", | ||
| "print(file_paths[0])\n" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Now, lets generate some responses" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 66, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "for file_path in file_paths:\n", | ||
| " response = client.responses.create(\n", | ||
| " input=[\n", | ||
| " {\"role\": \"user\",\n", | ||
| " \"content\": [\n", | ||
| " {\n", | ||
| " \"type\": \"input_text\",\n", | ||
| " \"text\": \"What does this file do?\"\n", | ||
| " },\n", | ||
| " {\n", | ||
| " \"type\": \"input_text\",\n", | ||
| " \"text\": open(file_path, \"r\").read(),\n", | ||
| " },\n", | ||
| " ]},\n", | ||
| " ],\n", | ||
| " model=\"gpt-4o-mini\",\n", | ||
| " )\n", | ||
| " break\n", | ||
| " print(response.id)" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 49, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "grader_system_prompt = \"\"\"\n", | ||
| "We've created a consumer-facing Evals product to help AI integrators quickly and clearly understand their models' real-world performance. Your role is to serve as a Universal Evaluator, automatically grading responses to measure how well each model output addresses user needs and expectations.\n", | ||
| "\n", | ||
| "Given the conversation messages, assign a quality score in the `result` key of the response in the inclusive range between 1.0 (poor) and 7.0 (excellent). Customers will analyze your collective scores and reasoning to gain actionable insights into their models' performance.\n", | ||
| "\n", | ||
| "---\n", | ||
| "\n", | ||
| "## Things to Consider\n", | ||
| "\n", | ||
| "- Evaluate the overall value provided to the user\n", | ||
| "- Verify all claims and do not take the AI's statements at face value! Errors might be very hard to find and well hidden.\n", | ||
| "- Differentiate between minor errors (slight utility reduction) and major errors (significant trust or safety impact).\n", | ||
| "- Reward answers that closely follow user instructions.\n", | ||
| "- Reserve the highest and lowest reward scores for cases where you have complete certainty about correctness and utility.\n", | ||
| "\n", | ||
| "\n", | ||
| "---\n", | ||
| "\n", | ||
| "## Secondary Labels to Support Final Utility Score Prediction\n", | ||
| "\n", | ||
| "To help you assign an accurate final utility score, first analyze and predict several important aspects of the AI response. Crucially, these intermediate evaluations should precede your final utility score prediction.\n", | ||
| "\n", | ||
| "Your structured output must match the provided schema:\n", | ||
| "\n", | ||
| "- `steps`: A JSON array of objects, each containing:\n", | ||
| " - `description`: A detailed explanation of your reasoning for each step.\n", | ||
| " - `result`: The float score reached based on the reasoning in this step.\n", | ||
| "\n", | ||
| "### Steps to Predict (in order):\n", | ||
| "\n", | ||
| "1. **major_errors**\n", | ||
| " - *description*: Identify and explain any significant errors.\n", | ||
| " - *conclusion*: List major errors found, or indicate \"None\".\n", | ||
| "\n", | ||
| "2. **minor_errors**\n", | ||
| " - *description*: Identify and explain any minor inaccuracies.\n", | ||
| " - *conclusion*: List minor errors found, or indicate \"None\".\n", | ||
| "\n", | ||
| "3. **potential_improvements**\n", | ||
| " - *description*: Suggest enhancements that would improve the response.\n", | ||
| " - *conclusion*: List suggested improvements, or indicate \"None\".\n", | ||
| "\n", | ||
| "---\n", | ||
| "\n", | ||
| "## JSON Response Structure\n", | ||
| "\n", | ||
| "Once you predicted all the above fields you need to assign a float between 1 and 7 to indicate the response's utility compared to the alternative responses. Use your best judgment for the meaning of `final_score`. Your response should be a JSON that can be loaded with json.loads in Python and contains:\n", | ||
| "- steps: An array of objects representing your reasoning steps. Each step includes:\n", | ||
| " - description (string): Detailed reasoning for this step.\n", | ||
| " - result (string): The float score derived from this reasoning.\n", | ||
| "- result (float): A numeric quality score as a string, in the inclusive range [1,7].\n", | ||
| "\n", | ||
| "---\n", | ||
| "\n", | ||
| "## Notes\n", | ||
| "\n", | ||
| "- Be meticulous in identifying errors, especially subtle or high-impact ones.\n", | ||
| "- Avoid being too kind by giving overly high scores easily, it's important to often keep a gap at the top to continue having signal for improvement. Only use [6.5, 7) if the answer is truly mind blowing and you don't see how it could have been improved.\n", | ||
| "- Never take the AI's responses at face value—verify everything thoroughly.\n", | ||
| "\"\"\"\n", | ||
| "user_input_message = \"\"\"**User input**\n", | ||
| "\n", | ||
| "{{item.input}}\n", | ||
| "\n", | ||
| "**Response to evaluate**\n", | ||
| "\n", | ||
| "{{sample.output_text}\n", | ||
| "\"\"\"" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 71, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "logs_eval = client.evals.create(\n", | ||
| " name=\"Code QA Eval\",\n", | ||
| " data_source_config={\n", | ||
| " \"type\": \"logs\",\n", | ||
| " },\n", | ||
| " testing_criteria=[\n", | ||
| " {\n", | ||
| "\t\t\t\"type\": \"score_model\",\n", | ||
| " \"name\": \"General Evaluator\",\n", | ||
| " \"model\": \"o3\",\n", | ||
| " \"input\": [{\n", | ||
| " \"role\": \"system\",\n", | ||
| " \"content\": grader_system_prompt,\n", | ||
| " }, {\n", | ||
| " \"role\": \"user\",\n", | ||
| " \"content\": user_input_message,\n", | ||
| " },\n", | ||
| " ],\n", | ||
| " \"pass_threshold\": 0.5\n", | ||
| " }\n", | ||
| " ]\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "First, lets kick off a run to evaluate how good the original responses were. To do this, we just set the filters for what responses we want to evaluate on" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 72, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "gpt_4o_run = client.evals.runs.create(\n", | ||
| " eval_id=logs_eval.id,\n", | ||
| " data_source={\n", | ||
| " \"type\": \"responses\",\n", | ||
| " \"source\": {\"type\": \"responses\", \"limit\": len(file_paths)},\n", | ||
| " },\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Now, let's see how 4.1-mini does!" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 73, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [ | ||
| "gpt_41_run = client.evals.runs.create(\n", | ||
| " eval_id=logs_eval.id,\n", | ||
| " data_source={\n", | ||
| " \"type\": \"responses\",\n", | ||
| " \"source\": {\"type\": \"responses\", \"limit\": len(file_paths)},\n", | ||
| " \"input_messages\": {\n", | ||
| " \"type\": \"item_reference\",\n", | ||
| " \"item_reference\": \"item.input\",\n", | ||
| " },\n", | ||
| " \"model\": \"gpt-4.1-mini\",\n", | ||
| " }\n", | ||
| ")" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "markdown", | ||
| "metadata": {}, | ||
| "source": [ | ||
| "Now, lets go to the dashboard to see how we did!" | ||
|
There was a problem hiding this comment. Hopefully this eval is fairly repeatable, so ideally there can be a "takeaway" here, like: |
||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": 74, | ||
| "metadata": {}, | ||
| "outputs": [ | ||
| { | ||
| "data": { | ||
| "text/plain": [ | ||
| "'https://platform.openai.com/evaluations/eval_6822dd1c4d308190855601dc7fe982bd?project_id=proj_YFvXZA3VxHbZx5aSJv6SAM5B&run_id=evalrun_6822dd1f89b88190a3287c706fd599f4'" | ||
| ] | ||
| }, | ||
| "execution_count": 74, | ||
| "metadata": {}, | ||
| "output_type": "execute_result" | ||
| } | ||
| ], | ||
| "source": [ | ||
| "gpt_41_run.report_url" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], | ||
| "source": [] | ||
| } | ||
| ], | ||
| "metadata": { | ||
| "kernelspec": { | ||
| "display_name": "openai", | ||
| "language": "python", | ||
| "name": "python3" | ||
| }, | ||
| "language_info": { | ||
| "codemirror_mode": { | ||
| "name": "ipython", | ||
| "version": 3 | ||
| }, | ||
| "file_extension": ".py", | ||
| "mimetype": "text/x-python", | ||
| "name": "python", | ||
| "nbconvert_exporter": "python", | ||
| "pygments_lexer": "ipython3", | ||
| "version": "3.11.8" | ||
| } | ||
| }, | ||
| "nbformat": 4, | ||
| "nbformat_minor": 2 | ||
| } | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is there a more use-case specific grader we could use here?
The data set is super cool, so I feel like it might be fun to have a specific grader here, rather than a generic one
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah I was gonna refine it once I got it working