wip

Author: willhath-openai

examples/evaluation/use-cases/responses-evaluation.ipynb

@@ -16,13 +16,14 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 68,
+   "execution_count": 30,
    "metadata": {},
    "outputs": [],
    "source": [
     "import openai\n",
     "import os\n",
     "\n",
+    "\n",
     "client = openai.OpenAI()"
    ]
   },
@@ -59,12 +60,12 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "Now, lets generate some responses"
+    "Now, lets generate some responses. "
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 66,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -85,89 +86,96 @@
     "        ],\n",
     "        model=\"gpt-4o-mini\",\n",
     "    )\n",
-    "    break\n",
-    "    print(response.id)"
+    "    print(response.output_text)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Note that in order for this to work, you'll have to be doing this on an org where data logging isn't disabled (through zdr, etc). If you aren't sure if this is the case for you, go to https://platform.openai.com/logs?api=responses and see if you can see the responses you just generated."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 49,
+   "execution_count": 31,
    "metadata": {},
    "outputs": [],
    "source": [
     "grader_system_prompt = \"\"\"\n",
-    "We've created a consumer-facing Evals product to help AI integrators quickly and clearly understand their models' real-world performance. Your role is to serve as a Universal Evaluator, automatically grading responses to measure how well each model output addresses user needs and expectations.\n",
-    "\n",
-    "Given the conversation messages, assign a quality score in the `result` key of the response in the inclusive range between 1.0 (poor) and 7.0 (excellent). Customers will analyze your collective scores and reasoning to gain actionable insights into their models' performance.\n",
-    "\n",
-    "---\n",
-    "\n",
-    "## Things to Consider\n",
-    "\n",
-    "- Evaluate the overall value provided to the user\n",
-    "-  Verify all claims and do not take the AI's statements at face value! Errors might be very hard to find and well hidden.\n",
-    "- Differentiate between minor errors (slight utility reduction) and major errors (significant trust or safety impact).\n",
-    "- Reward answers that closely follow user instructions.\n",
-    "- Reserve the highest and lowest reward scores for cases where you have complete certainty about correctness and utility.\n",
+    "You are **Code-Explanation Grader**, an expert software engineer and technical writer.  \n",
+    "Your job is to score how well *Model A* explained the purpose and behaviour of a given source-code file.\n",
     "\n",
+    "### What you receive\n",
+    "1. **File contents** – the full text of the code file (or a representative excerpt).  \n",
+    "2. **Candidate explanation** – the answer produced by Model A that tries to describe what the file does.\n",
     "\n",
-    "---\n",
+    "### What to produce\n",
+    "Return a single JSON object that can be parsed by `json.loads`, containing:\n",
+    "```json\n",
+    "{\n",
+    "  \"steps\": [\n",
+    "    { \"description\": \"...\", \"result\": \"float\" },\n",
+    "    { \"description\": \"...\", \"result\": \"float\" },\n",
+    "    { \"description\": \"...\", \"result\": \"float\" }\n",
+    "  ],\n",
+    "  \"result\": \"float\"\n",
+    "}\n",
+    "```\n",
+    "• Each object in `steps` documents your reasoning for one category listed under “Scoring dimensions”.  \n",
+    "• Place your final 1 – 7 quality score (inclusive) in the top-level `result` key as a **string** (e.g. `\"5.5\"`).\n",
     "\n",
-    "## Secondary Labels to Support Final Utility Score Prediction\n",
+    "### Scoring dimensions (evaluate in this order)\n",
     "\n",
-    "To help you assign an accurate final utility score, first analyze and predict several important aspects of the AI response. Crucially, these intermediate evaluations should precede your final utility score prediction.\n",
+    "1. **Correctness & Accuracy ≈ 45 %**  \n",
+    "   • Does the explanation match the actual code behaviour, interfaces, edge cases, and side effects?  \n",
+    "   • Fact-check every technical claim; penalise hallucinations or missed key functionality.\n",
     "\n",
-    "Your structured output must match the provided schema:\n",
+    "2. **Completeness & Depth ≈ 25 %**  \n",
+    "   • Are all major components, classes, functions, data flows, and external dependencies covered?  \n",
+    "   • Depth should be appropriate to the file’s size/complexity; superficial glosses lose points.\n",
     "\n",
-    "- `steps`: A JSON array of objects, each containing:\n",
-    "  - `description`: A detailed explanation of your reasoning for each step.\n",
-    "  - `result`: The float score reached based on the reasoning in this step.\n",
+    "3. **Clarity & Organization ≈ 20 %**  \n",
+    "   • Is the explanation well-structured, logically ordered, and easy for a competent developer to follow?  \n",
+    "   • Good use of headings, bullet lists, and concise language is rewarded.\n",
     "\n",
-    "### Steps to Predict (in order):\n",
+    "4. **Insight & Usefulness ≈ 10 %**  \n",
+    "   • Does the answer add valuable context (e.g., typical use cases, performance notes, risks) beyond line-by-line paraphrase?  \n",
+    "   • Highlighting **why** design choices matter is a plus.\n",
     "\n",
-    "1. **major_errors**\n",
-    "   - *description*: Identify and explain any significant errors.\n",
-    "   - *conclusion*: List major errors found, or indicate \"None\".\n",
+    "### Error taxonomy\n",
+    "• **Major error** – Any statement that materially misrepresents the file (e.g., wrong API purpose, inventing non-existent behaviour).  \n",
+    "• **Minor error** – Small omission or wording that slightly reduces clarity but doesn’t mislead.  \n",
+    "List all found errors in your `steps` reasoning.\n",
     "\n",
-    "2. **minor_errors**\n",
-    "   - *description*: Identify and explain any minor inaccuracies.\n",
-    "   - *conclusion*: List minor errors found, or indicate \"None\".\n",
+    "### Numeric rubric\n",
+    "1  Catastrophically wrong; mostly hallucination or irrelevant.  \n",
+    "2  Many major errors, few correct points.  \n",
+    "3  Several major errors OR pervasive minor mistakes; unreliable.  \n",
+    "4  Mostly correct but with at least one major gap or multiple minors; usable only with caution.  \n",
+    "5  Solid, generally correct; minor issues possible but no major flaws.  \n",
+    "6  Comprehensive, accurate, and clear; only very small nit-picks.  \n",
+    "7  Exceptional: precise, thorough, insightful, and elegantly presented; hard to improve.\n",
     "\n",
-    "3. **potential_improvements**\n",
-    "   - *description*: Suggest enhancements that would improve the response.\n",
-    "   - *conclusion*: List suggested improvements, or indicate \"None\".\n",
+    "Use the full scale. Reserve 6.5 – 7 only when you are almost certain the explanation is outstanding.\n",
     "\n",
-    "---\n",
+    "Then set `\"result\": \"4.0\"` (example).\n",
     "\n",
-    "## JSON Response Structure\n",
-    "\n",
-    "Once you predicted all the above fields you need to assign a float between 1 and 7 to indicate the response's utility compared to the alternative responses. Use your best judgment for the meaning of `final_score`. Your response should be a JSON that can be loaded with json.loads in Python and contains:\n",
-    "- steps: An array of objects representing your reasoning steps. Each step includes:\n",
-    "  - description (string): Detailed reasoning for this step.\n",
-    "  - result (string): The float score derived from this reasoning.\n",
-    "- result (float): A numeric quality score as a string, in the inclusive range [1,7].\n",
-    "\n",
-    "---\n",
-    "\n",
-    "## Notes\n",
-    "\n",
-    "- Be meticulous in identifying errors, especially subtle or high-impact ones.\n",
-    "- Avoid being too kind by giving overly high scores easily, it's important to often keep a gap at the top to continue having signal for improvement. Only use [6.5, 7) if the answer is truly mind blowing and you don't see how it could have been improved.\n",
-    "- Never take the AI's responses at face value—verify everything thoroughly.\n",
+    "Be rigorous and unbiased.\n",
     "\"\"\"\n",
     "user_input_message = \"\"\"**User input**\n",
     "\n",
     "{{item.input}}\n",
     "\n",
     "**Response to evaluate**\n",
     "\n",
-    "{{sample.output_text}\n",
+    "{{sample.output_text}}\n",
     "\"\"\""
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 71,
+   "execution_count": 25,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -189,7 +197,8 @@
     "                \"content\": user_input_message,\n",
     "            },\n",
     "            ],\n",
-    "            \"pass_threshold\": 0.5\n",
+    "            \"range\": [1, 7],\n",
+    "            \"pass_threshold\": 5.5,\n",
     "        }\n",
     "    ]\n",
     ")"
@@ -204,15 +213,16 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 72,
+   "execution_count": 26,
    "metadata": {},
    "outputs": [],
    "source": [
-    "gpt_4o_run = client.evals.runs.create(\n",
+    "gpt_4o_mini_run = client.evals.runs.create(\n",
+    "    name=\"gpt-4o-mini\",\n",
     "    eval_id=logs_eval.id,\n",
     "    data_source={\n",
     "        \"type\": \"responses\",\n",
-    "        \"source\": {\"type\": \"responses\", \"limit\": len(file_paths)},\n",
+    "        \"source\": {\"type\": \"responses\", \"limit\": len(file_paths)}, # just grab the most recent responses\n",
     "    },\n",
     ")"
    ]
@@ -226,11 +236,12 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 73,
+   "execution_count": 27,
    "metadata": {},
    "outputs": [],
    "source": [
-    "gpt_41_run = client.evals.runs.create(\n",
+    "gpt_41_mini_run = client.evals.runs.create(\n",
+    "    name=\"gpt-4.1-mini\",\n",
     "    eval_id=logs_eval.id,\n",
     "    data_source={\n",
     "        \"type\": \"responses\",\n",
@@ -253,35 +264,22 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 74,
+   "execution_count": null,
    "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "'https://platform.openai.com/evaluations/eval_6822dd1c4d308190855601dc7fe982bd?project_id=proj_YFvXZA3VxHbZx5aSJv6SAM5B&run_id=evalrun_6822dd1f89b88190a3287c706fd599f4'"
-      ]
-     },
-     "execution_count": 74,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
+   "outputs": [],
    "source": [
-    "gpt_41_run.report_url"
+    "gpt_4o_mini_run.report_url"
    ]
   },
   {
-   "cell_type": "code",
-   "execution_count": null,
+   "cell_type": "markdown",
    "metadata": {},
-   "outputs": [],
    "source": []
   }
  ],
  "metadata": {
   "kernelspec": {
-   "display_name": "openai",
+   "display_name": "Python 3",
    "language": "python",
    "name": "python3"
   },