added evals section

Author: corwin-oai

examples/Optimize Prompts.ipynb

@@ -6,7 +6,7 @@
    "source": [
     "# Optimize Prompts\n",
     "\n",
-    "This cookbook provides a look into an early version of OpenAI's prompt optimization system. Crafting effective prompts is a critical skill when working with AI models. Even experienced users can inadvertently introduce contradictions, ambiguities, or inconsistencies that lead to suboptimal results. The system demonstrated here helps identify and fix common issues, resulting in more reliable and effective prompts.\n",
+    "Crafting effective prompts is a critical skill when working with AI models. Even experienced users can inadvertently introduce contradictions, ambiguities, or inconsistencies that lead to suboptimal results. The system demonstrated here helps identify and fix common issues, resulting in more reliable and effective prompts.\n",
     "\n",
     "The optimization process uses a multi-agent approach with specialized AI agents collaborating to analyze and rewrite prompts. The system automatically identifies and addresses several types of common issues:\n",
     "\n",
@@ -16,16 +16,27 @@
     "\n",
     "---\n",
     "\n",
-    "**Objective**: This notebook demonstrates best practices for creating a useful and robust agent system and helps you develop more effective prompts for your applications.\n",
+    "**Objective**: This cookbook demonstrates best practices for using Agents SDK together with Evals to build an early version of OpenAI's prompt optimization system. You can optimize your prompt using this code or use the optimizer [in our playground!](https://platform.openai.com/playground/prompts)\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "\n",
+    "Ask ChatGPT\n",
+    "\n",
     "\n",
     "**Cookbook Structure**  \n",
     "This notebook follows this structure:\n",
     "\n",
     "- [Step 1. System Overview](#1-system-overview) - Learn how the prompt optimization system works  \n",
     "- [Step 2. Data Models](#2-data-models) - Understand the data structures used by the system\n",
     "- [Step 3. Defining the Agents](#3-defining-the-agents) - Look at agents that analyze and improve prompts\n",
-    "- [Step 4. Run Optimization Workflow](#4-run-optimization-workflow) - See how the workflow hands off the prompts\n",
-    "- [Step 5. Examples](#5-examples) - Explore real-world examples of prompt optimization\n",
+    "- [Step 4. Evaluations](#4-using-evaluations-to-arrive-at-these-agents) - Use Evals to verify our agent model choice and instructions\n",
+    "- [Step 5. Run Optimization Workflow](#4-run-optimization-workflow) - See how the workflow hands off the prompts\n",
+    "- [Step 6. Examples](#5-examples) - Explore real-world examples of prompt optimization\n",
     "\n",
     "**Prerequisites**\n",
     "- The `openai` Python package \n",
@@ -173,7 +184,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 7,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
@@ -182,7 +193,7 @@
     "    model=\"gpt-4.1\",\n",
     "    output_type=Issues,\n",
     "    instructions=\"\"\"\n",
-    "    You are **Dev-Contradiction-Checker-v2**.\n",
+    "    You are **Dev-Contradiction-Checker**.\n",
     "\n",
     "    Goal\n",
     "    Detect *genuine* self-contradictions or impossibilities **inside** the developer prompt supplied in the variable `DEVELOPER_MESSAGE`.\n",
@@ -215,7 +226,7 @@
     "    model=\"gpt-4.1\",\n",
     "    output_type=Issues,\n",
     "    instructions=\"\"\"\n",
-    "    You are Format-Checker-v2.\n",
+    "    You are Format-Checker.\n",
     "\n",
     "    Task\n",
     "    Decide whether the developer prompt requires a structured output (JSON/CSV/XML/Markdown table, etc.).\n",
@@ -246,7 +257,7 @@
     "    model=\"gpt-4.1\",\n",
     "    output_type=FewShotIssues,\n",
     "    instructions=\"\"\"\n",
-    "    You are FewShot-Consistency-Checker-v3.\n",
+    "    You are FewShot-Consistency-Checker.\n",
     "\n",
     "    Goal\n",
     "    Find conflicts between the DEVELOPER_MESSAGE rules and the accompanying **assistant** examples.\n",
@@ -306,7 +317,7 @@
     "    model=\"gpt-4.1\",\n",
     "    output_type=DevRewriteOutput,\n",
     "    instructions=\"\"\"\n",
-    "    You are Dev-Rewriter-v2.\n",
+    "    You are Dev-Rewriter.\n",
     "\n",
     "    You receive:\n",
     "    - ORIGINAL_DEVELOPER_MESSAGE\n",
@@ -338,7 +349,7 @@
     "    model=\"gpt-4.1\",\n",
     "    output_type=MessagesOutput,\n",
     "    instructions=\"\"\"\n",
-    "    You are FewShot-Rewriter-v2.\n",
+    "    You are FewShot-Rewriter.\n",
     "\n",
     "    Input payload\n",
     "    - NEW_DEVELOPER_MESSAGE (already optimized)\n",
@@ -373,9 +384,129 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 4. Run Optimization Workflow\n",
+    "## 4. Using Evaluations to Arrive at these Agents\n",
+    "\n",
+    "Let's see how we used our Evals to tune agent prompts + pick models. We constructed a set of golden examples: each one contains original messages (developer message + user/assistant message) and the changes our optimization workflow should make. Here are two example of golden pairs that we used:"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "vscode": {
+     "languageId": "javascript"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "[\n",
+    "    {\n",
+    "      \"focus\": \"contradiction_issues\",\n",
+    "      \"input_payload\": {\n",
+    "        \"developer_message\": \"Always answer in **English**.\\nNunca respondas en inglés.\",\n",
+    "        \"messages\": [\n",
+    "          {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"¿Qué hora es?\"\n",
+    "          }\n",
+    "        ]\n",
+    "      },\n",
+    "      \"golden_output\": {\n",
+    "        \"changes\": true,\n",
+    "        \"new_developer_message\": \"Always answer **in English**.\",\n",
+    "        \"new_messages\": [\n",
+    "          {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"¿Qué hora es?\"\n",
+    "          }\n",
+    "        ],\n",
+    "        \"contradiction_issues\": \"Developer message simultaneously insists on English and forbids it.\",\n",
+    "        \"few_shot_contradiction_issues\": \"\",\n",
+    "        \"format_issues\": \"\",\n",
+    "        \"general_improvements\": \"\"\n",
+    "      }\n",
+    "    },\n",
+    "    {\n",
+    "      \"focus\": \"few_shot_contradiction_issues\",\n",
+    "      \"input_payload\": {\n",
+    "        \"developer_message\": \"Respond with **only 'yes' or 'no'** – no explanations.\",\n",
+    "        \"messages\": [\n",
+    "          {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"Is the sky blue?\"\n",
+    "          },\n",
+    "          {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Yes, because wavelengths …\"\n",
+    "          },\n",
+    "          {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"Is water wet?\"\n",
+    "          },\n",
+    "          {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"Yes.\"\n",
+    "          }\n",
+    "        ]\n",
+    "      },\n",
+    "      \"golden_output\": {\n",
+    "        \"changes\": true,\n",
+    "        \"new_developer_message\": \"Respond with **only** the single word \\\"yes\\\" or \\\"no\\\".\",\n",
+    "        \"new_messages\": [\n",
+    "          {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"Is the sky blue?\"\n",
+    "          },\n",
+    "          {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"yes\"\n",
+    "          },\n",
+    "          {\n",
+    "            \"role\": \"user\",\n",
+    "            \"content\": \"Is water wet?\"\n",
+    "          },\n",
+    "          {\n",
+    "            \"role\": \"assistant\",\n",
+    "            \"content\": \"yes\"\n",
+    "          }\n",
+    "        ],\n",
+    "        \"contradiction_issues\": \"\",\n",
+    "        \"few_shot_contradiction_issues\": \"Assistant examples include explanations despite instruction not to.\",\n",
+    "        \"format_issues\": \"\",\n",
+    "        \"general_improvements\": \"\"\n",
+    "      }\n",
+    "    }\n",
+    "  ]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "From these 20 hand labelled golden outputs which cover a range of contradiction issues, few shot issues, format issues, no issues, or a combination of issues, we built a python string check grader to verify two things: whether an issue was detected for each golden pair and whether the detected issue matched the expected one. From this signal, we tuned the agent instructions and which model to use to maximize our accuracy across this evaluation. We landed on the 4.1 model as a balance between accuracy, cost, and speed. The specific prompts we used also follow the 4.1 prompting guide. As you can see, we achieve the correct labels on all 20 golden outputs: identifying the right issues and avoiding false positives. "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Accuracy for the golden set](../images/optimizepromptfig1.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "![Evaluation for the golden set](../images/optimizepromptfig2.png)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## 5. Run Optimization Workflow\n",
     "\n",
-    "Let's dive into how the optimization system actually works. The core workflow consists of multiple runs of the agents in parallel to efficiently process and optimize prompts."
+    "Let's dive into how the optimization system actually works end to end. The core workflow consists of multiple runs of the agents in parallel to efficiently process and optimize prompts."
    ]
   },
   {
@@ -484,7 +615,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "## 5. Examples\n",
+    "## 6. Examples\n",
     "\n",
     "Let's see the optimization system in action with some practical examples."
    ]