condensing for cleanness

Author: corwin-oai

examples/Prompt_migration_guide.ipynb

@@ -705,9 +705,9 @@
    "source": [
     "### Current Example\n",
     "\n",
-    "Let’s evaluate whether our current working prompt has improved as a result of prompt migration. The original prompt, drawn from this [paper](https://arxiv.org/pdf/2306.05685), is designed to serve as a judge between two assistants’ answers. Conveniently, the paper provides a set of human-annotated judgments and assesses the LLM judge based on its agreement with these human ground truths.\n",
+    "Let’s evaluate whether our current prompt migration has actually improved for the task of this judge. The original prompt, drawn from this [paper](https://arxiv.org/pdf/2306.05685), is designed to serve as a judge between two assistants’ answers. Conveniently, the paper provides a set of human-annotated ground truths, so we can measure how often the LLM judge agrees with the humans judgments.\n",
     "\n",
-    "Our goal here is to measure how closely the judgments generated by our migrated prompt align with human evaluations. For context, the benchmark we’re using is a subset of MT-Bench, which features multi-turn conversations. In this example, we’re evaluating 200 conversation rows, each comparing the performance of different model pairs.\n",
+    "Thus, our metric of success will be measuring how closely the judgments generated by our migrated prompt align with human evaluations compared to the judgments generated with our baseline prompt. For context, the benchmark we’re using is a subset of MT-Bench, which features multi-turn conversations. In this example, we’re evaluating 200 conversation rows, each comparing the performance of different model pairs.\n",
     "\n"
    ]
   },
@@ -716,65 +716,31 @@
    "id": "6f50f9a0",
    "metadata": {},
    "source": [
-    "On our evaluation subset, a useful reference point is human-human agreement, since each conversation is rated by multiple annotators. For turn 1 (without ties), humans agree with each other in 81% of cases, and for turn 2, in 76% of cases."
+    "On our evaluation subset, a useful reference anchor is human-human agreement, since each conversation is rated by multiple annotators. Humans do not always agree with each other on which assistant answer is better, so we wouldn't expect our judge to achieve perfect agreement either. For turn 1 (without ties), humans agree with each other in 81% of cases, and for turn 2, in 76% of cases."
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "20e4d610",
-   "metadata": {},
-   "source": [
-    "![Graph 1 for Model Agreement](../images/prompt_migrator_fig1.png)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "a3eccb6c",
-   "metadata": {},
-   "source": [
-    "Comparing this to our models before migration, GPT-4 (as used in the paper) achieves an agreement with human judgments of 74% on turn 1 and 71% on turn 2, which is not bad, but still below the human-human upper bound."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "91dc3d38",
-   "metadata": {},
-   "source": [
-    "![Graph 2 for Model Agreement](../images/prompt_migrator_fig2.png)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "9f7e206f",
+   "id": "7af0337b",
    "metadata": {},
    "source": [
-    "\n",
-    "Switching to GPT-4.1 (using the same prompt) improves the agreement: 78% (65/83) on turn 1 and 72% (61/85) on turn 2."
+    "![Graph 3 for Model Agreement](../images/prompt_migrator_fig3.png)"
    ]
   },
   {
    "cell_type": "markdown",
    "id": "800da674",
    "metadata": {},
    "source": [
-    "\n",
-    "Finally, after migrating and tuning our prompt specifically for GPT-4.1, the agreement climbs further, reaching 80% on turn 1 and 72% on turn 2, very close to matching the level of agreement seen between human annotators."
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "id": "7af0337b",
-   "metadata": {},
-   "source": [
-    "![Graph 3 for Model Agreement](../images/prompt_migrator_fig3.png)"
+    "Comparing this to our models before migration, GPT-4 (as used in the paper) achieves an agreement with human judgments of 74% on turn 1 and 71% on turn 2, which is not bad, but still below the human-human ceiling. Switching to GPT-4.1 (using the same prompt) improves the agreement: 77% on turn 1 and 72% on turn 2. Finally, after migrating and tuning our prompt specifically for GPT-4.1, the agreement climbs further, reaching 80% on turn 1 and 72% on turn 2, very close to matching the level of agreement seen between human annotators."
    ]
   },
   {
    "cell_type": "markdown",
    "id": "43ae2ba5",
    "metadata": {},
    "source": [
-    "Viewed all together, we can see that prompt migration and model upgrades improve agreement on our sample task. Go ahead and try it on yours!"
+    "Viewed all together, we can see that prompt migration and upgrading to more powerful models improve agreement on our sample task. Considering that the baseline prompt and the task were on the simpler side, there is less optimization to squeeze out of tuning the 4.1 instructions. However, on more complex tasks, you can expect a much larger bump! Go ahead and try it on your prompt now!"
    ]
   },
   {