eval driven system design cookbook updates

Author: shikhar-cyber

examples/partners/eval_driven_system_design/receipt_inspection.ipynb

@@ -112,8 +112,15 @@
    "source": [
     "## Project Lifecycle\n",
     "\n",
-    "Not every project will proceed in the same way, but projects generally have some common\n",
-    "important components.\n",
+    "Not every project will proceed in the same way, but projects generally have some \n",
+    "important components in common.\n",
+    "\n",
+    "![Project Lifecycle](../../../images/partner_project_lifecycle.png)\n",
+    "\n",
+    "The solid arrows show the primary progressions or steps, while the dotted line \n",
+    "represents the ongoing nature of problem understanding - uncovering more about\n",
+    "the customer domain will influence every step of the process. We wil examine \n",
+    "several of these iterative cycles of refinement in detail below. \n",
     "\n",
     "### 1. Understand the Problem\n",
     "\n",
@@ -133,10 +140,11 @@
     "It's very rare that a real-world project will start with all the data necessary to get\n",
     "to a satisfactory solution, much less to establish confidence.\n",
     "\n",
-    "In our case, we're going to assume that we have a decent sample of system *inputs*\n",
-    "(here, photographs of receipts), but start without any fully annotated data. We'll walk\n",
-    "through the process of incrementally expanding our test and training sets as we go along\n",
-    "and make our evals progressively more comprehensive.\n",
+    "In our case, we're going to assume that we have a decent sample of system *inputs*, \n",
+    "in the form of but receipt images, but start without any fully annotated data. We find \n",
+    "this is a not-unusual situation when automating an existing process. Instead, \n",
+    "we'll walk through the process of building that out as we go along by collaborating with\n",
+    "domain experts, and make our evals progressively more comprehensive.\n",
     "\n",
     "### 3. Build an End-to-End V0 System\n",
     "\n",
@@ -394,7 +402,7 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "<img src=\"../../../images/Supplies_20240322_220858_Raven_Scan_3_jpeg.rf.50852940734939c8838819d7795e1756.jpg\" alt=\"Walmart_image\" width=\"400\"/>"
+    "![Walmart_image](../../../images/Supplies_20240322_220858_Raven_Scan_3_jpeg.rf.50852940734939c8838819d7795e1756.jpg)"
    ]
   },
   {
@@ -497,8 +505,22 @@
    "source": [
     "### Action Decision\n",
     "\n",
-    "Next, we need to close the loop and get to an actual decision based on receipts. This\n",
-    "looks pretty similar, so we'll present the code without comment."
+    "Next, we need to close the loop and get to an actual decision based on receipts. \n",
+    "\n",
+    "Ordinarily one would start with the most capable model - `o3`, at this time - for a \n",
+    "first pass, and then once correctness is established experiment with different models\n",
+    "to analyze any tradeoffs for their business impact, and potentially consider whether \n",
+    "they are remediable with iteration. A client may be willing to take a certain accuracy \n",
+    "hit for lower latency or cost, or it may be more effective to change the architecture\n",
+    "to hit cost, latency, and accuracy goals. We'll get into how to make these tradeoffs\n",
+    "explicitly and objectively later on. \n",
+    "\n",
+    "For this cookbook, `o3` might be too good. We'll use `o4-mini` for our first pass, so \n",
+    "that we get a few reasoning errors we can use to illustrate the means of addressing\n",
+    "them when they occur.\n",
+    "\n",
+    "Otherwise, this is pretty similar to the last, so we'll present the code without \n",
+    "further comment."
    ]
   },
   {
@@ -887,7 +909,10 @@
    "metadata": {},
    "source": [
     "After you run that eval you'll be able to view it in the UI, and should see something\n",
-    "like:\n",
+    "like the below. \n",
+    "\n",
+    "(Note, if you have a Zero-Data-Retention agreement, this data is not stored\n",
+    "by OpenAI, so will not be available in this interface.)\n",
     "\n",
     "![Summary UI](../../../images/partner_summary_ui.png)\n",
     "\n",
@@ -1617,7 +1642,7 @@
     "ARE NOT TRAVEL-RELATED, THEN IT MUST BE AUDITED.\n",
     "```\n",
     "\n",
-    "3. We added three examples, JSON input/output pairs wrapped in XML tags.\n",
+    "4. We added three examples, JSON input/output pairs wrapped in XML tags.\n",
     "\n",
     "With our prompt revisions, we'll regenerate the data to evaluate and re-run the same\n",
     "eval to compare our results:"

images/partner_development_flywheel.png

@@ -9,8 +9,13 @@
   date: 2025-06-01
   authors:
     - shikhar-cyber
+    - moredatarequired
+    - tooluser
+    - eddiesiegel
   tags:
     - evals
+    - API Flywheel
+    - completions
     - responses
     - functions
     - tracing