Update to use model grader

Author: hendrytl

examples/evaluation/use-cases/EvalsAPI_Audio_Inputs.ipynb

@@ -6,9 +6,9 @@
    "source": [
     "# Evals API: Audio Inputs\n",
     "\n",
-    "This cookbook demonstrates how to use OpenAI's Evals framework for audio-based tasks. Leveraging the Evals API, we will grade model-generated responses to an audio message and prompt by using **sampling** to generate model responses and **string match grading** to score the model responses against the output audio transcript and reference answer. Note that grading will be on text outputs from the sampled response. Graders that can grade audio input are not currently supported.\n",
+    "This cookbook demonstrates how to use OpenAI's Evals framework for audio-based tasks. Leveraging the Evals API, we will grade model-generated responses to an audio message and prompt by using **sampling** to generate model responses and **model grading** to score the model responses against the output audio and reference answer. Note that grading will be on audio outputs from the sampled response.\n",
     "\n",
-    "Before audio support was added, in order to evaluate audio conversations, they needed to be first transcribed to text.  Now you can use the original audio and get samples from the model in audio as well. This will more accurately represent workflows such as a customer support agent where both the user and agent are using audio. For grading, we use the text transcript from the sampled audio so that we can leverage the existing suite of text graders. \n",
+    "Before audio support was added, in order to evaluate audio conversations, they needed to be first transcribed to text.  Now you can use the original audio and get samples from the model in audio as well. This will more accurately represent workflows such as a customer support agent where both the user and agent are using audio. For grading, we will use an audio model to grade the audio response with a model grader.  We could alternatively or in combination use the text transcript from the sampled audio and leverage the existing suite of text graders. \n",
     "\n",
     "In this example, we will evaluate how well our model can:\n",
     "1. **Generate appropriate responses** to user prompts about an audio message\n",
@@ -29,7 +29,7 @@
    "outputs": [],
    "source": [
     "# Install required packages\n",
-    "%pip install openai datasets pandas soundfile torch torchcodec --quiet"
+    "%pip install openai datasets pandas soundfile torch torchcodec pydub jiwer --quiet"
    ]
   },
   {
@@ -285,9 +285,9 @@
    "cell_type": "markdown",
    "metadata": {},
    "source": [
-    "For our testing criteria, we set up our grader config. In this example, it is a simple string_check grader that takes in the official answer and sampled model response (in the `sample` namespace), and then outputs a score between 0 and 1 based if the model response contains the reference answer. The response contains both audio and the text transcript of the audio. We will use the text transcript in the grader. For more info on graders, visit [API Grader docs](https://platform.openai.com/docs/api-reference/graders). \n",
+    "For our testing criteria, we set up our grader config. In this example, we use a score_model grader that takes in the official answer and sampled model response (in the `sample` namespace), and then outputs a score of 0 or 1 based if the model response matches the official answer. The response contains both audio and the text transcript of the audio. We will use the audio in the grader. For more info on graders, visit [API Grader docs](https://platform.openai.com/docs/api-reference/graders). \n",
     "\n",
-    "Getting both the data and the grader right are key for an effective evaluation. While this example uses a simple string check grader, a more powerful model grader could be used instead and you will likely want to iteratively refine the prompts for your graders. "
+    "Getting both the data and the grader right are key for an effective evaluation. You will likely want to iteratively refine the prompts for your graders. "
    ]
   },
   {
@@ -296,13 +296,48 @@
    "metadata": {},
    "outputs": [],
    "source": [
+    "grader_config = {\n",
+    "  \"type\": \"score_model\",\n",
+    "  \"name\": \"Reference answer audio model grader\",\n",
+    "  \"model\": \"gpt-4o-audio-preview\",\n",
+    "  \"input\": [\n",
+    "              {\n",
+    "                  \"role\": \"user\",\n",
+    "                  \"content\": [\n",
+    "                      {\n",
+    "                          \"type\": \"input_text\",\n",
+    "                          \"text\": \"Evaluate this audio clip to see if it reaches the same conclusion as the reference answer. Score the answer a 1 if it does, 0 if it does not. Reference answer: {{item.official_answer}}\",\n",
+    "                      },\n",
+    "                      {\n",
+    "                          \"type\": \"input_audio\",\n",
+    "                          \"input_audio\": {\n",
+    "                              \"data\": \"{{ sample.output_audio.data }}\",\n",
+    "                              \"format\": \"wav\",\n",
+    "                          },\n",
+    "                      },\n",
+    "                  ],\n",
+    "              },\n",
+    "          ],\n",
+    "          \"range\": [0, 1],\n",
+    "          \"pass_threshold\": 0.9,\n",
+    "}"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Alternatively we could use a string_check grader that takes in the official answer and sampled model response (in the `sample` namespace), and then outputs a score between 0 and 1 based on if the model response contains the reference answer. The response contains both audio and the text transcript of the audio. We will use the text transcript in the grader. \n",
+    "\n",
+    "```python\n",
     "grader_config = {\n",
     "  \"type\": \"string_check\",\n",
     "  \"name\": \"String check grader\",\n",
     "  \"input\": \"{{sample.output_text}}\",\n",
     "  \"reference\": \"{{item.official_answer}}\",\n",
     "  \"operation\": \"ilike\"\n",
-    "}"
+    "}\n",
+    "```"
    ]
   },
   {
@@ -401,8 +436,9 @@
     "            },\n",
     "            \"input_messages\": {\n",
     "                \"type\": \"template\", \n",
-    "                \"template\": sampling_messages}\n",
-    "        }\n",
+    "                \"template\": sampling_messages},\n",
+    "            \"modalities\": [\"audio\", \"text\"],\n",
+    "        },\n",
     "    )"
    ]
   },
@@ -479,10 +515,9 @@
    "source": [
     "## Conclusion\n",
     "\n",
-    "In this cookbook, we covered a workflow for evaluating native audio inputs to a model using the OpenAI Evals APIs. We demonstrated using a simple text grader to grade the text transcript of the audio response.\n",
+    "In this cookbook, we covered a workflow for evaluating native audio inputs to a model using the OpenAI Evals APIs. We demonstrated using a score model grader to grade the audio response.\n",
     "### Next steps\n",
     "- Convert this example to your use case.  \n",
-    "- Try using model based graders for additional flexibility in grading.\n",
     "- If you have large audio clips, try using the [uploads API](https://platform.openai.com/docs/api-reference/uploads/create) for support up to 8 GB.\n"
    ]
   }