Skip to content

Commit 7cfb312

Browse files
committed
address review comments
1 parent c01cd5e commit 7cfb312

File tree

1 file changed

+5
-7
lines changed

1 file changed

+5
-7
lines changed

examples/evaluation/use-cases/EvalsAPI_Audio_Inputs.ipynb

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -6,11 +6,11 @@
66
"source": [
77
"# Evals API: Audio Inputs\n",
88
"\n",
9-
"This cookbook demonstrates how to use OpenAI's Evals framework for audio-based tasks. Leveraging the Evals API, we will grade model-generated responses to an audio message and prompt by using **sampling** to generate model responses and **model grading** (LLM as a Judge) to score the model responses against the output audio transcript, prompt, and reference answer.\n",
9+
"This cookbook demonstrates how to use OpenAI's Evals framework for audio-based tasks. Leveraging the Evals API, we will grade model-generated responses to an audio message and prompt by using **sampling** to generate model responses and **model grading** (LLM as a Judge) to score the model responses against the output audio transcript, prompt, and reference answer. Note that grading will be on text outputs from the sampled response. Graders that can grade audio input are not currently supported.\n",
1010
"\n",
1111
"In this example, we will evaluate how well our model can:\n",
1212
"1. **Generate appropriate responses** to user prompts about an audio message\n",
13-
"3. **Align with reference answers** that represent high-quality responses"
13+
"2. **Align with reference answers** that represent high-quality responses"
1414
]
1515
},
1616
{
@@ -284,9 +284,9 @@
284284
"cell_type": "markdown",
285285
"metadata": {},
286286
"source": [
287-
"For our testing criteria, we set up our grader config. In this example, it is a simple string_check grader that takes in the official answer and sampled model response (in the `sample` namespace), and then outputs a score between 0 and 1 based if the model response matches the reference answer. For more info on graders, visit [API Grader docs](https://platform.openai.com/docs/api-reference/graders). \n",
287+
"For our testing criteria, we set up our grader config. In this example, it is a simple string_check grader that takes in the official answer and sampled model response (in the `sample` namespace), and then outputs a score between 0 and 1 based if the model response contains the reference answer. The response contains both audio and the text transcript of the audio. We will use the text transcript in the grader. For more info on graders, visit [API Grader docs](https://platform.openai.com/docs/api-reference/graders). \n",
288288
"\n",
289-
"Getting the both the data and the grader right are key for an effective evaluation. So, you will likely want to iteratively refine the prompts for your graders. "
289+
"Getting both the data and the grader right are key for an effective evaluation. While this example uses a simple string check grader, a more powerful model grader could be used instead and you will likely want to iteratively refine the prompts for your graders. "
290290
]
291291
},
292292
{
@@ -467,9 +467,7 @@
467467
"source": [
468468
"## Conclusion\n",
469469
"\n",
470-
"In this cookbook, we covered a workflow for evaluating an audio-based task using the OpenAI Evals API's. By using the audio input functionality, we were able to streamline our evals process for the task. It could also be useful to use the audio transcript as input to a model grader for additional flexibility in grading the response.\n",
471-
"\n",
472-
"We're excited to see you extend this to your own audio-based use cases!"
470+
"In this cookbook, we covered a workflow for evaluating native audio inputs to a model using the OpenAI Evals API's. We could additionally add model based graders for additional flexibility in grading in future.\n"
473471
]
474472
}
475473
],

0 commit comments

Comments
 (0)