Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion examples/evaluation/use-cases/EvalsAPI_Image_Inputs.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -245,7 +245,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"To create the run, we pass in the eval object id, the data source (i.e., the data we compiled earlier), and the chat message input we will use for sampling to generate the model response. While we won't dive into it in this cookbook, EvalsAPI also supports stored completions containing images as a data source. \n",
"To create the run, we pass in the eval object id, the data source (i.e., the data we compiled earlier), and the chat message input we will use for sampling to generate the model response. Note that EvalsAPI also supports stored completions and responses containing images as a data source. See the [Additional Info: Logs Data Source](#additional-info-logs-data-source) section for more info.\n",
"\n",
"Here's the sampling message input we'll use for this example."
]
Expand Down Expand Up @@ -509,6 +509,22 @@
"print(json.dumps(dict(first_item), indent=2, default=str))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Additional Info: Logs Data Source\n",
"\n",
"As mentioned earlier, EvalsAPI supports logs (i.e., stored completions or responses) containing images as a data source. To use this functionality, change your eval configurations as follows: \n",
"\n",
"Eval Creation\n",
" - set `data_source_config = { \"type\": \"logs\" }`\n",
" - revise templating in `grader_config` to use `{{item.input}}` and/or `{{sample.output_text}}`, denoting the input and output of the log\n",
"\n",
"Eval Run Creation\n",
" - specify the filters in the `data_source` field that will be used to obtain the corresponding logs for the eval run (see the [docs](https://platform.openai.com/docs/api-reference/evals/createRun) for more information)"
]
},
{
"cell_type": "markdown",
"metadata": {},
Expand Down