|
6 | 6 | "source": [
|
7 | 7 | "# Image Understanding with RAG using OpenAI's Vision & Responses APIs\n",
|
8 | 8 | "\n",
|
9 |
| - "Welcome! This notebook demonstrates how to build a Retrieval-Augmented Generation (RAG) system using OpenAI’s Vision and Responses APIs. It focuses on multimodal data, specifically, combining image and text inputs to analyze customer experiences. The system leverages GPT-4.1 and integrates image understanding with file search to provide context-aware responses.\n", |
| 9 | + "Welcome! This notebook demonstrates how to build a Retrieval-Augmented Generation (RAG) system using OpenAI’s Vision and Responses APIs. It focuses on multimodal data, combining image and text inputs to analyze customer experiences. The system leverages GPT-4.1 and integrates image understanding with file search to provide context-aware responses.\n", |
10 | 10 | "\n",
|
11 | 11 | "Multimodal datasets are increasingly common, particularly in domains like healthcare, where records often contain both visual data (e.g. radiology scans) and accompanying text (e.g. clinical notes). Real-world datasets also tend to be noisy, with incomplete or missing information, making it critical to analyze multiple modalities in tandem.\n",
|
12 | 12 | "\n",
|
13 |
| - "This guide focuses on a customer service use case: evaluating customer feedback that may include screenshots, photos, and written complaints. You’ll learn how to synthetically generate both image and text inputs, use file search for context retrieval, and apply the Evals API to assess how incorporating image understanding impacts overall performance.\n", |
| 13 | + "This guide focuses on a customer service use case: evaluating customer feedback that may include photos, and written reviews. You’ll learn how to synthetically generate both image and text inputs, use file search for context retrieval, and apply the Evals API to assess how incorporating image understanding impacts overall performance.\n", |
14 | 14 | "\n",
|
15 | 15 | "---\n",
|
16 | 16 | "\n",
|
|
251 | 251 | "cell_type": "markdown",
|
252 | 252 | "metadata": {},
|
253 | 253 | "source": [
|
254 |
| - "This example uses OpenAI's built-in vector store and file search capabilities to build a RAG system that can analyse customer experiences, from their feedback which can be both visual and text-based. We create two vector stores for comparisons, one with image understanding and one without." |
| 254 | + "This example uses OpenAI's built-in vector store and file search capabilities to build a RAG system that can analyse customer experiences from their feedback, which can be both visual and text-based. We create two vector stores for comparisons, one with image understanding and one without." |
255 | 255 | ]
|
256 | 256 | },
|
257 | 257 | {
|
|
323 | 323 | "outputs": [],
|
324 | 324 | "source": [
|
325 | 325 | "upload_files_to_vector_store(text_image_vector_store_id, df)\n",
|
326 |
| - "upload_files_to_vector_store(text_vector_store_id, df, column_name=\"text\") " |
| 326 | + "upload_files_to_vector_store(text_vector_store_id, df, column_name=\"text\")" |
327 | 327 | ]
|
328 | 328 | },
|
329 | 329 | {
|
|
332 | 332 | "source": [
|
333 | 333 | "# Retrieval and Filtering\n",
|
334 | 334 | "\n",
|
335 |
| - "We can analyse our dataset with natural language queries with the help of File Search. For the text-only dataset, we see that information is missing that could inform our analysis.\n" |
336 |
| - ] |
337 |
| - }, |
338 |
| - { |
339 |
| - "cell_type": "markdown", |
340 |
| - "metadata": {}, |
341 |
| - "source": [ |
342 |
| - "The only positive review for spaghetti in July has visual feedback and we can see the RAG system with only text based context available is uncertain about positive details. However with image context provided the second RAG system is able to provide a more accurate response." |
| 335 | + "We can analyse our dataset with natural language queries with the help of File Search. For the text-only dataset, we see that information is missing that could inform our analysis.\n", |
| 336 | + "\n", |
| 337 | + "The only positive review for spaghetti in July has visual feedback and we can see the RAG system with only text based context available is uncertain about positive details. However with image context provided the second RAG system is able to provide a more accurate response.\n" |
343 | 338 | ]
|
344 | 339 | },
|
345 | 340 | {
|
|
461 | 456 | "cell_type": "markdown",
|
462 | 457 | "metadata": {},
|
463 | 458 | "source": [
|
464 |
| - "Likewise we can test this for negative reviews in June." |
| 459 | + "Likewise we can test this for negative reviews in June concerning any burnt pizza." |
465 | 460 | ]
|
466 | 461 | },
|
467 | 462 | {
|
|
665 | 660 | "metadata": {},
|
666 | 661 | "outputs": [],
|
667 | 662 | "source": [
|
| 663 | + "# you may need to wait a few seconds before running this cell for the eval runs to finish up\n", |
| 664 | + "\n", |
668 | 665 | "text_only_run_output_items = client.evals.runs.output_items.list(eval_id=eval_id, run_id=text_only_run_id)\n",
|
669 | 666 | "text_image_run_output_items = client.evals.runs.output_items.list(eval_id=eval_id, run_id=text_image_run_id)"
|
670 | 667 | ]
|
|
772 | 769 | ")\n",
|
773 | 770 | "print(deleted_vector_store)"
|
774 | 771 | ]
|
775 |
| - }, |
776 |
| - { |
777 |
| - "cell_type": "code", |
778 |
| - "execution_count": null, |
779 |
| - "metadata": {}, |
780 |
| - "outputs": [], |
781 |
| - "source": [] |
782 | 772 | }
|
783 | 773 | ],
|
784 | 774 | "metadata": {
|
|
0 commit comments