Merge pull request #657 from HeidiSteen/heidist-rag

[azure search} RAG tutorial updates per MattG updates

Author: denrea

articles/search/tutorial-rag-build-solution-index-schema.md

@@ -8,7 +8,7 @@ author: HeidiSteen
 ms.author: heidist
 ms.service: cognitive-search
 ms.topic: tutorial
-ms.date: 10/01/2024
+ms.date: 10/04/2024
 
 ---
 
@@ -43,7 +43,7 @@ Chunks are the focus of the schema, and each chunk is the defining element of a
 
 ### Enhanced with generated data
 
-In this tutorial, sample data consists of PDFs and content from the [NASA Earth Book](https://www.nasa.gov/ebooks/earth/). This content is descriptive and informative, with numerous references to geographies, countries, and areas across the world. All of the textual content is captured in chunks, but these recurring instances of place names create an opportunity for adding structure to the index. Using skills, it's possible to recognize entities in the text and capture them in an index for use in queries and filters. In this tutorial, we include an [entity recognition skill](cognitive-search-skill-entity-recognition-v3.md) that recognizes and extracts location entities, loading it into a searchable and filterable `locations` field. Adding structured content to your index gives you more options for filtering, improved relevance, and more focused answers.
+In this tutorial, sample data consists of PDFs and content from the [NASA Earth Book](https://www.nasa.gov/ebooks/earth/). This content is descriptive and informative, with numerous references to geographies, countries, and areas across the world. All of the textual content is captured in chunks, but recurring instances of place names create an opportunity for adding structure to the index. Using skills, it's possible to recognize entities in the text and capture them in an index for use in queries and filters. In this tutorial, we include an [entity recognition skill](cognitive-search-skill-entity-recognition-v3.md) that recognizes and extracts location entities, loading it into a searchable and filterable `locations` field. Adding structured content to your index gives you more options for filtering, improved relevance, and more focused answers.
 
 ### Parent-child fields in one or two indexes?
 
@@ -61,11 +61,11 @@ In Azure AI Search, an index that works best for RAG workloads has these qualiti
 
 - Maintains a parent-child relationship between chunks of a document and the properties of the parent document, such as the file name, file type, title, author, and so forth. To answer a query, chunks could be pulled from anywhere in the index. Association with the parent document providing the chunk is useful for context, citations, and follow up queries.
 
-- Accommodates the queries you want create. You should have fields for vector and hybrid content, and those fields should be attributed to support specific query behaviors. You can only query one index at a time (no joins) so your fields collection should define all of your searchable content.
+- Accommodates the queries you want create. You should have fields for vector and hybrid content, and those fields should be attributed to support specific query behaviors, such as searchable or filterable. You can only query one index at a time (no joins) so your fields collection should define all of your searchable content.
 
 - Your schema should be flat (no complex types or structures). This requirement is specific to the RAG pattern in Azure AI Search.
 
-Although Azure AI Search can't join indexes, you can create indexes that preserve parent-child relationship, and then use sequential or parallel queries in your search logic to pull from both. This exercise includes templates for parent-child elements in the same index and in separate indexes, where information from the parent index is retrieved using a lookup query.
+<!-- Although Azure AI Search can't join indexes, you can create indexes that preserve parent-child relationship, and then use sequential queries in your search logic to pull from both (a query on the chunked data index, a lookup on the parent index). This exercise includes templates for parent-child elements in the same index and in separate indexes, where information from the parent index is retrieved using a lookup query. -->
 
 <!-- > [!NOTE]
 > Schema design affects storage and costs. This exercise is focused on schema fundamentals. In the [Minimize storage and costs](tutorial-rag-build-solution-minimize-storage.md) tutorial, you revisit schema design to consider narrow data types, attribution, and vector configurations that offer more efficient. -->
@@ -136,7 +136,7 @@ A minimal index for LLM is designed to store chunks of content. It typically inc
         SearchField(name="locations", type=SearchFieldDataType.Collection(SearchFieldDataType.String), filterable=True),
         SearchField(name="chunk_id", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True, analyzer_name="keyword"),  
         SearchField(name="chunk", type=SearchFieldDataType.String, sortable=False, filterable=False, facetable=False),  
-        SearchField(name="text_vector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), vector_search_dimensions=1536, vector_search_profile_name="myHnswProfile")
+        SearchField(name="text_vector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), vector_search_dimensions=1024, vector_search_profile_name="myHnswProfile")
         ]  
 
     # Configure the vector search configuration  
@@ -157,8 +157,8 @@ A minimal index for LLM is designed to store chunks of content. It typically inc
                 kind="azureOpenAI",  
                 parameters=AzureOpenAIVectorizerParameters(  
                     resource_url=AZURE_OPENAI_ACCOUNT,  
-                    deployment_name="text-embedding-ada-002",
-                    model_name="text-embedding-ada-002"
+                    deployment_name="text-embedding-3-large",
+                    model_name="text-embedding-3-large"
                 ),
             ),  
         ], 
@@ -170,7 +170,7 @@ A minimal index for LLM is designed to store chunks of content. It typically inc
     print(f"{result.name} created")  
     ```
 
-1. For an index schema that more closely mimics structured content, you would have separate indexes for parent and child (chunked) fields. You would need index projections to coordinate the indexing of the two indexes simultaneously. Queries execute against the child index. Query logic includes a lookup query, using the parent_idto retrieve content from the parent index.
+1. For an index schema that more closely mimics structured content, you would have separate indexes for parent and child (chunked) fields. You would need [index projections](index-projections-concept-intro.md) to coordinate the indexing of the two indexes simultaneously. Queries execute against the child index. Query logic includes a lookup query, using the parent_idt  retrieve content from the parent index.
 
    Fields in the child index:
 

articles/search/tutorial-rag-build-solution-maximize-relevance.md

@@ -27,7 +27,7 @@ In this tutorial, you modify the existing search index and queries to use:
 This tutorial updates the search index created by the [indexing pipeline](tutorial-rag-build-solution-pipeline.md). Updates don't affect the existing content, so no rebuild is necessary and you don't need to rerun the indexer.
 
 > [!NOTE]
-> There are more relevance features in preview, including vector query weighting and setting minimum thresholds, but we omit them from this tutorial becaues they aren't yet available in the Azure SDK for Python.
+> There are more relevance features in preview, including vector query weighting and setting minimum thresholds, but we omit them from this tutorial because they're in preview.
 
 ## Prerequisites
 
@@ -41,9 +41,78 @@ This tutorial updates the search index created by the [indexing pipeline](tutori
 
 The [sample notebook](https://github.com/Azure-Samples/azure-search-python-samples/blob/main/Tutorial-RAG/Tutorial-rag.ipynb) includes an updated index and query request.
 
+## Run a baseline query for comparison
+
+Let's start with a new query, "Are there any cloud formations specific to oceans and large bodies of water?".
+
+To compare outcomes after adding relevance features, run the query against the existing index schema, before you add semantic ranking or a scoring profile.
+
+```python
+from azure.search.documents import SearchClient
+from openai import AzureOpenAI
+
+token_provider = get_bearer_token_provider(credential, "https://cognitiveservices.azure.com/.default")
+openai_client = AzureOpenAI(
+     api_version="2024-06-01",
+     azure_endpoint=AZURE_OPENAI_ACCOUNT,
+     azure_ad_token_provider=token_provider
+ )
+
+deployment_name = "gpt-4o"
+
+search_client = SearchClient(
+     endpoint=AZURE_SEARCH_SERVICE,
+     index_name=index_name,
+     credential=credential
+ )
+
+GROUNDED_PROMPT="""
+You are an AI assistant that helps users learn from the information found in the source material.
+Answer the query using only the sources provided below.
+Use bullets if the answer has multiple points.
+If the answer is longer than 3 sentences, provide a summary.
+Answer ONLY with the facts listed in the list of sources below. Cite your source when you answer the question
+If there isn't enough information below, say you don't know.
+Do not generate answers that don't use the sources below.
+Query: {query}
+Sources:\n{sources}
+"""
+
+# Focused query on cloud formations and bodies of water
+query="Are there any cloud formations specific to oceans and large bodies of water?"
+vector_query = VectorizableTextQuery(text=query, k_nearest_neighbors=50, fields="text_vector")
+
+search_results = search_client.search(
+    search_text=query,
+    vector_queries= [vector_query],
+    select=["title", "chunk", "locations"],
+    top=5,
+)
+
+sources_formatted = "=================\n".join([f'TITLE: {document["title"]}, CONTENT: {document["chunk"]}, LOCATIONS: {document["locations"]}' for document in search_results])
+
+response = openai_client.chat.completions.create(
+    messages=[
+        {
+            "role": "user",
+            "content": GROUNDED_PROMPT.format(query=query, sources=sources_formatted)
+        }
+    ],
+    model=deployment_name
+)
+
+print(response.choices[0].message.content)
+```
+
+Output from this request might look like the following example.
+
+```
+Yes, there are cloud formations specific to oceans and large bodies of water. A notable example is "cloud streets," which are parallel rows of clouds that form over the Bering Strait in the Arctic Ocean. These cloud streets occur when wind blows from a cold surface like sea ice over warmer, moister air near the open ocean, leading to the formation of spinning air cylinders. Clouds form along the upward cycle of these cylinders, while skies remain clear along the downward cycle (Source: page-21.pdf).
+```
+
 ## Update the index for semantic ranking and scoring profiles
 
-In a previous tutorial, you [designed an index schema](tutorial-rag-build-solution-index-schema.md) for RAG workloads. We purposely omitted relevance enhancements from that schema so that you could focus on the fundamentals. Deferring relevance to a separate exercise also gives you a before-and-after comparison of the quality of search results after the updates are made.
+In a previous tutorial, you [designed an index schema](tutorial-rag-build-solution-index-schema.md) for RAG workloads. We purposely omitted relevance enhancements from that schema so that you could focus on the fundamentals. Deferring relevance to a separate exercise gives you a before-and-after comparison of the quality of search results after the updates are made.
 
 1. Update the import statements to include classes for semantic ranking and scoring profiles.
 
@@ -138,7 +207,7 @@ openai_client = AzureOpenAI(
      azure_ad_token_provider=token_provider
  )
 
-deployment_name = "gpt-35-turbo"
+deployment_name = "gpt-4o"
 
 search_client = SearchClient(
      endpoint=AZURE_SEARCH_SERVICE,
@@ -160,8 +229,8 @@ Sources:\n{sources}
 """
 
 # Queries are unchanged in this update
-query="how much of earth is covered by water"
-vector_query = VectorizableTextQuery(text=query, k_nearest_neighbors=1, fields="text_vector", exhaustive=True)
+query="Are there any cloud formations specific to oceans and large bodies of water?"
+vector_query = VectorizableTextQuery(text=query, k_nearest_neighbors=50, fields="text_vector")
 
 # Add query_type semantic and semantic_configuration_name
 # Add scoring_profile and scoring_parameters
@@ -175,7 +244,7 @@ search_results = search_client.search(
     select="title, chunk, locations",
     top=5,
 )
-sources_formatted = "\n".join([f'{document["title"]}:{document["chunk"]}:{document["locations"]}' for document in search_results])
+sources_formatted = "=================\n".join([f'TITLE: {document["title"]}, CONTENT: {document["chunk"]}, LOCATIONS: {document["locations"]}' for document in search_results])
 
 response = openai_client.chat.completions.create(
     messages=[
@@ -190,6 +259,24 @@ response = openai_client.chat.completions.create(
 print(response.choices[0].message.content)
 ```
 
+Output from a semantically ranked and boosted query might look like the following example.
+
+```
+Yes, there are specific cloud formations influenced by oceans and large bodies of water:
+
+- **Stratus Clouds Over Icebergs**: Low stratus clouds can frame holes over icebergs, such as Iceberg A-56 in the South Atlantic Ocean, likely due to thermal instability caused by the iceberg (source: page-39.pdf).
+
+- **Undular Bores**: These are wave structures in the atmosphere created by the collision of cool, dry air from a continent with warm, moist air over the ocean, as seen off the coast of Mauritania (source: page-23.pdf).
+
+- **Ship Tracks**: These are narrow clouds formed by water vapor condensing around tiny particles from ship exhaust. They are observed over the oceans, such as in the Pacific Ocean off the coast of California (source: page-31.pdf).
+
+These specific formations are influenced by unique interactions between atmospheric conditions and the presence of large water bodies or objects within them.
+```
+
+Adding semantic ranking and scoring profiles positively affects the response from the LLM by promoting results that meet scoring criteria and are semantically relevant. 
+
+Now that you have a better understanding of index and query design, let's move on to optimizing for speed and concision. We revisit the schema definition to implement quantization and storage reduction, but the rest of the pipeline and models remain intact.
+
 <!-- ## Update queries for minimum thresholds ** NOT AVAILABLE IN PYTHON SDK
 
 Keyword search only returns results if there's match found in the index, up to a maximum of 50 results by default. In contrast, vector search returns `k`-results every time, even if the matching vectors aren't a close match.