checkpoint

Author: HeidiSteen

articles/search/tutorial-rag-build-solution-index-schema.md

@@ -12,7 +12,7 @@ ms.date: 09/12/2024
 
 ---
 
-# Tutorial: Design an index (RAG in Azure AI Search)
+# Tutorial: Design an index for RAG in Azure AI Search
 
 An index contains searchable text and vector content, plus configurations. In a RAG pattern that uses a chat model for responses, you want an index that contains chunks of content that can be passed to an LLM at query time. 
 
@@ -35,29 +35,29 @@ The output of this exercise is an index definition in JSON. At this point, it's
 
 In conversational search, LLMs compose the response that the user sees, not the search engine, so you don't need to think about what fields to show in your search results, and whether the representations of individual search documents are coherent to the user. Depending on the question, the LLM might return verbatim content from your index, or more likely, repackage the content for a better answer.
 
-### Focus on chunks
+### Organized around chunks
 
 When LLMs generate a response, they operate on chunks of content for message inputs, and while they need to know where the chunk came from for citation purposes, what matters most is the quality of message inputs and its relevance to the user's question. Whether the chunks come from one document or a thousand, the LLM ingests the information or *grounding data*, and formulates the response using instructions provided in a system prompt.
 
 Chunks are the focus of the schema, and each chunk is the defining element of a search document in a RAG pattern. You can think of your index as a large collection of chunks, as opposed to traditional search documents that probably have more structure, such as fields containing uniform content for a name, descriptions, categories, and addresses.
 
-### Focus on content
+### Content-aware
 
 In addition to structural considerations, like chunked content, you also want to consider the substance of your content because it also informs what fields are indexed.
 
-In this tutorial, we use PDFs and content from the NASA Earth Book. This content is descriptive and informative, with numerous references to geographies, countries, and areas across the world. To capture this information in our index and potentially use it in queries, we can include skills in our indexing pipeline that recognize and extract this information, loading it into a searchable and filterable `locations` field.
+In this tutorial, sample data consists of PDFs and content from the NASA Earth Book. This content is descriptive and informative, with numerous references to geographies, countries, and areas across the world. To capture this information in our index and potentially use it in queries, we include skills in our indexing pipeline that recognize and extract this information, loading it into a searchable and filterable `locations` field.
 
 The original ebook is large, over 100 pages and 35 MB in size. We broke it up into smaller PDFs, one per page of text, to stay under the REST API payload limit of 16 MB per API call.
 
 For simplicity, we omit image vectorization for this exercise.
 
-### Focus on parent-child indexes
+### Parent-child fields in one or two indexes
 
-Chunked content typically derives from a larger document. And although the schema is organized around chunks, you also want to capture properties and content at the parent level. Examples of these properties might include the parent file path, title, authors, publication date, summary.
+Chunked content typically derives from a larger document. And although the schema is organized around chunks, you also want to capture properties and content at the parent level. Examples of these properties might include the parent file path, title, authors, publication date, or a summary.
 
 An inflection point in schema design is whether to have two indexes for parent and child/chunked content, or a single index that repeats parent elements for each chunk.
 
-In this tutorial, because all of the chunks of text originate from a single parent (NASA Earth Book), you don't need a separate index dedicated to up level parent fields. If you index from multiple parent PDFs, you might want a parent-child index pair to capture level-specific fields and then send lookup queries to the parent index to retrieve those fields relevant to each chunk. We include an example of that parent-child index template in this exercise for comparison.
+In this tutorial, because all of the chunks of text originate from a single parent (NASA Earth Book), you don't need a separate index dedicated to up level parent fields. However, if you index from multiple parent PDFs, you might want a parent-child index pair to capture level-specific fields and then send lookup queries to the parent index to retrieve those fields relevant to each chunk.
 
 ### Checklist of schema considerations
 
@@ -76,13 +76,13 @@ Although Azure AI Search can't join indexes, you can create indexes that preserv
 > [!NOTE]
 > Schema design affects storage and costs. This exercise is focused on schema fundamentals. In the [Minimize storage and costs](tutorial-rag-build-solution-minimize-storage.md) tutorial, you revisit schema design to consider narrow data types, attribution, and vector configurations that offer more efficient.
 
-## Create a basic index
+## Create an index for RAG workloads
 
-A minimal index for LLM is designed to store chunks of content. It includes vector fields if you want similarity search for highly relevant results, and nonvector fields for human-readable inputs to the LLM for conversational search. Nonvector chunked content in the search results becomes the grounding data sent to the LLM.
+A minimal index for LLM is designed to store chunks of content. It typically includes vector fields if you want similarity search for highly relevant results. It also includes nonvector fields for human-readable inputs to the LLM for conversational search. Nonvector chunked content in the search results becomes the grounding data sent to the LLM.
 
 1. Open Visual Studio Code and create a new file. It doesn't have to be a Python file type for this exercise.
 
-1. Here's a minimal index definition for RAG solutions that support vector and hybrid search. Review it for an introduction to required elements: name, fields, and a `vectorSearch` configuration for the vector fields.
+1. Here's a minimal index definition for RAG solutions that support vector and hybrid search. Review it for an introduction to required elements: index name, fields, and a configuration section for vector fields.
 
     ```json
     {
@@ -91,7 +91,7 @@ A minimal index for LLM is designed to store chunks of content. It includes vect
         { "name": "id", "type": "Edm.String", "key": true },
         { "name": "chunked_content", "type": "Edm.String", "searchable": true, "retrievable": true },
         { "name": "chunked_content_vectorized", "type": "Edm.Single", "dimensions": 1536, "vectorSearchProfile": "my-vector-profile", "searchable": true, "retrievable": false, "stored": false },
-        { "name": "metadata", "type": "Edm.String", "retrievable": true, "searchable": true }
+        { "name": "metadata", "type": "Edm.String", "retrievable": true, "searchable": true, "filterable": true }
       ],
       "vectorSearch": {
           "algorithms": [
@@ -104,13 +104,62 @@ A minimal index for LLM is designed to store chunks of content. It includes vect
     }
     ```
 
-   Fields must include key field (`"id"`) and should include vector chunks for similarity search, and nonvector chunks for the LLM. Metadata about the source file might be file path, creation date, or content type.
-
-   Vector fields have [specific types](/rest/api/searchservice/supported-data-types#edm-data-types-for-vector-fields) and extra attributes for embedding model dimensions and configuration. `Edm.Single` is a data type that works for the more commonly used LLMs. For more information about vector fields, see [Create a vector index](vector-search-how-to-create-index.md).
-
-1. Here's the index schema for the tutorial and the Earth Book content. It's similar to the basic schema, but adds a parent ID, metadata (`title`), strings (`chunks`), and vectors for similarity search (`text_vectors`). It also includes a `locations` field for storing generated content that's created in the [indexing pipeline](tutorial-rag-build-solution-pipeline.md).
+   Fields must include key field (`"id"`) and should include vector chunks for similarity search, and nonvector chunks for inputs to the LLM. 
+
+   Vector fields have [specific types](/rest/api/searchservice/supported-data-types#edm-data-types-for-vector-fields) and extra attributes for embedding model dimensions and configuration. `Edm.Single` is a data type that works for commonly used LLMs. For more information about vector fields, see [Create a vector index](vector-search-how-to-create-index.md).
+
+   Metadata fields might be file path, creation date, or content type and are useful for [filters](vector-search-filters.md).
+
+1. Here's the index schema for the [tutorial source code](https://github.com/Azure-Samples/azure-search-python-samples/blob/main/Tutorial-RAG/Tutorial-rag.ipynb) and the [Earth Book content](https://github.com/Azure-Samples/azure-search-sample-data/tree/main/nasa-e-book/earth_book_2019_text_pages). 
+
+   Like the basic schema, it's organized around chunks. The `chunk_id` uniquely identifies each chunk. The `text_vector` field is an embedding of the chunk. The nonvector `chunk` field is a readable string. The `title` maps to a unique metadata storage path for the blobs. The `parent_id` is the only parent-level field, and it's a base64-encoded version of the parent file URI. 
+
+   The schema also includes a `locations` field for storing generated content that's created by the [indexing pipeline](tutorial-rag-build-solution-pipeline.md).
+
+   ```python
+    index_name = "py-rag-tutorial-idx"
+    index_client = SearchIndexClient(endpoint=AZURE_SEARCH_SERVICE, credential=AZURE_SEARCH_CREDENTIAL)  
+    fields = [
+        SearchField(name="parent_id", type=SearchFieldDataType.String),  
+        SearchField(name="title", type=SearchFieldDataType.String),
+        SearchField(name="locations", type=SearchFieldDataType.Collection(SearchFieldDataType.String), filterable=True),
+        SearchField(name="chunk_id", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True, analyzer_name="keyword"),  
+        SearchField(name="chunk", type=SearchFieldDataType.String, sortable=False, filterable=False, facetable=False),  
+        SearchField(name="text_vector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), vector_search_dimensions=1536, vector_search_profile_name="myHnswProfile")
+        ]  
+      
+    # Configure the vector search configuration  
+    vector_search = VectorSearch(  
+        algorithms=[  
+            HnswAlgorithmConfiguration(name="myHnsw"),
+        ],  
+        profiles=[  
+            VectorSearchProfile(  
+                name="myHnswProfile",  
+                algorithm_configuration_name="myHnsw",  
+                vectorizer="myOpenAI",  
+            )
+        ],  
+        vectorizers=[  
+            AzureOpenAIVectorizer(  
+                name="myOpenAI",  
+                kind="azureOpenAI",  
+                azure_open_ai_parameters=AzureOpenAIParameters(  
+                    resource_uri=AZURE_OPENAI_ACCOUNT,  
+                    deployment_id="text-embedding-ada-002",
+                    model_name="text-embedding-ada-002"
+                ),
+            ),  
+        ],  
+    )  
+      
+    # Create the search index
+    index = SearchIndex(name=index_name, fields=fields, vector_search=vector_search)  
+    result = index_client.create_or_update_index(index)  
+    print(f"{result.name} created")  
+    ```
 
-    ```json
+<!--     ```json
     {
       "name": "rag-tutorial-earth-book",
       "defaultScoringProfile": null,
@@ -223,7 +272,21 @@ A minimal index for LLM is designed to store chunks of content. It includes vect
         "compressions": []
       }
     }
+    ``` -->
+
+1. For an index schema that more closely mimics structured content, you would have separate indexes for parent and child (chunked) fields. You would need index projections to coordinate the indexing of the two indexes simultaneously. Queries execute against the child index. Query logic includes a lookup query, using the parent_idto retrieve content from the parent index.
+
+   Fields in the child index:
+
+   - id
+   - chunk
+   - chunkVectcor
+   - parent_id
 
+   Fields in the parent index (everything that you want "one of"):
+
+   - parent_id
+   - parent-level fields (name, title, category)
 
 <!-- Objective:
 
@@ -251,22 +314,7 @@ Tasks:
 <!-- 
 
 ps 2: A richer index has more fields and configurations, and is often better because extra fields support richer queries and more opportunities for relevance tuning. Filters and scoring profiles for boosting apply to nonvector fields. If you have content that should be matched precisely and not similarly, such as a name or employee number, then create fields to contain that information.*
-
-## BLOCKED: Index for hybrid queries and relevance tuning
-
-Per Carey, you would want a couple indexes for this scenario - parent index, chunked/child index linked to parent - with queries that include lookup to access fields at the parent level. You would need index projections to "project" to coordiate the indexing of the two indexes simultaneously.
-
-child index:
-- id
-- chunk
-- chunkVectcor
-- parentId
-
-parent index (everything that you want "one of"):
-- fields for verbatim matching (name, title, category)
-- fields for filters or boosting (dates, geo coordates)
-
-This is probably out of scope for this tutorial, but could be an extension. -->
+ -->
 
 ## Next step
 

articles/search/tutorial-rag-build-solution-minimize-storage.md

@@ -12,7 +12,7 @@ ms.date: 09/12/2024
 
 ---
 
-# Tutorial: Minimize storage and costs using vector compression and narrow data types (RAG in Azure AI Search)
+# Tutorial: Minimize storage and costs (RAG in Azure AI Search)
 
 In this tutorial, learn the techniques for reducing index size, with a focus on vector compression and storage. 
 

articles/search/tutorial-rag-build-solution-models.md

@@ -13,7 +13,7 @@ ms.date: 09/12/2024
 
 ---
 
-# Tutorial: Choose embedding and chat models (RAG in Azure AI Search)
+# Tutorial: Choose embedding and chat models for RAG in Azure AI Search
 
 A RAG solution built on Azure AI Search takes a dependency on embedding models for vectorization, and on chat models for conversational search over your data.