checkpoint

Author: HeidiSteen

articles/search/tutorial-rag-build-solution-index-schema.md

@@ -8,7 +8,7 @@ author: HeidiSteen
 ms.author: heidist
 ms.service: cognitive-search
 ms.topic: tutorial
-ms.date: 10/01/2024
+ms.date: 10/04/2024
 
 ---
 
@@ -43,7 +43,7 @@ Chunks are the focus of the schema, and each chunk is the defining element of a
 
 ### Enhanced with generated data
 
-In this tutorial, sample data consists of PDFs and content from the [NASA Earth Book](https://www.nasa.gov/ebooks/earth/). This content is descriptive and informative, with numerous references to geographies, countries, and areas across the world. All of the textual content is captured in chunks, but these recurring instances of place names create an opportunity for adding structure to the index. Using skills, it's possible to recognize entities in the text and capture them in an index for use in queries and filters. In this tutorial, we include an [entity recognition skill](cognitive-search-skill-entity-recognition-v3.md) that recognizes and extracts location entities, loading it into a searchable and filterable `locations` field. Adding structured content to your index gives you more options for filtering, improved relevance, and more focused answers.
+In this tutorial, sample data consists of PDFs and content from the [NASA Earth Book](https://www.nasa.gov/ebooks/earth/). This content is descriptive and informative, with numerous references to geographies, countries, and areas across the world. All of the textual content is captured in chunks, but recurring instances of place names create an opportunity for adding structure to the index. Using skills, it's possible to recognize entities in the text and capture them in an index for use in queries and filters. In this tutorial, we include an [entity recognition skill](cognitive-search-skill-entity-recognition-v3.md) that recognizes and extracts location entities, loading it into a searchable and filterable `locations` field. Adding structured content to your index gives you more options for filtering, improved relevance, and more focused answers.
 
 ### Parent-child fields in one or two indexes?
 
@@ -61,11 +61,11 @@ In Azure AI Search, an index that works best for RAG workloads has these qualiti
 
 - Maintains a parent-child relationship between chunks of a document and the properties of the parent document, such as the file name, file type, title, author, and so forth. To answer a query, chunks could be pulled from anywhere in the index. Association with the parent document providing the chunk is useful for context, citations, and follow up queries.
 
-- Accommodates the queries you want create. You should have fields for vector and hybrid content, and those fields should be attributed to support specific query behaviors. You can only query one index at a time (no joins) so your fields collection should define all of your searchable content.
+- Accommodates the queries you want create. You should have fields for vector and hybrid content, and those fields should be attributed to support specific query behaviors, such as searchable or filterable. You can only query one index at a time (no joins) so your fields collection should define all of your searchable content.
 
 - Your schema should be flat (no complex types or structures). This requirement is specific to the RAG pattern in Azure AI Search.
 
-Although Azure AI Search can't join indexes, you can create indexes that preserve parent-child relationship, and then use sequential or parallel queries in your search logic to pull from both. This exercise includes templates for parent-child elements in the same index and in separate indexes, where information from the parent index is retrieved using a lookup query.
+<!-- Although Azure AI Search can't join indexes, you can create indexes that preserve parent-child relationship, and then use sequential queries in your search logic to pull from both (a query on the chunked data index, a lookup on the parent index). This exercise includes templates for parent-child elements in the same index and in separate indexes, where information from the parent index is retrieved using a lookup query. -->
 
 <!-- > [!NOTE]
 > Schema design affects storage and costs. This exercise is focused on schema fundamentals. In the [Minimize storage and costs](tutorial-rag-build-solution-minimize-storage.md) tutorial, you revisit schema design to consider narrow data types, attribution, and vector configurations that offer more efficient. -->
@@ -136,7 +136,7 @@ A minimal index for LLM is designed to store chunks of content. It typically inc
         SearchField(name="locations", type=SearchFieldDataType.Collection(SearchFieldDataType.String), filterable=True),
         SearchField(name="chunk_id", type=SearchFieldDataType.String, key=True, sortable=True, filterable=True, facetable=True, analyzer_name="keyword"),  
         SearchField(name="chunk", type=SearchFieldDataType.String, sortable=False, filterable=False, facetable=False),  
-        SearchField(name="text_vector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), vector_search_dimensions=1536, vector_search_profile_name="myHnswProfile")
+        SearchField(name="text_vector", type=SearchFieldDataType.Collection(SearchFieldDataType.Single), vector_search_dimensions=1024, vector_search_profile_name="myHnswProfile")
         ]  
 
     # Configure the vector search configuration  
@@ -157,8 +157,8 @@ A minimal index for LLM is designed to store chunks of content. It typically inc
                 kind="azureOpenAI",  
                 parameters=AzureOpenAIVectorizerParameters(  
                     resource_url=AZURE_OPENAI_ACCOUNT,  
-                    deployment_name="text-embedding-ada-002",
-                    model_name="text-embedding-ada-002"
+                    deployment_name="text-embedding-3-large",
+                    model_name="text-embedding-3-large"
                 ),
             ),  
         ], 
@@ -170,7 +170,7 @@ A minimal index for LLM is designed to store chunks of content. It typically inc
     print(f"{result.name} created")  
     ```
 
-1. For an index schema that more closely mimics structured content, you would have separate indexes for parent and child (chunked) fields. You would need index projections to coordinate the indexing of the two indexes simultaneously. Queries execute against the child index. Query logic includes a lookup query, using the parent_idto retrieve content from the parent index.
+1. For an index schema that more closely mimics structured content, you would have separate indexes for parent and child (chunked) fields. You would need [index projections](index-projections-concept-intro.md) to coordinate the indexing of the two indexes simultaneously. Queries execute against the child index. Query logic includes a lookup query, using the parent_idt  retrieve content from the parent index.
 
    Fields in the child index:
 

articles/search/tutorial-rag-build-solution-maximize-relevance.md

@@ -27,7 +27,7 @@ In this tutorial, you modify the existing search index and queries to use:
 This tutorial updates the search index created by the [indexing pipeline](tutorial-rag-build-solution-pipeline.md). Updates don't affect the existing content, so no rebuild is necessary and you don't need to rerun the indexer.
 
 > [!NOTE]
-> There are more relevance features in preview, including vector query weighting and setting minimum thresholds, but we omit them from this tutorial becaues they aren't yet available in the Azure SDK for Python.
+> There are more relevance features in preview, including vector query weighting and setting minimum thresholds, but we omit them from this tutorial because they're in preview.
 
 ## Prerequisites
 

articles/search/tutorial-rag-build-solution-models.md

@@ -9,7 +9,7 @@ ms.author: heidist
 ms.service: cognitive-search
 ms.topic: tutorial
 ms.custom: references_regions
-ms.date: 10/01/2024
+ms.date: 10/04/2024
 
 ---
 
@@ -48,7 +48,7 @@ If you don't have an Azure subscription, create a [free account](https://azure.m
 
   - [Azure AI Studio](/azure/ai-studio/reference/region-support) regions. 
 
-  Azure AI Search is currently facing limited availability in some regions, such as West Europe and West US 2/3. Check the [Azure AI Search region list](search-region-support.md) to confirm region status.
+  Azure AI Search is currently facing limited availability in some regions, such as West Europe and West US 2/3. To confirm region status, check the [Azure AI Search region list](search-region-support.md).
 
 > [!TIP]
 > Currently, the following regions provide the most overlap among the model providers and have the most capacity: **East US2** and **South Central** in the Americas; **France Central** or **Switzerland North** in Europe; **Australia East** in Asia Pacific.
@@ -59,7 +59,7 @@ If you don't have an Azure subscription, create a [free account](https://azure.m
 
 Vectorized content improves the query results in a RAG solution. Azure AI Search supports a built-in vectorization action in an indexing pipeline. It also supports vectorization at query time, converting text or image inputs into embeddings for a vector search. In this step, identify an embedding model that works for your content and queries. If you're providing raw vector data and raw vector queries, or if your RAG solution doesn't include vector data, skip this step.
 
-Vector queries that include a text-to-vector conversion step must use the same embedding model that was used during indexing. The search engine won't throw an error if you use different models, but you'll get poor results.
+Vector queries that include a text-to-vector conversion step must use the same embedding model that was used during indexing. The search engine doesn't throw an error if you use different models, but you get poor results.
 
 To meet the same-model requirement, choose embedding models that can be referenced through *skills* during indexing and through *vectorizers* during query execution. The following table lists the skill and vectorizer pairs. To see how the embedding models are used, skip ahead to [Create an indexing pipeline](tutorial-rag-build-solution-pipeline.md) for code that calls an embedding skill and a matching vectorizer. 
 
@@ -75,7 +75,7 @@ Azure AI Search provides skill and vectorizer support for the following embeddin
 
 <sup>2</sup> Deployed models in the model catalog are accessed over an AML endpoint. We use the existing AML skill for this connection.
 
-You can use other models besides those listed here. For more information, see [Use non-Azure models for embeddings](#use-non-azure-models-for-embeddings) in this article.
+You can use other models besides the ones listed here. For more information, see [Use non-Azure models for embeddings](#use-non-azure-models-for-embeddings) in this article.
 
 > [!NOTE]
 > Inputs to an embedding models are typically chunked data. In an Azure AI Search RAG pattern, chunking is handled in the indexer pipeline, covered in [another tutorial](tutorial-rag-build-solution-pipeline.md) in this series.
@@ -90,16 +90,18 @@ The following models are commonly used for a chat search experience:
 |--------|------------|
 | Azure OpenAI | GPT-35-Turbo, <br>GPT-4, <br>GPT-4o, <br>GPT-4 Turbo |
 
-GPT-35-Turbo and GPT-4 models are optimized to work with inputs formatted as a conversation. 
+GPT-35-Turbo and GPT-4 models are optimized to work with inputs formatted as a conversation.
+
+We use GPT-4o in this tutorial. During testing, we found that it's less likely to supplement with its own training data. For example, given the query "how much of the earth is covered by water?", GPT-35-Turbo answered using its built-in knowledge of earth to state that 71% of the earth is covered by water, even though the sample data doesn't provide that fact. In contrast, GPT-4o responded (correctly) with "I don't know".
 
 ## Deploy models and collect information
 
-Models must be deployed and accessible through an endpoint. Both embedding-related skills and vectorizers need the number of dimensions and the model name. Other details about your model might be required by the client used on the connection.
+Models must be deployed and accessible through an endpoint. Both embedding-related skills and vectorizers need the number of dimensions and the model name. 
 
 This tutorial series uses the following models and model providers:
 
-- Text-embedding-ada-02 on Azure OpenAI for embeddings
-- GPT-35-Turbo on Azure OpenAI for chat completion
+- Text-embedding-3-large on Azure OpenAI for embeddings
+- GPT-4o on Azure OpenAI for chat completion
 
 You must have [**Cognitive Services OpenAI Contributor**]( /azure/ai-services/openai/how-to/role-based-access-control#cognitive-services-openai-contributor) or higher to deploy models in Azure OpenAI.
 
@@ -109,17 +111,17 @@ You must have [**Cognitive Services OpenAI Contributor**]( /azure/ai-services/op
 
 1. Select **Deploy model** > **Deploy base model**.
 
-1. Select **text-embedding-ada-02** from the dropdown list and confirm the selection.
+1. Select **text-embedding-3-large** from the dropdown list and confirm the selection.
 
-1. Specify a deployment name. We recommend "text-embedding-ada-002".
+1. Specify a deployment name. We recommend "text-embedding-3-large".
 
 1. Accept the defaults.
 
 1. Select **Deploy**.
 
-1. Repeat the previous steps for **gpt-35-turbo**.
+1. Repeat the previous steps for **gpt-4o**.
 
-1. Make a note of the model names and endpoint. Embedding skills and vectorizers assemble the full endpoint internally, so you only need the resource URI. For example, given `https://MY-FAKE-ACCOUNT.openai.azure.com/openai/deployments/text-embedding-ada-002/embeddings?api-version=2023-05-15`, the endpoint you should provide in skill and vectorizer definitions is `https://MY-FAKE-ACCOUNT.openai.azure.com`.
+1. Make a note of the model names and endpoint. Embedding skills and vectorizers assemble the full endpoint internally, so you only need the resource URI. For example, given `https://MY-FAKE-ACCOUNT.openai.azure.com/openai/deployments/text-embedding-3-large/embeddings?api-version=2024-06-01`, the endpoint you should provide in skill and vectorizer definitions is `https://MY-FAKE-ACCOUNT.openai.azure.com`.
 
 ## Configure search engine access to Azure models
 
@@ -138,10 +140,13 @@ Assign yourself and the search service identity permissions on Azure OpenAI. The
 1. Select **Add role assignment**.
 
 1. Select [**Cognitive Services OpenAI User**](/azure/ai-services/openai/how-to/role-based-access-control#cognitive-services-openai-userpermissions).
+
 1. Select **Managed identity** and then select **Members**. Find the system-managed identity for your search service in the dropdown list.
 
 1. Next, select **User, group, or service principal** and then select **Members**. Search for your user account and then select it from the dropdown list.
 
+1. Make sure you have two security principals assigned to the role.
+
 1. Select **Review and Assign** to create the role assignments.
 
 For access to models on Azure AI Vision, assign **Cognitive Services OpenAI User**. For Azure AI Studio, assign **Azure AI Developer**.