checkpoint

Author: HeidiSteen

articles/search/tutorial-rag-build-solution-index-schema.md

@@ -21,9 +21,9 @@ In this tutorial, you:
 > [!div class="checklist"]
 > - Learn the characteristics of an index schema built for RAG
 > - Create an index that accommodate vectors and hybrid queries
-> - Add semantic ranking and filters
 > - Add vector profiles and configurations
 > - Add structured data
+> - Add filters
 
 ## Prerequisites
 
@@ -37,25 +37,31 @@ In conversational search, LLMs compose the response that the user sees, not the
 
 To generate a response, LLMs operate on chunks of content, and while they need to know where the chunk came from for citation purposes, what matters most is the quality of message inputs and its relevance to the user's question. Whether the chunks come from one document or a thousand, the LLM ingests the information or *grounding data*, and formulates the response using instructions provided in a system prompt. 
 
-A minimal index for LLM is designed to store chunks of content. It includes vector fields if you want similarity search, and nonvector fields for human-readable results. Nonvector "chunked" content in the search results becomes the grounding data sent to the LLM.
+A minimal index for LLM is designed to store chunks of content. It includes vector fields if you want similarity search for highly relevant results, and nonvector fields for human-readable inputs to the LLM for conversational search. Nonvector "chunked" content in the search results becomes the grounding data sent to the LLM.
 
-A schema that works best for RAG workloads:
+An index that works best for RAG workloads has these qualities:
 
-- Returns chunks of content in search results. 
-- Search results go to an LLM, which can handle a certain level of redundancy and incomplete strings.
-- Accommodates the queries you want create. You should have fields for vector and hybrid content, and those fields should be attributed to support specific query behaviors.
-- Your schema should be flat (no complex types or structures).
-- You can only query one index at a time (no joins), but you can create indexes that preserve parent-child relationship, and then use nested queries or parallel queries in your search logic to pull from both. This exercise includes templates for both.
+- Returns chunks that are relevant to the query and readable to the LLM.
 
-Schema design has an impact on storage and costs. This exercise is focused on schema fundamentals. In the [Minimize storage and costs](tutorial-rag-build-solution-optimize.md) tutorial, we revisit schema design to consider narrow data types, attribution, and vector configurations that are more efficient.
+  LLMs can handle a certain level of dirty data in chunked data, such as mark up, redundancy, and incomplete strings. While chunks need to be readable, they don't need to be pristine.
+
+- Maintains a relationship between chunks of a document and properties of the parent document. For example, file name, file type, title, author, and so forth. Chunks in search results could be pulled from anywhere in the index. Association with the parent document that provides the chunk is useful for context, citations, and follow up queries.
+
+- Accommodates the queries you want create. You should have fields for vector and hybrid content, and those fields should be attributed to support specific query behaviors. You can only query one index at a time (no joins) so your field collection should define all of your searchable content.
+
+- Your schema should be flat (no complex types or structures). This requirement is specific to the RAG pattern in Azure AI Search.
+
+Although Azure AI Search can't join indexes, you can create indexes that preserve parent-child relationship, and then use nested queries or parallel queries in your search logic to pull from both. This exercise includes templates for both parent-child elements in the same index, or parent-child elements in separate indexes that are connected at query time through sequential queries to different indexes.
+
+Schema design affects storage and costs. This exercise is focused on schema fundamentals. In the [Minimize storage and costs](tutorial-rag-build-solution-optimize.md) tutorial, we revisit schema design to consider narrow data types, attribution, and vector configurations that are more efficient.
 
 ## Create a basic index
 
 Here's a minimal index definition for RAG solutions that supports vector and hybrid search. If you aren't using vectors, the index can be even simpler (see [Quickstart: Generative search (RAG)](search-get-started-rag.md) for an example).
 
 1. Open Visual Studio Code and create a new file. It doesn't have to be a Python file type for this exercise.
 
-1. Review the following example for an introduction to required elements. These elements consist of a name, key field ("id"), non-vector chunks for the LLM, vector chunks for similarity search by the search engine, and a `vectorSearch` configuration for the vector fields. 
+1. Review the following example for an introduction to required elements. In a RAG pattern, elements consist of a name, key field (`"id"`), non-vector chunks for the LLM, vector chunks for similarity search by the search engine, and a `vectorSearch` configuration for the vector fields. 
 
    Vector fields have [specific types](/rest/api/searchservice/supported-data-types#edm-data-types-for-vector-fields) and extra attributes for embedding model dimensions and configuration. `Edm.Single` is a data type that works for the more commonly used LLMs. For more information about vector fields, see [Create a vector index](vector-search-how-to-create-index.md).
 
@@ -79,7 +85,6 @@ Here's a minimal index definition for RAG solutions that supports vector and hyb
     }
     ```
 
-1. 
 <!-- Objective:
 
 - Design an index schema that generates results in a format that works for LLMs.
@@ -105,44 +110,9 @@ Tasks:
 
 <!-- 
 
-ps: We have another physical resource limit for our services: vector index size. HNSW requires vector indices to reside entirely in memory. "Vector index size" is our customer-facing resource limit that governs the memory consumed by their vector data. (and this is a big reason why the beefiest VMs have 512 GB of RAM). Increasing partitions also increases the amount of vector quota for customers as well. 
-
-
-## Old introduction
-
-
-`sidenote: the following applies to the non-basic index, which might be out of scope`.
-*A richer index has more fields and configurations, and is often better because extra fields support richer queries and more opportunities for relevance tuning. Filters and scoring profiles for boosting apply to nonvector fields. If you have content that should be matched precisely and not similarly, such as a name or employee number, then create fields to contain that information.*
-
-
-## Create a basic index
-
-1. Create an index definition with required elements. The index requires a key field ("id"). It includes vector and nonvector chunks of text. Vector content is used for similarity search. Nonvector content is returned in results and will be passed in messages to the LLM. The vector search configuration defines the algorithms used for a vector query.
-
-```http
-### Create an index for RAG scenarios
-POST {{baseUrl}}/indexes?api-version=2024-05-01-preview  HTTP/1.1
-  Content-Type: application/json
-  Authorization: Bearer {{token}}
+ps 1: We have another physical resource limit for our services: vector index size. HNSW requires vector indices to reside entirely in memory. "Vector index size" is our customer-facing resource limit that governs the memory consumed by their vector data. (and this is a big reason why the beefiest VMs have 512 GB of RAM). Increasing partitions also increases the amount of vector quota for customers as well. 
 
-    {
-      "name": "demo-rag-index",
-      "fields": [
-        { "name": "id", "type": "Edm.String", "key": true },
-        { "name": "chunked_content", "type": "Edm.String", "searchable": true, "retrievable": true },
-        { "name": "chunked_content_vectorized", "type": "Edm.Single", "dimensions": 1536, "vectorSearchProfile": "my-vector-profile", "searchable": true, "retrievable": false, "stored": false },
-        { "name": "metadata", "type": "Edm.String", "retrievable": true, "searchable": true }
-      ],
-      "vectorSearch": {
-         "algorithms": [
-             { "name": "my-algo-config", "kind": "hnsw", "hnswParameters": { }  }
-         ],
-         "profiles": [ 
-            { "name": "my-vector-profile", "algorithm": "my-algo-config" }
-         ]
-     }
-    }
-```
+ps 2: A richer index has more fields and configurations, and is often better because extra fields support richer queries and more opportunities for relevance tuning. Filters and scoring profiles for boosting apply to nonvector fields. If you have content that should be matched precisely and not similarly, such as a name or employee number, then create fields to contain that information.*
 
 ## BLOCKED: Index for hybrid queries and relevance tuning
 

articles/search/tutorial-rag-build-solution-models.md

@@ -21,32 +21,45 @@ In this tutorial, you:
 
 > [!div class="checklist"]
 > - Learn which models in the Azure cloud work with built-in integration
+> - Learn about the Azure models used for chat
 > - Deploy models and collect model information for your code
 > - Configure search engine access to Azure models
 > - Learn about custom skills and vectorizers for attaching non-Azure models
 
+If you don't have an Azure subscription, create a [free account](https://azure.microsoft.com/free/?WT.mc_id=A261C142F) before you begin.
+
 ## Prerequisites
 
 - Use the Azure portal to deploy models and configure role assignments in the Azure cloud.
 
-- A model provider, such as [Azure OpenAI](/azure/ai-services/openai/how-to/create-resource) (used in this tutorial), Azure AI Vision created using [a multi-service account](/azure/ai-services/multi-service-resource), or [Azure AI Studio](https://ai.azure.com/).
+- An **Owner** role on your Azure subscription is necessary for creating role assignments. Your model provider imposes more permissions or requirements for deploying and accessing models.
+
+- A model provider, such as [Azure OpenAI](/azure/ai-services/openai/how-to/create-resource) (used in this tutorial), Azure AI Vision created using [a multi-service account](/azure/ai-services/multi-service-resource), or [Azure AI Studio](https://ai.azure.com/). We use Azure OpenAI in this tutorial, but list the other Azure resources so that you know your options.
+
+- Azure AI Search, Basic tier or higher provides a [managed identity](search-howto-managed-identities-data-sources.md) used in role assignments. 
+
+To complete all of the tutorials in this series, the region must also support both Azure AI Search and the model provider. See supported regions for:
+
+- [Azure OpenAI regions](/azure/ai-services/openai/concepts/models#model-summary-table-and-region-availability)
+
+- [Azure AI Vision regions](/azure/ai-services/computer-vision/overview-image-analysis?tabs=4-0#region-availability)
 
-- Azure AI Search, Basic tier or higher provides a [managed identity](search-howto-managed-identities-data-sources.md) used in role assignments. To complete all of the tutorials in this series, the region must also support the model provider (see supported regions for [Azure OpenAI](/azure/ai-services/openai/concepts/models#model-summary-table-and-region-availability), [Azure AI Vision](/azure/ai-services/computer-vision/overview-image-analysis?tabs=4-0#region-availability), [Azure AI Studio](/azure/ai-studio/reference/region-support)). Azure AI Search is currently facing limited availability in some regions. Check the [Azure AI Search region list](search-region-support.md) to confirm availability.
+- [Azure AI Studio](/azure/ai-studio/reference/region-support) regions. 
 
-- An **Owner** role on your Azure subscription is necessary for creating role assignments. Your model provider might impose more permissions or requirements for deploying and accessing models.
+Azure AI Search is currently facing limited availability in some regions. Check the [Azure AI Search region list](search-region-support.md) to confirm availability.
 
 > [!TIP]
-> If you're creating Azure resources for this tutorial, you just need Azure AI Search and Azure OpenAI. If you want to try other model providers, the following regions provide the most overlap and have the most capacity: East US, East US2, and South Central in the Americas; France Central or Switzerland North in Europe; Australia East in Asia Pacific. 
+> Currently, the following regions provide the most overlap and have the most capacity: **East US**, **East US2**, and **South Central** in the Americas; **France Central** or **Switzerland North** in Europe; **Australia East** in Asia Pacific.
 
 ## Review models supporting built-in vectorization
 
-Azure AI Search supports an embedding action in an indexing pipeline. It also supports an embedding action at query time, converting text or image inputs into vectors for a vector search. In this step, identify an embedding model that works for your content and queries. If you're providing raw vector data and vector queries, or if your RAG solution doesn't include vector data, skip this step.
+Azure AI Search supports an embedding action in an indexing pipeline. It also supports an embedding action at query time, converting text or image inputs into vectors for a vector search. In this step, identify an embedding model that works for your content and queries. If you're providing raw vector data and raw vector queries, or if your RAG solution doesn't include vector data, skip this step.
 
 Vector queries work best when you use the same embedding model for both indexing and query input conversions. Using different embedding models for each action typically results in poor query outcomes.
 
 To meet the same-model requirement, choose embedding models that can be referenced through *skills* during indexing and through *vectorizers* during query execution. Review [Create an indexing pipeline](tutorial-rag-build-solution-pipeline.md) for code that calls an embedding skill and a matching vectorizer. 
 
-The following embedding models have skills and vectorizer support in Azure AI Search.
+Azure AI Search provides skill and vectorizer support for the following embedding models in the Azure cloud.
 
 | Client | Embedding models | Skill | Vectorizer |
 |--------|------------------|-------|------------|
@@ -58,7 +71,7 @@ The following embedding models have skills and vectorizer support in Azure AI Se
 
 <sup>2</sup> Deployed models in the model catalog are accessed over an AML endpoint. We use the existing AML skill for this connection.
 
-You can use other models besides those listed here. For more information, see [Use non-Azure models for embeddings and chat](#use-non-azure-models-for-embeddings-and-chat) in this article.
+You can use other models besides those listed here. For more information, see [Use non-Azure models for embeddings](#use-non-azure-models-for-embeddings-and-chat) in this article.
 
 > [!NOTE]
 > Inputs to an embedding models are typically chunked data. In an Azure AI Search RAG pattern, chunking is handled in the indexer pipeline, covered in [another tutorial](tutorial-rag-build-solution-pipeline.md) in this series.
@@ -98,7 +111,7 @@ You must have **Cognitive Services AI User** permissions to deploy models in Azu
 
 ## Configure search engine access to Azure models
 
-For pipeline and query execution, this tutorial uses Micrsoft Entra ID for authentication and roles for authorization. 
+For pipeline and query execution, this tutorial uses Microsoft Entra ID for authentication and roles for authorization. 
 
 Assign yourself and the search service identity permissions on Azure OpenAI. The code for this tutorial runs locally. Requests to Azure OpenAI originate from your system. Also, search results from the search engine are passed to Azure OpenAI. For these reasons, both you and the search service need permissions on Azure OpenAI.