Merge pull request #267279 from HeidiSteen/heidist-langchain

[azure search] vector store updates

Author: AnnaMHuff

articles/search/media/search-howto-run-reset-indexers/search-units.png

@@ -9,7 +9,7 @@ ms.service: cognitive-search
 ms.custom:
   - ignite-2023
 ms.topic: how-to
-ms.date: 12/18/2023
+ms.date: 02/26/2024
 ---
 
 # Run or reset indexers, skills, or documents
@@ -24,7 +24,15 @@ This article explains how to run indexers on demand, with and without a reset. I
 
 ## Indexer execution
 
-You can run multiple indexers at one time assuming you sufficient replicas (one indexer job per replica), but each indexer itself is single-instance. Starting a new instance while the indexer is already in execution produces this error: `"Failed to run indexer "<indexer name>" error: "Another indexer invocation is currently in progress; concurrent invocations are not allowed."`
+A search service runs one indexer job per [search unit](search-capacity-planning.md#concepts-search-units-replicas-partitions-shards). Every search service starts with one search unit, but each new partition or replica increases the search units of your service. You can check the search unit count in the portal's Essential section of the **Overview** page. If you need concurrent processing, make sure you have sufficient replicas. Indexers don't run in the background, so you might detect more query throttling than usual if the service is under pressure.
+
+The following screenshot shows the number of search units, which determines how many indexers can run at once.
+
+:::image type="content" source="media/search-howto-run-reset-indexers/search-units.png" alt-text="Screenshot of the essentials section of the overview page, showing search units.":::
+
+Once indexer execution starts, you can't pause or stop it. Indexer execution stops when there are no more documents to load or refresh, or when the [maximum running time limit](search-limits-quotas-capacity.md#indexer-limits) is reached.
+
+You can run multiple indexers at one time assuming sufficient capacity, but each indexer itself is single-instance. Starting a new instance while the indexer is already in execution produces this error: `"Failed to run indexer "<indexer name>" error: "Another indexer invocation is currently in progress; concurrent invocations are not allowed."`
 
 An indexer job runs in a managed execution environment. Currently, there are two environments. You can't control or configure which environment is used. Azure AI Search determines the environment based on job composition and the ability of the service to move an indexer job onto a content processor (some [security features](search-indexer-securing-resources.md#indexer-execution-environment) block the multitenant environment).
 

articles/search/search-howto-run-reset-indexers.md

@@ -29,9 +29,9 @@ Considerations for vector storage include the following points:
 
 In Azure AI Search, there are two patterns for working with search results. 
 
-+ Generative search. Language models formulate a response to the user's query using data from Azure AI Search. This pattern usually includes an orchestration layer to coordinate prompts and maintain context. In this pattern, results are fed into prompt flows, received by chat models like GPT and Text-Davinci. This approach is based on [**Retrieval augmented generation (RAG)**](retrieval-augmented-generation-overview.md) architecture, where the search index provides the grounding data.
++ Generative search. Language models formulate a response to the user's query using data from Azure AI Search. This pattern includes an orchestration layer to coordinate prompts and maintain context. In this pattern, search results are fed into prompt flows, received by chat models like GPT and Text-Davinci. This approach is based on [**Retrieval augmented generation (RAG)**](retrieval-augmented-generation-overview.md) architecture, where the search index provides the grounding data.
 
-+ Classic search. Search engine formulates a response based on content in your index, and you render those results in a client app. In a direct response from the search engine, results are returned in a flattened row set, and you can choose which fields are passed to the client app. It's expected that you would populate the vector store (search index) with nonvector content that's human readable in your response. The search engine matches on vectors, but can return nonvector values from the same search document. [**Vector queries**](vector-search-how-to-query.md) and [**hybrid queries**](hybrid-search-how-to-query.md) cover the types of requests.
++ Classic search using a search bar, query input string, and rendered results. The search engine accepts and executes the vector query, formulates a response, and you render those results in a client app. In Azure AI Search, results are returned in a flattened row set, and you can choose which fields to include search results. Since there's no chat model, it's expected that you would populate the vector store (search index) with nonvector content that's human readable in your response. Although the search engine matches on vectors, you should use nonvector values to populate the search results. [**Vector queries**](vector-search-how-to-query.md) and [**hybrid queries**](hybrid-search-how-to-query.md) cover the types of query requests you can formulate for classic search scenarios.
 
 Your index schema should reflect your primary use case. The following section highlights the differences in field composition for solutions built for generative AI or classic search.
 
@@ -41,7 +41,7 @@ An index schema for a vector store requires a name, a key field (string), one or
 
 ### Basic vector field configuration
 
-A vector field, such as `"content_vector"` in the following example, is of type `Collection(Edm.Single)`. It must be searchable and retrievable. It can't be filterable, facetable, or sortable, and it can't have analyzers, normalizers, or synonym map assignments. It must have dimensions set to the number of embeddings generated by the embedding model. For instance, if you're using text-embedding-ada-002, it generates 1,536 embeddings. A vector search profile is specified in a separate [vector search configuration](vector-search-how-to-create-index.md) and assigned to a vector field using a profile name.
+Vector fields are distinguished by their data type and vector-specific properties. Here's what a vector field looks like in a fields collection:
 
 ```json
 {
@@ -54,11 +54,17 @@ A vector field, such as `"content_vector"` in the following example, is of type
 }
 ```
 
-### Fields collection for basic vector workloads
+Vector fields are of type `Collection(Edm.Single)`. 
+
+Vector fields must be searchable and retrievable, but they can't be filterable, facetable, or sortable, or have analyzers, normalizers, or synonym map assignments. 
+
+Vector fields must have `dimensions` set to the number of embeddings generated by the embedding model. For example, text-embedding-ada-002 generates 1,536 embeddings for each chunk of text. 
+
+Vector fields are indexed using algorithms indicated by a *vector search profile*, which is defined elsewhere in the index and thus not shown in the example. For more information, see [vector search configuration](vector-search-how-to-create-index.md).
 
-Here's an example showing a vector field in context, with other fields in a collection. 
+### Fields collection for basic vector workloads
 
-The key field (required) is `"id"` in this example. The `"content"` field is the human readable equivalent of the `"content_vector"` field. Although if you're using language models exclusively for response formulation, you can skip nonvector content fields. Metadata fields are useful for filters, especially if metadata includes origin information about the source document. You can't filter on a vector field directly, but you can set prefilter or postfilter modes to filter before or after vector query execution.
+Vector stores require more fields besides vector fields. For example, a key field (`"id"` in this example) is an index requirement. 
 
 ```json
 "name": "example-basic-vector-idx",
@@ -70,6 +76,10 @@ The key field (required) is `"id"` in this example. The `"content"` field is the
 ]
 ```
 
+Other fields, such as the `"content"` field, provide the human readable equivalent of the `"content_vector"` field. If you're using language models exclusively for response formulation, you can omit nonvector content fields, but solutions that push search results directly to client apps should have nonvector content. 
+
+Metadata fields are useful for filters, especially if metadata includes origin information about the source document. You can't filter on a vector field directly, but you can set prefilter or postfilter modes to filter before or after vector query execution.
+
 ### Schema generated by the Import and vectorize data wizard
 
 We recommend the [Import and vectorize data wizard](search-get-started-portal-import-vectors.md) for evaluation and proof-of-concept testing. The wizard generates the example schema in this section.
@@ -182,11 +192,13 @@ All vector indexing and query requests target an index. Endpoints are usually on
 | `<your-service>.search.windows.net/indexes` | Targets the indexes collection. Used when creating, listing, or deleting an index. Admin rights are required for these operations, available through admin [API keys](search-security-api-keys.md) or a [Search Contributor role](search-security-rbac.md#built-in-roles-used-in-search). |
 | `<your-service>.search.windows.net/indexes/<your-index>/docs` | Targets the documents collection of a single index. Used when querying an index or data refresh. For queries, read rights are sufficient, and available through query API keys or a data reader role. For data refresh, admin rights are required. |
 
-#### How to connect to Azure AI Search
+### How to connect to Azure AI Search
+
+1. [Make sure you have permissions](search-security-rbac.md) or an [API access key](search-security-api-keys.md). Unless you're querying an existing index, you need admin rights or a contributor role assignment to manage and view content on a search service.
 
-1. [Start with the Azure portal](https://portal.azure.com). Azure subscribers, or the person who created the search service, can manage the search service in the Azure portal. An Azure subscription requires Contributor or above permissions to create or delete services. This permission level is sufficient for fully managing a search service in the Azure portal.
+1. [Start with the Azure portal](https://portal.azure.com). The person who created the search service can view and manage the search service, including granting access to others through the **Access control (IAM)** page.
 
-1. Try other clients for programmatic access. We recommend the quickstarts and samples for first steps:
+1. Move on to other clients for programmatic access. We recommend the quickstarts and samples for first steps:
 
    + [Quickstart: REST](search-get-started-vector.md)
    + [Vector samples](https://github.com/Azure/azure-search-vector-samples/blob/main/README.md)