You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: articles/search/query-odata-filter-orderby-syntax.md
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -7,7 +7,7 @@ author: bevloh
7
7
ms.author: beloh
8
8
ms.service: cognitive-search
9
9
ms.topic: conceptual
10
-
ms.date: 06/06/2024
10
+
ms.date: 08/19/2024
11
11
---
12
12
13
13
# OData language overview for `$filter`, `$orderby`, and `$select` in Azure AI Search
@@ -27,6 +27,8 @@ Once you understand these common concepts, you can continue with the top-level s
27
27
28
28
The syntax of these expressions is distinct from the [simple](query-simple-syntax.md) or [full](query-lucene-syntax.md) query syntax used in the **search** parameter, although there's some overlap in the syntax for referencing fields.
29
29
30
+
For examples in other languages such as Python or C#, see the examples in the [azure-search-vector-samples](https://github.com/Azure/azure-search-vector-samples) repository.
31
+
30
32
> [!NOTE]
31
33
> Terminology in Azure AI Search differs from the [OData standard](https://www.odata.org/documentation/) in a few ways. What we call a **field** in Azure AI Search is called a **property** in OData, and similarly for **field path** versus **property path**. An **index** containing **documents** in Azure AI Search is referred to more generally in OData as an **entity set** containing **entities**. The Azure AI Search terminology is used throughout this reference.
# Add a filter in a vector query in Azure AI Search
16
16
17
-
You can set a vector filter modes on a vector query to specify whether you want filtering before or after query execution.
17
+
You can define a vector query request that includes a [filter expression](search-filters.md) to add inclusion or exclusion criteria to your queries. In this article, learn how to:
18
+
19
+
> [!div class="checklist"]
20
+
> -[Define a `filter` expression](#define-a-filter)
21
+
> -[Set the `vectorFilterMode` for pre-query or post-query filtering](#set-the-vectorfiltermode)
22
+
23
+
This article uses REST for illustration. For code samples in other languages, see the [azure-search-vector-samples](https://github.com/Azure/azure-search-vector-samples) GitHub repository for end-to-end solutions that include vector queries.
24
+
25
+
You can also use [Search Explorer](search-get-started-portal-import-vectors.md#check-results) in the Azure portal to query vector content. If you use the JSON view, you can add filters and specify the filter mode.
26
+
27
+
## How filtering works in a vector query
28
+
29
+
Filters apply to `filterable` nonvector fields, either a string field or numeric, to include or exclude search documents based on filter criteria. Although a vector field isn't filterable itself, filters can be applied to other fields in the same index, including or excluding the documents that also contain vector fields.
30
+
31
+
Filters are applied before or after query execution based on the `vectorFilterMode` parameter.
32
+
33
+
## Define a filter
18
34
19
35
Filters determine the scope of a vector query. Filters are set on and iterate over nonvector string and numeric fields attributed as `filterable` in the index, but the purpose of a filter determines *what* the vector query executes over: the entire searchable space, or the contents of a search result.
20
36
21
-
This article provides conceptual information, describing each filter mode and providing guidance on when to use each one.
37
+
If you don't have source fields with text or numeric values, check for document metadata, such as LastModified or CreatedBy properties, that might be useful in a metadata filter.
38
+
39
+
### [**2024-07-01**](#tab/filter-2024-07-01)
40
+
41
+
[**2024-07-01**](/rest/api/searchservice/search-service-api-versions#2024-07-01) is the stable version for this API. It has:
42
+
43
+
-`vectorFilterMode` for prefilter (default) or postfilter [filtering modes](vector-search-filters.md).
44
+
-`filter` provides the criteria.
45
+
46
+
In the following example, the vector is a representation of this query string: "what Azure services support full text search". The query targets the `contentVector` field. The actual vector has 1536 embeddings, so it's trimmed in this example for readability.
47
+
48
+
The filter criteria are applied to a filterable text field (`category` in this example) before the search engine executes the vector query.
49
+
50
+
```http
51
+
POST https://{{search-service-name}}.search.windows.net/indexes/{{index-name}}/docs/search?api-version=2024-07-01
[**2024-05-01-preview**](/rest/api/searchservice/search-service-api-versions#2024-05-01-preview) introduces filter options. This version adds:
80
+
81
+
-`vectorFilterMode` for prefilter (default) or postfilter [filtering modes](vector-search-filters.md).
82
+
-`filter` provides the criteria.
83
+
84
+
In the following example, the vector is a representation of this query string: "what Azure services support full text search". The query targets the `contentVector` field. The actual vector has 1536 embeddings, so it's trimmed in this example for readability.
85
+
86
+
The filter criteria are applied to a filterable text field (`category` in this example) before the search engine executes the vector query.
87
+
88
+
```http
89
+
POST https://{{search-service-name}}.search.windows.net/indexes/{{index-name}}/docs/search?api-version=2024-05-01-preview
90
+
Content-Type: application/json
91
+
api-key: {{admin-api-key}}
92
+
{
93
+
"count": true,
94
+
"select": "title, content, category",
95
+
"filter": "category eq 'Databases'",
96
+
"vectorFilterMode": "preFilter",
97
+
"vectorQueries": [
98
+
{
99
+
"kind": "vector",
100
+
"vector": [
101
+
-0.009154141,
102
+
0.018708462,
103
+
. . .
104
+
-0.02178128,
105
+
-0.00086512347
106
+
],
107
+
"exhaustive": true,
108
+
"fields": "contentVector",
109
+
"k": 5
110
+
}
111
+
]
112
+
}
113
+
```
114
+
115
+
---
116
+
117
+
## Set the vectorFilterMode
118
+
119
+
The vectorFilterMode query parameter determines whether the filter is applied before or after vector query execution.
22
120
23
-
For instructions on setting up the vector filter in your query, see [Vector query with filter](vector-search-how-to-query.md#vector-query-with-filter).
121
+
### Use prefilter mode
24
122
25
-
## Prefilter mode
123
+
Prefiltering applies filters before query execution, reducing the search surface area over which the vector search algorithm looks for similar content.
26
124
27
-
Prefiltering applies filters before query execution, reducing the search surface area over which the vector search algorithm looks for similar content. In a vector query, `preFilter` is the default.
125
+
In a vector query, `preFilter` is the default.
28
126
29
127
:::image type="content" source="media/vector-search-filters/pre-filter.svg" alt-text="Diagram of prefilters." border="true" lightbox="media/vector-search-filters/pre-filter.png":::
30
128
31
-
##Postfilter mode
129
+
### Use postfilter mode
32
130
33
131
Post-filtering applies filters after query execution, narrowing the search results.
34
132
35
133
:::image type="content" source="media/vector-search-filters/post-filter.svg" alt-text="Diagram of post-filters." border="true" lightbox="media/vector-search-filters/post-filter.png":::
36
134
37
-
## Benchmark testing of vector filter modes
135
+
###Benchmark testing of vector filter modes
38
136
39
137
To understand the conditions under which one filter mode performs better than the other, we ran a series of tests to evaluate query outcomes over small, medium, and large indexes.
40
138
@@ -93,7 +191,7 @@ Outcomes were measured in Queries Per Second (QPS).
93
191
+ Postfiltering is for customers who:
94
192
95
193
+ value speed over selection (postfiltering can return fewer than `k` results)
96
-
+ use filters that are not overly selective
194
+
+ use filters that aren't overly selective
97
195
+ have indexes of sufficient size such that prefiltering performance is unacceptable
Copy file name to clipboardExpand all lines: articles/search/vector-search-how-to-create-index.md
+3-3Lines changed: 3 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -12,9 +12,9 @@ ms.date: 08/05/2024
12
12
13
13
# Create a vector index
14
14
15
-
In Azure AI Search, a *vector store* has an index schema that defines vector and nonvector fields, a vector configuration for algorithms that create and compress the embedding space, and settings on vector field definitions that are used in query requests. The [Create or Update Index](/rest/api/searchservice/indexes/create-or-update) API creates the vector store.
15
+
In Azure AI Search, a *vector store* has an index schema that defines vector and nonvector fields, a vector configuration for algorithms that create and compress the embedding space, and settings on vector field definitions that are used in query requests.
16
16
17
-
Follow these steps to index vector data:
17
+
The [Create or Update Index](/rest/api/searchservice/indexes/create-or-update) API creates the vector store. Follow these steps to index vector data:
18
18
19
19
> [!div class="checklist"]
20
20
> + Define a schema with vector algorithms and optional compression
@@ -24,7 +24,7 @@ Follow these steps to index vector data:
24
24
This article explains the workflow and uses REST for illustration. Once you understand the basic workflow, continue with the Azure SDK code samples in the [azure-search-vector-samples](https://github.com/Azure/azure-search-vector-samples) repository for guidance on using these features in test and production code.
25
25
26
26
> [!TIP]
27
-
> Use the Azure portal to [create a vector index](search-get-started-portal-import-vectors.md) and try out integrated vectorization.
27
+
> Use the Azure portal to [create a vector index](search-get-started-portal-import-vectors.md) and try integrated data chunking and vectorization.
Copy file name to clipboardExpand all lines: articles/search/vector-search-how-to-query.md
+1-89Lines changed: 1 addition & 89 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ ms.service: cognitive-search
9
9
ms.custom:
10
10
- build-2024
11
11
ms.topic: how-to
12
-
ms.date: 08/05/2024
12
+
ms.date: 08/19/2024
13
13
---
14
14
15
15
# Create a vector query in Azure AI Search
@@ -18,7 +18,6 @@ In Azure AI Search, if you have a [vector index](vector-search-how-to-create-ind
18
18
19
19
> [!div class="checklist"]
20
20
> +[Query vector fields](#vector-query-request)
21
-
> +[Filter a vector query](#vector-query-with-filter)
22
21
> +[Query multiple vector fields at once](#multiple-vector-fields)
23
22
> +[Set vector weights](#vector-weighting)
24
23
> +[Query with integrated vectorization](#query-with-integrated-vectorization)
@@ -256,93 +255,6 @@ If you do want vector fields in the result, here's an example of the response st
256
255
257
256
+ Fields in search results are either all `retrievable` fields, or fields in a `select` clause. During vector query execution, the match is made on vector data alone. However, a response can include any `retrievable` field in an index. Because there's no facility for decoding a vector field result, the inclusion of nonvector text fields is helpful for their human readable values.
258
257
259
-
## Vector query with filter
260
-
261
-
A query request can include a vector query and a [filter expression](search-filters.md). Filters apply to `filterable` nonvector fields, either a string field or numeric, and are useful for including or excluding search documents based on filter criteria. Although a vector field isn't filterable itself, filters can be applied to other fields in the same index.
262
-
263
-
You can apply filters as exclusion criteria before the query executes, or after query execution to filter search results. For a comparison of each mode and the expected performance based on index size, see [Filters in vector queries](vector-search-filters.md).
264
-
265
-
> [!TIP]
266
-
> If you don't have source fields with text or numeric values, check for document metadata, such as LastModified or CreatedBy properties, that might be useful in a metadata filter.
267
-
268
-
### [**2024-07-01**](#tab/filter-2024-07-01)
269
-
270
-
[**2024-07-01**](/rest/api/searchservice/search-service-api-versions#2024-07-01) is the stable version for this API. It has:
271
-
272
-
+`vectorFilterMode` for prefilter (default) or postfilter [filtering modes](vector-search-filters.md).
273
-
+`filter` provides the criteria.
274
-
275
-
In the following example, the vector is a representation of this query string: "what Azure services support full text search". The query targets the `contentVector` field. The actual vector has 1536 embeddings, so it's trimmed in this example for readability.
276
-
277
-
The filter criteria are applied to a filterable text field (`category` in this example) before the search engine executes the vector query.
278
-
279
-
```http
280
-
POST https://{{search-service-name}}.search.windows.net/indexes/{{index-name}}/docs/search?api-version=2024-07-01
[**2024-05-01-preview**](/rest/api/searchservice/search-service-api-versions#2024-05-01-preview) introduces filter options. This version adds:
309
-
310
-
+`vectorFilterMode` for prefilter (default) or postfilter [filtering modes](vector-search-filters.md).
311
-
+`filter` provides the criteria.
312
-
313
-
In the following example, the vector is a representation of this query string: "what Azure services support full text search". The query targets the `contentVector` field. The actual vector has 1536 embeddings, so it's trimmed in this example for readability.
314
-
315
-
The filter criteria are applied to a filterable text field (`category` in this example) before the search engine executes the vector query.
316
-
317
-
```http
318
-
POST https://{{search-service-name}}.search.windows.net/indexes/{{index-name}}/docs/search?api-version=2024-05-01-preview
319
-
Content-Type: application/json
320
-
api-key: {{admin-api-key}}
321
-
{
322
-
"count": true,
323
-
"select": "title, content, category",
324
-
"filter": "category eq 'Databases'",
325
-
"vectorFilterMode": "preFilter",
326
-
"vectorQueries": [
327
-
{
328
-
"kind": "vector",
329
-
"vector": [
330
-
-0.009154141,
331
-
0.018708462,
332
-
. . .
333
-
-0.02178128,
334
-
-0.00086512347
335
-
],
336
-
"exhaustive": true,
337
-
"fields": "contentVector",
338
-
"k": 5
339
-
}
340
-
]
341
-
}
342
-
```
343
-
344
-
---
345
-
346
258
## Multiple vector fields
347
259
348
260
You can set the "vectorQueries.fields" property to multiple vector fields. The vector query executes against each vector field that you provide in the `fields` list. When querying multiple vector fields, make sure each one contains embeddings from the same embedding model, and that the query is also generated from the same embedding model.
0 commit comments