Skip to content

Commit 680ab12

Browse files
jcodellajcodella
authored
Updated to include HSNW and filtered vector search
1 parent 856ba3d commit 680ab12

File tree

1 file changed

+165
-117
lines changed

1 file changed

+165
-117
lines changed

‎articles/cosmos-db/mongodb/vcore/vector-search.md

Lines changed: 165 additions & 117 deletions
Original file line numberDiff line numberDiff line change
@@ -25,9 +25,72 @@ Vector search is a method that helps you find similar items based on their data
2525

2626
By integrating vector search capabilities natively, you can unlock the full potential of your data in applications that are built on top of the [OpenAI API](../../../ai-services/openai/concepts/understand-embeddings.md). You can also create custom-built solutions that use vector embeddings.
2727

28-
## Use the createIndexes template to create a vector index
28+
## Create a vector index
29+
To perform vector similiarity search over vector properties in your documents, you'll have to first create a _vector index_.
2930

30-
To create a vector index, use the following `createIndexes` template:
31+
### Create a vector index using HNSW
32+
33+
You can create (Hierarchical Navigable Small World) indexes on M40 cluster tiers and higher. To use create the HSNW index, you need to create a vector index with the `"kind"` parameter set to `"vector-hnsw"` following the template below:
34+
35+
```javascript
36+
{
37+
"createIndexes": "<collection_name>",
38+
"indexes": [
39+
{
40+
"name": "<index_name>",
41+
"key": {
42+
"<path_to_property>": "cosmosSearch"
43+
},
44+
"cosmosSearchOptions": {
45+
"kind": "vector-hnsw",
46+
"m": <integer_value>,
47+
"efConstruction": <integer_value>,
48+
"similarity": "<string_value>",
49+
"dimensions": <integer_value>
50+
}
51+
}
52+
]
53+
}
54+
```
55+
56+
|Field |Type |Description |
57+
|---------|---------|---------|
58+
| `kind` | string | Type of vector index to create. Type of vector index to create. Primarily, `vector-ivf` is supported. `vector-hnsw` is available as a preview feature that requires enablement via [Azure Feature Enablement Control](../../../azure-resource-manager/management/preview-features.md).|
59+
|`m` |integer |The max number of connections per layer (`16` by default, minimum value is `2`, maximum value is `100`). Higher m is suitable for datasets with high dimensionality and/or high accuracy requirements. |
60+
|`efConstruction` |integer |the size of the dynamic candidate list for constructing the graph (`64` by default, minimum value is `4`, maximum value is `1000`). Higher `efConstruction` will result in better index quality and higher accuracy, but it will also increase the time required to build the index. `efConstruction` has to be at least `2 * m` |
61+
|`similarity` |string |Similarity metric to use with the index. Possible options are `COS` (cosine distance), `L2` (Euclidean distance), and `IP` (inner product). |
62+
|`dimensions` |integer |Number of dimensions for vector similarity. The maximum number of supported dimensions is `2000`. |
63+
64+
65+
> [!WARNING]
66+
> Using the HSNW vector index (preview) with large datasets can result in resource running out of memory, or reducing the performance of other operations running on your database. To reduce the chance of this happening, we recommend to:
67+
> - Only use HNSW indexes on a cluster tier of M40 or higher.
68+
> - Scale to a higher cluster tier or reduce the size of the database if your encounter errors.
69+
70+
### Perform a vector search with HNSW
71+
To perform a vector search, use the `$search` aggregation pipeline stage the query with the `cosmosSearch` operator.
72+
```javascript
73+
{
74+
"$search": {
75+
"cosmosSearch": {
76+
"vector": <query_vector>,
77+
"path": "<path_to_property>",
78+
"k": <num_results_to_return>,
79+
"efSearch": <integer_value>
80+
},
81+
}
82+
}
83+
}
84+
85+
```
86+
|Field |Type |Description |
87+
|---------|---------|---------|
88+
|`efSearch` |integer |The size of the dynamic candidate list for search (`40` by default). A higher value provides better recall at the cost of speed. |
89+
|`k` |integer |The number of results to return. it should be less than or equal to `efSearch` |
90+
91+
### Create an vector index using IVF
92+
93+
To create a vector index using the IVF (Inverted File) algorithm, use the following `createIndexes` template and set the `"kind"` paramter to `"vector-ivf"`:
3194

3295
```json
3396
{
@@ -65,40 +128,68 @@ To create a vector index, use the following `createIndexes` template:
65128
>
66129
> If you're experimenting with a new scenario or creating a small demo, you can start with `numLists` set to `1` to perform a brute-force search across all vectors. This should provide you with the most accurate results from the vector search, however be aware that the search speed and latency will be slow. After your initial setup, you should go ahead and tune the `numLists` parameter using the above guidance.
67130
131+
### Perform a vector search with IVF
132+
133+
To perform a vector search, use the `$search` aggregation pipeline stage in a MongoDB query. To use the `cosmosSearch` index, use the new `cosmosSearch` operator.
134+
135+
```json
136+
{
137+
{
138+
"$search": {
139+
"cosmosSearch": {
140+
"vector": <query_vector>,
141+
"path": "<path_to_property>",
142+
"k": <num_results_to_return>,
143+
},
144+
"returnStoredSource": True }},
145+
{
146+
"$project": { "<custom_name_for_similarity_score>": {
147+
"$meta": "searchScore" },
148+
"document" : "$$ROOT"
149+
}
150+
}
151+
}
152+
```
153+
To retrieve the similarity score (`searchScore`) along with the documents found by the vector search, use the `$project` operator to include `searchScore` and rename it as `<custom_name_for_similarity_score>` in the results. Then the document is also projected as nested object. Note that the similarity score is calculated using the metric defined in the vector index.
154+
155+
68156
> [!IMPORTANT]
69157
> Vectors must be a `number[]` to be indexed. Using another type, such as `double[]`, prevents the document from being indexed. Non-indexed documents won't be returned in the result of a vector search.
70158
71159
## Examples
72160

73161
The following examples show you how to index vectors, add documents that have vector properties, perform a vector search, and retrieve the index configuration.
74162

75-
### Create a vector index
163+
```javascript
164+
use test;
165+
166+
db.createCollection("exampleCollection");
76167

77168
```javascript
78169
use test;
79170
80171
db.createCollection("exampleCollection");
81172
82-
db.runCommand({
83-
createIndexes: 'exampleCollection',
84-
indexes: [
85-
{
86-
name: 'vectorSearchIndex',
87-
key: {
88-
"vectorContent": "cosmosSearch"
89-
},
90-
cosmosSearchOptions: {
91-
kind: 'vector-ivf',
92-
numLists: 3,
93-
similarity: 'COS',
94-
dimensions: 3
95-
}
96-
}
97-
]
173+
db.runCommand({
174+
"createIndexes": "exampleCollection",
175+
"indexes": [
176+
{
177+
"name": "VectorSearchIndex",
178+
"key": {
179+
"contentVector": "cosmosSearch"
180+
},
181+
"cosmosSearchOptions": {
182+
"kind": "vector-hnsw",
183+
"m": 16,
184+
"efConstruction": 64,
185+
"similarity": "COS",
186+
"dimensions": 3
187+
}
188+
}
189+
]
98190
});
99191
```
100-
101-
This command creates a `vector-ivf` index against the `vectorContent` property in the documents that are stored in the specified collection, `exampleCollection`. The `cosmosSearchOptions` property specifies the parameters for the IVF vector index. If your document has the vector stored in a nested property, you can set this property by using a dot notation path. For example, you might use `text.vectorContent` if `vectorContent` is a subproperty of `text`.
192+
This command creates an HNSW index against the `contentVector` property in the documents that are stored in the specified collection, `exampleCollection`. The `cosmosSearchOptions` property specifies the parameters for the HNSW vector index. If your document has the vector stored in a nested property, you can set this property by using a dot notation path. For example, you might use `text.contentVector` if `contentVector` is a subproperty of `text`.
102193

103194
### Add vectors to your database
104195

@@ -115,50 +206,23 @@ db.exampleCollection.insertMany([
115206
116207
### Perform a vector search
117208
118-
To perform a vector search, use the `$search` aggregation pipeline stage in a MongoDB query. To use the `cosmosSearch` index, use the new `cosmosSearch` operator.
119-
120-
```json
121-
{
122-
{
123-
"$search": {
124-
"cosmosSearch": {
125-
"vector": <vector_to_search>,
126-
"path": "<path_to_property>",
127-
"k": <num_results_to_return>,
128-
},
129-
"returnStoredSource": True }},
130-
{
131-
"$project": { "<custom_name_for_similarity_score>": {
132-
"$meta": "searchScore" },
133-
"document" : "$$ROOT"
134-
}
135-
}
136-
}
137-
```
138-
To retrieve the similarity score (`searchScore`) along with the documents found by the vector search, use the `$project` operator to include `searchScore` and rename it as `<custom_name_for_similarity_score>` in the results. Then the document is also projected as nested object. Note that the similarity score is calculated using the metric defined in the vector index.
139-
140-
### Query vectors and vector distances (aka similarity scores) using $search"
141-
142-
Continuing with the last example, create another vector, `queryVector`. Vector search measures the distance between `queryVector` and the vectors in the `vectorContent` path of your documents. You can set the number of results that the search returns by setting the parameter `k`, which is set to `2` here. You can also set `nProbes`, which is an integer that controls the number of nearby clusters that are inspected in each search. A higher value may improve accuracy, however the search will be slower as a result. This is an optional parameter with a default value of 1 and cannot be larger than the `numLists` value specified in the vector index.
209+
Continuing with the last example, create another vector, `queryVector`. Vector search measures the distance between `queryVector` and the vectors in the `contentVector` path of your documents. You can set the number of results that the search returns by setting the parameter `k`, which is set to `2` here. You can also set `efSearch`, which is an integer that controls the size of the candidate vector list. A higher value may improve accuracy, however the search will be slower as a result. This is an optional parameter with a default value of 40.
143210
144211
145212
```javascript
146213
const queryVector = [0.52, 0.28, 0.12];
147214
db.exampleCollection.aggregate([
148215
{
149-
$search: {
150-
"cosmosSearch": {
151-
"vector": queryVector,
152-
"path": "vectorContent",
153-
"k": 2
154-
},
155-
"returnStoredSource": true }},
156-
{
157-
"$project": { "similarityScore": {
158-
"$meta": "searchScore" },
159-
"document" : "$$ROOT"
160-
}
216+
"$search": {
217+
"cosmosSearch": {
218+
"vector": "queryVector",
219+
"path": "contentVector",
220+
"k": 2,
221+
"efSearch": 40
222+
},
223+
}
161224
}
225+
}
162226
]);
163227
```
164228
@@ -202,11 +266,12 @@ In this example, `vectorIndex` is returned with all the `cosmosSearch` parameter
202266
{ v: 2, key: { _id: 1 }, name: '_id_', ns: 'test.exampleCollection' },
203267
{
204268
v: 2,
205-
key: { vectorContent: 'cosmosSearch' },
269+
key: { contentVector: 'cosmosSearch' },
206270
name: 'vectorSearchIndex',
207271
cosmosSearch: {
208-
kind: 'vector-ivf',
209-
numLists: 3,
272+
kind: 'vector-hnsw',
273+
m: 40,
274+
efConstruction: 64
210275
similarity: 'COS',
211276
dimensions: 3
212277
},
@@ -215,74 +280,57 @@ In this example, `vectorIndex` is returned with all the `cosmosSearch` parameter
215280
]
216281
```
217282
218-
## HNSW vector index (preview)
219-
220-
HNSW stands for Hierarchical Navigable Small World, a graph-based data structure that partitions vectors into clusters and subclusters. With HNSW, you can perform fast approximate nearest neighbor search at higher speeds with greater accuracy.
283+
## Filtered vector search (preview)
284+
You can now execute vector searches with any supported query filter such as `$lt, $lte, $eq, $neq, $gte, $gt, $in, $nin, and $regex`. Enable the "filtering vector search" feature in the "Preview Features" tab of your Azure Subscription. Learn more about preview features [here](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/preview-features?tabs=azure-portal).
221285
222-
As a preview feature, this must be enabled using Azure Feature Enablement Control (AFEC) by selecting the "mongoHnswIndex" feature. For more information, see [enable preview features](../../../azure-resource-manager/management/preview-features.md).
286+
First, you'll need to define an index for your filter in addition to a vector index. For example, you can define the filter index on a property
223287

224-
### Create an HNSW vector index
288+
```javascript
289+
db.runCommand({
290+
"createIndexes": "<collection_name",
291+
"indexes": [ {
292+
"key": {
293+
"<property_to_filter>": 1
294+
},
295+
"name": "<name_of_filter_index>"
296+
}
297+
]
298+
});
299+
```
225300

226-
To use HNSW as your index algorithm, you need to create a vector index with the `kind` parameter set to "vector-hnsw" following the template below:
301+
Next, you can add the `"filter"` term to your vector search as shown below. In this example the filter is looking for documents where the `"title"` property is not in the list of `["not in this text", "or this text"]`.
227302

228303
```javascript
229-
{
230-
"createIndexes": "<collection_name>",
231-
"indexes": [
232-
{
233-
"name": "<index_name>",
234-
"key": {
235-
"<path_to_property>": "cosmosSearch"
236-
},
237-
"cosmosSearchOptions": {
238-
"kind": "vector-hnsw",
239-
"m": <integer_value>,
240-
"efConstruction": <integer_value>,
241-
"similarity": "<string_value>",
242-
"dimensions": <integer_value>
243-
}
244-
}
245-
]
304+
305+
db.exampleCollection.aggregate([
306+
{
307+
'$search': {
308+
"cosmosSearch": {
309+
"vector": "<query_vector>",
310+
"path": <path_to_vector>,
311+
"k": num_results,
312+
"filter": {<property_to_filter>: {"$nin": ["not in this text", "or this text"]}}
313+
},
314+
"returnStoredSource": True }},
315+
{'$project': { 'similarityScore': { '$meta': 'searchScore' }, 'document' : '$$ROOT' }
246316
}
317+
]);
247318
```
319+
> [!IMPORTANT]
320+
> While in preview, filtered vector search may require you to adjust your vector index parameters to achieve higher accuracy. For example, increasing `m`, `efConstruction`, or `efSearch` when using HNSW, or `numLists`, or `nProbes` when using IVF, may lead to better results. You should test your configuration before use to ensure that the results are satisfactory.
248321

249-
|Field |Type |Description |
250-
|---------|---------|---------|
251-
| `kind` | string | Type of vector index to create. Type of vector index to create. Primarily, `vector-ivf` is supported. `vector-hnsw` is available as a preview feature that requires enablement via [Azure Feature Enablement Control](../../../azure-resource-manager/management/preview-features.md).|
252-
|`m` |integer |The max number of connections per layer (`16` by default, minimum value is `2`, maximum value is `100`). Higher m is suitable for datasets with high dimensionality and/or high accuracy requirements. |
253-
|`efConstruction` |integer |the size of the dynamic candidate list for constructing the graph (`64` by default, minimum value is `4`, maximum value is `1000`). Higher `efConstruction` will result in better index quality and higher accuracy, but it will also increase the time required to build the index. `efConstruction` has to be at least `2 * m` |
254-
|`similarity` |string |Similarity metric to use with the index. Possible options are `COS` (cosine distance), `L2` (Euclidean distance), and `IP` (inner product). |
255-
|`dimensions` |integer |Number of dimensions for vector similarity. The maximum number of supported dimensions is `2000`. |
256-
322+
## Use LLM Orchestration tools such
257323

258-
> [!WARNING]
259-
> Using the HSNW vector index (preview) with large datasets can result in resource running out of memory, or reducing the performance of other operations running on your database. To reduce the chance of this happening, we recommend to:
260-
> - Only use HNSW indexes on a cluster tier of M40 or higher.
261-
> - Scale to a higher cluster tier or reduce the size of the database if your encounter errors.
324+
### Use as a vector database with Semantic Kernel
325+
Use Semantic Kernel to orchestrate your information retrieval from Azure Cosmos DB for MongoDB vCore and your LLM. Learn more [here]([https://python.langchain.com/docs/integrations/vectorstores/azure_cosmos_db](https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/connectors/memory/azure_cosmosdb)).
262326

263-
### Perform a vector search with HNSW
264-
To perform a vector search, use the `$search` aggregation pipeline stage the query with the `cosmosSearch` operator.
265-
```javascript
266-
{
267-
"$search": {
268-
"cosmosSearch": {
269-
"vector": <vector_to_search>,
270-
"path": "<path_to_property>",
271-
"k": <num_results_to_return>,
272-
"efSearch": <integer_value>
273-
},
274-
}
275-
}
276-
}
327+
https://github.com/microsoft/semantic-kernel/tree/main/python/semantic_kernel/connectors/memory/azure_cosmosdb
277328

278-
```
279-
|Field |Type |Description |
280-
|---------|---------|---------|
281-
|`efSearch` |integer |The size of the dynamic candidate list for search (`40` by default). A higher value provides better recall at the cost of speed. |
282-
|`k` |integer |The number of results to return. it should be less than or equal to `efSearch` |
329+
### Use as a vector database with LangChain
330+
Use LangChain to orchestrate your information retrieval from Azure Cosmos DB for MongoDB vCore and your LLM. Learn more [here](https://python.langchain.com/docs/integrations/vectorstores/azure_cosmos_db).
283331

284-
## Use as a vector database with LangChain
285-
You can now use LangChain to orchestrate your information retrieval from Azure Cosmos DB for MongoDB vCore and your LLM. Learn more [here](https://python.langchain.com/docs/integrations/vectorstores/azure_cosmos_db).
332+
### Use as a semantic cache with LangChain
333+
Use LangChain and Azure Cosmos DB for MongoDB (vCore) to orchestrate Semantic Caching, using previously recocrded LLM respones that can save you LLM API costs and reduce latency for responses. Learn more [here](https://python.langchain.com/docs/integrations/llms/llm_caching#azure-cosmos-db-semantic-cache)
286334

287335
## Features and limitations
288336

0 commit comments

Comments
 (0)