|
| 1 | += Vector Search Index Architecture |
| 2 | +:page-topic-type: concept |
| 3 | +:description: Vector Search indexes use features from traditional Search indexes, with unique indexing algorithms and features that allow you to compare vectors in nearest neighbor searches. |
| 4 | +:page-toclevels: 3 |
| 5 | + |
| 6 | +[abstract] |
| 7 | +{description} |
| 8 | + |
| 9 | +A Vector Search index still relies on <<sync,>> and uses <<segments,>> to manage merging and persisting data to disk in your cluster. |
| 10 | +All changes from Database Change Protocol (DCP) and the Data Service are introduced to a Search index in batches, which are further managed by segments. |
| 11 | + |
| 12 | +[#sync] |
| 13 | +== Synchronization with Database Change Protocol (DCP) and the Data Service |
| 14 | + |
| 15 | +The Search Service uses batches to process data that comes in from xref:server:learn:clusters-and-availability/intra-cluster-replication.adoc#database-change-protocol[DCP] and the xref:server:learn:services-and-indexes:services/data-service.adoc[Data Service]. |
| 16 | +DCP and Data Service changes are introduced gradually, based on available memory on Search Service nodes, until reindexing operations for an index are complete. |
| 17 | + |
| 18 | +The Search Service can merge batches into a single batch before they're sent to the disk write queue, to reduce the resources required for batch processing. |
| 19 | + |
| 20 | +The Search Service maintains index snapshots on each Search index partition. |
| 21 | +These snapshots contain a representation of document mutations on either a write queue, or in storage. |
| 22 | + |
| 23 | +If the Search Service loses connection to the Data Service, the Search Service compares its rollback sequence numbers in its snapshots with the Data Service when the connection is reestablished. |
| 24 | +If the index snapshots on the Search Service are too far ahead, the Search Service performs a full rollback to get back in sync with the Data Service. |
| 25 | + |
| 26 | +[#segments] |
| 27 | +== Search Index Segments |
| 28 | + |
| 29 | +Search and Vector Search indexes in Couchbase Server are built with segments. |
| 30 | + |
| 31 | +All Search indexes contain a root segment, which includes all data for the Search index but excludes any segments that might be stale. |
| 32 | +Stale segments are eventually removed by the Search Services's persister or merger routines. |
| 33 | + |
| 34 | +The persister reads in-memory segments from the disk write queue and flushes them to disk, completing batch operations as part of <<sync,>>. |
| 35 | +The merger works with the persister to consolidate flushed files and flush the consolidated results back through the persister - while purging the smaller, older files. |
| 36 | + |
| 37 | +The persister and merger interact to continuously flush and merge new in-memory segments to disk, and remove stale segments. |
| 38 | + |
| 39 | +Segments are marked as stale when they're replaced by a new merged segment created by the merger. |
| 40 | +Stale segments are deleted when they're no longer used by any new queries. |
| 41 | + |
| 42 | +As smaller segments are merged together through the merger routine, the Search Service automatically runs any needed retraining for Vector Search indexes. |
| 43 | +The segments for a Vector Search index can contain different index types and use a separate indexing pipeline, choosing the appropriate indexing algorithm based on the size of your available documents. |
| 44 | + |
| 45 | +== Vector Search and FAISS |
| 46 | + |
| 47 | +Vector Search specifically uses https://faiss.ai/index.html[FAISS^] indexes. |
| 48 | +Any vectors inside your documents are indexed using FAISS, to create a new query vector that can be searched for similar vectors inside your Vector Search index. |
| 49 | + |
| 50 | +Vector Search chooses the best https://github.com/facebookresearch/faiss/wiki/Faiss-indexes[FAISS index class^], or vector search algorithm, for your data, and automatically tunes parameters to provide a balance of recall and latency. |
| 51 | +You can choose to prioritize recall, latency, or memory efficiency with the xref:search:child-field-options-reference.adoc#optimized[Optimized For] setting on your index. |
| 52 | +You can also choose to xref:fine-tune-vector-search.adoc[fine tune your Vector Search queries] to override the default balancing for your index, and change the number of centroids or probes searched in a query. |
| 53 | + |
| 54 | +The FAISS indexes created for your vector data can be: |
| 55 | + |
| 56 | +* <<flat,>> |
| 57 | +* <<ivf,>> |
| 58 | + |
| 59 | +The specific type of index used depends on the number of vectors in your dataset: |
| 60 | + |
| 61 | +|==== |
| 62 | +| Vector Count | Index Types | Description |
| 63 | + |
| 64 | +| >=10,000 |
| 65 | +| IVF with scalar quantization |
| 66 | +a| Vectors are indexed with <<ivf,>> indexes and <<scalar-quant,>>. |
| 67 | + |
| 68 | +If xref:search:child-field-options-reference.adoc#optimized[Optimized For] is set to *recall* or *latency*, Vector Search uses 8bit scalar quantization. |
| 69 | +If set to *memory-efficient*, Vector Search uses 4bit scalar quantization. |
| 70 | + |
| 71 | +| >=1000 |
| 72 | +| IVF with Flat |
| 73 | +| Vectors are indexed with <<ivf,>> combined with <<flat,>>. |
| 74 | +Indexes do not use <<scalar-quant,>>. |
| 75 | + |
| 76 | +| <1000 |
| 77 | +| Flat |
| 78 | +| Vectors are indexed with <<flat,>>. |
| 79 | +Indexes do not use <<scalar-quant,>>. |
| 80 | +|==== |
| 81 | + |
| 82 | +[#flat] |
| 83 | +=== FLAT Indexes |
| 84 | + |
| 85 | +The most basic kind of index that Vector Search can use for your vectors is a flat index. |
| 86 | + |
| 87 | +Vector Search uses flat indexes for data that contains less than 1000 vectors. |
| 88 | + |
| 89 | +Flat indexes are a list of vectors. |
| 90 | +Searches run on a nearest neighbor process, based on examining the query vector against each vector in the index and calculating the distance. |
| 91 | +Results for flat indexes are very accurate, but performance does not scale well as a dataset grows. |
| 92 | + |
| 93 | +If a Vector Search index uses only flat indexes, no training is required - IDs are mapped directly to vectors with exact vector comparisons, with no need for preprocessing or learning on the data. |
| 94 | + |
| 95 | +[#ivf] |
| 96 | +=== Inverted File Index (IVF) |
| 97 | + |
| 98 | +For reduced latency, Vector Search can also use Inverted File Indexes (IVF). |
| 99 | + |
| 100 | +Vector Search uses a combination of IVF and flat indexes for data that contains between 1000 and 9999 vectors. |
| 101 | +For even larger datasets, Vector Search uses IVF indexes with <<scalar-quant,>>. |
| 102 | + |
| 103 | +IVF creates partitions called Voronoi cells in an index. |
| 104 | +The total number of cells is the *nlist* parameter. |
| 105 | + |
| 106 | +Every cell has a centroid. |
| 107 | +Every vector in the processed dataset is assigned to a cell that corresponds to its nearest centroid. |
| 108 | + |
| 109 | +In an IVF index, Vector Search first tries to find a centroid vector closest to the query vector. |
| 110 | +After finding this closest centroid vector, Vector Search uses the default `nprobe` and `max_codes` values to search over adjoining cells to the closest centroid and finds the top `k` number of vectors. |
| 111 | + |
| 112 | +IVF index searches are not exhaustive searches. |
| 113 | +You can increase accuracy by changing the `max_nprobe_pct` parameter or `max_codes_pct` when you xref:fine-tune-vector-search.adoc[fine tune your Vector Search queries]. |
| 114 | + |
| 115 | +The Search Service automatically trains larger IVF indexes to learn the data distribution of your vectors, and the centroids of cells in your dataset. |
| 116 | +The training data helps to encode and compress the vectors in your index with <<scalar-quant,>>. |
| 117 | +All training occurs during building and merging <<segments,>>. |
| 118 | + |
| 119 | +IVF indexes that also use flat indexing automatically train to determine the centroids of cells, but still use exact vector comparisons within each cell. |
| 120 | +Training still occurs while building and merging <<segments,>>. |
| 121 | + |
| 122 | +[#scalar-quant] |
| 123 | +==== Scalar Quantization |
| 124 | + |
| 125 | +Vector Search uses scalar quantization on large datasets to reduce the size of your indexes. |
| 126 | + |
| 127 | +Scalar quantization is an important data compression technique that turns the floating point values that could be present in a large vector into low-dimensional integers. |
| 128 | +For example, a float32 value could be reduced to an int8 value. |
| 129 | + |
| 130 | +Scalar quantization in Vector Search does not have a significant effect on the recall, or accuracy, of query results on large datasets. |
| 131 | + |
| 132 | +Vector Search uses both 8bit and 4bit scalar quantization for indexes, based on your xref:search:child-field-options-reference.adoc#optimized[Optimized For] setting. |
| 133 | + |
| 134 | +== Search Request Processing |
| 135 | + |
| 136 | +The Search Service uses a scatter-gather process for running all Search queries, when there are multiple nodes in the cluster running the Search Service. |
| 137 | + |
| 138 | +The Search Service node that receives the Search request is assigned as the coordinating node. |
| 139 | +Using https://grpc.io/[gRPC^], the coordinating node scatters the request to all other partitions for the Search or Vector Search index in the request across other nodes. |
| 140 | +The coordinating node applies filters to the results received from the other partitions, and returns the final result set. |
| 141 | + |
| 142 | +Results are scored, and based on the xref:search:search-request-params.adoc#sort[Sort Object] provided in the Search request, returned in a list. |
| 143 | + |
| 144 | +For a Vector Search query, search results include the top `k` nearest neighbor vectors to the vector in the Search query. |
| 145 | +For more information about how results are scored and returned for Search requests, see xref:search:run-searches.adoc#scoring[Scoring for Search Queries]. |
| 146 | + |
| 147 | +== See Also |
| 148 | + |
| 149 | +* xref:fine-tune-vector-search.adoc[] |
| 150 | +* xref:search:search-request-params.adoc[] |
| 151 | +* xref:create-vector-search-index-rest-api.adoc[] |
| 152 | +* xref:create-vector-search-index-ui.adoc[] |
0 commit comments