You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/indexing/index.mdx
+1-2Lines changed: 1 addition & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -47,7 +47,7 @@ Vector indexes can use different quantization methods to compress vectors and im
47
47
| :----------- | :------- | :---------- |
48
48
|`PQ` (Product Quantization) | Default choice for most vector search scenarios. Use when you need to balance index size and recall. | Divides vectors into subvectors and quantizes each subvector independently. Provides a good balance between compression ratio and search accuracy. |
49
49
|`SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
50
-
|`RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. |
50
+
|`RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. |
51
51
|`None/Flat`| Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. |
52
52
53
53
## Understanding the IVF-PQ Index
@@ -149,4 +149,3 @@ Then the greedy search routine operates as follows:
149
149
* At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer.
150
150
* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method.
151
151
* Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10).
Copy file name to clipboardExpand all lines: docs/indexing/quantization.mdx
+6-2Lines changed: 6 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -15,7 +15,7 @@ Use quantization when:
15
15
16
16
LanceDB currently exposes multiple quantized vector index types, including:
17
17
-`IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples.
18
-
-`IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). See [below](#rabitq-quantization) for details.
18
+
-`IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details.
19
19
20
20
`IVF_PQ` is the default indexing option in LanceDB and works well in many cases. However, in cases where more drastic compression is needed, RaBitQ is also a reasonable option.
21
21
@@ -42,6 +42,11 @@ For a deeper dive into the theory and some benchmark results, see the blog post:
42
42
### Using RaBitQ
43
43
44
44
You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`.
45
+
46
+
<Notetitle="Dimension requirement">
47
+
When using `IVF_RQ`, vector dimensions must be divisible by `8`.
48
+
</Note>
49
+
45
50
`num_bits` controls how many bits per dimension are used:
46
51
47
52
## API Reference
@@ -62,4 +67,3 @@ The full list of parameters to the algorithm are listed below.
62
67
Number of samples per partition during training. Higher values may improve accuracy but increase training time.
63
68
-`target_partition_size`: Optional[int], defaults to None
64
69
Target number of vectors per partition. Adjust to control partition granularity and memory usage.
0 commit comments