Skip to content

Commit b9c62d5

Browse files
committed
Add IVF_RQ dimension note
1 parent 4ada458 commit b9c62d5

File tree

2 files changed

+7
-4
lines changed

2 files changed

+7
-4
lines changed

docs/indexing/index.mdx

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ Vector indexes can use different quantization methods to compress vectors and im
4747
| :----------- | :------- | :---------- |
4848
| `PQ` (Product Quantization) | Default choice for most vector search scenarios. Use when you need to balance index size and recall. | Divides vectors into subvectors and quantizes each subvector independently. Provides a good balance between compression ratio and search accuracy. |
4949
| `SQ` (Scalar Quantization) | Use when you need faster indexing or when vector dimensions have consistent value ranges. | Quantizes each dimension independently. Simpler than PQ but typically provides less compression. |
50-
| `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. |
50+
| `RQ` (RabitQ Quantization) | Use when you need maximum compression or have specific per-dimension requirements. | Per-dimension quantization using a RabitQ codebook. Provides fine-grained control over compression per dimension. For `IVF_RQ`, vector dimensions must be divisible by `8`. |
5151
| `None/Flat` | Use for binary vectors (with `hamming` distance) or when you need maximum recall and have sufficient storage. | No quantization—stores raw vectors. Provides the highest accuracy but requires more storage and memory. |
5252

5353
## Understanding the IVF-PQ Index
@@ -149,4 +149,3 @@ Then the greedy search routine operates as follows:
149149
* At the top layer (using an arbitrary vertex as an entry point), use the greedy local search routine on the k-ANN graph to get an approximate nearest neighbor at that layer.
150150
* Using the approximate nearest neighbor found in the previous layer as an entry point, find an approximate nearest neighbor in the next layer with the same method.
151151
* Repeat until the bottom-most layer is reached. Then use the entry point to find multiple nearest neighbors (e.g. top 10).
152-

docs/indexing/quantization.mdx

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ Use quantization when:
1515

1616
LanceDB currently exposes multiple quantized vector index types, including:
1717
- `IVF_PQ` -- Inverted File index with Product Quantization (default). See the [vector indexing guide](/indexing/vector-index) for `IVF_PQ` examples.
18-
- `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). See [below](#rabitq-quantization) for details.
18+
- `IVF_RQ` -- Inverted File index with **RaBitQ** quantization (binary, 1 bit per dimension). Requires vector dimensions divisible by `8`. See [below](#rabitq-quantization) for details.
1919

2020
`IVF_PQ` is the default indexing option in LanceDB and works well in many cases. However, in cases where more drastic compression is needed, RaBitQ is also a reasonable option.
2121

@@ -42,6 +42,11 @@ For a deeper dive into the theory and some benchmark results, see the blog post:
4242
### Using RaBitQ
4343

4444
You can create an RaBitQ-backed vector index by setting `index_type="IVF_RQ"` when calling `create_index`.
45+
46+
<Note title="Dimension requirement">
47+
When using `IVF_RQ`, vector dimensions must be divisible by `8`.
48+
</Note>
49+
4550
`num_bits` controls how many bits per dimension are used:
4651

4752
## API Reference
@@ -62,4 +67,3 @@ The full list of parameters to the algorithm are listed below.
6267
Number of samples per partition during training. Higher values may improve accuracy but increase training time.
6368
- `target_partition_size`: Optional[int], defaults to None
6469
Target number of vectors per partition. Adjust to control partition granularity and memory usage.
65-

0 commit comments

Comments
 (0)