vectors - design doc draft #9317

alphaprinz · 2025-11-26T17:28:45Z

No description provided.

coderabbitai · 2025-11-26T17:28:52Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

Signed-off-by: Amit Prinz Setter <[email protected]>

…tion details options Signed-off-by: Amit Prinz Setter <[email protected]>

Signed-off-by: Amit Prinz Setter <[email protected]>

docs/design/vectors.md

guymguym · 2026-01-12T13:13:02Z

docs/design/vectors.md

+4. Hopefully will be able to reuse current bucket/object implementation with some adaptation.
+   1. Eg add "bucket content" field to indicate whether bucket is object or vector instead of creating a new “vector bucket” entity.
+
+### Terminology


add also the noobaa terminology here for a reader that is less familiar with it -
NB vs FS bucketspace
NB vs FS namespace
refer to https://github.com/noobaa/noobaa-core/blob/master/docs/bucket-types.md

docs/design/vectors.md

guymguym · 2026-01-12T14:18:18Z

docs/design/vectors.md

+### CRDs
+We need to specify, at least in Lance client case, two kinds of independent parameters:
+-A storage connection - Can be an S3 account (endpoint url, secret id, secret key), or a FS based.
+This can be the alread-existing CRDs Backingstore and NamespaceStore.


I think this should be based on a BucketClass, and not on the underlying BackingStore/NSStore.

IIUC BucketClass has a bunch of stuff relevant to objects but not for the vector use case, eg tiers and mirroring?
I just need a directory mounted on the endpoints, or an s3 connection credentials.

guymguym · 2026-01-12T14:20:01Z

docs/design/vectors.md

+We need to specify, at least in Lance client case, two kinds of independent parameters:
+-A storage connection - Can be an S3 account (endpoint url, secret id, secret key), or a FS based.
+This can be the alread-existing CRDs Backingstore and NamespaceStore.
+For FS we can also use a pvc directly, without NamespaceStore?


I think this is eventually what has to happen, because the DB itself will need to be initialized with the configuration of the storage which is either a FS path or an S3 connection.

Signed-off-by: Amit Prinz Setter <[email protected]>

… add custom header options, fix typos Signed-off-by: Amit Prinz Setter <[email protected]>

guymguym · 2026-01-15T07:06:04Z

docs/design/vectors.md

+    A2[Inference/Agent]
+    B[S3 Vector Endpoint]
+    B1[BucketSpace]
+    C{{Vector Plugin System}}


Suggested change

C{{Vector Plugin System}}

C{{VectorSDK}}

guymguym · 2026-01-15T07:06:31Z

docs/design/vectors.md

+    E2-.->G
+```
+
+### Vector Bucket


Suggested change

### Vector Bucket

### Operations on Vector Buckets

guymguym · 2026-01-15T07:12:15Z

docs/design/vectors.md

+NB shall add "deleted" field to the relevant db row.
+FS shall delete the relevant json file.
+
+For NB, actual deletion will be implemeted in standard "DB Cleaner" BG worker.


Empirically we can check how aws responds to deleting non-empty index. The doc might not be complete.

guymguym · 2026-01-15T07:13:49Z

docs/design/vectors.md

+
+For NB, actual deletion will be implemeted in standard "DB Cleaner" BG worker.
+
+### Vector


Suggested change

### Vector

### Operations on Vectors

guymguym · 2026-01-15T07:18:10Z

docs/design/vectors.md

+-translate keys into Lance "ids in" filter.
+-execute delete
+
+### Index


Suggested change

### Index

### Operations on Vector Indexes

guymguym · 2026-01-15T07:34:14Z

docs/design/vectors.md

+In Lance case, the technical difficulty here is to translate find a place for the Lance parameters in the AWS request body type.
+Since 
+-Lance index and S3 vectors index are essentially different AND
+-S3 vectors' CreateIndex request type is restricted,
+we can utilize the generic Tags field to put all data.
+
+https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_CreateIndex.html
+https://lancedb.github.io/lancedb/js/interfaces/IndexOptions/
+
+Other than that, Index API is a simple CRUD.


For now we should add configs to pick the index type (HNSW / IVF) and configs to other default parameters in addition to the ones sent by S3 API (dataType, dimensions, distanceMetric).

Can we call createIndex eachtime?

Here is what I found on when to use LanceDB without an Index -

Brute-force (no-index) search is suitable in specific scenarios:

Small datasets: For datasets with up to around 100,000 records, a vector index is usually not necessary as the latency is acceptable (often under 100ms).

Guaranteed 100% recall: When you need to find the exact nearest neighbors without any approximation.

Immediate searchability of new data: When new data is added to a table, it is immediately searchable via brute-force even before a background indexing job has run.

I think these are not our goals in S3 Vectors, so by default I would define an index.

S3 CreateVectorIndex should be translated to Lance createTable + createIndex.

guymguym · 2026-01-15T07:47:05Z

docs/design/vectors.md

+    G[S3 Backend]
+
+    A2-->|Query|B
+    A1-->|Ingest|B


Suggested change

A1-->|Ingest|B

A1-->|Put|B

guymguym · 2026-01-15T07:50:35Z

docs/design/vectors.md

+3. Use account-level default (similar to account's default resource).
+This enables account-level granularity control (not per-bucket as needed).
+Can be combine as a default fall-back with above "Pure s3-compatible" option.
+
+4. New actions in operator cli (similar to OB in ODF, manage_nsfs in NSFS). Allows control on parameter names and values. Eg
+nb vector-bucket create --vector-storage=NS1 --vector-storage-path='/vectors1' --vector-plugin Lance
+We can add a new "VectorBucket" CRD or add fields to existing bucket's CRD.


these two should be the first options to implement.

guymguym · 2026-01-15T08:11:16Z

docs/design/vectors.md

+#### Query
+https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_QueryVectors.html
+
+In Lance:


missing "filter" "returnMetadata"

guymguym · 2026-01-15T08:13:04Z

docs/design/vectors.md

+In Lance case, the technical difficulty here is to translate find a place for the Lance parameters in the AWS request body type.
+Since 
+-Lance index and S3 vectors index are essentially different AND
+-S3 vectors' CreateIndex request type is restricted,
+we can utilize the generic Tags field to put all data.
+
+https://docs.aws.amazon.com/AmazonS3/latest/API/API_S3VectorBuckets_CreateIndex.html
+https://lancedb.github.io/lancedb/js/interfaces/IndexOptions/
+
+Other than that, Index API is a simple CRUD.


S3 CreateVectorIndex should be translated to Lance createTable + createIndex.

guymguym · 2026-01-15T08:15:39Z

docs/design/vectors.md

+https://aws.amazon.com/s3/features/vectors/
+https://docs.aws.amazon.com/AmazonS3/latest/API/API_Operations_Amazon_S3_Vectors.html
+
+#### LanceDB


need to mention the gap that lance metadata is columns with schema, whereas s3vectors metadata is a json (free schema), and we need to decide which solution to use to map between those.

guymguym · 2026-01-15T08:28:20Z

docs/design/vectors.md

+VectorPlugin is abstract, with a concrete implementation per vector backend (Lance, Davinci, etc).
+VectorPlugin translates parameters to vector backend api and calls appropriate apis on vector backend client.
+
+## Vector storage


Suggested change

## Vector storage

## Vector bucket configurations

Signed-off-by: Amit Prinz Setter <[email protected]>

alphaprinz requested review from dannyzaken, guymguym and nimrod-becker November 26, 2025 17:28

pull-request-size bot added the size/M label Nov 26, 2025

nimrod-becker approved these changes Dec 2, 2025

View reviewed changes

vectors - design doc draft

e758333

Signed-off-by: Amit Prinz Setter <[email protected]>

alphaprinz force-pushed the vectors_design_md branch from e258949 to 9a8c413 Compare January 8, 2026 21:56

pull-request-size bot added size/L and removed size/M labels Jan 8, 2026

alphaprinz added 2 commits January 9, 2026 16:51

vectors - design - flesh out api/buckets, middle layer and add connec…

1b20153

…tion details options Signed-off-by: Amit Prinz Setter <[email protected]>

vectors - design - add vectors api

de76727

Signed-off-by: Amit Prinz Setter <[email protected]>

alphaprinz force-pushed the vectors_design_md branch from 9a8c413 to de76727 Compare January 10, 2026 00:53

nimrod-becker approved these changes Jan 11, 2026

View reviewed changes

guymguym reviewed Jan 12, 2026

View reviewed changes

alphaprinz added 3 commits January 12, 2026 11:24

vectors - design - PR comments 12/01/26

98a855c

Signed-off-by: Amit Prinz Setter <[email protected]>

vectors - design - add index/tag/policy, remove old Lance section

4b595b5

Signed-off-by: Amit Prinz Setter <[email protected]>

vectors - design - refine NamespaceStore usage, remove BackingsStore,…

4cfff75

… add custom header options, fix typos Signed-off-by: Amit Prinz Setter <[email protected]>

guymguym reviewed Jan 15, 2026

View reviewed changes

docs/design/vectors.md

G[S3 Backend]

A2-->|Query|B

A1-->|Ingest|B

Copy link

Member

guymguym Jan 15, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

A1-->|Ingest|B

A1-->|Put|B

guymguym reviewed Jan 15, 2026

View reviewed changes

vectors - design - add aws index- lance table, pagination

2344aee

Signed-off-by: Amit Prinz Setter <[email protected]>


		For NB, actual deletion will be implemeted in standard "DB Cleaner" BG worker.

		### Vector

vectors - design doc draft #9317

Are you sure you want to change the base?

vectors - design doc draft #9317

Uh oh!

Conversation

alphaprinz commented Nov 26, 2025

Uh oh!

coderabbitai bot commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitai bot commented Nov 26, 2025 •

edited

Loading