Merge pull request #171 from HarperDB/vector-indexing

Add documentation on how to define fields with vector indexing

Author: kriszyp

docs/developers/applications/defining-schemas.md

@@ -171,7 +171,51 @@ The `@primaryKey` directive specifies that an attribute is the primary key for a
 
 #### `@indexed`
 
-The `@indexed` directive specifies that an attribute should be indexed. This is necessary if you want to execute queries using this attribute (whether that is through RESTful query parameters, SQL, or NoSQL operations).
+The `@indexed` directive specifies that an attribute should be indexed. When an attribute is indexed, Harper will create secondary index from the data in this field for fast/efficient querying using this field. This is necessary if you want to execute queries using this attribute (whether that is through RESTful query parameters, SQL, or NoSQL operations).
+
+A standard index will index the values in each field, so you can query directly by those values. If the field's value is an array, each of the values in the array will be indexed (you can query by any individual value).
+
+#### Vector Indexing
+
+The `@indexed` directive can also specify a `type`. To use vector indexing, you can specify the `type` as `HNSW` for Hierarchical Navigable Small World indexing. This will create a vector index for the attribute. For example:
+```graphql
+type Product @table {
+	id: Long @primaryKey
+	textEmbeddings: [Float] @indexed(type: "HNSW")
+}
+```
+
+HNSW indexing finds the nearest neighbors to a search vector. To use this, you can query with a `sort` parameter, for example:
+```javascript
+let results = Product.search({
+  sort: { attribute: 'textEmbeddings', target: searchVector },
+  limit: 5 // get the five nearest neighbors
+})
+```
+This can be used in combination with other conditions as well, for example:
+```javascript
+let results = Product.search({
+  conditions: [{ attribute: 'price', comparator: 'lt', value: 50 }],
+  sort: { attribute: 'textEmbeddings', target: searchVector },
+  limit: 5 // get the five nearest neighbors
+})
+```
+
+HNSW supports several additional arguments to the `@indexed` directive to adjust the HNSW parameters:
+* `distance` - Define the distance function. This can be set to 'euclidean' or 'cosine' (uses negative of cosine similarity). The default is cosine.
+* `efConstruction` - Maximum number of nodes to keep in the list for finding nearest neighbors. A higher value can yield better recall, and a lower value can have better performance. If `efSearchConstruction` is set, this is only applied to indexing. The default is 100.
+* `M` - The preferred number of connections at each layer in the HNSW graph. A higher number uses more space but can be helpful when the intrinsic dimensionality of the data is higher. A lower number can be more efficient. The default is 16.
+* `optimizeRouting` - This uses a heuristic to avoid graph connections that match existing indirect connections (connections through another node). This can yield more efficient graph traversals for the same M setting. This is a number between 0 and 1 and a higher value will more aggressively omit connections with alternate paths. Setting this to 0 will disable route optimizing and follow the traditional HNSW algorithm for creating connections. The default is 0.5.
+* `mL` - The normalization factor for level generation, by default this is computed from `M`.
+* `efSearchConstruction` - Maximum number of nodes to keep in the list for finding nearest neighbors for searching. The default is 50.
+ 
+For exmpale
+```graphql
+type Product @table {
+  id: Long @primaryKey
+  textEmbeddings: [Float] @indexed(type: "HNSW", distance: "euclidean", optimizeRouting: 0, efSearchConstruction: 100)
+}
+```
 
 #### `@createdTime`
 

docs/technical-details/release-notes/4.tucker/4.6.0.md

@@ -5,7 +5,7 @@
 6/13/2025
 
 ### Vector Indexing: Hierarchical Navigable Small World
-Harper 4.6 now includes support for vector indexing, which allows for efficient and fast queries on large semantic data sets. Vector indexing is powered by the [Hierarchical Navigable Small World (HNSW) algorithm](https://arxiv.org/abs/1603.09320) and can be used to index any vector-valued property, and is particularly useful for vector text-embedding data. HNSW is a preferred algorithm for vector indexing and searching because it provides an excellent balance of recall and performance.
+Harper 4.6 now includes support for vector indexing, which allows for efficient and fast queries on large semantic data sets. Vector indexing is powered by the [Hierarchical Navigable Small World (HNSW) algorithm](https://arxiv.org/abs/1603.09320) and can be used to index any vector-valued property, and is particularly useful for vector text-embedding data. This provides powerful efficient vector-based searching for semantic and AI-based querying functionality. HNSW is a preferred algorithm for vector indexing and searching because it provides an excellent balance of recall and performance.
 
 ### New Extension API with support for dynamic reloading
 4.6 introduces a new extension API with significant ergonomic improvements for creating new extension components that are more robust and dynamic. The new API also provides a mechanism for dynamic reloading of some files and configuration without restarts.
@@ -21,4 +21,3 @@ An important change is that logging to standard out/error will _not_ include the
 
 ### Resource API Upgrades
 4.6 includes an upgraded form of the Resource API that can be selected with significant improvements in ease of use.
-