diff --git a/content/commands/vadd/index.md b/content/commands/vadd/index.md new file mode 100644 index 0000000000..2e5d65c2f7 --- /dev/null +++ b/content/commands/vadd/index.md @@ -0,0 +1,119 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(log(N)) for each element added, where N is the number of elements in the vector set. +description: Add a new element to a vector set, or update its vector if it already exists. +group: vector_set +hidden: false +linkTitle: VADD +since: 8.0.0 +summary: Add a new element to a vector set, or update its vector if it already exists. +syntax_fmt: "VADD key [REDUCE dim] (FP32 | VALUES num) vector element [CAS] [NOQUANT | Q8 | BIN]\n [EF build-exploration-factor] [SETATTR attributes] [M numlinks]" +title: VADD +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Add a new element into the vector set specified by `key`. The vector can be provided as 32-bit floating point (`FP32`) blob of values, or as floating point numbers as strings, prefixed by the number of elements (3 in the example below): + +``` +VADD mykey VALUES 3 0.1 1.2 0.5 my-element +``` + +## Required arguments + +
+key + +is the name of the key that will hold the vector set data. +
+ +
+FP32 vector or VALUES num vector + +either a 32-bit floating point (FP32) blob of values or `num` floating point numbers as strings. +
+ +
+element + +is the name of the element that is being added to the vector set. +
+ +## Optional arguments + +
+REDUCE dim + +implements random projection to reduce the dimensionality of the vector. The projection matrix is saved and reloaded along with the vector set. Please note that the REDUCE option must be passed immediately before the vector. For example, + +``` +VADD mykey REDUCE 50 VALUES ... +``` +
+ +
+CAS + +performs the operation partially using threads, in a check-and-set style. The neighbor candidates collection, which is slow, is performed in the background, while the command is executed in the main thread. +
+ +
+NOQUANT + +in the first VADD call for a given key, NOQUANT forces the vector to be created without int8 quantization, which is otherwise the default. +
+ +
+BIN + +forces the vector to use binary quantization instead of int8. This is much faster and uses less memory, but impacts the recall quality. +
+ +
+Q8 + +forces the vector to use signed 8-bit quantization. This is the default, and the option only exists to make sure to check at insertion time that the vector set is of the same format. +
+ +{{< note >}} +`NOQUANT`, `Q8`, and `BIN` are mutually exclusive. + +{{< /note >}} + +
+EF build-exploration-factor + +plays a role in the effort made to find good candidates when connecting the new node to the existing Hierarchical Navigable Small World (HNSW) graph. The default is 200. Using a larger value may help in achieving a better recall. To improve the recall it is also possible to increase EF during VSIM searches. +
+ +
+SETATTR attributes + +associates attributes in the form of a JavaScript object to the newly created entry or updates the attributes (if they already exist). +It is the same as calling the VSETATTR command separately. +
+ +
+M numlinks + +is the maximum number of connections that each node of the graph will have with other nodes. The default is 16. More connections means more memory, but provides for more efficient graph exploration. Nodes at layer zero (every node exists at least at layer zero) have `M * 2` connections, while the other layers only have `M` connections. For example, setting `M` to `64` will use at least 1024 bytes of memory for layer zero. That's `M * 2` connections times 8 bytes (pointers), or `128 * 8 = 1024`. For higher layers, consider the following: + +- Each node appears in ~1.33 layers on average (empirical observation from HNSW papers), which works out to be 0.33 higher layers per node. +- Each of those higher layers has `M = 64` connections. + +So, the additional amount of memory is approximately `0.33 × 64 × 8 ≈ 169.6` bytes per node, bringing the total memory to ~1193 bytes. + +If you don't have a recall quality problem, the default is acceptable, and uses a minimal amount of memory. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) \ No newline at end of file diff --git a/content/commands/vcard/index.md b/content/commands/vcard/index.md new file mode 100644 index 0000000000..2aa7868318 --- /dev/null +++ b/content/commands/vcard/index.md @@ -0,0 +1,41 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(1) +description: Return the number of elements in a vector set. +group: vector_set +hidden: false +linkTitle: VCARD +since: 8.0.0 +summary: Return the number of elements in a vector set. +syntax_fmt: "VCARD key" +title: VCARD +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Return the number of elements in the specified vector set. + +```shell +VCARD word_embeddings +(integer) 3000000 +``` + +## Required arguments + +
+key + +is the name of the key that holds the vector set. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/commands/vdim/index.md b/content/commands/vdim/index.md new file mode 100644 index 0000000000..d033a5ddcb --- /dev/null +++ b/content/commands/vdim/index.md @@ -0,0 +1,43 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(1) +description: Return the dimension of vectors in the vector set. +group: vector_set +hidden: false +linkTitle: VDIM +since: 8.0.0 +summary: Return the dimension of vectors in the vector set. +syntax_fmt: "VDIM key" +title: VDIM +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Return the number of dimensions of the vectors in the specified vector set. + +```shell +VDIM word_embeddings +(integer) 300 +``` + +If the vector set was created using the `REDUCE` option for dimensionality reduction, this command reports the reduced dimension. However, you must still use full-size vectors when performing queries with the `VSIM` command. + +## Required arguments + +
+key + +is the name of the key that holds the vector set. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/commands/vemb/index.md b/content/commands/vemb/index.md new file mode 100644 index 0000000000..861ae8769b --- /dev/null +++ b/content/commands/vemb/index.md @@ -0,0 +1,72 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(1) +description: Return the vector associated with an element. +group: vector_set +hidden: false +linkTitle: VEMB +since: 8.0.0 +summary: Return the vector associated with an element. +syntax_fmt: "VEMB key element [RAW]" +title: VEMB +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Return the approximate vector associated with a given element in the vector set. + +```shell +VEMB word_embeddings SQL +1) "0.18208661675453186" +2) "0.08535309880971909" +3) "0.1365649551153183" +4) "-0.16501599550247192" +5) "0.14225517213344574" +... 295 more elements ... +``` + +Vector sets normalize and may quantize vectors on insertion. `VEMB` reverses this process to approximate the original vector by de-normalizing and de-quantizing it. + +To retrieve the raw internal representation, use the `RAW` option: + +```shell +VEMB word_embeddings apple RAW +1) int8 +2) "\xf1\xdc\xfd\x1e\xcc%E...\xde\x1f\xfbN" # artificially shortened for this example +3) "3.1426539421081543" +4) "0.17898885905742645" +``` + +## Required arguments + +
+key + +is the name of the key that holds the vector set. +
+ +
+element + +is the name of the element whose vector you want to retrieve. +
+ +## Optional arguments + +
+RAW + +returns the raw vector data, its quantization type, and metadata such as norm and range. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/commands/vgetattr/index.md b/content/commands/vgetattr/index.md new file mode 100644 index 0000000000..68184c7fc7 --- /dev/null +++ b/content/commands/vgetattr/index.md @@ -0,0 +1,46 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(1) +description: Retrieve the JSON attributes of elements. +group: vector_set +hidden: false +linkTitle: VGETATTR +since: 8.0.0 +summary: Retrieve the JSON attributes of elements. +syntax_fmt: "VGETATTR key element" +title: VGETATTR +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Return the JSON attributes associated with an element in a vector set. + +```shell +VGETATTR key element +``` + +## Required arguments + +
+key + +is the name of the key that holds the vector set. +
+ +
+element + +is the name of the element whose attributes you want to retrieve. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/commands/vinfo/index.md b/content/commands/vinfo/index.md new file mode 100644 index 0000000000..7044fdf7ad --- /dev/null +++ b/content/commands/vinfo/index.md @@ -0,0 +1,52 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(1) +description: Return information about a vector set. +group: vector_set +hidden: false +linkTitle: VINFO +since: 8.0.0 +summary: Return information about a vector set. +syntax_fmt: "VINFO key" +title: VINFO +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Return metadata and internal details about a vector set, including size, dimensions, quantization type, and graph structure. + +```shell +VINFO word_embeddings +1) quant-type +2) int8 +3) vector-dim +4) (integer) 300 +5) size +6) (integer) 3000000 +7) max-level +8) (integer) 12 +9) vset-uid +10) (integer) 1 +11) hnsw-max-node-uid +12) (integer) 3000000 +``` + +## Required arguments + +
+key + +is the name of the key that holds the vector set. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/commands/vlinks/index.md b/content/commands/vlinks/index.md new file mode 100644 index 0000000000..5c88a1e927 --- /dev/null +++ b/content/commands/vlinks/index.md @@ -0,0 +1,60 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(1) +description: Return the neighbors of an element at each layer in the HNSW graph. +group: vector_set +hidden: false +linkTitle: VLINKS +since: 8.0.0 +summary: Return the neighbors of an element at each layer in the HNSW graph. +syntax_fmt: "VLINKS key element [WITHSCORES]" +title: VLINKS +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Return the neighbors of a specified element in a vector set. The command shows the connections for each layer of the HNSW graph. + +```shell +VLINKS key element +``` + +Use the `WITHSCORES` option to include similarity scores for each neighbor. + +```shell +VLINKS key element WITHSCORES +``` + +## Required arguments + +
+key + +is the name of the key that holds the vector set. +
+ +
+element + +is the name of the element whose HNSW neighbors you want to inspect. +
+ +## Optional arguments + +
+WITHSCORES + +includes similarity scores for each neighbor. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/commands/vrandmember/index.md b/content/commands/vrandmember/index.md new file mode 100644 index 0000000000..bf8ea842b1 --- /dev/null +++ b/content/commands/vrandmember/index.md @@ -0,0 +1,113 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(N) where N is the absolute value of the count argument. +description: Return one or multiple random members from a vector set. +group: vector_set +hidden: false +linkTitle: VRANDMEMBER +since: 8.0.0 +summary: Return one or multiple random members from a vector set. +syntax_fmt: "VRANDMEMBER key [count]" +title: VRANDMEMBER +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Return one or more random elements from a vector set. + +The behavior is similar to the `SRANDMEMBER` command: + +- When called without a count, returns a single element as a bulk string. +- When called with a positive count, returns up to that many distinct elements (no duplicates). +- When called with a negative count, returns that many elements, possibly with duplicates. +- If the count exceeds the number of elements, the entire set is returned. +- If the key does not exist, the command returns `null` if no count is given, or an empty array if a count is provided. + +```shell +VADD vset VALUES 3 1 0 0 elem1 +VADD vset VALUES 3 0 1 0 elem2 +VADD vset VALUES 3 0 0 1 elem3 +``` + +Return a single random element: + +```shell +VRANDMEMBER vset +"elem2" +``` + +Return two distinct random elements: + +```shell +VRANDMEMBER vset 2 +1) "elem1" +2) "elem3" +``` + +Return 3 random elements with possible duplicates: + +```shell +VRANDMEMBER vset -3 +1) "elem2" +2) "elem2" +3) "elem1" +``` + +Request more elements than exist in the set: + +```shell +VRANDMEMBER vset 10 +1) "elem1" +2) "elem2" +3) "elem3" +``` + +When the key doesn't exist: + +```shell +VRANDMEMBER nonexistent +(nil) +``` + +```shell +VRANDMEMBER nonexistent 3 +(empty array) +``` + +This command is useful for: + +- Sampling elements for testing or training. +- Generating random queries for performance testing. + +Internally: + +- For small counts (less than 20% of the set size), a dictionary is used to ensure uniqueness. +- For large counts (more than 20% of the set size), a linear scan provides faster performance, though results may be less random. + +## Required arguments + +
+key + +is the name of the key that holds the vector set. +
+ +## Optional arguments + +
+count + +specifies the number of elements to return. Positive values return distinct elements; negative values allow duplicates. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/commands/vrem/index.md b/content/commands/vrem/index.md new file mode 100644 index 0000000000..208caae252 --- /dev/null +++ b/content/commands/vrem/index.md @@ -0,0 +1,59 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(log(N)) for each element removed, where N is the number of elements in the vector set. +description: Remove one or more elements from a vector set. +group: vector_set +hidden: false +linkTitle: VREM +since: 8.0.0 +summary: Remove one or more elements from a vector set. +syntax_fmt: "VREM key element" +title: VREM +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Remove an element from a vector set. + +```shell +VADD vset VALUES 3 1 0 1 bar +(integer) 1 +``` + +```shell +VREM vset bar +(integer) 1 +``` + +```shell +VREM vset bar +(integer) 0 +``` + +`VREM` reclaims memory immediately. It does not use tombstones or logical deletions, making it safe to use in long-running applications that frequently update the same vector set. + +## Required arguments + +
+key + +is the name of the key that holds the vector set. +
+ +
+element + +is the name of the element to remove from the vector set. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/commands/vsetattr/index.md b/content/commands/vsetattr/index.md new file mode 100644 index 0000000000..ca4833b90b --- /dev/null +++ b/content/commands/vsetattr/index.md @@ -0,0 +1,60 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(1) +description: Associate or remove the JSON attributes of elements. +group: vector_set +hidden: false +linkTitle: VSETATTR +since: 8.0.0 +summary: Associate or remove the JSON attributes of elements. +syntax_fmt: "VSETATTR key element \"{ JSON obj }\"" +title: VSETATTR +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Associate a JSON object with an element in a vector set. Use this command to store attributes that can be used in filtered similarity searches with `VSIM`. + +You can also update existing attributes or delete them by setting an empty string. + +```shell +VSETATTR key element "{\"type\": \"fruit\", \"color\": \"red\"}" +``` + +To remove attributes, pass an empty JSON string: + +```shell +VSETATTR key element "" +``` + +## Required arguments + +
+key + +is the name of the key that holds the vector set. +
+ +
+element + +is the name of the element whose attributes you want to set or remove. +
+ +
+json + +is a valid JSON string. Use an empty string (`""`) to delete the attributes. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/commands/vsim/index.md b/content/commands/vsim/index.md new file mode 100644 index 0000000000..a6bbfa991d --- /dev/null +++ b/content/commands/vsim/index.md @@ -0,0 +1,122 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +complexity: O(log(N)) where N is the number of elements in the vector set. +description: Return elements by vector similarity. +group: vector_set +hidden: false +linkTitle: VSIM +since: 8.0.0 +summary: Return elements by vector similarity. +syntax_fmt: "VSIM key (ELE | FP32 | VALUES num) (vector | element) [WITHSCORES] [COUNT num] [EF search-exploration-factor]\n [FILTER expression] [FILTER-EF max-filtering-effort] [TRUTH] [NOTHREAD]" +title: VSIM +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +--- + +Return elements similar to a given vector or element. Use this command to perform approximate or exact similarity searches within a vector set. + +You can query using either a vector (via `FP32` or `VALUES num`) or by referencing another element (using `ELE`). Optional parameters let you control the search behavior, such as score output, result count, and filtering options. + +```shell +VSIM word_embeddings ELE apple +1) "apple" +2) "apples" +3) "pear" +4) "fruit" +5) "berry" +6) "pears" +7) "strawberry" +8) "peach" +9) "potato" +10) "grape" +``` + +You can include similarity scores and limit the number of results: + +```shell +VSIM word_embeddings ELE apple WITHSCORES COUNT 3 +1) "apple" +2) "0.9998867657923256" +3) "apples" +4) "0.8598527610301971" +5) "pear" +6) "0.8226882219314575" +``` + +Set the `EF` (exploration factor) to improve recall at the cost of performance. Use the `TRUTH` option to perform an exact linear scan, useful for benchmarking. The `NOTHREAD` option runs the search in the main thread and may increase server latency. + +## Required arguments + +
+key + +is the name of the key that holds the vector set data. +
+ +
+ELE | FP32 | VALUES num + +specifies how the input vector is provided. Use `ELE` to refer to an existing element, `FP32` for binary float format, or `VALUES num` for a list of stringified float values. +
+ +
+vector or element + +is either the vector data (for `FP32` or `VALUES`) or the name of the element (for `ELE`) to use as the similarity reference. +
+ +## Optional arguments + +
+WITHSCORES + +returns the similarity score (from 1 to 0) alongside each result. A score of 1 is identical; 0 is the opposite. +
+ +
+COUNT num + +limits the number of returned results to `num`. +
+ +
+EF search-exploration-factor + +controls the search effort. Higher values explore more nodes, improving recall at the cost of speed. Typical values range from 50 to 1000. +
+ +
+FILTER expression + +applies a filter expression to restrict matching elements. See the filtered search section for syntax details. +
+ +
+FILTER-EF max-filtering-effort + +limits the number of filtering attempts for the `FILTER` expression. See the filtered search section for more. +
+ +
+TRUTH + +forces an exact linear scan of all elements, bypassing the HNSW graph. Use for benchmarking or to calculate recall. This is significantly slower (O(N)). +
+ +
+NOTHREAD + +executes the search in the main thread instead of a background thread. Useful for small vector sets or benchmarks. This may block the server during execution. +
+ +## Related topics + +- [Vector sets]({{< relref "/develop/data-types/vector-sets" >}}) diff --git a/content/develop/data-types/vector-sets/_index.md b/content/develop/data-types/vector-sets/_index.md new file mode 100644 index 0000000000..cf223ce794 --- /dev/null +++ b/content/develop/data-types/vector-sets/_index.md @@ -0,0 +1,40 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +description: Introduction to Redis vector sets +linkTitle: Vector sets +title: Redis vector sets +weight: 55 +bannerText: Vector set is a new data type that is currently in preview and may be subject to change. +bannerChildren: true +--- + +Vector sets are a data type similar to sorted sets, but instead of a score, vector set elements have a string representation of a vector. +Vector sets allow you to add items to a set, and then either: + +* retrieve a subset of items that are the most similar to a specified vector, or +* retrieve a subset of items that are the most similar to the vector of an element that is already part of the vector set. + +Vector sets also provide for optional [filtered search]({{< relref "/develop/data-types/vector-sets/filtered-search" >}}). You can associate attributes with all or some elements in a vector set, and then use the `FILTER` option of the [`VSIM`]({{< relref "/commands/vsim" >}}) command to retrieve items similar to a given vector while applying simple mathematical filters to those attributes. Here's a sample filter: `".year > 1950"`. + +The following commands are available for vector sets: + +- [VADD]({{< relref "/commands/vadd" >}}) - add an element to a vector set, creating a new set if it didn't already exist. +- [VCARD]({{< relref "/commands/vcard" >}}) - retrieve the number of elements in a vector set. +- [VDIM]({{< relref "/commands/vdim" >}}) - retrieve the dimension of the vectors in a vector set. +- [VEMB]({{< relref "/commands/vemb" >}}) - retrieve the approximate vector associated with a vector set element. +- [VGETATTR]({{< relref "/commands/vgetattr" >}}) - retrieve the attributes of a vector set element. +- [VINFO]({{< relref "/commands/vinfo" >}}) - retrieve metadata and internal details about a vector set, including size, dimensions, quantization type, and graph structure. +- [VLINKS]({{< relref "/commands/vlinks" >}}) - retrieve the neighbors of a specified element in a vector set; the connections for each layer of the HNSW graph. +- [VRANDMEMBER]({{< relref "/commands/vrandmember" >}}) - retrieve random elements of a vector set. +- [VREM]({{< relref "/commands/vrem" >}}) - remove an element from a vector set. +- [VSETATTR]({{< relref "/commands/vsetattr" >}}) - set or replace attributes on a vector set element. +- [VSIM]({{< relref "/commands/vsim" >}}) - retrieve elements similar to a given vector or element with optional filtering. diff --git a/content/develop/data-types/vector-sets/filtered-search.md b/content/develop/data-types/vector-sets/filtered-search.md new file mode 100644 index 0000000000..396f550966 --- /dev/null +++ b/content/develop/data-types/vector-sets/filtered-search.md @@ -0,0 +1,119 @@ +--- +categories: +- docs +- develop +- stack +- rs +- rc +- oss +- kubernetes +- clients +description: Use filter expressions to refine vector similarity results with Redis vector sets +linkTitle: Filter expressions +title: Filter expressions +weight: 10 +--- + +## Overview + +Filtered search lets you combine vector similarity search with scalar filtering. You can associate JSON attributes with elements in a vector set, and then filter results using those attributes during [`VSIM`]({{< relref "/commands/vsim" >}}) queries. + +This allows queries such as: + +```bash +VSIM movies VALUES 3 0.5 0.8 0.2 FILTER '.year >= 1980 and .rating > 7' +``` + +## Assigning attributes + +You can associate attributes when adding a new vector using the `SETATTR` argument: + +```bash +VADD vset VALUES 3 1 1 1 a SETATTR '{"year": 1950}' +``` + +Or update them later with the [`VSETATTR`]({{< relref "/commands/vsetattr" >}}) command: + +```bash +VSETATTR vset a '{"year": 1960}' +``` + +You can retrieve attributes with the [`VGETATTR`]({{< relref "/commands/vgetattr" >}}) command: + +```bash +VGETATTR vset a +``` + +## Filtering during similarity search + +To filter by attributes, pass the `FILTER` option to the [`VSIM`]({{< relref "/commands/vsim" >}}) command: + +```bash +VSIM vset VALUES 3 0 0 0 FILTER '.year > 1950' +``` + +This returns only elements that match both the vector similarity and the filter expression. + +## Expression syntax + +Expressions support familiar JavaScript-like syntax: + +- Arithmetic: `+`, `-`, `*`, `/`, `%`, `**` +- Comparison: `==`, `!=`, `>`, `<`, `>=`, `<=` +- Logical: `and`, `or`, `not` (or `&&`, `||`, `!`) +- Containment: `in` +- Grouping: Parentheses `()` + +Use dot notation to access attribute fields, for example, `.year`, `.rating`. + +> Only top-level fields are supported (for example, `.genre`, but not `.movie.genre`). + +## Supported data types + +- Numbers +- Strings +- Booleans (converted to 1 or 0) +- Arrays (for `in`) + +If a field is missing or invalid, the element is skipped without error. + +## FILTER-EF + +The `FILTER-EF` option controls how many candidate nodes the engine inspects to find enough filtered results. The defaults is `COUNT * 100`. + +```bash +VSIM vset VALUES 3 0 0 0 COUNT 10 FILTER '.year > 2000' FILTER-EF 500 +``` + +- Use a higher value for rare filters. +- Use `FILTER-EF 0` to scan as many as needed to fulfill the request. +- The engine will stop early if enough high-quality results are found. + +## Examples + +```bash +# Filter by year range +VSIM movies VALUES 3 0.5 0.8 0.2 FILTER '.year >= 1980 and .year < 1990' + +# Filter by genre and rating +VSIM movies VALUES 3 0.5 0.8 0.2 FILTER '.genre == "action" and .rating > 8.0' + +# Use IN with array +VSIM movies VALUES 3 0.5 0.8 0.2 FILTER '.director in ["Spielberg", "Nolan"]' + +# Math and logic +VSIM movies VALUES 3 0.5 0.8 0.2 FILTER '(.year - 2000) ** 2 < 100 and .rating / 2 > 4' +``` + +## Tips + +- Missing attributes are treated as non-matching. +- Use `FILTER-EF` to tune recall vs performance. +- Combine multiple attributes for fine-grained filtering. + +## See also + +- [VSIM]({{< relref "/commands/vsim" >}}) +- [VADD]({{< relref "/commands/vadd" >}}) +- [VSETATTR]({{< relref "/commands/vsetattr" >}}) +- [VGETATTR]({{< relref "/commands/vgetattr" >}}) \ No newline at end of file diff --git a/content/develop/data-types/vector-sets/memory.md b/content/develop/data-types/vector-sets/memory.md new file mode 100644 index 0000000000..af56e9d44b --- /dev/null +++ b/content/develop/data-types/vector-sets/memory.md @@ -0,0 +1,88 @@ +--- +categories: +- docs +- develop +- stack +- rs +- rc +- oss +- kubernetes +- clients +description: Learn how to optimize memory consumption in Redis vector sets +linkTitle: Memory optimization +title: Memory optimization +weight: 25 +--- + +## Overview + +Redis vector sets are efficient, but vector similarity indexing and graph traversal require memory tradeoffs. This guide helps you manage memory use through quantization, graph tuning, and attribute choices. + +## Quantization modes + +Vector sets support three quantization levels: + +| Mode | Memory usage | Recall | Notes | +|------------|---------------|--------|---------------------------------| +| `Q8` | 4x smaller | High | Default, fast and accurate | +| `BIN` | 32x smaller | Lower | Fastest, best for coarse search | +| `NOQUANT` | Full size | Highest| Best precision, slowest | + +Use `Q8` unless your use case demands either ultra-precision (use `NOQUANT`) or ultra-efficiency (use `BIN`). + +## Graph structure memory + +HNSW graphs store multiple connections per node. Each node: + +- Has an average of `M * 2 + M * 0.33` pointers (default M = 16). +- Stores pointers using 8 bytes each. +- Allocates ~1.33 layers per node. + +> A single node with M = 64 may consume ~1 KB in links alone. + +To reduce memory: + +- Lower `M` to shrink per-node connections. +- Avoid unnecessarily large values for `M` unless recall needs to be improved. + +## Attribute and label size + +Each node stores: + +- A string label (element name) +- Optional JSON attribute string + +Tips: + +- Use short, fixed-length strings for labels. +- Keep attribute JSON minimal and flat. For example, use `{"year":2020}` instead of nested data. + +## Vector dimension + +High-dimensional vectors increase storage: + +- 300 components at `FP32` = 1200 bytes/vector +- 300 components at `Q8` = 300 bytes/vector + +You can reduce this using the `REDUCE` option during [`VADD`]({{< relref "/commands/vadd" >}}), which applies random projection. + +```bash +VADD vset REDUCE 100 VALUES 300 ... item1 +``` + +This projects a 300-dimensional vector into 100 dimensions, reducing size and improving speed at the cost of some recall. + +## Summary + +| Strategy | Effect | +|---------------------|------------------------------------------| +| Use `Q8` | Best tradeoff for most use cases | +| Use `BIN` | Minimal memory, fastest search | +| Lower `M` | Shrinks HNSW link graph size | +| Reduce dimensions | Cuts memory per vector | +| Minimize JSON | Smaller attributes, less memory per node | + +## See also + +- [Performance]({{< relref "/develop/data-types/vector-sets/performance" >}}) +- [Scalability]({{< relref "/develop/data-types/vector-sets/scalability" >}}) diff --git a/content/develop/data-types/vector-sets/performance.md b/content/develop/data-types/vector-sets/performance.md new file mode 100644 index 0000000000..9b0f07ea17 --- /dev/null +++ b/content/develop/data-types/vector-sets/performance.md @@ -0,0 +1,73 @@ +--- +categories: +- docs +- develop +- stack +- rs +- rc +- oss +- kubernetes +- clients +description: Learn how Redis vector sets behave under load and how to optimize for speed and recall +linkTitle: Performance +title: Performance +weight: 15 +--- + +## Query performance + +Vector similarity queries using the [`VSIM`]({{< relref "/commands/vsim" >}}) are threaded by default. Redis uses up to 32 threads to process these queries in parallel. + +- `VSIM` performance scales nearly linearly with available CPU cores. +- Expect ~50,000 similarity queries per second for a 3M-item set with 300-dim vectors using int8 quantization. +- Performance depends heavily on the `EF` parameter: + - Higher `EF` improves recall, but slows down search. + - Lower `EF` returns faster results with reduced accuracy. + +## Insertion performance + +Inserting vectors with the [`VADD`]({{< relref "/commands/vadd" >}}) command is more computationally expensive than querying: + +- Insertion is single-threaded by default. +- Use the `CAS` option to offload candidate graph search to a background thread. +- Expect a few thousand insertions per second on a single node. + +## Quantization effects + +Quantization greatly impacts both speed and memory: + +- `Q8` (default): 4x smaller than `FP32`, high recall, high speed +- `BIN` (binary): 32x smaller than `FP32`, lower recall, fastest search +- `NOQUANT` (`FP32`): Full precision, slower performance, highest memory use + +Use the quantization mode that best fits your tradeoff between precision and efficiency. + +## Deletion performance + +Deleting large vector sets using the [`DEL`]({{< relref "/commands/del" >}}) can cause latency spikes: + +- Redis must unlink and restructure many graph nodes. +- Latency is most noticeable when deleting millions of elements. + +## Save and load performance + +Vector sets save and load the full HNSW graph structure: + +- When reloading from disk is fast and there's no need to rebuild the graph. + +Example: A 3M vector set with 300 components loads in ~15 seconds. + +## Summary of tuning tips + +| Factor | Effect on performance | Tip | +|------------|-------------------------------------|------------------------------------------------| +| `EF` | Slower queries but higher recall | Start low (for example, 200) and tune upward | +| `M` | More memory per node, better recall | Use defaults unless recall is too low | +| Quant type | Binary is fastest, `FP32` is slowest| Use `Q8` or `BIN` unless full precision needed | +| `CAS` | Faster insertions with threading | Use when high write throughput is needed | + +## See also + +- [Memory usage]({{< relref "/develop/data-types/vector-sets/memory" >}}) +- [Scalability]({{< relref "/develop/data-types/vector-sets/scalability" >}}) +- [Filtered search]({{< relref "/develop/data-types/vector-sets/filtered-search" >}}) diff --git a/content/develop/data-types/vector-sets/scalability.md b/content/develop/data-types/vector-sets/scalability.md new file mode 100644 index 0000000000..d1e9cfac15 --- /dev/null +++ b/content/develop/data-types/vector-sets/scalability.md @@ -0,0 +1,83 @@ +--- +categories: +- docs +- develop +- stack +- rs +- rc +- oss +- kubernetes +- clients +description: Scale Redis vector sets to handle larger data sets and workloads +linkTitle: Scalability +title: Scalability +weight: 20 +--- + +## Multi-instance scalability + +Vector sets can scale horizontally by sharding your data across multiple Redis instances. This is done by partitioning the dataset manually across keys and nodes. + +### Example strategy + +You can shard data using a consistent hash: + +```python +key_index = crc32(item) % 3 +key = f"vset:{key_index}" +``` + +Then add elements into different keys: + +```bash +VADD vset:0 VALUES 3 0.1 0.2 0.3 item1 +VADD vset:1 VALUES 3 0.4 0.5 0.6 item2 +``` + +To run a similarity search across all shards, send [`VSIM`]({{< relref "/commands/vsim" >}}) commands to each key and then merge the results client-side: + +```bash +VSIM vset:0 VALUES ... WITHSCORES +VSIM vset:1 VALUES ... WITHSCORES +VSIM vset:2 VALUES ... WITHSCORES +``` + +Then combine and sort the results by score. + +## Key properties + +- Write operations ([`VADD`]({{< relref "/commands/vadd" >}}), [`VREM`]({{< relref "/commands/vrem" >}})) scale linearly—you can insert in parallel across instances. +- Read operations ([`VSIM`]({{< relref "/commands/vsim" >}})) do not scale linearly—you must query all shards for a full result set. +- Smaller vector sets yield faster queries, so distributing them helps reduce query time per node. +- Merging results client-side keeps logic simple and doesn't add server-side overhead. + +## Availability benefits + +This sharding model also improves fault tolerance: + +- If one instance is down, you can still retrieve partial results from others. +- Use timeouts and partial fallbacks to increase resilience. + +## Latency considerations + +To avoid additive latency across N instances: + +- Send queries to all shards in parallel. +- Wait for the slowest response. + +This makes total latency close to the worst-case shard time, not the sum of all times. + +## Summary + +| Goal | Approach | +|---------------------------|---------------------------------------------------| +| Scale inserts | Split data across keys and instances | +| Scale reads | Query all shards and merge results | +| High availability | Accept partial results when some shards fail | +| Maintain performance | Use smaller shards for faster per-node traversal | + +## See also + +- [Performance]({{< relref "/develop/data-types/vector-sets/performance" >}}) +- [Filtered search]({{< relref "/develop/data-types/vector-sets/filtered-search" >}}) +- [Memory usage]({{< relref "/develop/data-types/vector-sets/memory" >}}) diff --git a/content/develop/data-types/vector-sets/troubleshooting.md b/content/develop/data-types/vector-sets/troubleshooting.md new file mode 100644 index 0000000000..eadaf8be4e --- /dev/null +++ b/content/develop/data-types/vector-sets/troubleshooting.md @@ -0,0 +1,102 @@ +--- +categories: +- docs +- develop +- stack +- rs +- rc +- oss +- kubernetes +- clients +description: Diagnose and debug issues when working with Redis vector sets +linkTitle: Troubleshooting +title: Troubleshooting +weight: 30 +--- + +## Common challenges + +Vector sets are approximate by design. That makes debugging trickier than with exact match queries. This section helps you understand issues with recall, filtering, and graph structure. + +## Low recall or missing results + +If [`VSIM`]({{< relref "/commands/vsim" >}}) doesn't return expected items: + +- Increase the `EF` parameter: + + ```bash + VSIM myset VALUES 3 ... COUNT 10 EF 1000 + ``` + +- Check quantization mode. Binary quantization (`BIN`) trades accuracy for speed. +- Use `TRUTH` to compare results against a linear scan: + + ```bash + VSIM myset VALUES 3 ... COUNT 10 TRUTH + ``` + + This gives you the most accurate results for validation, but it's slow. + +## Filtering issues + +Filters silently exclude items if: + +- A field is missing from the element’s attributes +- The JSON is invalid +- A type doesn’t match the expression (for example, `.rating > 8` when `.rating` is a string) + +Try retrieving the attributes with [`VGETATTR`]({{< relref "/commands/vgetattr" >}}): + +```bash +VGETATTR myset myelement +``` + +Double-check field names, JSON validity, and value types. + +## Unexpected memory usage + +Memory issues may arise from: + +- Large vectors (use `REDUCE` to project down) +- High `M` values inflating link graphs +- Large or deeply nested JSON attributes +- Storing raw `FP32` vectors (`NOQUANT`) + +Use default `Q8` quantization and compact attributes to save space. + +## Inspecting the graph + +Use [`VLINKS`]({{< relref "/commands/vlinks" >}}) to examine a node’s connections: + +```bash +VLINKS myset myelement WITHSCORES +``` + +- Helps you verify whether isolated or weakly connected nodes exist. +- Useful for explaining poor recall. + +## Deletion spikes + +Large sets deleted using the `DEL` command can briefly spike latency as Redis reclaims memory and rebuilds HNSW linkages. + +## Replication quirks + +- `VADD` with `REDUCE` does not replicate the random projection matrix. +- Replicas will produce different projected vectors for the same inputs. + +This doesn't affect similarity searches but does affect [`VEMB`]({{< relref "/commands/vemb" >}}) output. + +## Summary + +| Symptom | Try this | +|----------------------------------|-----------------------------------------------------------| +| Poor recall | Use higher `EF`, check quantization, use `TRUTH` | +| Filters exclude too much | Validate attributes with `VGETATTR`, simplify expressions | +| Memory spikes | Use `REDUCE`, `Q8`, smaller `M`, compact JSON | +| Replication mismatch with REDUCE | Avoid relying on projected vectors from replicas | + +## See also + +- [Filtered Search]({{< relref "/develop/data-types/vector-sets/filtered-search" >}}) +- [Memory Usage]({{< relref "/develop/data-types/vector-sets/memory" >}}) +- [Performance]({{< relref "/develop/data-types/vector-sets/performance" >}}) diff --git a/content/operate/oss_and_stack/install/archive/_index.md b/content/operate/oss_and_stack/install/archive/_index.md index fb45c3a516..b8c76ce808 100644 --- a/content/operate/oss_and_stack/install/archive/_index.md +++ b/content/operate/oss_and_stack/install/archive/_index.md @@ -17,7 +17,7 @@ You can install [Redis](https://redis.io/about/) or [Redis Stack](https://redis. Here are the installation instructions: -* [Install Redis]({{< relref "/operate/oss_and_stack/install/install-redis" >}}) -* [Install Redis Stack]({{< relref "/operate/oss_and_stack/install/install-stack" >}}) +* [Install Redis]({{< relref "/operate/oss_and_stack/install/archive/install-redis" >}}) +* [Install Redis Stack]({{< relref "/operate/oss_and_stack/install/archive/install-stack" >}}) While you can install Redis or Redis Stack locally, you might also consider using Redis Cloud by creating a [free account](https://redis.com/try-free/?utm_source=redisio&utm_medium=referral&utm_campaign=2023-09-try_free&utm_content=cu-redis_cloud_users). diff --git a/data/resp2_replies.json b/data/resp2_replies.json index a2c3bafb37..ac37ede34e 100644 --- a/data/resp2_replies.json +++ b/data/resp2_replies.json @@ -1204,6 +1204,66 @@ "UNWATCH": [ "[Simple string reply](../../develop/reference/protocol-spec#simple-strings): `OK`." ], + "VADD": [ + "One of the following:", + "* [Integer reply](../../develop/reference/protocol-spec#integers): 1 if key was added; 0 if key was not added.", + "* [Simple error reply](../../develop/reference/protocol-spec#simple-errors): if the command was malformed." + ], + "VCARD": [ + "[Integer reply](../../develop/reference/protocol-spec#integers): 0 if the key doesn't exist or the number of elements contained in the vector set." + ], + "VDIM": [ + "One of the following:", + "* [Integer reply](../../develop/reference/protocol-spec#integers): the number of vector set elements.", + "* [Simple error reply](../../develop/reference/protocol-spec#simple-errors): if the key does not exist." + ], + "VEMB": [ + "One of the following:", + "* [Array reply](../../develop/reference/protocol-spec#arrays): of real numbers as [bulk strings](../../develop/reference/protocol-spec#bulk-strings), representing the vector.", + "* [Array reply](../../develop/reference/protocol-spec#arrays): consisting of the following elements:", + " 1. The quantization type as a [simple string](../../develop/reference/protocol-spec#simple-strings): `fp32`, `bin`, or `q8`.", + " 1. A [bulk string](../../develop/reference/protocol-spec#bulk-strings) blob with the following raw data:", + " * 4-byte floats for fp32", + " * A bitmap for binary quantization", + " * A byte array for q8", + " 1. The L2 norm, as a [simple string](../../develop/reference/protocol-spec#simple-strings), of the vector before normalization.", + " 1. (Only for q8): The quantization range as a [simple string](../../develop/reference/protocol-spec#simple-strings). Multiply this by integer components to recover normalized values." + ], + "VGETATTR": [ + "One of the following:", + "* [Simple string reply](../../develop/reference/protocol-spec#simple-strings) containing the JSON attribute(s).", + "* [Bulk string reply](../../develop/reference/protocol-spec#bulk-strings) (null bulk string) for unknown key or element, or when no attributes exist for the given key/element pair." + ], + "VINFO": [ + "One of the following:", + "* [Array reply](../../develop/reference/protocol-spec#arrays) containing metadata and internal details about a vector set, including size, dimensions, quantization type, and graph structure.", + "* [Array reply](../../develop/reference/protocol-spec#arrays) (null array reply) for unknown key." + ], + "VLINKS": [ + "One of the following:", + "* [Array reply](../../develop/reference/protocol-spec#arrays) containing the names of adjacent elements as [strings](../../develop/reference/protocol-spec#simple-strings); interleved with scores when using the WITHSCORES option.", + "* [Bulk string reply](../../develop/reference/protocol-spec#bulk-strings) (null bulk string) for unknown keys and/or elements." + ], + "VRANDMEMBER": [ + "One of the following:", + "* [Array reply](../../develop/reference/protocol-spec#arrays) containing the names of count random elements as [strings](../../develop/reference/protocol-spec#simple-strings).", + "* [Bulk string reply](../../develop/reference/protocol-spec#bulk-strings) (null bulk string) for unknown keys.", + "* [Array reply](../../develop/reference/protocol-spec#arrays) (empty array) for unknown keys when a count is specified." + ], + "VREM": [ + "[Integer reply](../../develop/reference/protocol-spec#integers): 0 if either element or key do not exist; 1 if the element was removed." + ], + "VSETATTR": [ + "One of the following:", + "* [Integer reply](../../develop/reference/protocol-spec#integers): 0 if either the key or element does not exist; 1 if the attributes were successfully added to the element.", + "* [Simple error reply](develop/reference/protocol-spec/#simple-errors) for improperly specified attribute string." + ], + "VSIM": [ + "One of the following:", + "* [Simple error reply](develop/reference/protocol-spec/#simple-errors) for unknown element.", + "* [Array reply](../../develop/reference/protocol-spec#arrays) (empty array) for unknown key.", + "* [Array reply](../../develop/reference/protocol-spec#arrays) with matching elements; juxtaposed with scores when used with the WITHSCORES option." + ], "WAIT": [ "[Integer reply](../../develop/reference/protocol-spec#integers): the command returns the number of replicas reached by all the writes performed in the context of the current connection." ], diff --git a/data/resp3_replies.json b/data/resp3_replies.json index 3f9321866d..650108d53d 100644 --- a/data/resp3_replies.json +++ b/data/resp3_replies.json @@ -1269,6 +1269,68 @@ "UNWATCH": [ "[Simple string reply](../../develop/reference/protocol-spec#simple-strings): `OK`." ], + "VADD": [ + "One of the following:", + "* [Boolean reply](../../develop/reference/protocol-spec#booleans): true if key was added; false if key was not added.", + "* [Simple error reply](../../develop/reference/protocol-spec#simple-errors): if the command was malformed." + ], + "VCARD": [ + "[Integer reply](../../develop/reference/protocol-spec#integers): 0 if the key doesn't exist or the number of elements contained in the vector set." + ], + "VDIM": [ + "One of the following:", + "* [Integer reply](../../develop/reference/protocol-spec#integers): the number of vector set elements.", + "* [Simple error reply](../../develop/reference/protocol-spec#simple-errors): if the key does not exist." + ], + "VEMB": [ + "One of the following:", + "* [Array reply](../../develop/reference/protocol-spec#arrays): of [doubles](../../develop/reference/protocol-spec#doubles), representing the vector.", + "* [Array reply](../../develop/reference/protocol-spec#arrays): consisting of the following elements:", + " 1. The quantization type as a [simple string](../../develop/reference/protocol-spec#simple-strings): `fp32`, `bin`, or `q8`.", + " 1. A [bulk string](../../develop/reference/protocol-spec#bulk-strings) blob with the following raw data:", + " * 4-byte floats for fp32", + " * A bitmap for binary quantization", + " * A byte array for q8", + " 1. The [double](../../develop/reference/protocol-spec#doubles) L2 norm of the vector before normalization.", + " 1. (Only for q8): The quantization range as a [double](../../develop/reference/protocol-spec#doubles). Multiply this by integer components to recover normalized values." + ], + "VGETATTR": [ + "One of the following:", + "* [Simple string reply](../../develop/reference/protocol-spec#simple-strings) containing the JSON attribute(s).", + "* [Null reply](../../develop/reference/protocol-spec#nulls) for unknown key or element, or when no attributes exist for the given key/element pair." + ], + "VINFO": [ + "One of the following:", + "* [Array reply](../../develop/reference/protocol-spec#arrays) containing metadata and internal details about a vector set, including size, dimensions, quantization type, and graph structure.", + "* [Null reply](../../develop/reference/protocol-spec#nulls) for unknown key." + ], + "VLINKS": [ + "One of the following:", + "* [Array reply](../../develop/reference/protocol-spec#arrays) containing the names of adjacent elements as [strings](../../develop/reference/protocol-spec#simple-strings) when used without the WITHSCORES option.", + "* [Map reply](../../develop/reference/protocol-spec#maps) containing the names of adjecent elements as [strings](../../develop/reference/protocol-spec#simple-strings), together with their scores as [doubles](../../develop/reference/protocol-spec#doubles) when used with the WITHSCORES option.", + "* [Null reply](../../develop/reference/protocol-spec#nulls) for unknown keys and/or elements." + ], + "VRANDMEMBER": [ + "One of the following:", + "* [Array reply](../../develop/reference/protocol-spec#arrays) containing the names of *count* random elements as [strings](../../develop/reference/protocol-spec#simple-strings).", + "* [Null reply](../../develop/reference/protocol-spec#nulls) for unknown keys.", + "* [Array reply](../../develop/reference/protocol-spec#arrays) (empty array) for unknown keys when a count is specified." + ], + "VREM": [ + "[Boolean reply](../../develop/reference/protocol-spec#booleans): false if either element or key do not exist; true if the element was removed." + ], + "VSETATTR": [ + "One of the following:", + "* [Boolean reply](../../develop/reference/protocol-spec#booleans): false if either the key or element does not exist; true if the attributes were successfully added to the element.", + "* [Simple error reply](develop/reference/protocol-spec/#simple-errors) for improperly specified attribute string." + ], + "VSIM": [ + "One of the following:", + "* [Simple error reply](develop/reference/protocol-spec/#simple-errors) for unknown element.", + "* [Array reply](../../develop/reference/protocol-spec#arrays) (empty array) for unknown key.", + "* [Array reply](../../develop/reference/protocol-spec#arrays) with matching elements.", + "* [Map reply](../../develop/reference/protocol-spec#maps) with matching elements and [double](../../develop/reference/protocol-spec#doubles) scores when used with the WITHSCORES option." + ], "WAIT": [ "[Integer reply](../../develop/reference/protocol-spec#integers): the number of replicas reached by all the writes performed in the context of the current connection." ], diff --git a/layouts/commands/list.html b/layouts/commands/list.html index 2f93f0d386..e19ce379e1 100644 --- a/layouts/commands/list.html +++ b/layouts/commands/list.html @@ -54,6 +54,7 @@

Commands

+