From 1c129211878e070507cd0adfaca1a450f2311e07 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Mon, 17 Feb 2025 09:20:29 +0000 Subject: [PATCH 1/5] DOC-4837 started Jedis vector example --- content/develop/clients/jedis/produsage.md | 2 +- content/develop/clients/jedis/transpipe.md | 2 +- content/develop/clients/jedis/vecsearch.md | 234 +++++++++++++++++++++ 3 files changed, 236 insertions(+), 2 deletions(-) create mode 100644 content/develop/clients/jedis/vecsearch.md diff --git a/content/develop/clients/jedis/produsage.md b/content/develop/clients/jedis/produsage.md index 8cf2518859..eadd8c10bc 100644 --- a/content/develop/clients/jedis/produsage.md +++ b/content/develop/clients/jedis/produsage.md @@ -12,7 +12,7 @@ categories: description: Get your Jedis app ready for production linkTitle: Production usage title: Production usage -weight: 3 +weight: 6 --- The following sections explain how to handle situations that may occur diff --git a/content/develop/clients/jedis/transpipe.md b/content/develop/clients/jedis/transpipe.md index 4b7b66cdfa..d26b595f92 100644 --- a/content/develop/clients/jedis/transpipe.md +++ b/content/develop/clients/jedis/transpipe.md @@ -12,7 +12,7 @@ categories: description: Learn how to use Redis pipelines and transactions linkTitle: Pipelines/transactions title: Pipelines and transactions -weight: 2 +weight: 5 --- Redis lets you send a sequence of commands to the server together in a batch. diff --git a/content/develop/clients/jedis/vecsearch.md b/content/develop/clients/jedis/vecsearch.md new file mode 100644 index 0000000000..c4f0bd71d4 --- /dev/null +++ b/content/develop/clients/jedis/vecsearch.md @@ -0,0 +1,234 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +description: Learn how to index and query vector embeddings with Redis +linkTitle: Index and query vectors +title: Index and query vectors +weight: 3 +--- + +[Redis Query Engine]({{< relref "/develop/interact/search-and-query" >}}) +lets you index vector fields in [hash]({{< relref "/develop/data-types/hashes" >}}) +or [JSON]({{< relref "/develop/data-types/json" >}}) objects (see the +[Vectors]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors" >}}) +reference page for more information). +Among other things, vector fields can store *text embeddings*, which are AI-generated vector +representations of the semantic information in pieces of text. The +[vector distance]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#distance-metrics" >}}) +between two embeddings indicates how similar they are semantically. By comparing the +similarity of an embedding generated from some query text with embeddings stored in hash +or JSON fields, Redis can retrieve documents that closely match the query in terms +of their meaning. + +In the example below, we use the +[`sentence-transformers`](https://pypi.org/project/sentence-transformers/) +library to generate vector embeddings to store and index with +Redis Query Engine. + +## Initialize + +Install [`redis-py`]({{< relref "/develop/clients/redis-py" >}}) if you +have not already done so. Also, install `sentence-transformers` with the +following command: + +```bash +pip install sentence-transformers +``` + +In a new Python source file, start by importing the required classes: + +```python +from sentence_transformers import SentenceTransformer +from redis.commands.search.query import Query +from redis.commands.search.field import TextField, TagField, VectorField +from redis.commands.search.indexDefinition import IndexDefinition, IndexType + +import numpy as np +import redis +``` + +The first of these imports is the +`SentenceTransformer` class, which generates an embedding from a section of text. +Here, we create an instance of `SentenceTransformer` that uses the +[`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) +model for the embeddings. This model generates vectors with 384 dimensions, regardless +of the length of the input text, but note that the input is truncated to 256 +tokens (see +[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6) +at the [Hugging Face](https://huggingface.co/) docs to learn more about the way tokens +are related to the original text). + +```python +model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") +``` + +## Create the index + +Connect to Redis and delete any index previously created with the +name `vector_idx`. (The `dropindex()` call throws an exception if +the index doesn't already exist, which is why you need the +`try: except:` block.) + +```python +r = redis.Redis(decode_responses=True) + +try: + r.ft("vector_idx").dropindex(True) +except redis.exceptions.ResponseError: + pass +``` + +Next, we create the index. +The schema in the example below specifies hash objects for storage and includes +three fields: the text content to index, a +[tag]({{< relref "/develop/interact/search-and-query/advanced-concepts/tags" >}}) +field to represent the "genre" of the text, and the embedding vector generated from +the original text content. The `embedding` field specifies +[HNSW]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#hnsw-index" >}}) +indexing, the +[L2]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#distance-metrics" >}}) +vector distance metric, `Float32` values to represent the vector's components, +and 384 dimensions, as required by the `all-MiniLM-L6-v2` embedding model. + +```python +schema = ( + TextField("content"), + TagField("genre"), + VectorField("embedding", "HNSW", { + "TYPE": "FLOAT32", + "DIM": 384, + "DISTANCE_METRIC":"L2" + }) +) + +r.ft("vector_idx").create_index( + schema, + definition=IndexDefinition( + prefix=["doc:"], index_type=IndexType.HASH + ) +) +``` + +## Add data + +You can now supply the data objects, which will be indexed automatically +when you add them with [`hset()`]({{< relref "/commands/hset" >}}), as long as +you use the `doc:` prefix specified in the index definition. + +Use the `model.encode()` method of `SentenceTransformer` +as shown below to create the embedding that represents the `content` field. +The `astype()` option that follows the `model.encode()` call specifies that +we want a vector of `float32` values. The `tobytes()` option encodes the +vector components together as a single binary string rather than the +default Python list of `float` values. +Use the binary string representation when you are indexing hash objects +(as we are here), but use the default list of `float` for JSON objects. + +```python +content = "That is a very happy person" + +r.hset("doc:0", mapping={ + "content": content, + "genre": "persons", + "embedding": model.encode(content).astype(np.float32).tobytes(), +}) + +content = "That is a happy dog" + +r.hset("doc:1", mapping={ + "content": content, + "genre": "pets", + "embedding": model.encode(content).astype(np.float32).tobytes(), +}) + +content = "Today is a sunny day" + +r.hset("doc:2", mapping={ + "content": content, + "genre": "weather", + "embedding": model.encode(content).astype(np.float32).tobytes(), +}) +``` + +## Run a query + +After you have created the index and added the data, you are ready to run a query. +To do this, you must create another embedding vector from your chosen query +text. Redis calculates the similarity between the query vector and each +embedding vector in the index as it runs the query. It then ranks the +results in order of this numeric similarity value. + +The code below creates the query embedding using `model.encode()`, as with +the indexing, and passes it as a parameter when the query executes +(see +[Vector search]({{< relref "/develop/interact/search-and-query/query/vector-search" >}}) +for more information about using query parameters with embeddings). + +```python +q = Query( + "*=>[KNN 3 @embedding $vec AS vector_distance]" +).return_field("score").dialect(2) + +query_text = "That is a happy person" + +res = r.ft("vector_idx").search( + q, query_params={ + "vec": model.encode(query_text).astype(np.float32).tobytes() + } +) + +print(res) +``` + +The code is now ready to run, but note that it may take a while to complete when +you run it for the first time (which happens because RedisVL must download the +`all-MiniLM-L6-v2` model data before it can +generate the embeddings). When you run the code, it outputs the following result +object (slightly formatted here for clarity): + +```Python +Result{ + 3 total, + docs: [ + Document { + 'id': 'doc:0', + 'payload': None, + 'vector_distance': '0.114169985056', + 'content': 'That is a very happy person' + }, + Document { + 'id': 'doc:1', + 'payload': None, + 'vector_distance': '0.610845386982', + 'content': 'That is a happy dog' + }, + Document { + 'id': 'doc:2', + 'payload': None, + 'vector_distance': '1.48624813557', + 'content': 'Today is a sunny day' + } + ] +} +``` + +Note that the results are ordered according to the value of the `vector_distance` +field, with the lowest distance indicating the greatest similarity to the query. +As you would expect, the result for `doc:0` with the content text *"That is a very happy person"* +is the result that is most similar in meaning to the query text +*"That is a happy person"*. + +## Learn more + +See +[Vector search]({{< relref "/develop/interact/search-and-query/query/vector-search" >}}) +for more information about the indexing options, distance metrics, and query format +for vectors. From 48e6af3b8213a0893931c44814f87ce06ed14c1f Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Fri, 21 Feb 2025 16:19:33 +0000 Subject: [PATCH 2/5] DOC-4837 added imports --- content/develop/clients/jedis/vecsearch.md | 54 +++++++++++++++++----- 1 file changed, 42 insertions(+), 12 deletions(-) diff --git a/content/develop/clients/jedis/vecsearch.md b/content/develop/clients/jedis/vecsearch.md index c4f0bd71d4..a11e20c951 100644 --- a/content/develop/clients/jedis/vecsearch.md +++ b/content/develop/clients/jedis/vecsearch.md @@ -35,26 +35,56 @@ Redis Query Engine. ## Initialize -Install [`redis-py`]({{< relref "/develop/clients/redis-py" >}}) if you -have not already done so. Also, install `sentence-transformers` with the -following command: +Install [`jedis`]({{< relref "/develop/clients/jedis" >}}) if you +have not already done so. + +If you are using [Maven](https://maven.apache.org/), add the following +dependencies to your `pom.xml` file: + +```xml + + redis.clients + jedis + 5.2.0 + + + ai.djl.huggingface + tokenizers + 0.24.0 + +``` + +If you are using [Gradle](https://gradle.org/), add the following +dependencies to your `build.gradle` file: ```bash -pip install sentence-transformers +implementation 'redis.clients:jedis:5.2.0' +implementation 'ai.djl.huggingface:tokenizers:0.24.0' ``` -In a new Python source file, start by importing the required classes: +## Import dependencies -```python -from sentence_transformers import SentenceTransformer -from redis.commands.search.query import Query -from redis.commands.search.field import TextField, TagField, VectorField -from redis.commands.search.indexDefinition import IndexDefinition, IndexType +Import the following classes in your source file: -import numpy as np -import redis +```java +// Jedis client and query engine classes. +import redis.clients.jedis.UnifiedJedis; +import redis.clients.jedis.search.*; +import redis.clients.jedis.search.schemafields.*; +import redis.clients.jedis.search.schemafields.VectorField.VectorAlgorithm; +import redis.clients.jedis.exceptions.JedisDataException; + +// Data manipulation. +import java.nio.ByteBuffer; +import java.nio.ByteOrder; +import java.util.Map; +import java.util.List; + +// Tokenizer to generate the vector embeddings. +import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer; ``` + The first of these imports is the `SentenceTransformer` class, which generates an embedding from a section of text. Here, we create an instance of `SentenceTransformer` that uses the From 48c40d4643bbb1b480a938c2ee282d420cdb001d Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Mon, 24 Feb 2025 10:13:48 +0000 Subject: [PATCH 3/5] DOC-4837 added index creation code --- content/develop/clients/jedis/vecsearch.md | 127 +++++++++++---------- 1 file changed, 65 insertions(+), 62 deletions(-) diff --git a/content/develop/clients/jedis/vecsearch.md b/content/develop/clients/jedis/vecsearch.md index a11e20c951..8018fe2580 100644 --- a/content/develop/clients/jedis/vecsearch.md +++ b/content/develop/clients/jedis/vecsearch.md @@ -84,41 +84,36 @@ import java.util.List; import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer; ``` +## Create a tokenizer instance -The first of these imports is the -`SentenceTransformer` class, which generates an embedding from a section of text. -Here, we create an instance of `SentenceTransformer` that uses the -[`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) -model for the embeddings. This model generates vectors with 384 dimensions, regardless -of the length of the input text, but note that the input is truncated to 256 -tokens (see -[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6) -at the [Hugging Face](https://huggingface.co/) docs to learn more about the way tokens -are related to the original text). +We will use the +[`all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) +tokenizer to generate the embeddings. The vectors that represent the +embeddings have 768 components, regardless of the length of the input +text. -```python -model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2") +```java +HuggingFaceTokenizer sentenceTokenizer = HuggingFaceTokenizer.newInstance( + "sentence-transformers/all-mpnet-base-v2", + Map.of("maxLength", "768", "modelMaxLength", "768") +); ``` ## Create the index Connect to Redis and delete any index previously created with the -name `vector_idx`. (The `dropindex()` call throws an exception if +name `vector_idx`. (The `ftDropIndex()` call throws an exception if the index doesn't already exist, which is why you need the -`try: except:` block.) +`try...catch` block.) -```python -r = redis.Redis(decode_responses=True) +```java +UnifiedJedis jedis = new UnifiedJedis("redis://localhost:6379"); -try: - r.ft("vector_idx").dropindex(True) -except redis.exceptions.ResponseError: - pass +try {jedis.ftDropIndex("vector_idx");} catch (JedisDataException j){} ``` Next, we create the index. -The schema in the example below specifies hash objects for storage and includes -three fields: the text content to index, a +The schema in the example below includes three fields: the text content to index, a [tag]({{< relref "/develop/interact/search-and-query/advanced-concepts/tags" >}}) field to represent the "genre" of the text, and the embedding vector generated from the original text content. The `embedding` field specifies @@ -126,27 +121,41 @@ the original text content. The `embedding` field specifies indexing, the [L2]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#distance-metrics" >}}) vector distance metric, `Float32` values to represent the vector's components, -and 384 dimensions, as required by the `all-MiniLM-L6-v2` embedding model. +and 768 dimensions, as required by the `all-mpnet-base-v2` embedding model. -```python -schema = ( - TextField("content"), - TagField("genre"), - VectorField("embedding", "HNSW", { - "TYPE": "FLOAT32", - "DIM": 384, - "DISTANCE_METRIC":"L2" - }) -) +The `FTCreateParams` object specifies hash objects for storage and a +prefix `doc:` that identifies the hash objects we want to index. -r.ft("vector_idx").create_index( - schema, - definition=IndexDefinition( - prefix=["doc:"], index_type=IndexType.HASH - ) -) +```java +SchemaField[] schema = { + TextField.of("content"), + TagField.of("genre"), + VectorField.builder() + .fieldName("embedding") + .algorithm(VectorAlgorithm.HNSW) + .attributes( + Map.of( + "TYPE", "FLOAT32", + "DIM", 768, + "DISTANCE_METRIC", "L2", + "INITIAL_CAP", 3 + ) + ) + .build() +}; + +jedis.ftCreate("vector_idx", + FTCreateParams.createParams() + .addPrefix("doc:") + .on(IndexDataType.HASH), + schema +); ``` +## Define some helper methods + + + ## Add data You can now supply the data objects, which will be indexed automatically @@ -162,30 +171,24 @@ default Python list of `float` values. Use the binary string representation when you are indexing hash objects (as we are here), but use the default list of `float` for JSON objects. -```python -content = "That is a very happy person" - -r.hset("doc:0", mapping={ - "content": content, - "genre": "persons", - "embedding": model.encode(content).astype(np.float32).tobytes(), -}) - -content = "That is a happy dog" - -r.hset("doc:1", mapping={ - "content": content, - "genre": "pets", - "embedding": model.encode(content).astype(np.float32).tobytes(), -}) - -content = "Today is a sunny day" +```java +String sentence1 = "That is a very happy person"; +jedis.hset("doc:1", Map.of( "content", sentence1, "genre", "persons")); +jedis.hset( + "doc:1".getBytes(), + "embedding".getBytes(), + longArrayToByteArray(sentenceTokenizer.encode(sentence1).getIds()) +); + +String sentence2 = "That is a happy dog"; +jedis.hset("doc:2", Map.of( "content", sentence2, "genre", "pets")); +jedis.hset("doc:2".getBytes(), "embedding".getBytes(), longArrayToByteArray(sentenceTokenizer.encode(sentence2).getIds())); + +String sentence3 = "Today is a sunny day"; +Map doc3 = Map.of( "content", sentence3, "genre", "weather"); +jedis.hset("doc:3", doc3); +jedis.hset("doc:3".getBytes(), "embedding".getBytes(), longArrayToByteArray(sentenceTokenizer.encode(sentence3).getIds())); -r.hset("doc:2", mapping={ - "content": content, - "genre": "weather", - "embedding": model.encode(content).astype(np.float32).tobytes(), -}) ``` ## Run a query From be874f5b02398d727d1c219335def05cc1483954 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Mon, 24 Feb 2025 15:23:33 +0000 Subject: [PATCH 4/5] DOC-4837 added remaining code examples --- content/develop/clients/jedis/vecsearch.md | 130 +++++++++++---------- 1 file changed, 69 insertions(+), 61 deletions(-) diff --git a/content/develop/clients/jedis/vecsearch.md b/content/develop/clients/jedis/vecsearch.md index 8018fe2580..78c85d780d 100644 --- a/content/develop/clients/jedis/vecsearch.md +++ b/content/develop/clients/jedis/vecsearch.md @@ -28,10 +28,9 @@ similarity of an embedding generated from some query text with embeddings stored or JSON fields, Redis can retrieve documents that closely match the query in terms of their meaning. -In the example below, we use the -[`sentence-transformers`](https://pypi.org/project/sentence-transformers/) -library to generate vector embeddings to store and index with -Redis Query Engine. +In the example below, we use the [HuggingFace](https://huggingface.co/) model +[`all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) +to generate the vector embeddings to store and index with Redis Query Engine. ## Initialize @@ -152,9 +151,28 @@ jedis.ftCreate("vector_idx", ); ``` -## Define some helper methods +## Define a helper method + +The embedding model represents the vectors as an array of `long` integer values, +but Redis Query Engine expects the vector components to be `float` values. +Also, when you store vectors in a hash object, you must encode the vector +array as a `byte` string. To simplify this situation, we declare a helper +method `longsToFloatsByteString()` that takes the `long` array that the +embedding model returns, converts it to an array of `float` values, and +then encodes the `float` array as a `byte` string: +```java +public static byte[] longsToFloatsByteString(long[] input) { + float[] floats = new float[input.length]; + for (int i = 0; i < input.length; i++) { + floats[i] = input[i]; + } + byte[] bytes = new byte[Float.BYTES * floats.length]; + ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer().put(floats); + return bytes; +} +``` ## Add data @@ -162,14 +180,12 @@ You can now supply the data objects, which will be indexed automatically when you add them with [`hset()`]({{< relref "/commands/hset" >}}), as long as you use the `doc:` prefix specified in the index definition. -Use the `model.encode()` method of `SentenceTransformer` +Use the `encode()` method of the `sentenceTokenizer` object as shown below to create the embedding that represents the `content` field. -The `astype()` option that follows the `model.encode()` call specifies that -we want a vector of `float32` values. The `tobytes()` option encodes the -vector components together as a single binary string rather than the -default Python list of `float` values. -Use the binary string representation when you are indexing hash objects -(as we are here), but use the default list of `float` for JSON objects. +The `getIds()` method that follows the `encode()` call obtains the vector +of `long` values which we then convert to a `float` array stored as a `byte` +string. Use the `byte` string representation when you are indexing hash +objects (as we are here), but use the default list of `float` for JSON objects. ```java String sentence1 = "That is a very happy person"; @@ -177,18 +193,25 @@ jedis.hset("doc:1", Map.of( "content", sentence1, "genre", "persons")); jedis.hset( "doc:1".getBytes(), "embedding".getBytes(), - longArrayToByteArray(sentenceTokenizer.encode(sentence1).getIds()) + longsToFloatsByteString(sentenceTokenizer.encode(sentence1).getIds()) ); String sentence2 = "That is a happy dog"; jedis.hset("doc:2", Map.of( "content", sentence2, "genre", "pets")); -jedis.hset("doc:2".getBytes(), "embedding".getBytes(), longArrayToByteArray(sentenceTokenizer.encode(sentence2).getIds())); +jedis.hset( + "doc:2".getBytes(), + "embedding".getBytes(), + longsToFloatsByteString(sentenceTokenizer.encode(sentence2).getIds()) +); String sentence3 = "Today is a sunny day"; Map doc3 = Map.of( "content", sentence3, "genre", "weather"); jedis.hset("doc:3", doc3); -jedis.hset("doc:3".getBytes(), "embedding".getBytes(), longArrayToByteArray(sentenceTokenizer.encode(sentence3).getIds())); - +jedis.hset( + "doc:3".getBytes(), + "embedding".getBytes(), + longsToFloatsByteString(sentenceTokenizer.encode(sentence3).getIds()) +); ``` ## Run a query @@ -199,58 +222,43 @@ text. Redis calculates the similarity between the query vector and each embedding vector in the index as it runs the query. It then ranks the results in order of this numeric similarity value. -The code below creates the query embedding using `model.encode()`, as with -the indexing, and passes it as a parameter when the query executes -(see +The code below creates the query embedding using the `encode()` method, as with +the indexing, and passes it as a parameter when the query executes (see [Vector search]({{< relref "/develop/interact/search-and-query/query/vector-search" >}}) for more information about using query parameters with embeddings). -```python -q = Query( - "*=>[KNN 3 @embedding $vec AS vector_distance]" -).return_field("score").dialect(2) - -query_text = "That is a happy person" - -res = r.ft("vector_idx").search( - q, query_params={ - "vec": model.encode(query_text).astype(np.float32).tobytes() - } -) - -print(res) +```java +String sentence = "That is a happy person"; + +int K = 3; +Query q = new Query("*=>[KNN $K @embedding $BLOB AS score]"). + returnFields("content", "score"). + addParam("K", K). + addParam( + "BLOB", + longsToFloatsByteString( + sentenceTokenizer.encode(sentence).getIds() + ) + ). + dialect(2); + +List docs = jedis.ftSearch("vector_idx", q).getDocuments(); + +for (Document doc: docs) { + System.out.println(doc); +} ``` The code is now ready to run, but note that it may take a while to complete when -you run it for the first time (which happens because RedisVL must download the -`all-MiniLM-L6-v2` model data before it can +you run it for the first time (which happens because the tokenizer must download the +`all-mpnet-base-v2` model data before it can generate the embeddings). When you run the code, it outputs the following result -object (slightly formatted here for clarity): - -```Python -Result{ - 3 total, - docs: [ - Document { - 'id': 'doc:0', - 'payload': None, - 'vector_distance': '0.114169985056', - 'content': 'That is a very happy person' - }, - Document { - 'id': 'doc:1', - 'payload': None, - 'vector_distance': '0.610845386982', - 'content': 'That is a happy dog' - }, - Document { - 'id': 'doc:2', - 'payload': None, - 'vector_distance': '1.48624813557', - 'content': 'Today is a sunny day' - } - ] -} +objects: + +``` +id:doc:1, score: 1.0, properties:[score=9301635, content=That is a very happy person] +id:doc:2, score: 1.0, properties:[score=1411344, content=That is a happy dog] +id:doc:3, score: 1.0, properties:[score=67178800, content=Today is a sunny day] ``` Note that the results are ordered according to the value of the `vector_distance` From 194273e21b10e7e68b820f2ead62a1b33fd00478 Mon Sep 17 00:00:00 2001 From: Andy Stark Date: Tue, 25 Feb 2025 12:43:14 +0000 Subject: [PATCH 5/5] DOC-4837 tidied text and examples --- content/develop/clients/jedis/vecsearch.md | 130 ++++++++++++--------- 1 file changed, 72 insertions(+), 58 deletions(-) diff --git a/content/develop/clients/jedis/vecsearch.md b/content/develop/clients/jedis/vecsearch.md index 78c85d780d..6b4eccb62f 100644 --- a/content/develop/clients/jedis/vecsearch.md +++ b/content/develop/clients/jedis/vecsearch.md @@ -34,9 +34,6 @@ to generate the vector embeddings to store and index with Redis Query Engine. ## Initialize -Install [`jedis`]({{< relref "/develop/clients/jedis" >}}) if you -have not already done so. - If you are using [Maven](https://maven.apache.org/), add the following dependencies to your `pom.xml` file: @@ -83,6 +80,33 @@ import java.util.List; import ai.djl.huggingface.tokenizers.HuggingFaceTokenizer; ``` +## Define a helper method + +Our embedding model represents the vectors as an array of `long` integer values, +but Redis Query Engine expects the vector components to be `float` values. +Also, when you store vectors in a hash object, you must encode the vector +array as a `byte` string. To simplify this situation, we declare a helper +method `longsToFloatsByteString()` that takes the `long` array that the +embedding model returns, converts it to an array of `float` values, and +then encodes the `float` array as a `byte` string: + +```java +public static byte[] longsToFloatsByteString(long[] input) { + float[] floats = new float[input.length]; + for (int i = 0; i < input.length; i++) { + floats[i] = input[i]; + } + + byte[] bytes = new byte[Float.BYTES * floats.length]; + ByteBuffer + .wrap(bytes) + .order(ByteOrder.LITTLE_ENDIAN) + .asFloatBuffer() + .put(floats); + return bytes; +} +``` + ## Create a tokenizer instance We will use the @@ -136,8 +160,7 @@ SchemaField[] schema = { Map.of( "TYPE", "FLOAT32", "DIM", 768, - "DISTANCE_METRIC", "L2", - "INITIAL_CAP", 3 + "DISTANCE_METRIC", "L2" ) ) .build() @@ -151,29 +174,6 @@ jedis.ftCreate("vector_idx", ); ``` -## Define a helper method - -The embedding model represents the vectors as an array of `long` integer values, -but Redis Query Engine expects the vector components to be `float` values. -Also, when you store vectors in a hash object, you must encode the vector -array as a `byte` string. To simplify this situation, we declare a helper -method `longsToFloatsByteString()` that takes the `long` array that the -embedding model returns, converts it to an array of `float` values, and -then encodes the `float` array as a `byte` string: - -```java -public static byte[] longsToFloatsByteString(long[] input) { - float[] floats = new float[input.length]; - for (int i = 0; i < input.length; i++) { - floats[i] = input[i]; - } - - byte[] bytes = new byte[Float.BYTES * floats.length]; - ByteBuffer.wrap(bytes).order(ByteOrder.LITTLE_ENDIAN).asFloatBuffer().put(floats); - return bytes; -} -``` - ## Add data You can now supply the data objects, which will be indexed automatically @@ -182,14 +182,17 @@ you use the `doc:` prefix specified in the index definition. Use the `encode()` method of the `sentenceTokenizer` object as shown below to create the embedding that represents the `content` field. -The `getIds()` method that follows the `encode()` call obtains the vector +The `getIds()` method that follows `encode()` obtains the vector of `long` values which we then convert to a `float` array stored as a `byte` -string. Use the `byte` string representation when you are indexing hash -objects (as we are here), but use the default list of `float` for JSON objects. +string using our helper method. Use the `byte` string representation when you are +indexing hash objects (as we are here), but use the default list of `float` for +JSON objects. Note that when we set the `embedding` field, we must use an overload +of `hset()` that requires `byte` arrays for each of the key, the field name, and +the value, which is why we include the `getBytes()` calls on the strings. ```java String sentence1 = "That is a very happy person"; -jedis.hset("doc:1", Map.of( "content", sentence1, "genre", "persons")); +jedis.hset("doc:1", Map.of("content", sentence1, "genre", "persons")); jedis.hset( "doc:1".getBytes(), "embedding".getBytes(), @@ -197,7 +200,7 @@ jedis.hset( ); String sentence2 = "That is a happy dog"; -jedis.hset("doc:2", Map.of( "content", sentence2, "genre", "pets")); +jedis.hset("doc:2", Map.of("content", sentence2, "genre", "pets")); jedis.hset( "doc:2".getBytes(), "embedding".getBytes(), @@ -205,8 +208,7 @@ jedis.hset( ); String sentence3 = "Today is a sunny day"; -Map doc3 = Map.of( "content", sentence3, "genre", "weather"); -jedis.hset("doc:3", doc3); +jedis.hset("doc:3", Map.of("content", sentence3, "genre", "weather")); jedis.hset( "doc:3".getBytes(), "embedding".getBytes(), @@ -218,53 +220,65 @@ jedis.hset( After you have created the index and added the data, you are ready to run a query. To do this, you must create another embedding vector from your chosen query -text. Redis calculates the similarity between the query vector and each -embedding vector in the index as it runs the query. It then ranks the -results in order of this numeric similarity value. +text. Redis calculates the vector distance between the query vector and each +embedding vector in the index as it runs the query. We can request the results to be +sorted to rank them in order of ascending distance. The code below creates the query embedding using the `encode()` method, as with the indexing, and passes it as a parameter when the query executes (see [Vector search]({{< relref "/develop/interact/search-and-query/query/vector-search" >}}) for more information about using query parameters with embeddings). +The query is a +[K nearest neighbors (KNN)]({{< relref "/develop/interact/search-and-query/advanced-concepts/vectors#knn-vector-search" >}}) +search that sorts the results in order of vector distance from the query vector. ```java String sentence = "That is a happy person"; int K = 3; -Query q = new Query("*=>[KNN $K @embedding $BLOB AS score]"). - returnFields("content", "score"). - addParam("K", K). - addParam( - "BLOB", - longsToFloatsByteString( - sentenceTokenizer.encode(sentence).getIds() - ) - ). - dialect(2); +Query q = new Query("*=>[KNN $K @embedding $BLOB AS distance]") + .returnFields("content", "distance") + .addParam("K", K) + .addParam( + "BLOB", + longsToFloatsByteString( + sentenceTokenizer.encode(sentence)..getIds() + ) + ) + .setSortBy("distance", true) + .dialect(2); List docs = jedis.ftSearch("vector_idx", q).getDocuments(); for (Document doc: docs) { - System.out.println(doc); + System.out.println( + String.format( + "ID: %s, Distance: %s, Content: %s", + doc.getId(), + doc.get("distance"), + doc.get("content") + ) + ); } ``` -The code is now ready to run, but note that it may take a while to complete when +Assuming you have added the code from the steps above to your source file, +it is now ready to run, but note that it may take a while to complete when you run it for the first time (which happens because the tokenizer must download the `all-mpnet-base-v2` model data before it can -generate the embeddings). When you run the code, it outputs the following result -objects: +generate the embeddings). When you run the code, it outputs the following result text: ``` -id:doc:1, score: 1.0, properties:[score=9301635, content=That is a very happy person] -id:doc:2, score: 1.0, properties:[score=1411344, content=That is a happy dog] -id:doc:3, score: 1.0, properties:[score=67178800, content=Today is a sunny day] +Results: +ID: doc:2, Distance: 1411344, Content: That is a happy dog +ID: doc:1, Distance: 9301635, Content: That is a very happy person +ID: doc:3, Distance: 67178800, Content: Today is a sunny day ``` -Note that the results are ordered according to the value of the `vector_distance` +Note that the results are ordered according to the value of the `distance` field, with the lowest distance indicating the greatest similarity to the query. -As you would expect, the result for `doc:0` with the content text *"That is a very happy person"* -is the result that is most similar in meaning to the query text +For this model, the text *"That is a happy dog"* +is the result judged to be most similar in meaning to the query text *"That is a happy person"*. ## Learn more