Skip to content

DOC-5552 Jedis vector set embedding examples #2142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 17, 2025
Merged

DOC-5552 Jedis vector set embedding examples #2142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
d6e4711
DOC-5552 update Jedis vector search example with
andy-stark-redis Sep 17, 2025
1a79e4b
DOC-5552 added Jedis vector set embedding examples
andy-stark-redis Sep 17, 2025
88466ea
DOC-5552 applied PR feedback
andy-stark-redis Sep 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 52 additions & 44 deletions content/develop/clients/jedis/vecsearch.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ similarity of an embedding generated from some query text with embeddings stored
or JSON fields, Redis can retrieve documents that closely match the query in terms
of their meaning.

In the example below, we use the [HuggingFace](https://huggingface.co/) model
[`all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
The example below uses the [HuggingFace](https://huggingface.co/) model
[`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
to generate the vector embeddings to store and index with Redis Query Engine.
The code is first demonstrated for hash documents with a
separate section to explain the
Expand All @@ -53,21 +53,33 @@ dependencies to your `pom.xml` file:
<dependency>
<groupId>redis.clients</groupId>
<artifactId>jedis</artifactId>
<version>5.2.0</version>
<version>6.1.0</version>
</dependency>
<dependency>
<groupId>ai.djl.huggingface</groupId>
<artifactId>tokenizers</artifactId>
<version>0.24.0</version>
<version>0.33.0</version>
</dependency>
<dependency>
<groupId>ai.djl.pytorch</groupId>
<artifactId>pytorch-model-zoo</artifactId>
<version>0.33.0</version>
</dependency>
<dependency>
<groupId>ai.djl</groupId>
<artifactId>api</artifactId>
<version>0.33.0</version>
</dependency>
```

If you are using [Gradle](https://gradle.org/), add the following
dependencies to your `build.gradle` file:

```bash
implementation 'redis.clients:jedis:5.2.0'
implementation 'ai.djl.huggingface:tokenizers:0.24.0'
implementation 'redis.clients:jedis:6.1.0'
implementation 'ai.djl.huggingface:tokenizers:0.33.0'
implementation 'ai.djl.pytorch:pytorch-model-zoo:0.33.0'
implementation 'ai.djl:api:0.33.0'
```

## Import dependencies
Expand All @@ -79,24 +91,25 @@ Import the following classes in your source file:

## Define a helper method

Our embedding model represents the vectors as an array of `long` integer values,
but Redis Query Engine expects the vector components to be `float` values.
The embedding model represents the vectors as an array of `float` values,
which is the format required by Redis Query Engine.
Also, when you store vectors in a hash object, you must encode the vector
array as a `byte` string. To simplify this situation, we declare a helper
method `longsToFloatsByteString()` that takes the `long` array that the
embedding model returns, converts it to an array of `float` values, and
then encodes the `float` array as a `byte` string:
array as a `byte` string. To simplify this situation, you can declare a helper
method `floatsToByteString()` that takes the `float` array that the
embedding model returns and encodes it as a `byte` string:

{{< clients-example set="HomeQueryVec" step="helper_method" lang_filter="Java-Sync" >}}
{{< /clients-example >}}

## Create a tokenizer instance

We will use the
[`all-mpnet-base-v2`](https://huggingface.co/sentence-transformers/all-mpnet-base-v2)
tokenizer to generate the embeddings. The vectors that represent the
embeddings have 768 components, regardless of the length of the input
text.
The next step is to generate the embeddings using the
[`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
model. The vectors that represent the
embeddings have 384 components, regardless of the length of the input
text, but note that the input is truncated to 256
tokens (see
[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6)

{{< clients-example set="HomeQueryVec" step="tokenizer" lang_filter="Java-Sync" >}}
{{< /clients-example >}}
Expand All @@ -111,7 +124,7 @@ the index doesn't already exist, which is why you need the
{{< clients-example set="HomeQueryVec" step="connect" lang_filter="Java-Sync" >}}
{{< /clients-example >}}

Next, we create the index.
Next, create the index.
The schema in the example below includes three fields: the text content to index, a
[tag]({{< relref "/develop/ai/search-and-query/advanced-concepts/tags" >}})
field to represent the "genre" of the text, and the embedding vector generated from
Expand All @@ -120,10 +133,10 @@ the original text content. The `embedding` field specifies
indexing, the
[L2]({{< relref "/develop/ai/search-and-query/vectors#distance-metrics" >}})
vector distance metric, `Float32` values to represent the vector's components,
and 768 dimensions, as required by the `all-mpnet-base-v2` embedding model.
and 384 dimensions, as required by the `all-MiniLM-L6-v2` embedding model.

The `FTCreateParams` object specifies hash objects for storage and a
prefix `doc:` that identifies the hash objects we want to index.
prefix `doc:` that identifies the hash objects to index.

{{< clients-example set="HomeQueryVec" step="create_index" lang_filter="Java-Sync" >}}
{{< /clients-example >}}
Expand All @@ -134,16 +147,15 @@ You can now supply the data objects, which will be indexed automatically
when you add them with [`hset()`]({{< relref "/commands/hset" >}}), as long as
you use the `doc:` prefix specified in the index definition.

Use the `encode()` method of the `sentenceTokenizer` object
Use the `predict()` method of the `Predictor` object
as shown below to create the embedding that represents the `content` field.
The `getIds()` method that follows `encode()` obtains the vector
of `long` values which we then convert to a `float` array stored as a `byte`
string using our helper method. Use the `byte` string representation when you are
indexing hash objects (as we are here), but use an array of `float` for
The `predict()` method returns a `float[]` array which is then converted to a `byte`
string using the helper method. Use the `byte` string representation when you are
indexing hash objects (as in this example), but use the array of `float` directly for
JSON objects (see [Differences with JSON objects](#differences-with-json-documents)
below). Note that when we set the `embedding` field, we must use an overload
below). Note that when you set the `embedding` field, you must use an overload
of `hset()` that requires `byte` arrays for each of the key, the field name, and
the value, which is why we include the `getBytes()` calls on the strings.
the value, which is why you must include the `getBytes()` calls on the strings.

{{< clients-example set="HomeQueryVec" step="add_data" lang_filter="Java-Sync" >}}
{{< /clients-example >}}
Expand All @@ -153,10 +165,10 @@ the value, which is why we include the `getBytes()` calls on the strings.
After you have created the index and added the data, you are ready to run a query.
To do this, you must create another embedding vector from your chosen query
text. Redis calculates the vector distance between the query vector and each
embedding vector in the index as it runs the query. We can request the results to be
embedding vector in the index as it runs the query. You can request the results to be
sorted to rank them in order of ascending distance.

The code below creates the query embedding using the `encode()` method, as with
The code below creates the query embedding using the `predict()` method, as with
the indexing, and passes it as a parameter when the query executes (see
[Vector search]({{< relref "/develop/ai/search-and-query/query/vector-search" >}})
for more information about using query parameters with embeddings).
Expand All @@ -170,19 +182,19 @@ search that sorts the results in order of vector distance from the query vector.
Assuming you have added the code from the steps above to your source file,
it is now ready to run, but note that it may take a while to complete when
you run it for the first time (which happens because the tokenizer must download the
`all-mpnet-base-v2` model data before it can
`all-MiniLM-L6-v2` model data before it can
generate the embeddings). When you run the code, it outputs the following result text:

```
Results:
ID: doc:2, Distance: 1411344, Content: That is a happy dog
ID: doc:1, Distance: 9301635, Content: That is a very happy person
ID: doc:3, Distance: 67178800, Content: Today is a sunny day
ID: doc:1, Distance: 0.114169836044, Content: That is a very happy person
ID: doc:2, Distance: 0.610845506191, Content: That is a happy dog
ID: doc:3, Distance: 1.48624765873, Content: Today is a sunny day
```

Note that the results are ordered according to the value of the `distance`
field, with the lowest distance indicating the greatest similarity to the query.
For this model, the text *"That is a happy dog"*
As expected, the text *"That is a very happy person"*
is the result judged to be most similar in meaning to the query text
*"That is a happy person"*.

Expand All @@ -202,12 +214,8 @@ the one created previously for hashes:
{{< /clients-example >}}

An important difference with JSON indexing is that the vectors are
specified using arrays of `float` instead of binary strings. This requires
a modified version of the `longsToFloatsByteString()` method
used previously:

{{< clients-example set="HomeQueryVec" step="json_helper_method" lang_filter="Java-Sync" >}}
{{< /clients-example >}}
specified using arrays of `float` instead of binary strings, so you don't need
a helper method to convert the array to a binary string.

Use [`jsonSet()`]({{< relref "/commands/json.set" >}}) to add the data
instead of [`hset()`]({{< relref "/commands/hset" >}}). Use instances
Expand All @@ -221,7 +229,7 @@ The query is almost identical to the one for the hash documents. This
demonstrates how the right choice of aliases for the JSON paths can
save you having to write complex queries. An important thing to notice
is that the vector parameter for the query is still specified as a
binary string (using the `longsToFloatsByteString()` method), even though
binary string (created using the `floatsToByteString()` method), even though
the data for the `embedding` field of the JSON was specified as an array.

{{< clients-example set="HomeQueryVec" step="json_query" lang_filter="Java-Sync" >}}
Expand All @@ -232,9 +240,9 @@ query is the same as for hash:

```
Results:
ID: jdoc:2, Distance: 1411344, Content: That is a happy dog
ID: jdoc:1, Distance: 9301635, Content: That is a very happy person
ID: jdoc:3, Distance: 67178800, Content: Today is a sunny day
ID: jdoc:1, Distance: 0.114169836044, Content: That is a very happy person
ID: jdoc:2, Distance: 0.610845506191, Content: That is a happy dog
ID: jdoc:3, Distance: 1.48624765873, Content: Today is a sunny day
```

## Learn more
Expand Down
Loading