@@ -28,10 +28,9 @@ similarity of an embedding generated from some query text with embeddings stored
2828or JSON fields, Redis can retrieve documents that closely match the query in terms
2929of their meaning.
3030
31- In the example below, we use the
32- [ ` sentence-transformers ` ] ( https://pypi.org/project/sentence-transformers/ )
33- library to generate vector embeddings to store and index with
34- Redis Query Engine.
31+ In the example below, we use the [ HuggingFace] ( https://huggingface.co/ ) model
32+ [ ` all-mpnet-base-v2 ` ] ( https://huggingface.co/sentence-transformers/all-mpnet-base-v2 )
33+ to generate the vector embeddings to store and index with Redis Query Engine.
3534
3635## Initialize
3736
@@ -152,43 +151,67 @@ jedis.ftCreate("vector_idx",
152151);
153152```
154153
155- ## Define some helper methods
154+ ## Define a helper method
155+
156+ The embedding model represents the vectors as an array of ` long ` integer values,
157+ but Redis Query Engine expects the vector components to be ` float ` values.
158+ Also, when you store vectors in a hash object, you must encode the vector
159+ array as a ` byte ` string. To simplify this situation, we declare a helper
160+ method ` longsToFloatsByteString() ` that takes the ` long ` array that the
161+ embedding model returns, converts it to an array of ` float ` values, and
162+ then encodes the ` float ` array as a ` byte ` string:
156163
164+ ``` java
165+ public static byte [] longsToFloatsByteString(long [] input) {
166+ float [] floats = new float [input. length];
167+ for (int i = 0 ; i < input. length; i++ ) {
168+ floats[i] = input[i];
169+ }
157170
171+ byte [] bytes = new byte [Float . BYTES * floats. length];
172+ ByteBuffer . wrap(bytes). order(ByteOrder . LITTLE_ENDIAN ). asFloatBuffer(). put(floats);
173+ return bytes;
174+ }
175+ ```
158176
159177## Add data
160178
161179You can now supply the data objects, which will be indexed automatically
162180when you add them with [ ` hset() ` ] ({{< relref "/commands/hset" >}}), as long as
163181you use the ` doc: ` prefix specified in the index definition.
164182
165- Use the ` model. encode()` method of ` SentenceTransformer `
183+ Use the ` encode() ` method of the ` sentenceTokenizer ` object
166184as shown below to create the embedding that represents the ` content ` field.
167- The ` astype() ` option that follows the ` model.encode() ` call specifies that
168- we want a vector of ` float32 ` values. The ` tobytes() ` option encodes the
169- vector components together as a single binary string rather than the
170- default Python list of ` float ` values.
171- Use the binary string representation when you are indexing hash objects
172- (as we are here), but use the default list of ` float ` for JSON objects.
185+ The ` getIds() ` method that follows the ` encode() ` call obtains the vector
186+ of ` long ` values which we then convert to a ` float ` array stored as a ` byte `
187+ string. Use the ` byte ` string representation when you are indexing hash
188+ objects (as we are here), but use the default list of ` float ` for JSON objects.
173189
174190``` java
175191String sentence1 = " That is a very happy person" ;
176192jedis. hset(" doc:1" , Map . of( " content" , sentence1, " genre" , " persons" ));
177193jedis. hset(
178194 " doc:1" . getBytes(),
179195 " embedding" . getBytes(),
180- longArrayToByteArray (sentenceTokenizer. encode(sentence1). getIds())
196+ longsToFloatsByteString (sentenceTokenizer. encode(sentence1). getIds())
181197);
182198
183199String sentence2 = " That is a happy dog" ;
184200jedis. hset(" doc:2" , Map . of( " content" , sentence2, " genre" , " pets" ));
185- jedis. hset(" doc:2" . getBytes(), " embedding" . getBytes(), longArrayToByteArray(sentenceTokenizer. encode(sentence2). getIds()));
201+ jedis. hset(
202+ " doc:2" . getBytes(),
203+ " embedding" . getBytes(),
204+ longsToFloatsByteString(sentenceTokenizer. encode(sentence2). getIds())
205+ );
186206
187207String sentence3 = " Today is a sunny day" ;
188208Map<String , String > doc3 = Map . of( " content" , sentence3, " genre" , " weather" );
189209jedis. hset(" doc:3" , doc3);
190- jedis. hset(" doc:3" . getBytes(), " embedding" . getBytes(), longArrayToByteArray(sentenceTokenizer. encode(sentence3). getIds()));
191-
210+ jedis. hset(
211+ " doc:3" . getBytes(),
212+ " embedding" . getBytes(),
213+ longsToFloatsByteString(sentenceTokenizer. encode(sentence3). getIds())
214+ );
192215```
193216
194217## Run a query
@@ -199,58 +222,43 @@ text. Redis calculates the similarity between the query vector and each
199222embedding vector in the index as it runs the query. It then ranks the
200223results in order of this numeric similarity value.
201224
202- The code below creates the query embedding using ` model.encode() ` , as with
203- the indexing, and passes it as a parameter when the query executes
204- (see
225+ The code below creates the query embedding using the ` encode() ` method, as with
226+ the indexing, and passes it as a parameter when the query executes (see
205227[ Vector search] ({{< relref "/develop/interact/search-and-query/query/vector-search" >}})
206228for more information about using query parameters with embeddings).
207229
208- ``` python
209- q = Query(
210- " *=>[KNN 3 @embedding $vec AS vector_distance]"
211- ).return_field(" score" ).dialect(2 )
212-
213- query_text = " That is a happy person"
214-
215- res = r.ft(" vector_idx" ).search(
216- q, query_params = {
217- " vec" : model.encode(query_text).astype(np.float32).tobytes()
218- }
219- )
220-
221- print (res)
230+ ``` java
231+ String sentence = " That is a happy person" ;
232+
233+ int K = 3 ;
234+ Query q = new Query (" *=>[KNN $K @embedding $BLOB AS score]" ).
235+ returnFields(" content" , " score" ).
236+ addParam(" K" , K ).
237+ addParam(
238+ " BLOB" ,
239+ longsToFloatsByteString(
240+ sentenceTokenizer. encode(sentence). getIds()
241+ )
242+ ).
243+ dialect(2 );
244+
245+ List<Document > docs = jedis. ftSearch(" vector_idx" , q). getDocuments();
246+
247+ for (Document doc: docs) {
248+ System . out. println(doc);
249+ }
222250```
223251
224252The code is now ready to run, but note that it may take a while to complete when
225- you run it for the first time (which happens because RedisVL must download the
226- ` all-MiniLM-L6 -v2 ` model data before it can
253+ you run it for the first time (which happens because the tokenizer must download the
254+ ` all-mpnet-base -v2 ` model data before it can
227255generate the embeddings). When you run the code, it outputs the following result
228- object (slightly formatted here for clarity):
229-
230- ``` Python
231- Result{
232- 3 total,
233- docs: [
234- Document {
235- ' id' : ' doc:0' ,
236- ' payload' : None ,
237- ' vector_distance' : ' 0.114169985056' ,
238- ' content' : ' That is a very happy person'
239- },
240- Document {
241- ' id' : ' doc:1' ,
242- ' payload' : None ,
243- ' vector_distance' : ' 0.610845386982' ,
244- ' content' : ' That is a happy dog'
245- },
246- Document {
247- ' id' : ' doc:2' ,
248- ' payload' : None ,
249- ' vector_distance' : ' 1.48624813557' ,
250- ' content' : ' Today is a sunny day'
251- }
252- ]
253- }
256+ objects:
257+
258+ ```
259+ id:doc:1, score: 1.0, properties:[score=9301635, content=That is a very happy person]
260+ id:doc:2, score: 1.0, properties:[score=1411344, content=That is a happy dog]
261+ id:doc:3, score: 1.0, properties:[score=67178800, content=Today is a sunny day]
254262```
255263
256264Note that the results are ordered according to the value of the ` vector_distance `
0 commit comments