Skip to content

Commit b332cd8

Browse files
andy-stark-redisandy-stark-redis
committed
DOC-5533 added Go vector set page and code
1 parent 4ea1a9c commit b332cd8

File tree

2 files changed

+459
-0
lines changed

2 files changed

+459
-0
lines changed

‎content/develop/clients/go/vecsets.md

Lines changed: 174 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,174 @@
1+
---
2+
categories:
3+
- docs
4+
- develop
5+
- stack
6+
- oss
7+
- rs
8+
- rc
9+
- oss
10+
- kubernetes
11+
- clients
12+
description: Index and query embeddings with Redis vector sets
13+
linkTitle: Vector set embeddings
14+
title: Vector set embeddings
15+
weight: 40
16+
bannerText: Vector set is a new data type that is currently in preview and may be subject to change.
17+
bannerChildren: true
18+
---
19+
20+
A Redis [vector set]({{< relref "/develop/data-types/vector-sets" >}}) lets
21+
you store a set of unique keys, each with its own associated vector.
22+
You can then retrieve keys from the set according to the similarity between
23+
their stored vectors and a query vector that you specify.
24+
25+
You can use vector sets to store any type of numeric vector but they are
26+
particularly optimized to work with text embedding vectors (see
27+
[Redis for AI]({{< relref "/develop/ai" >}}) to learn more about text
28+
embeddings). The example below shows how to use the
29+
[`Hugot`](https://github.com/knights-analytics/hugot)
30+
library to generate vector embeddings and then
31+
store and retrieve them using a vector set with `go-redis`.
32+
33+
## Initialize
34+
35+
Start by [installing]({{< relref "/develop/clients/go#install" >}}) `go-redis` if you haven't already done so. Note that you need `go-redis`
36+
[v9.10.0](https://github.com/redis/go-redis/releases/tag/v9.10.0)
37+
or later to use vector sets.
38+
39+
Also, install `hugot`:
40+
41+
```bash
42+
go get github.com/knights-analytics/hugot
43+
```
44+
45+
In a new Go file, add the required imports:
46+
47+
{{< clients-example set="home_vecsets" step="import" lang_filter="Go" >}}
48+
{{< /clients-example >}}
49+
50+
These include the `Hugot` library to generate an embedding from a section of text.
51+
This example uses an instance of the `SentenceTransformer` model
52+
[`all-MiniLM-L6-v2`](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2)
53+
for the embeddings. This model generates vectors with 384 dimensions, regardless
54+
of the length of the input text, but note that the input is truncated to 256
55+
tokens (see
56+
[Word piece tokenization](https://huggingface.co/learn/nlp-course/en/chapter6/6)
57+
at the [Hugging Face](https://huggingface.co/) docs to learn more about the way tokens
58+
are related to the original text).
59+
60+
{{< clients-example set="home_vecsets" step="model" lang_filter="Go" >}}
61+
{{< /clients-example >}}
62+
63+
## Create the data
64+
65+
The example data is contained in a map with some brief
66+
descriptions of famous people:
67+
68+
{{< clients-example set="home_vecsets" step="data" lang_filter="Go" >}}
69+
{{< /clients-example >}}
70+
71+
## Add the data to a vector set
72+
73+
The next step is to connect to Redis and add the data to a new vector set.
74+
75+
The code below iterates through the `peopleData` map and adds corresponding
76+
elements to a vector set called `famousPeople`.
77+
78+
Use the
79+
`RunPipeline()` method of `embeddingPipeline` to generate the
80+
embedding as an array of `float32` values, then use a loop like the one
81+
shown below to convert the `float32` array to a `float64` array.
82+
You can then pass this array to the
83+
[`VAdd()`]({{< relref "/commands/vadd" >}}) command to set the embedding.
84+
85+
The call to `VAdd()` also adds the `born` and `died` values from the
86+
original map as attribute data. You can access this during a query
87+
or by using the [`VGetAttr()`]({{< relref "/commands/vgetattr" >}}) method.
88+
89+
{{< clients-example set="home_vecsets" step="add_data" lang_filter="Go" >}}
90+
{{< /clients-example >}}
91+
92+
## Query the vector set
93+
94+
You can now query the data in the set. The basic approach is to use the
95+
`RunPipeline()` method to generate another embedding vector for the query text.
96+
(This is the same method used to add the elements to the set.) Then, pass
97+
the query vector to [`VSim()`]({{< relref "/commands/vsim" >}}) to return elements
98+
of the set, ranked in order of similarity to the query.
99+
100+
Start with a simple query for "actors":
101+
102+
{{< clients-example set="home_vecsets" step="basic_query" lang_filter="Go" >}}
103+
{{< /clients-example >}}
104+
105+
This returns the following list of elements (formatted slightly for clarity):
106+
107+
```
108+
'actors': Masako Natsume, Chaim Topol, Linus Pauling, Marie Fredriksson,
109+
Maryam Mirzakhani, Marie Curie, Freddie Mercury, Paul Erdos
110+
```
111+
112+
The first two people in the list are the two actors, as expected, but none of the
113+
people from Linus Pauling onward was especially well-known for acting (and there certainly
114+
isn't any information about that in the short description text).
115+
As it stands, the search attempts to rank all the elements in the set, based
116+
on the information contained in the embedding model.
117+
You can use the `Count` parameter of `VSimWithArgs()` to limit the list of elements
118+
to just the most relevant few items:
119+
120+
{{< clients-example set="home_vecsets" step="limited_query" lang_filter="Go" >}}
121+
{{< /clients-example >}}
122+
123+
The reason for using text embeddings rather than simple text search
124+
is that the embeddings represent semantic information. This allows a query
125+
to find elements with a similar meaning even if the text is
126+
different. For example, the word "entertainer" doesn't appear in any of the
127+
descriptions but if you use it as a query, the actors and musicians are ranked
128+
highest in the results list:
129+
130+
{{< clients-example set="home_vecsets" step="entertainer_query" lang_filter="Go" >}}
131+
{{< /clients-example >}}
132+
133+
Similarly, if you use "science" as a query, you get the following results:
134+
135+
```
136+
'science': Marie Curie, Linus Pauling, Maryam Mirzakhani, Paul Erdos,
137+
Marie Fredriksson, Freddie Mercury, Masako Natsume, Chaim Topol
138+
```
139+
140+
The scientists are ranked highest but they are then followed by the
141+
mathematicians. This seems reasonable given the connection between mathematics
142+
and science.
143+
144+
You can also use
145+
[filter expressions]({{< relref "/develop/data-types/vector-sets/filtered-search" >}})
146+
with `VSimWithArgs()` to restrict the search further. For example,
147+
repeat the "science" query, but this time limit the results to people
148+
who died before the year 2000:
149+
150+
{{< clients-example set="home_vecsets" step="filtered_query" lang_filter="Go" >}}
151+
{{< /clients-example >}}
152+
153+
Note that the boolean filter expression is applied to items in the list
154+
before the vector distance calculation is performed. Items that don't
155+
pass the filter test are removed from the results completely, rather
156+
than just reduced in rank. This can help to improve the performance of the
157+
search because there is no need to calculate the vector distance for
158+
elements that have already been filtered out of the search.
159+
160+
## More information
161+
162+
See the [vector sets]({{< relref "/develop/data-types/vector-sets" >}})
163+
docs for more information and code examples. See the
164+
[Redis for AI]({{< relref "/develop/ai" >}}) section for more details
165+
about text embeddings and other AI techniques you can use with Redis.
166+
167+
You may also be interested in
168+
[vector search]({{< relref "/develop/clients/go/vecsearch" >}}).
169+
This is a feature of the
170+
[Redis query engine]({{< relref "/develop/ai/search-and-query" >}})
171+
that lets you retrieve
172+
[JSON]({{< relref "/develop/data-types/json" >}}) and
173+
[hash]({{< relref "/develop/data-types/hashes" >}}) documents based on
174+
vector data stored in their fields.

0 commit comments

Comments
 (0)