You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -25,6 +26,10 @@ Examples for application of these technique includes for instance customer servi
25
26
26
27
In the context of Retrieval-Augmented Generation (RAG), knowledge seeding involves incorporating external information from pre-existing sources into the generative process, while querying refers to the mechanism of retrieving relevant knowledge from these sources to inform the generation of coherent and contextually accurate text. Both are shown below.
27
28
29
+
:::note[Looking for a managed option?]
30
+
[AutoRAG](/autorag) offers a fully managed way to build RAG pipelines on Cloudflare, handling ingestion, indexing, and querying out of the box. [Get started](/autorag/get-started/).
Build full-stack AI applications with Vectorize, Cloudflare's powerful vector database.
18
26
19
-
20
27
</Description>
21
28
22
29
Vectorize is a globally distributed vector database that enables you to build full-stack, AI-powered applications with [Cloudflare Workers](/workers/). Vectorize makes querying embeddings — representations of values or objects like text, images, audio that are designed to be consumed by machine learning models and semantic search algorithms — faster, easier and more affordable.
23
30
24
31
<Renderfile="vectorize-ga" />
25
32
26
-
27
33
For example, by storing the embeddings (vectors) generated by a machine learning model, including those built-in to [Workers AI](/workers-ai/) or by bringing your own from platforms like [OpenAI](#), you can build applications with powerful search, similarity, recommendation, classification and/or anomaly detection capabilities based on your own data.
28
34
29
35
The vectors returned can reference images stored in Cloudflare R2, documents in KV, and/or user profiles stored in D1 — enabling you to go from vector search result to concrete object all within the Workers platform, and without standing up additional infrastructure.
30
36
31
-
***
37
+
---
32
38
33
39
## Features
34
40
35
41
<Featureheader="Vector database"href="/vectorize/get-started/intro/"cta="Create your Vector database">
36
42
37
43
Learn how to create your first Vectorize database, upload vector embeddings, and query those embeddings from [Cloudflare Workers](/workers/).
38
44
39
-
40
45
</Feature>
41
46
42
47
<Featureheader="Vector embeddings using Workers AI"href="/vectorize/get-started/embeddings/"cta="Create vector embeddings using Workers AI">
43
48
44
49
Learn how to use Vectorize to generate vector embeddings using Workers AI.
45
50
51
+
</Feature>
52
+
53
+
<Featureheader="Search using Vectorize and AutoRAG"href="/autorag"cta="Build a RAG with Vectorize">
54
+
55
+
Learn how to automatically index your data and store it in Vectorize, then query it to generate context-aware responses using AutoRAG.
Connect with the Workers community on Discord to ask questions, join the `#vectorize` channel to show what you are building, and discuss the platform with other developers.
103
+
<LinkTitleCard
104
+
title="Developer Discord"
105
+
href="https://discord.cloudflare.com"
106
+
icon="discord"
107
+
>
108
+
Connect with the Workers community on Discord to ask questions, join the
109
+
`#vectorize` channel to show what you are building, and discuss the platform
Copy file name to clipboardExpand all lines: src/content/docs/vectorize/reference/what-is-a-vector-database.mdx
+12-12Lines changed: 12 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -3,7 +3,6 @@ title: Vector databases
3
3
pcx_content_type: concept
4
4
sidebar:
5
5
order: 2
6
-
7
6
---
8
7
9
8
Vector databases are a key part of building scalable AI-powered applications. Vector databases provide long term memory, on top of an existing machine learning model.
@@ -14,10 +13,10 @@ Without a vector database, you would need to train your model (or models) or re-
14
13
15
14
A vector database determines what other data (represented as vectors) is near your input query. This allows you to build different use-cases on top of a vector database, including:
16
15
17
-
* Semantic search, used to return results similar to the input of the query.
18
-
* Classification, used to return the grouping (or groupings) closest to the input query.
19
-
* Recommendation engines, used to return content similar to the input based on different criteria (for example previous product sales, or user history).
20
-
* Anomaly detection, used to identify whether specific data points are similar to existing data, or different.
16
+
- Semantic search, used to return results similar to the input of the query.
17
+
- Classification, used to return the grouping (or groupings) closest to the input query.
18
+
- Recommendation engines, used to return content similar to the input based on different criteria (for example previous product sales, or user history).
19
+
- Anomaly detection, used to identify whether specific data points are similar to existing data, or different.
21
20
22
21
Vector databases can also power [Retrieval Augmented Generation](https://arxiv.org/abs/2005.11401) (RAG) tasks, which allow you to bring additional context to LLMs (Large Language Models) by using the context from a vector search to augment the user prompt.
23
22
@@ -44,16 +43,17 @@ Instead of passing the prompt directly to the LLM, in the RAG approach you:
44
43
1. Generate vector embeddings from an existing dataset or corpus (for example, the dataset you want to use to add additional context to the LLMs response). An existing dataset or corpus could be a product documentation, research data, technical specifications, or your product catalog and descriptions.
45
44
2. Store the output embeddings in a Vectorize database index.
46
45
47
-
When a user initiates a prompt, instead of passing it (without additional context) to the LLM, you *augment* it with additional context:
46
+
When a user initiates a prompt, instead of passing it (without additional context) to the LLM, you _augment_ it with additional context:
48
47
49
48
1. The user prompt is passed into the same ML model used for your dataset, returning a vector embedding representation of the query.
50
49
2. This embedding is used as the query (semantic search) against the vector database, which returns similar vectors.
51
50
3. These vectors are used to look up the content they relate to (if not embedded directly alongside the vectors as metadata).
52
51
4. This content is provided as context alongside the original user prompt, providing additional context to the LLM and allowing it to return an answer that is likely to be far more contextual than the standalone prompt.
53
52
54
-
Refer to the [RAG using Workers AI tutorial](/workers-ai/tutorials/build-a-retrieval-augmented-generation-ai/) to learn how to combine Workers AI and Vectorize for generative AI use-cases.
53
+
Create a RAG today with [AutoRAG](/autorag) to deploy a fully managed RAG pipeline in just a few clicks. It automatically sets up Vectorize, handles continuous indexing, and serves responses through a single API.
55
54
56
-
<sup>1</sup> You can learn more about the theory behind RAG by reading the [RAG paper](https://arxiv.org/abs/2005.11401).
55
+
<sup>1</sup> You can learn more about the theory behind RAG by reading the [RAG
56
+
paper](https://arxiv.org/abs/2005.11401).
57
57
58
58
## Terminology
59
59
@@ -85,9 +85,9 @@ Refer to the [dimensions](/vectorize/best-practices/create-indexes/#dimensions)
85
85
86
86
The distance metric is an index used for vector search. It defines how it determines how close your query vector is to other vectors within the index.
87
87
88
-
* Distance metrics determine how the vector search engine assesses similarity between vectors.
89
-
* Cosine, Euclidean (L2), and Dot Product are the most commonly used distance metrics in vector search.
90
-
* The machine learning model and type of embedding you use will determine which distance metric is best suited for your use-case.
91
-
* Different metrics determine different scoring characteristics. For example, the `cosine` distance metric is well suited to text, sentence similarity and/or document search use-cases. `euclidean` can be better suited for image or speech recognition use-cases.
88
+
- Distance metrics determine how the vector search engine assesses similarity between vectors.
89
+
- Cosine, Euclidean (L2), and Dot Product are the most commonly used distance metrics in vector search.
90
+
- The machine learning model and type of embedding you use will determine which distance metric is best suited for your use-case.
91
+
- Different metrics determine different scoring characteristics. For example, the `cosine` distance metric is well suited to text, sentence similarity and/or document search use-cases. `euclidean` can be better suited for image or speech recognition use-cases.
92
92
93
93
Refer to the [distance metrics](/vectorize/best-practices/create-indexes/#distance-metrics) documentation to learn how to configure a distance metric when creating a Vectorize index.
0 commit comments