You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[Pinecone](https://www.pinecone.io/) is the default vector store used in this project. It's a cloud-native vector database that's optimized for machine learning applications. You'll need to create a Pinecone account and get an API key to use it.
61
+
62
+
Once you have your Pinecone account set up, you'll need to store your API key and index name as a ZenML secret (with name `pinecone-zenml`). You can do this by running the following command:
The `pinecone_index` value you specify will be used for all your development pipeline runs. When you promote your ZenML model to production and run your ingestion pipeline again, it will automatically create a new production index called `<YOUR_INDEX_NAME>-prod`. This separation ensures that your development and production environments remain isolated.
69
+
70
+
### Choosing Your Vector Store
71
+
72
+
While Pinecone is the default vector store, this project supports multiple vector stores. You can choose between:
73
+
74
+
1.**Pinecone** (default): A cloud-native vector database optimized for machine learning applications
75
+
2.**PostgreSQL with pgvector**: A local or cloud PostgreSQL database with vector similarity search capabilities
76
+
3.**Elasticsearch**: A distributed search engine with vector search support
77
+
78
+
To switch between vector stores, you need to create or modify a pipeline configuration file (e.g., `configs/dev/rag.yaml`) and set the `index_type` parameter for the `index_generator` step. For example:
This configuration will be used by both the basic RAG and RAG pipelines. Each vector store requires its own setup and credentials as described in their respective sections below.
88
+
89
+
### Alternative: Setting up Supabase
90
+
91
+
While Pinecone is the default vector store, you can still use Supabase's PostgreSQL database as an alternative.
59
92
60
93
[Supabase](https://supabase.com/) is a cloud provider that offers a PostgreSQL
61
94
database. It's simple to use and has a free tier that should be sufficient for
@@ -76,7 +109,7 @@ string from the Supabase dashboard.
76
109
77
110

78
111
79
-
In case Supabase is not an option for you, you can use a different database as the backend.
112
+
In case neither Pinecone nor Supabase is an option for you, you can use a different database as the backend.
80
113
81
114
### Running the RAG pipeline
82
115
@@ -89,12 +122,12 @@ python run.py rag
89
122
```
90
123
91
124
This will run the basic RAG pipeline, which scrapes the ZenML documentation and
92
-
stores the embeddings in the Supabase database.
125
+
stores the embeddings in your configured vector store (Pinecone by default).
93
126
94
127
### Querying your RAG pipeline assets
95
128
96
-
Once the pipeline has run successfully, you can query the assets in the Supabase
97
-
database using the `--query` flag as well as passing in the model you'd like to
129
+
Once the pipeline has run successfully, you can query the assets in your vector store
130
+
using the `--query` flag as well as passing in the model you'd like to
98
131
use for the LLM.
99
132
100
133
When you're ready to make the query, run the following command:
0 commit comments