Skip to content

Commit a597a43

Browse files
committed
update readme with pinecone info
1 parent 3edd521 commit a597a43

File tree

1 file changed

+39
-6
lines changed

1 file changed

+39
-6
lines changed

llm-complete-guide/README.md

Lines changed: 39 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ using ZenML, enabling you to build powerful, scalable, and maintainable
1818
LLM-powered applications.
1919

2020
This project contains all the pipeline and step code necessary to follow along
21-
with the guide. You'll need a PostgreSQL database to store the embeddings; full
21+
with the guide. You'll need a vector store to store the embeddings; full
2222
instructions are provided below for how to set that up.
2323

2424
## 📽️ Watch the webinars
@@ -55,7 +55,40 @@ zenml secret create llm-complete --openai_api_key=<your-openai-api-key>
5555
export ZENML_PROJECT_SECRET_NAME=llm-complete
5656
```
5757

58-
### Setting up Supabase
58+
### Setting up Pinecone
59+
60+
[Pinecone](https://www.pinecone.io/) is the default vector store used in this project. It's a cloud-native vector database that's optimized for machine learning applications. You'll need to create a Pinecone account and get an API key to use it.
61+
62+
Once you have your Pinecone account set up, you'll need to store your API key and index name as a ZenML secret (with name `pinecone-zenml`). You can do this by running the following command:
63+
64+
```shell
65+
zenml secret create pinecone-zenml --pinecone_api_key=<YOUR_PINECONE_API_KEY> --pinecone_env=<YOUR_PINECONE_ENV> --pinecone_index=<YOUR_INDEX_NAME>
66+
```
67+
68+
The `pinecone_index` value you specify will be used for all your development pipeline runs. When you promote your ZenML model to production and run your ingestion pipeline again, it will automatically create a new production index called `<YOUR_INDEX_NAME>-prod`. This separation ensures that your development and production environments remain isolated.
69+
70+
### Choosing Your Vector Store
71+
72+
While Pinecone is the default vector store, this project supports multiple vector stores. You can choose between:
73+
74+
1. **Pinecone** (default): A cloud-native vector database optimized for machine learning applications
75+
2. **PostgreSQL with pgvector**: A local or cloud PostgreSQL database with vector similarity search capabilities
76+
3. **Elasticsearch**: A distributed search engine with vector search support
77+
78+
To switch between vector stores, you need to create or modify a pipeline configuration file (e.g., `configs/dev/rag.yaml`) and set the `index_type` parameter for the `index_generator` step. For example:
79+
80+
```yaml
81+
steps:
82+
index_generator:
83+
parameters:
84+
index_type: pinecone # Options: pinecone, postgres, elasticsearch
85+
```
86+
87+
This configuration will be used by both the basic RAG and RAG pipelines. Each vector store requires its own setup and credentials as described in their respective sections below.
88+
89+
### Alternative: Setting up Supabase
90+
91+
While Pinecone is the default vector store, you can still use Supabase's PostgreSQL database as an alternative.
5992
6093
[Supabase](https://supabase.com/) is a cloud provider that offers a PostgreSQL
6194
database. It's simple to use and has a free tier that should be sufficient for
@@ -76,7 +109,7 @@ string from the Supabase dashboard.
76109
77110
![](.assets/supabase-connection-string.png)
78111
79-
In case Supabase is not an option for you, you can use a different database as the backend.
112+
In case neither Pinecone nor Supabase is an option for you, you can use a different database as the backend.
80113
81114
### Running the RAG pipeline
82115
@@ -89,12 +122,12 @@ python run.py rag
89122
```
90123

91124
This will run the basic RAG pipeline, which scrapes the ZenML documentation and
92-
stores the embeddings in the Supabase database.
125+
stores the embeddings in your configured vector store (Pinecone by default).
93126

94127
### Querying your RAG pipeline assets
95128

96-
Once the pipeline has run successfully, you can query the assets in the Supabase
97-
database using the `--query` flag as well as passing in the model you'd like to
129+
Once the pipeline has run successfully, you can query the assets in your vector store
130+
using the `--query` flag as well as passing in the model you'd like to
98131
use for the LLM.
99132

100133
When you're ready to make the query, run the following command:

0 commit comments

Comments
 (0)