Skip to content

Commit 632c4b4

Browse files
zackproserzackproser
authored
Merge pull request #364 from junefish/langchain-prefix
Create example notebook for using ID prefixes with LangChain
2 parents 8aab26b + aa1366f commit 632c4b4

File tree

1 file changed

+389
-0
lines changed

1 file changed

+389
-0
lines changed

‎learn/generation/langchain/managing_rag_docs_langchain.ipynb

Lines changed: 389 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,389 @@
1+
{
2+
"nbformat": 4,
3+
"nbformat_minor": 0,
4+
"metadata": {
5+
"colab": {
6+
"provenance": []
7+
},
8+
"kernelspec": {
9+
"name": "python3",
10+
"display_name": "Python 3"
11+
},
12+
"language_info": {
13+
"name": "python"
14+
}
15+
},
16+
"cells": [
17+
{
18+
"cell_type": "markdown",
19+
"metadata": {},
20+
"source": [
21+
"[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/experimental/merge-namespaces/merge-namespaces.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/experimental/merge-namespaces/merge-namespaces.ipynb)"
22+
]
23+
},
24+
{
25+
"cell_type": "markdown",
26+
"source": [
27+
"# Managing RAG Documents with LangChain"
28+
],
29+
"metadata": {
30+
"id": "6pAOU9GazGY2"
31+
}
32+
},
33+
{
34+
"cell_type": "markdown",
35+
"source": [
36+
"When upserting documents with LangChain's [`PineconeVectorStore`](https://api.python.langchain.com/en/latest/vectorstores/langchain_pinecone.vectorstores.PineconeVectorStore.html#langchain_pinecone.vectorstores.PineconeVectorStore) method, by default the vector IDs generated are random UUIDs. As a best practice when [managing RAG documents](https://docs.pinecone.io/guides/data/manage-rag-documents), ID prefixes should be used.\n",
37+
"\n",
38+
"This notebook gives an example of specifying ID prefixes when upserting to a Pinecone index with LangChain."
39+
],
40+
"metadata": {
41+
"id": "ZdCSDz10zbB4"
42+
}
43+
},
44+
{
45+
"cell_type": "markdown",
46+
"source": [
47+
"\n",
48+
"## Setup"
49+
],
50+
"metadata": {
51+
"id": "7AdAv8Jg4U27"
52+
}
53+
},
54+
{
55+
"cell_type": "code",
56+
"execution_count": 2,
57+
"metadata": {
58+
"colab": {
59+
"base_uri": "https://localhost:8080/"
60+
},
61+
"id": "BSUyXksS3bkk",
62+
"outputId": "559433f2-ed32-48f3-fbad-22be3a286a2f"
63+
},
64+
"outputs": [
65+
{
66+
"output_type": "stream",
67+
"name": "stdout",
68+
"text": [
69+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m974.6/974.6 kB\u001b[0m \u001b[31m8.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
70+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.2/2.2 MB\u001b[0m \u001b[31m16.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
71+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m315.6/315.6 kB\u001b[0m \u001b[31m16.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
72+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m215.9/215.9 kB\u001b[0m \u001b[31m17.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
73+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m325.5/325.5 kB\u001b[0m \u001b[31m20.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
74+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m22.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
75+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m125.2/125.2 kB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
76+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.2/49.2 kB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
77+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m145.0/145.0 kB\u001b[0m \u001b[31m15.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
78+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m75.6/75.6 kB\u001b[0m \u001b[31m9.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
79+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m77.9/77.9 kB\u001b[0m \u001b[31m9.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
80+
"\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m7.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
81+
"\u001b[?25h"
82+
]
83+
}
84+
],
85+
"source": [
86+
"%pip install --upgrade --quiet \\\n",
87+
" langchain-pinecone \\\n",
88+
" langchain-openai \\\n",
89+
" langchain \\\n",
90+
" langchain-community \\\n",
91+
" pinecone-notebooks"
92+
]
93+
},
94+
{
95+
"cell_type": "code",
96+
"source": [
97+
"import getpass\n",
98+
"openai_api_key = getpass.getpass(prompt='Enter your OpenAI API key:')"
99+
],
100+
"metadata": {
101+
"id": "tjnKw_Qf36KK",
102+
"colab": {
103+
"base_uri": "https://localhost:8080/"
104+
},
105+
"outputId": "e46b4c3a-5105-4ba9-8a95-5864965d92e1"
106+
},
107+
"execution_count": 11,
108+
"outputs": [
109+
{
110+
"name": "stdout",
111+
"output_type": "stream",
112+
"text": [
113+
"Enter your OpenAI API key:··········\n"
114+
]
115+
}
116+
]
117+
},
118+
{
119+
"cell_type": "markdown",
120+
"source": [
121+
"## Chunk the file"
122+
],
123+
"metadata": {
124+
"id": "6ulCGTfj4OkW"
125+
}
126+
},
127+
{
128+
"cell_type": "code",
129+
"source": [
130+
"filepath = \"/content/sample_data/state_of_the_union.txt\""
131+
],
132+
"metadata": {
133+
"id": "VUNVpZiO30tc"
134+
},
135+
"execution_count": 16,
136+
"outputs": []
137+
},
138+
{
139+
"cell_type": "code",
140+
"source": [
141+
"from langchain_community.document_loaders import TextLoader\n",
142+
"from langchain_openai import OpenAIEmbeddings\n",
143+
"from langchain_text_splitters import CharacterTextSplitter\n",
144+
"\n",
145+
"loader = TextLoader(filepath)\n",
146+
"documents = loader.load()\n",
147+
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
148+
"docs = text_splitter.split_documents(documents)\n",
149+
"\n",
150+
"embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)"
151+
],
152+
"metadata": {
153+
"id": "d_gZtjGO3j2d"
154+
},
155+
"execution_count": 17,
156+
"outputs": []
157+
},
158+
{
159+
"cell_type": "markdown",
160+
"source": [
161+
"## Connect to Pinecone"
162+
],
163+
"metadata": {
164+
"id": "TAdv4PJj4Y0x"
165+
}
166+
},
167+
{
168+
"cell_type": "code",
169+
"source": [
170+
"from pinecone_notebooks.colab import Authenticate\n",
171+
"\n",
172+
"Authenticate()"
173+
],
174+
"metadata": {
175+
"colab": {
176+
"base_uri": "https://localhost:8080/",
177+
"height": 247
178+
},
179+
"id": "i0dIZL6d4MOK",
180+
"outputId": "3bc6cbb4-aa99-4fa2-9e58-2d569fee7771"
181+
},
182+
"execution_count": 9,
183+
"outputs": [
184+
{
185+
"output_type": "display_data",
186+
"data": {
187+
"text/plain": [
188+
"<IPython.core.display.HTML object>"
189+
],
190+
"text/html": [
191+
"<script type=\"text/javascript\" src=\"https://connect.pinecone.io/embed.js\"></script>"
192+
]
193+
},
194+
"metadata": {}
195+
}
196+
]
197+
},
198+
{
199+
"cell_type": "code",
200+
"source": [
201+
"import os\n",
202+
"pinecone_api_key = os.environ.get(\"PINECONE_API_KEY\")\n",
203+
"\n",
204+
"import time\n",
205+
"\n",
206+
"from pinecone import Pinecone, ServerlessSpec\n",
207+
"\n",
208+
"pc = Pinecone(api_key=pinecone_api_key)"
209+
],
210+
"metadata": {
211+
"id": "P7voylv74ezR"
212+
},
213+
"execution_count": 18,
214+
"outputs": []
215+
},
216+
{
217+
"cell_type": "code",
218+
"source": [
219+
"import time\n",
220+
"\n",
221+
"index_name = \"langchain-id-test\" # change to match an existing index\n",
222+
"\n",
223+
"existing_indexes = [index_info[\"name\"] for index_info in pc.list_indexes()]\n",
224+
"\n",
225+
"if index_name not in existing_indexes:\n",
226+
" pc.create_index(\n",
227+
" name=index_name,\n",
228+
" dimension=1536,\n",
229+
" metric=\"cosine\",\n",
230+
" spec=ServerlessSpec(cloud=\"aws\", region=\"us-east-1\"),\n",
231+
" )\n",
232+
" while not pc.describe_index(index_name).status[\"ready\"]:\n",
233+
" time.sleep(1)\n",
234+
"\n",
235+
"index = pc.Index(index_name)"
236+
],
237+
"metadata": {
238+
"id": "XWddh1Lq4h5Z"
239+
},
240+
"execution_count": 19,
241+
"outputs": []
242+
},
243+
{
244+
"cell_type": "markdown",
245+
"source": [
246+
"## Upsert data"
247+
],
248+
"metadata": {
249+
"id": "x4KaMq5P4tjU"
250+
}
251+
},
252+
{
253+
"cell_type": "markdown",
254+
"source": [
255+
"### Generate IDs with the specified prefix"
256+
],
257+
"metadata": {
258+
"id": "1lKRfEYJ0GKj"
259+
}
260+
},
261+
{
262+
"cell_type": "code",
263+
"source": [
264+
"prefix = \"sotu\" # change to reflect your document\n",
265+
"\n",
266+
"ids = []\n",
267+
"for i in range(len(docs)):\n",
268+
" ids.append(prefix+\"#\"+str(i))"
269+
],
270+
"metadata": {
271+
"id": "nrONNB070LRj"
272+
},
273+
"execution_count": null,
274+
"outputs": []
275+
},
276+
{
277+
"cell_type": "markdown",
278+
"source": [
279+
"### Upsert to Pinecone"
280+
],
281+
"metadata": {
282+
"id": "yx5yPTIj0M_H"
283+
}
284+
},
285+
{
286+
"cell_type": "code",
287+
"source": [
288+
"from langchain_pinecone import PineconeVectorStore\n",
289+
"\n",
290+
"docsearch = PineconeVectorStore.from_documents(docs, embeddings, index_name=index_name)\n",
291+
"\n",
292+
"vectorstore = PineconeVectorStore(index_name=index_name, embedding=embeddings)\n",
293+
"vectorstore.add_documents(docs, ids=ids) # prints IDs of upserted vectors"
294+
],
295+
"metadata": {
296+
"id": "2OhSEwVY4ut6",
297+
"colab": {
298+
"base_uri": "https://localhost:8080/"
299+
},
300+
"outputId": "23f077c4-a80d-4bf0-9555-027038e5d152"
301+
},
302+
"execution_count": 26,
303+
"outputs": [
304+
{
305+
"output_type": "execute_result",
306+
"data": {
307+
"text/plain": [
308+
"['sotu#0',\n",
309+
" 'sotu#1',\n",
310+
" 'sotu#2',\n",
311+
" 'sotu#3',\n",
312+
" 'sotu#4',\n",
313+
" 'sotu#5',\n",
314+
" 'sotu#6',\n",
315+
" 'sotu#7',\n",
316+
" 'sotu#8',\n",
317+
" 'sotu#9',\n",
318+
" 'sotu#10',\n",
319+
" 'sotu#11',\n",
320+
" 'sotu#12',\n",
321+
" 'sotu#13',\n",
322+
" 'sotu#14',\n",
323+
" 'sotu#15',\n",
324+
" 'sotu#16',\n",
325+
" 'sotu#17',\n",
326+
" 'sotu#18',\n",
327+
" 'sotu#19',\n",
328+
" 'sotu#20',\n",
329+
" 'sotu#21',\n",
330+
" 'sotu#22',\n",
331+
" 'sotu#23',\n",
332+
" 'sotu#24',\n",
333+
" 'sotu#25',\n",
334+
" 'sotu#26',\n",
335+
" 'sotu#27',\n",
336+
" 'sotu#28',\n",
337+
" 'sotu#29',\n",
338+
" 'sotu#30',\n",
339+
" 'sotu#31',\n",
340+
" 'sotu#32',\n",
341+
" 'sotu#33',\n",
342+
" 'sotu#34',\n",
343+
" 'sotu#35',\n",
344+
" 'sotu#36',\n",
345+
" 'sotu#37',\n",
346+
" 'sotu#38',\n",
347+
" 'sotu#39',\n",
348+
" 'sotu#40',\n",
349+
" 'sotu#41']"
350+
]
351+
},
352+
"metadata": {},
353+
"execution_count": 26
354+
}
355+
]
356+
},
357+
{
358+
"cell_type": "code",
359+
"source": [
360+
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
361+
"docs = docsearch.similarity_search(query)\n",
362+
"print(docs[0].page_content)"
363+
],
364+
"metadata": {
365+
"colab": {
366+
"base_uri": "https://localhost:8080/"
367+
},
368+
"id": "l3KkioA24z6-",
369+
"outputId": "627683e5-e8c9-4b35-b799-50967ce7f61e"
370+
},
371+
"execution_count": 24,
372+
"outputs": [
373+
{
374+
"output_type": "stream",
375+
"name": "stdout",
376+
"text": [
377+
"Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \n",
378+
"\n",
379+
"Tonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n",
380+
"\n",
381+
"One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n",
382+
"\n",
383+
"And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.\n"
384+
]
385+
}
386+
]
387+
}
388+
]
389+
}

0 commit comments

Comments
 (0)