![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To start, we need a RAG app to evaluate. Let's create one using LangChain and connect it with Redis as the vector DB.\n",
- "\n",
- "## Init redis, data prep, and populating the vector DB"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.2\u001b[0m\n",
- "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
- "Note: you may need to restart the kernel to use updated packages.\n"
- ]
- }
- ],
- "source": [
- "%pip install -q redis \"unstructured[pdf]\" sentence-transformers langchain langchain-redis langchain-huggingface langchain-openai ragas datasets"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "#### Running Redis in Colab\n",
- "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "# NBVAL_SKIP\n",
- "%%sh\n",
- "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
- "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
- "sudo apt-get update > /dev/null 2>&1\n",
- "sudo apt-get install redis-stack-server > /dev/null 2>&1\n",
- "redis-stack-server --daemonize yes"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "#### For Alternative Environments\n",
- "There are many ways to get the necessary redis-stack instance running\n",
- "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
- "own version of Redis Enterprise running, that works too!\n",
- "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
- "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import warnings\n",
- "warnings.filterwarnings('ignore')\n",
- "\n",
- "# Replace values below with your own if using Redis Cloud instance\n",
- "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
- "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n",
- "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n",
- "\n",
- "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n",
- "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\""
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
- "from langchain_community.document_loaders import PyPDFLoader\n",
- "\n",
- "CHUNK_SIZE = 2500\n",
- "CHUNK_OVERLAP = 0\n",
- "\n",
- "# pdf to load\n",
- "path = 'resources/nke-10k-2023.pdf'\n",
- "assert os.path.exists(path), f\"File not found: {path}\"\n",
- "\n",
- "# load and split\n",
- "loader = PyPDFLoader(path)\n",
- "pages = loader.load()\n",
- "text_splitter = RecursiveCharacterTextSplitter(chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
- "chunks = text_splitter.split_documents(pages)\n",
- "\n",
- "print(\"Done preprocessing. Created\", len(chunks), \"chunks of the original pdf\", path)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 95,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"Table of ContentsUNITED STATESSECURITIES AND EXCHANGE COMMISSIONWashington, D.C. 20549FORM 10-K(Mark One)☑ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934FOR THE FISCAL YEAR ENDED MAY 31, 2023OR☐ TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934FOR THE TRANSITION PERIOD FROM TO .Commission File No. 1-10635\\n\\nAs of November 30, 2022, the aggregate market values of the Registrant's Common Stock held by non-affiliates were:Class A$7,831,564,572 Class B136,467,702,472 $144,299,267,044\\n\\nNIKE, Inc.(Exact name of Registrant as specified in its charter)Oregon93-0584541(State or other jurisdiction of incorporation)(IRS Employer Identification No.)One Bowerman Drive, Beaverton, Oregon 97005-6453(Address of principal executive offices and zip code)(503) 671-6453(Registrant's telephone number, including area code)SECURITIES REGISTERED PURSUANT TO SECTION 12(B) OF THE ACT:Class B Common StockNKENew York Stock Exchange(Title of each class)(Trading symbol)(Name of each exchange on which registered)SECURITIES REGISTERED PURSUANT TO SECTION 12(G) OF THE ACT:NONE\")"
- ]
- },
- "execution_count": 95,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "chunks[0]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 96,
- "metadata": {},
- "outputs": [],
- "source": [
- "from langchain_huggingface import HuggingFaceEmbeddings\n",
- "\n",
- "embeddings = HuggingFaceEmbeddings(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 97,
- "metadata": {},
- "outputs": [],
- "source": [
- "from langchain_redis import RedisVectorStore\n",
- "\n",
- "# set the index name for this example\n",
- "index_name = \"ragas_ex\"\n",
- "\n",
- "# construct the vector store class from texts and metadata\n",
- "rds = RedisVectorStore.from_documents(\n",
- " chunks,\n",
- " embeddings,\n",
- " index_name=index_name,\n",
- " redis_url=REDIS_URL,\n",
- " metadata_schema=[\n",
- " {\n",
- " \"name\": \"source\",\n",
- " \"type\": \"text\"\n",
- " },\n",
- " ]\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Test the vector store"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 98,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "'As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, our operating segments are evidence of the structure of the Company\\'s internal organization. The NIKE Brand segments are defined by geographic regions for operations participating in NIKE Brand sales activity.\\n\\nThe breakdown of Revenues is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023 FISCAL 2022\\n\\n% CHANGE\\n\\n% CHANGE EXCLUDING CURRENCY (1) CHANGES FISCAL 2021\\n\\n% CHANGE\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n21,608 $ 13,418 7,248\\n\\n18,353 12,479 7,547\\n\\n18 % 8 % -4 %\\n\\n18 % $ 21 % 4 %\\n\\n17,179 11,456 8,290\\n\\n7 % 9 % -9 %\\n\\nAsia Pacific & Latin America Global Brand Divisions\\n\\n(3)\\n\\n(2)\\n\\n6,431 58\\n\\n5,955 102\\n\\n8 % -43 %\\n\\n17 % -43 %\\n\\n5,343 25\\n\\n11 % 308 %\\n\\nTOTAL NIKE BRAND Converse\\n\\n$\\n\\n48,763 $ 2,427\\n\\n44,436 2,346\\n\\n10 % 3 %\\n\\n16 % $ 8 %\\n\\n42,293 2,205\\n\\n5 % 6 %\\n\\n(4)\\n\\nCorporate TOTAL NIKE, INC. REVENUES\\n\\n$\\n\\n27\\n\\n51,217 $\\n\\n(72) 46,710\\n\\n— 10 %\\n\\n— 16 % $\\n\\n40 44,538\\n\\n— 5 %\\n\\n(1) The percent change excluding currency changes represents a non-GAAP financial measure. For further information, see \"Use of Non-GAAP Financial Measures\".\\n\\n(2) For additional information on the transition of our NIKE Brand businesses within our CASA territory to a third-party distributor, see Note 18 — Acquisitions and Divestitures of the Notes to Consolidated\\n\\nFinancial Statements contained in Item 8 of this Annual Report.\\n\\n(3) Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment.\\n\\n(4) Corporate revenues primarily consist of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse, but\\n\\nmanaged through our central foreign exchange risk management program.\\n\\nThe primary financial measure used by the Company to evaluate performance is Earnings Before Interest and Taxes (\"EBIT\"). As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, certain corporate costs are not included in EBIT.\\n\\nThe breakdown of EBIT is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023\\n\\nFISCAL 2022\\n\\n% CHANGE\\n\\nFISCAL 2021\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n5,454 3,531 2,283\\n\\n$\\n\\n5,114 3,293 2,365\\n\\n7 % $ 7 % -3 %\\n\\n5,089 2,435 3,243\\n\\nAsia Pacific & Latin America Global Brand Divisions (1)'"
- ]
- },
- "execution_count": 98,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "rds.similarity_search(\"What was nike's revenue last year?\")[0].page_content"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup RAG\n",
- "\n",
- "Now that the vector db is populated let's initialize our RAG app."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 99,
- "metadata": {},
- "outputs": [],
- "source": [
- "import getpass\n",
- "from langchain_openai import ChatOpenAI\n",
- "\n",
- "if \"OPENAI_API_KEY\" not in os.environ:\n",
- " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OPENAI_API_KEY\")\n",
- "\n",
- "llm = ChatOpenAI(\n",
- " openai_api_key=os.environ[\"OPENAI_API_KEY\"],\n",
- " model=\"gpt-3.5-turbo-16k\",\n",
- " max_tokens=None\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 108,
- "metadata": {},
- "outputs": [],
- "source": [
- "from langchain_core.prompts import ChatPromptTemplate\n",
- "\n",
- "system_prompt = \"\"\"\n",
- " Use the following pieces of context from financial 10k filings data to answer the user question at the end. \n",
- " If you don't know the answer, say that you don't know, don't try to make up an answer.\n",
- "\n",
- " Context:\n",
- " ---------\n",
- " {context}\n",
- "\"\"\"\n",
- "\n",
- "def format_docs(docs):\n",
- " return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
- "\n",
- "prompt = ChatPromptTemplate.from_messages(\n",
- " [\n",
- " (\"system\", system_prompt),\n",
- " (\"human\", \"{input}\")\n",
- " ]\n",
- ")\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Test it out"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 109,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "{'input': \"What was nike's revenue last year?\",\n",
- " 'context': [Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content='As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, our operating segments are evidence of the structure of the Company\\'s internal organization. The NIKE Brand segments are defined by geographic regions for operations participating in NIKE Brand sales activity.\\n\\nThe breakdown of Revenues is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023 FISCAL 2022\\n\\n% CHANGE\\n\\n% CHANGE EXCLUDING CURRENCY (1) CHANGES FISCAL 2021\\n\\n% CHANGE\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n21,608 $ 13,418 7,248\\n\\n18,353 12,479 7,547\\n\\n18 % 8 % -4 %\\n\\n18 % $ 21 % 4 %\\n\\n17,179 11,456 8,290\\n\\n7 % 9 % -9 %\\n\\nAsia Pacific & Latin America Global Brand Divisions\\n\\n(3)\\n\\n(2)\\n\\n6,431 58\\n\\n5,955 102\\n\\n8 % -43 %\\n\\n17 % -43 %\\n\\n5,343 25\\n\\n11 % 308 %\\n\\nTOTAL NIKE BRAND Converse\\n\\n$\\n\\n48,763 $ 2,427\\n\\n44,436 2,346\\n\\n10 % 3 %\\n\\n16 % $ 8 %\\n\\n42,293 2,205\\n\\n5 % 6 %\\n\\n(4)\\n\\nCorporate TOTAL NIKE, INC. REVENUES\\n\\n$\\n\\n27\\n\\n51,217 $\\n\\n(72) 46,710\\n\\n— 10 %\\n\\n— 16 % $\\n\\n40 44,538\\n\\n— 5 %\\n\\n(1) The percent change excluding currency changes represents a non-GAAP financial measure. For further information, see \"Use of Non-GAAP Financial Measures\".\\n\\n(2) For additional information on the transition of our NIKE Brand businesses within our CASA territory to a third-party distributor, see Note 18 — Acquisitions and Divestitures of the Notes to Consolidated\\n\\nFinancial Statements contained in Item 8 of this Annual Report.\\n\\n(3) Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment.\\n\\n(4) Corporate revenues primarily consist of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse, but\\n\\nmanaged through our central foreign exchange risk management program.\\n\\nThe primary financial measure used by the Company to evaluate performance is Earnings Before Interest and Taxes (\"EBIT\"). As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, certain corporate costs are not included in EBIT.\\n\\nThe breakdown of EBIT is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023\\n\\nFISCAL 2022\\n\\n% CHANGE\\n\\nFISCAL 2021\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n5,454 3,531 2,283\\n\\n$\\n\\n5,114 3,293 2,365\\n\\n7 % $ 7 % -3 %\\n\\n5,089 2,435 3,243\\n\\nAsia Pacific & Latin America Global Brand Divisions (1)'),\n",
- " Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"NIKE, INC. CONSOLIDATED STATEMENTS OF INCOME\\n\\n(In millions, except per share data)\\n\\nRevenues Cost of sales\\n\\nGross profit\\n\\nDemand creation expense Operating overhead expense\\n\\nTotal selling and administrative expense\\n\\nInterest expense (income), net\\n\\nOther (income) expense, net Income before income taxes\\n\\nIncome tax expense NET INCOME\\n\\nEarnings per common share:\\n\\nBasic Diluted\\n\\nWeighted average common shares outstanding:\\n\\nBasic Diluted\\n\\nThe accompanying Notes to the Consolidated Financial Statements are an integral part of this statement.\\n\\n$\\n\\n$\\n\\n$ $\\n\\nYEAR ENDED MAY 31,\\n\\n2023\\n\\n2022\\n\\n2021\\n\\n51,217 $ 28,925\\n\\n46,710 $ 25,231\\n\\n44,538 24,576\\n\\n22,292 4,060 12,317\\n\\n21,479 3,850 10,954\\n\\n19,962 3,114 9,911\\n\\n16,377 (6)\\n\\n14,804 205\\n\\n13,025 262\\n\\n(280) 6,201\\n\\n(181) 6,651\\n\\n14 6,661\\n\\n1,131 5,070 $\\n\\n605 6,046 $\\n\\n934 5,727\\n\\n3.27 $ 3.23 $\\n\\n3.83 $ 3.75 $\\n\\n3.64 3.56\\n\\n1,551.6 1,569.8\\n\\n1,578.8 1,610.8\\n\\n1,573.0 1,609.4\\n\\n2023 FORM 10-K 55\\n\\nTable of Contents\\n\\nNIKE, INC. CONSOLIDATED STATEMENTS OF COMPREHENSIVE INCOME\\n\\nYEAR ENDED MAY 31,\\n\\n(Dollars in millions)\\n\\n2023\\n\\n2022\\n\\nNet income Other comprehensive income (loss), net of tax:\\n\\n$\\n\\n5,070 $\\n\\n6,046 $\\n\\nChange in net foreign currency translation adjustment\\n\\n267\\n\\n(522)\\n\\nChange in net gains (losses) on cash flow hedges Change in net gains (losses) on other\\n\\n(348) (6)\\n\\n1,214 6\\n\\nTotal other comprehensive income (loss), net of tax TOTAL COMPREHENSIVE INCOME\\n\\n$\\n\\n(87) 4,983 $\\n\\n698 6,744 $\\n\\nThe accompanying Notes to the Consolidated Financial Statements are an integral part of this statement.\\n\\n2023 FORM 10-K 56\\n\\n2021\\n\\n5,727\\n\\n496\\n\\n(825) 5\\n\\n(324) 5,403\\n\\nTable of Contents\\n\\nNIKE, INC. CONSOLIDATED BALANCE SHEETS\\n\\n(In millions)\\n\\nASSETS\\n\\nCurrent assets:\\n\\nCash and equivalents Short-term investments\\n\\nAccounts receivable, net Inventories Prepaid expenses and other current assets\\n\\nTotal current assets\\n\\nProperty, plant and equipment, net\\n\\nOperating lease right-of-use assets, net Identifiable intangible assets, net Goodwill\\n\\nDeferred income taxes and other assets\\n\\nTOTAL ASSETS\\n\\nLIABILITIES AND SHAREHOLDERS' EQUITY Current liabilities:\\n\\nCurrent portion of long-term debt Notes payable Accounts payable\\n\\nCurrent portion of operating lease liabilities Accrued liabilities Income taxes payable\\n\\nTotal current liabilities\\n\\nLong-term debt\\n\\nOperating lease liabilities Deferred income taxes and other liabilities Commitments and contingencies (Note 16)\\n\\nRedeemable preferred stock Shareholders' equity: Common stock at stated value:\"),\n",
- " Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"Tax (expense) benefit Gain (loss) net of tax\\n\\n5 (14)\\n\\n(9) 22\\n\\nTotal net gain (loss) reclassified for the period\\n\\n$\\n\\n463 $\\n\\n30\\n\\n2023 FORM 10-K 82\\n\\nTable of Contents\\n\\nNOTE 14 — REVENUES\\n\\nDISAGGREGATION OF REVENUES The following tables present the Company's Revenues disaggregated by reportable operating segment, major product line and distribution channel:\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\\n\\nYEAR ENDED MAY 31, 2023 ASIA PACIFIC & LATIN (1)\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE\\n\\nAMERICA\\n\\nBRAND CONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by: Footwear\\n\\n$\\n\\n14,897 $\\n\\n8,260 $\\n\\n5,435 $\\n\\n4,543 $\\n\\n— $\\n\\n33,135 $\\n\\n2,155 $\\n\\n— $\\n\\n35,290\\n\\nApparel Equipment Other\\n\\n5,947 764 —\\n\\n4,566 592 —\\n\\n1,666 147 —\\n\\n1,664 224 —\\n\\n— — 58\\n\\n13,843 1,727 58\\n\\n90 28 154\\n\\n— — 27\\n\\n13,933 1,755 239\\n\\nTOTAL REVENUES\\n\\n$\\n\\n21,608 $\\n\\n13,418 $\\n\\n7,248 $\\n\\n6,431 $\\n\\n58 $\\n\\n48,763 $\\n\\n2,427 $\\n\\n27 $\\n\\n51,217\\n\\nRevenues by:\\n\\nSales to Wholesale Customers Sales through Direct to Consumer\\n\\n$\\n\\n11,273 $ 10,335\\n\\n8,522 $ 4,896\\n\\n3,866 $ 3,382\\n\\n3,736 $ 2,695\\n\\n— $ —\\n\\n27,397 $ 21,308\\n\\n1,299 $ 974\\n\\n— $ —\\n\\n28,696 22,282\\n\\nOther\\n\\nTOTAL REVENUES\\n\\n$\\n\\n—\\n\\n21,608 $\\n\\n—\\n\\n13,418 $\\n\\n— 7,248 $\\n\\n— 6,431 $\\n\\n58 58 $\\n\\n58\\n\\n48,763 $\\n\\n154 2,427 $\\n\\n27 27 $\\n\\n239 51,217\\n\\n(1) Refer to Note 18 — Acquisitions and Divestitures for additional information on the transition of the Company's NIKE Brand businesses in its CASA territory to third-party distributors.\\n\\nYEAR ENDED MAY 31, 2022\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\\n\\nASIA PACIFIC & LATIN AMERICA\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE\\n\\nBRAND CONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by: Footwear Apparel\\n\\n$\\n\\n12,228 $ 5,492\\n\\n7,388 $ 4,527\\n\\n5,416 $ 1,938\\n\\n4,111 $ 1,610\\n\\n— $ —\\n\\n29,143 $ 13,567\\n\\n2,094 $ 103\\n\\n— $ —\\n\\n31,237 13,670\\n\\nEquipment Other\\n\\n633 —\\n\\n564 —\\n\\n193 —\\n\\n234 —\\n\\n— 102\\n\\n1,624 102\\n\\n26 123\\n\\n— (72)\\n\\n1,650 153\\n\\nTOTAL REVENUES Revenues by:\\n\\n$\\n\\n18,353 $\\n\\n12,479 $\\n\\n7,547 $\\n\\n5,955 $\\n\\n102 $\\n\\n44,436 $\\n\\n2,346 $\\n\\n(72) $\\n\\n46,710\\n\\nSales to Wholesale Customers Sales through Direct to Consumer Other\\n\\n$\\n\\n9,621 $ 8,732 —\\n\\n8,377 $ 4,102 —\\n\\n4,081 $ 3,466 —\\n\\n3,529 $ 2,426 —\\n\\n— $ — 102\\n\\n25,608 $ 18,726 102\\n\\n1,292 $ 931 123\\n\\n— $ — (72)\\n\\n26,900 19,657 153\\n\\nTOTAL REVENUES\\n\\n$\\n\\n18,353 $\\n\\n12,479 $\\n\\n7,547 $\\n\\n5,955 $\\n\\n102 $\\n\\n44,436 $\\n\\n2,346 $\\n\\n(72) $\\n\\n46,710\\n\\n2023 FORM 10-K 83\\n\\nTable of Contents\\n\\nYEAR ENDED MAY 31, 2021\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\"),\n",
- " Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"ASIA PACIFIC & LATIN AMERICA\\n\\n(1)\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE BRAND\\n\\nCONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by:\\n\\nFootwear Apparel Equipment\\n\\n$\\n\\n11,644 $ 5,028 507\\n\\n6,970 $ 3,996 490\\n\\n5,748 $ 2,347 195\\n\\n3,659 $ 1,494 190\\n\\n— $ — —\\n\\n28,021 $ 12,865 1,382\\n\\n1,986 $ 104 29\\n\\n— $ — —\\n\\n30,007 12,969 1,411\\n\\nOther\\n\\nTOTAL REVENUES\\n\\n$\\n\\n—\\n\\n17,179 $\\n\\n—\\n\\n11,456 $\\n\\n— 8,290 $\\n\\n— 5,343 $\\n\\n25 25 $\\n\\n25\\n\\n42,293 $\\n\\n86 2,205 $\\n\\n40 40 $\\n\\n151 44,538\\n\\nRevenues by:\\n\\nSales to Wholesale Customers $\\n\\n10,186 $\\n\\n7,812 $\\n\\n4,513 $\\n\\n3,387 $\\n\\n— $\\n\\n25,898 $\\n\\n1,353 $\\n\\n— $\\n\\n27,251\\n\\nSales through Direct to Consumer Other\\n\\n6,993 —\\n\\n3,644 —\\n\\n3,777 —\\n\\n1,956 —\\n\\n— 25\\n\\n16,370 25\\n\\n766 86\\n\\n— 40\\n\\n17,136 151\\n\\nTOTAL REVENUES\\n\\n$\\n\\n17,179 $\\n\\n11,456 $\\n\\n8,290 $\\n\\n5,343 $\\n\\n25 $\\n\\n42,293 $\\n\\n2,205 $\\n\\n40 $\\n\\n44,538\\n\\n(1) Refer to Note 18 — Acquisitions and Divestitures for additional information on the transition of the Company's NIKE Brand business in Brazil to a third-party distributor.\\n\\nFor the fiscal years ended May 31, 2023, 2022 and 2021, Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment. Converse Other revenues were primarily attributable to licensing businesses. Corporate revenues primarily consisted of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse but managed through the Company's central foreign exchange risk management program.\\n\\nAs of May 31, 2023 and 2022, the Company did not have any contract assets and had an immaterial amount of contract liabilities recorded in Accrued liabilities on the Consolidated Balance Sheets.\\n\\nSALES-RELATED RESERVES\\n\\nAs of May 31, 2023 and 2022, the Company's sales-related reserve balance, which includes returns, post-invoice sales discounts and miscellaneous claims, was $994 million and $1,015 million, respectively, recorded in Accrued liabilities on the Consolidated Balance Sheets. The estimated cost of inventory for expected product returns was $226 million and $194 million as of May 31, 2023 and 2022, respectively, and was recorded in Prepaid expenses and other current assets on the Consolidated Balance Sheets.\\n\\nNOTE 15 — OPERATING SEGMENTS AND RELATED INFORMATION\")],\n",
- " 'answer': \"Nike's revenue last year was $51,217 million.\"}"
- ]
- },
- "execution_count": 109,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from langchain.chains import create_retrieval_chain\n",
- "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
- "\n",
- "question_answer_chain = create_stuff_documents_chain(llm, prompt)\n",
- "rag_chain = create_retrieval_chain(rds.as_retriever(), question_answer_chain)\n",
- "\n",
- "rag_chain.invoke({\"input\": \"What was nike's revenue last year?\"})"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## (Optional) Creating a test set\n",
- "\n",
- "Now that our setup is complete and we have our RAG app to evaluate we need a test set to evaluate against. The ragas library provides a helpful class for generating a synthetic test set given our data as input that we will use here. The output of this generation is a set of `questions`, `contexts`, and `ground_truth`. \n",
- "\n",
- "The questions are generated by an LLM based on slices of context from the provided doc and the ground_truth is determined via a critic LLM. Note there is nothing special about this data itself and you can provide your own `questions` and `ground_truth` for evaluation purposes. When starting a project however, there is often a lack of quality human labeled data to be used for evaluation and a synthetic dataset is a valuable place to start if pre live user/process data (which should be incorporated as an ultimate goal).\n",
- "\n",
- "For more detail see [the docs](https://docs.ragas.io/en/stable/concepts/testset_generation.html)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [],
- "source": [
- "# NBVAL_SKIP\n",
- "# source: https://docs.ragas.io/en/latest/getstarted/testset_generation.html\n",
- "from ragas.testset.generator import TestsetGenerator\n",
- "from ragas.testset.evolutions import simple, reasoning, multi_context\n",
- "from ragas.run_config import RunConfig\n",
- "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
- "\n",
- "run_config = RunConfig(\n",
- " timeout=200,\n",
- " max_wait=160,\n",
- " max_retries=3,\n",
- ")\n",
- "\n",
- "# generator with openai models\n",
- "generator_llm = ChatOpenAI(model=\"gpt-3.5-turbo-16k\")\n",
- "critic_llm = ChatOpenAI(model=\"gpt-4o-mini\")\n",
- "embeddings = OpenAIEmbeddings()\n",
- "\n",
- "generator = TestsetGenerator.from_langchain(\n",
- " generator_llm,\n",
- " critic_llm,\n",
- " embeddings,\n",
- " run_config=run_config,\n",
- ")\n",
- "\n",
- "testset = generator.generate_with_langchain_docs(\n",
- " chunks,\n",
- " test_size=10,\n",
- " distributions={\n",
- " simple: 0.5,\n",
- " reasoning: 0.25,\n",
- " multi_context: 0.25\n",
- " },\n",
- " run_config=run_config\n",
- ")\n",
- "\n",
- "# save to csv since this can be a time consuming process\n",
- "testset.to_pandas().to_csv(\"resources/new_testset.csv\", index=False)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Evaluation helper functions\n",
- "\n",
- "The following code takes a RetrievalQA chain, testset dataframe, and the metrics to be evaluated and returns a dataframe including the metrics calculated."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 110,
- "metadata": {},
- "outputs": [],
- "source": [
- "import pandas as pd\n",
- "from datasets import Dataset\n",
- "from ragas import evaluate\n",
- "from ragas.run_config import RunConfig\n",
- "\n",
- "def parse_contexts(source_docs):\n",
- " return [doc.page_content for doc in source_docs]\n",
- "\n",
- "def create_evaluation_dataset(chain, testset):\n",
- " res_set = {\n",
- " \"question\": [],\n",
- " \"answer\": [],\n",
- " \"contexts\": [],\n",
- " \"ground_truth\": []\n",
- " }\n",
- "\n",
- " for _, row in testset.iterrows():\n",
- " result = chain.invoke({\"input\": row[\"question\"]})\n",
- "\n",
- " res_set[\"question\"].append(row[\"question\"])\n",
- " res_set[\"answer\"].append(result[\"answer\"])\n",
- "\n",
- " contexts = parse_contexts(result[\"context\"])\n",
- "\n",
- " if not len(contexts):\n",
- " print(f\"no contexts found for question: {row['question']}\")\n",
- " res_set[\"contexts\"].append(contexts)\n",
- " res_set[\"ground_truth\"].append(str(row[\"ground_truth\"]))\n",
- "\n",
- " return Dataset.from_dict(res_set)\n",
- "\n",
- "def evaluate_dataset(eval_dataset, metrics, llm, embeddings):\n",
- "\n",
- " run_config = RunConfig(max_retries=1) # see ragas docs for more run_config options\n",
- "\n",
- " eval_result = evaluate(\n",
- " eval_dataset,\n",
- " metrics=metrics,\n",
- " run_config=run_config,\n",
- " llm=llm,\n",
- " embeddings=embeddings\n",
- " )\n",
- "\n",
- " eval_df = eval_result.to_pandas()\n",
- " return eval_df"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Create the evaluation data\n",
- "\n",
- "Input: chain to be evaluated and a pregenerated test set\n", - "Output: dataset formatted for use with ragas evaluation function" - ] - }, - { - "cell_type": "code", - "execution_count": 111, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n",
- "\n",
- "\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " question \\\n",
- "0 What are short-term investments and how are th... \n",
- "1 What are some of the risks and uncertainties a... \n",
- "2 What is NIKE's policy regarding securities ana... \n",
- "3 What are the revenues for the Footwear and App... \n",
- "4 How do master netting arrangements impact the ... \n",
- "\n",
- " contexts \\\n",
- "0 [\"CASH AND EQUIVALENTS Cash and equivalents re... \n",
- "1 ['Our NIKE Direct operations, including our re... \n",
- "2 [\"Investors should also be aware that while NI... \n",
- "3 ['(Dollars in millions, except per share data)... \n",
- "4 [\"The Company records the assets and liabiliti... \n",
- "\n",
- " ground_truth evolution_type \\\n",
- "0 Short-term investments are highly liquid inves... simple \n",
- "1 Many factors unique to retail operations, some... simple \n",
- "2 NIKE's policy is to not disclose any material ... simple \n",
- "3 The revenues for the Footwear and Apparel cate... simple \n",
- "4 The Company records the assets and liabilities... simple \n",
- "\n",
- " metadata episode_done \n",
- "0 [{'source': 'resources/nke-10k-2023.pdf'}] True \n",
- "1 [{'source': 'resources/nke-10k-2023.pdf'}] True \n",
- "2 [{'source': 'resources/nke-10k-2023.pdf'}] True \n",
- "3 [{'source': 'resources/nke-10k-2023.pdf'}] True \n",
- "4 [{'source': 'resources/nke-10k-2023.pdf'}] True "
- ]
- },
- "execution_count": 111,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "testset_df = pd.read_csv(\"resources/testset_15.csv\")\n",
- "testset_df.head()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 112,
- "metadata": {},
- "outputs": [],
- "source": [
- "eval_dataset = create_evaluation_dataset(rag_chain, testset_df)\n",
- "eval_dataset.to_pandas().shape"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Evaluate generation metrics\n",
- "Generation metrics quantify how well the RAG app did creating answers to the provided questions (i.e. the G in **R**etrival **A**ugments **G**eneration). We will calculate the generation metrics **faithfulness** and **answer relevancy** for this example.\n",
- "\n",
- "The ragas libary conveniently abstracts the calculation of these metrics so we don't have to write redundant code but please review the following definitions in order to build intuition around what these metrics actually measure.\n",
- "\n",
- "Note: the following examples are paraphrased from the [ragas docs](https://docs.ragas.io/en/stable/concepts/metrics/index.html)\n",
- "\n",
- "------\n",
- "\n",
- "### Faithfulness\n",
- "\n",
- "An answer to a question can be said to be \"faithful\" if the **claims** that are made in the answer **can be inferred** from the **context**.\n",
- "\n",
- "#### Mathematically:\n",
- "\n",
- "$$\n",
- "Faithfullness\\ score = \\frac{Number\\ of\\ claims\\ in\\ the\\ generated\\ answer\\ that\\ can\\ be\\ inferred\\ from\\ the\\ given\\ context}{Total\\ number\\ of\\ claim\\ in\\ the\\ generated\\ answer}\n",
- "$$\n",
- "\n",
- "#### Example process:\n",
- "\n",
- "> Question: Where and when was Einstein born?\n",
- "> \n",
- "> Context: Albert Einstein (born 14 March 1879) was a German-born theoretical physicist, widely held to be one of the greatest and most influential scientists of all time\n",
- ">\n",
- "> answer: Einstein was born in Germany on 20th March 1879.\n",
- "\n",
- "Step 1: Use LLM to break generated answer into individual statements.\n",
- "- “Einstein was born in Germany.”\n",
- "- “Einstein was born on 20th March 1879.”\n",
- "\n",
- "Step 2: For each statement use LLM to verify if it can be inferred from the context.\n",
- "- “Einstein was born in Germany.” => yes. \n",
- "- “Einstein was born on 20th March 1879.” => no.\n",
- "\n",
- "Step 3: plug into formula\n",
- "\n",
- "Number of claims inferred from context = 1\n",
- "Total number of claims = 2\n",
- "Faithfulness = 1/2\n",
- "\n",
- "### Answer Relevance\n",
- "\n",
- "An answer can be said to be relevant if it directly addresses the question (intuitively).\n",
- "\n",
- "#### Example process:\n",
- "\n",
- "1. Use an LLM to generate \"hypothetical\" questions to a given answer with the following prompt:\n",
- "\n",
- " > Generate a question for the given answer.\n",
- " > answer: [answer]\n",
- "\n",
- "2. Embed the generated \"hypothetical\" questions as vectors.\n",
- "3. Calculate the cosine similarity of the hypothetical questions and the original question, sum those similarities, and divide by n.\n",
- "\n",
- "With data:\n",
- "\n",
- "> Question: Where is France and what is it’s capital?\n",
- "> \n",
- "> answer: France is in western Europe.\n",
- "\n",
- "Step 1 - use LLM to create 'n' variants of question from the generated answer.\n",
- "\n",
- "- “In which part of Europe is France located?”\n",
- "- “What is the geographical location of France within Europe?”\n",
- "- “Can you identify the region of Europe where France is situated?”\n",
- "\n",
- "Step 2 - Calculate the mean cosine similarity between the generated questions and the actual question.\n",
- "\n",
- "## Now let's implement using our helper functions\n",
- "\n"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 114,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "dd9cabb4b0c448b08cad96d2ef3391a2",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "Evaluating: 0%| | 0/15 [00:00\n",
- "\n",
- "| \n", - " | question | \n", - "contexts | \n", - "ground_truth | \n", - "evolution_type | \n", - "metadata | \n", - "episode_done | \n", - "
|---|---|---|---|---|---|---|
| 0 | \n", - "What are short-term investments and how are th... | \n", - "[\"CASH AND EQUIVALENTS Cash and equivalents re... | \n", - "Short-term investments are highly liquid inves... | \n", - "simple | \n", - "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", - "True | \n", - "
| 1 | \n", - "What are some of the risks and uncertainties a... | \n", - "['Our NIKE Direct operations, including our re... | \n", - "Many factors unique to retail operations, some... | \n", - "simple | \n", - "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", - "True | \n", - "
| 2 | \n", - "What is NIKE's policy regarding securities ana... | \n", - "[\"Investors should also be aware that while NI... | \n", - "NIKE's policy is to not disclose any material ... | \n", - "simple | \n", - "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", - "True | \n", - "
| 3 | \n", - "What are the revenues for the Footwear and App... | \n", - "['(Dollars in millions, except per share data)... | \n", - "The revenues for the Footwear and Apparel cate... | \n", - "simple | \n", - "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", - "True | \n", - "
| 4 | \n", - "How do master netting arrangements impact the ... | \n", - "[\"The Company records the assets and liabiliti... | \n", - "The Company records the assets and liabilities... | \n", - "simple | \n", - "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", - "True | \n", - "
| \n", - " | faithfulness | \n", - "answer_relevancy | \n", - "
|---|---|---|
| count | \n", - "15.000000 | \n", - "15.000000 | \n", - "
| mean | \n", - "0.781229 | \n", - "0.938581 | \n", - "
| std | \n", - "0.362666 | \n", - "0.085342 | \n", - "
| min | \n", - "0.000000 | \n", - "0.736997 | \n", - "
| 25% | \n", - "0.652778 | \n", - "0.926596 | \n", - "
| 50% | \n", - "1.000000 | \n", - "0.975230 | \n", - "
| 75% | \n", - "1.000000 | \n", - "0.994168 | \n", - "
| max | \n", - "1.000000 | \n", - "1.000000 | \n", - "
| \n", - " | context_recall | \n", - "context_precision | \n", - "
|---|---|---|
| count | \n", - "15.000000 | \n", - "15.000000 | \n", - "
| mean | \n", - "0.966667 | \n", - "0.925926 | \n", - "
| std | \n", - "0.129099 | \n", - "0.145352 | \n", - "
| min | \n", - "0.500000 | \n", - "0.500000 | \n", - "
| 25% | \n", - "1.000000 | \n", - "0.916667 | \n", - "
| 50% | \n", - "1.000000 | \n", - "1.000000 | \n", - "
| 75% | \n", - "1.000000 | \n", - "1.000000 | \n", - "
| max | \n", - "1.000000 | \n", - "1.000000 | \n", - "
\n",
- "\n",
- "\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " context_recall context_precision faithfulness answer_relevancy\n",
- "count 15.000000 15.000000 15.000000 15.000000\n",
- "mean 0.966667 0.925926 0.781229 0.938581\n",
- "std 0.129099 0.145352 0.362666 0.085342\n",
- "min 0.500000 0.500000 0.000000 0.736997\n",
- "25% 1.000000 0.916667 0.652778 0.926596\n",
- "50% 1.000000 1.000000 1.000000 0.975230\n",
- "75% 1.000000 1.000000 1.000000 0.994168\n",
- "max 1.000000 1.000000 1.000000 1.000000"
- ]
- },
- "execution_count": 121,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "metrics.describe()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Analysis\n",
- "Overall our RAG app showed pretty good performance. All values indicated above 0.6, which from anecdotal experience, is a reasonable lower-bound for performance however obviously higher values are more ideal. It is worth noting that generation metrics can be a bit more hazy in terms of ideal ranges since the LLM evaluation cannot yet capture the way a response feels to a user. For these metrics it's important to make sure they are not severely low however blind optimization to the top can result in a very uncreative chat experience which may or may not be ideal for the intended use case.\n",
- "\n",
- "## Review\n",
- "\n",
- "- we initialized our RAG app with data from a 10k document\n",
- "- generated a testset to evaluate \n",
- "- calculated both retrieval and generation metrics\n",
- "\n",
- "## Next steps\n",
- "\n",
- "Now that we know how to measure our system we can quickly and easily experiment with different techniques with a baseline in place to improve our systems.\n",
- "\n",
- "## Cleanup"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 122,
- "metadata": {},
- "outputs": [],
- "source": [
- "from redisvl.index import SearchIndex\n",
- "\n",
- "idx = SearchIndex.from_existing(\n",
- " index_name,\n",
- " redis_url=REDIS_URL\n",
- ")\n",
- "\n",
- "idx.delete()"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.11.9"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 2
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "# Evaluating RAG\n",
+ "\n",
+ "This notebook uses the [ragas library](https://docs.ragas.io/en/stable/getstarted/index.html) and [Redis](https://redis.com) to evaluate the performance of sample RAG application. Also see the original [source paper](https://arxiv.org/pdf/2309.15217) to build a more detailed understanding.\n",
+ "\n",
+ "## Let's Begin!\n",
+ "| \n", - " | context_recall | \n", - "context_precision | \n", - "faithfulness | \n", - "answer_relevancy | \n", - "
|---|---|---|---|---|
| count | \n", - "15.000000 | \n", - "15.000000 | \n", - "15.000000 | \n", - "15.000000 | \n", - "
| mean | \n", - "0.966667 | \n", - "0.925926 | \n", - "0.781229 | \n", - "0.938581 | \n", - "
| std | \n", - "0.129099 | \n", - "0.145352 | \n", - "0.362666 | \n", - "0.085342 | \n", - "
| min | \n", - "0.500000 | \n", - "0.500000 | \n", - "0.000000 | \n", - "0.736997 | \n", - "
| 25% | \n", - "1.000000 | \n", - "0.916667 | \n", - "0.652778 | \n", - "0.926596 | \n", - "
| 50% | \n", - "1.000000 | \n", - "1.000000 | \n", - "1.000000 | \n", - "0.975230 | \n", - "
| 75% | \n", - "1.000000 | \n", - "1.000000 | \n", - "1.000000 | \n", - "0.994168 | \n", - "
| max | \n", - "1.000000 | \n", - "1.000000 | \n", - "1.000000 | \n", - "1.000000 | \n", - "
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "To start, we need a RAG app to evaluate. Let's create one using LangChain and connect it with Redis as the vector DB.\n",
+ "\n",
+ "## Init redis, data prep, and populating the vector DB"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.2\u001b[0m\n",
+ "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
+ "Note: you may need to restart the kernel to use updated packages.\n"
+ ]
+ }
+ ],
+ "source": [
+ "%pip install -q redis \"unstructured[pdf]\" sentence-transformers langchain \"langchain-redis>=0.2.0\" langchain-huggingface langchain-openai ragas datasets"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Running Redis in Colab\n",
+ "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NBVAL_SKIP\n",
+ "%%sh\n",
+ "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
+ "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
+ "sudo apt-get update > /dev/null 2>&1\n",
+ "sudo apt-get install redis-stack-server > /dev/null 2>&1\n",
+ "redis-stack-server --daemonize yes"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### For Alternative Environments\n",
+ "There are many ways to get the necessary redis-stack instance running\n",
+ "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
+ "own version of Redis Enterprise running, that works too!\n",
+ "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
+ "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "import warnings\n",
+ "warnings.filterwarnings('ignore')\n",
+ "\n",
+ "# Replace values below with your own if using Redis Cloud instance\n",
+ "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
+ "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n",
+ "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n",
+ "\n",
+ "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n",
+ "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+ "from langchain_community.document_loaders import PyPDFLoader\n",
+ "\n",
+ "CHUNK_SIZE = 2500\n",
+ "CHUNK_OVERLAP = 0\n",
+ "\n",
+ "# pdf to load\n",
+ "path = 'resources/nke-10k-2023.pdf'\n",
+ "assert os.path.exists(path), f\"File not found: {path}\"\n",
+ "\n",
+ "# load and split\n",
+ "loader = PyPDFLoader(path)\n",
+ "pages = loader.load()\n",
+ "text_splitter = RecursiveCharacterTextSplitter(chunk_size=CHUNK_SIZE, chunk_overlap=CHUNK_OVERLAP)\n",
+ "chunks = text_splitter.split_documents(pages)\n",
+ "\n",
+ "print(\"Done preprocessing. Created\", len(chunks), \"chunks of the original pdf\", path)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 95,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"Table of ContentsUNITED STATESSECURITIES AND EXCHANGE COMMISSIONWashington, D.C. 20549FORM 10-K(Mark One)☑ ANNUAL REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934FOR THE FISCAL YEAR ENDED MAY 31, 2023OR☐ TRANSITION REPORT PURSUANT TO SECTION 13 OR 15(D) OF THE SECURITIES EXCHANGE ACT OF 1934FOR THE TRANSITION PERIOD FROM TO .Commission File No. 1-10635\\n\\nAs of November 30, 2022, the aggregate market values of the Registrant's Common Stock held by non-affiliates were:Class A$7,831,564,572 Class B136,467,702,472 $144,299,267,044\\n\\nNIKE, Inc.(Exact name of Registrant as specified in its charter)Oregon93-0584541(State or other jurisdiction of incorporation)(IRS Employer Identification No.)One Bowerman Drive, Beaverton, Oregon 97005-6453(Address of principal executive offices and zip code)(503) 671-6453(Registrant's telephone number, including area code)SECURITIES REGISTERED PURSUANT TO SECTION 12(B) OF THE ACT:Class B Common StockNKENew York Stock Exchange(Title of each class)(Trading symbol)(Name of each exchange on which registered)SECURITIES REGISTERED PURSUANT TO SECTION 12(G) OF THE ACT:NONE\")"
+ ]
+ },
+ "execution_count": 95,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "chunks[0]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 96,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from langchain_huggingface import HuggingFaceEmbeddings\n",
+ "\n",
+ "embeddings = HuggingFaceEmbeddings(model_name=\"sentence-transformers/all-MiniLM-L6-v2\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 97,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from langchain_redis import RedisVectorStore\n",
+ "\n",
+ "# set the index name for this example\n",
+ "index_name = \"ragas_ex\"\n",
+ "\n",
+ "# construct the vector store class from texts and metadata\n",
+ "rds = RedisVectorStore.from_documents(\n",
+ " chunks,\n",
+ " embeddings,\n",
+ " index_name=index_name,\n",
+ " redis_url=REDIS_URL,\n",
+ " metadata_schema=[\n",
+ " {\n",
+ " \"name\": \"source\",\n",
+ " \"type\": \"text\"\n",
+ " },\n",
+ " ]\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Test the vector store"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 98,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, our operating segments are evidence of the structure of the Company\\'s internal organization. The NIKE Brand segments are defined by geographic regions for operations participating in NIKE Brand sales activity.\\n\\nThe breakdown of Revenues is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023 FISCAL 2022\\n\\n% CHANGE\\n\\n% CHANGE EXCLUDING CURRENCY (1) CHANGES FISCAL 2021\\n\\n% CHANGE\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n21,608 $ 13,418 7,248\\n\\n18,353 12,479 7,547\\n\\n18 % 8 % -4 %\\n\\n18 % $ 21 % 4 %\\n\\n17,179 11,456 8,290\\n\\n7 % 9 % -9 %\\n\\nAsia Pacific & Latin America Global Brand Divisions\\n\\n(3)\\n\\n(2)\\n\\n6,431 58\\n\\n5,955 102\\n\\n8 % -43 %\\n\\n17 % -43 %\\n\\n5,343 25\\n\\n11 % 308 %\\n\\nTOTAL NIKE BRAND Converse\\n\\n$\\n\\n48,763 $ 2,427\\n\\n44,436 2,346\\n\\n10 % 3 %\\n\\n16 % $ 8 %\\n\\n42,293 2,205\\n\\n5 % 6 %\\n\\n(4)\\n\\nCorporate TOTAL NIKE, INC. REVENUES\\n\\n$\\n\\n27\\n\\n51,217 $\\n\\n(72) 46,710\\n\\n— 10 %\\n\\n— 16 % $\\n\\n40 44,538\\n\\n— 5 %\\n\\n(1) The percent change excluding currency changes represents a non-GAAP financial measure. For further information, see \"Use of Non-GAAP Financial Measures\".\\n\\n(2) For additional information on the transition of our NIKE Brand businesses within our CASA territory to a third-party distributor, see Note 18 — Acquisitions and Divestitures of the Notes to Consolidated\\n\\nFinancial Statements contained in Item 8 of this Annual Report.\\n\\n(3) Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment.\\n\\n(4) Corporate revenues primarily consist of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse, but\\n\\nmanaged through our central foreign exchange risk management program.\\n\\nThe primary financial measure used by the Company to evaluate performance is Earnings Before Interest and Taxes (\"EBIT\"). As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, certain corporate costs are not included in EBIT.\\n\\nThe breakdown of EBIT is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023\\n\\nFISCAL 2022\\n\\n% CHANGE\\n\\nFISCAL 2021\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n5,454 3,531 2,283\\n\\n$\\n\\n5,114 3,293 2,365\\n\\n7 % $ 7 % -3 %\\n\\n5,089 2,435 3,243\\n\\nAsia Pacific & Latin America Global Brand Divisions (1)'"
+ ]
+ },
+ "execution_count": 98,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "rds.similarity_search(\"What was nike's revenue last year?\")[0].page_content"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Setup RAG\n",
+ "\n",
+ "Now that the vector db is populated let's initialize our RAG app."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 99,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import getpass\n",
+ "from langchain_openai import ChatOpenAI\n",
+ "\n",
+ "if \"OPENAI_API_KEY\" not in os.environ:\n",
+ " os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OPENAI_API_KEY\")\n",
+ "\n",
+ "llm = ChatOpenAI(\n",
+ " openai_api_key=os.environ[\"OPENAI_API_KEY\"],\n",
+ " model=\"gpt-3.5-turbo-16k\",\n",
+ " max_tokens=None\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 108,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from langchain_core.prompts import ChatPromptTemplate\n",
+ "\n",
+ "system_prompt = \"\"\"\n",
+ " Use the following pieces of context from financial 10k filings data to answer the user question at the end. \n",
+ " If you don't know the answer, say that you don't know, don't try to make up an answer.\n",
+ "\n",
+ " Context:\n",
+ " ---------\n",
+ " {context}\n",
+ "\"\"\"\n",
+ "\n",
+ "def format_docs(docs):\n",
+ " return \"\\n\\n\".join(doc.page_content for doc in docs)\n",
+ "\n",
+ "prompt = ChatPromptTemplate.from_messages(\n",
+ " [\n",
+ " (\"system\", system_prompt),\n",
+ " (\"human\", \"{input}\")\n",
+ " ]\n",
+ ")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Test it out"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 109,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'input': \"What was nike's revenue last year?\",\n",
+ " 'context': [Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content='As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, our operating segments are evidence of the structure of the Company\\'s internal organization. The NIKE Brand segments are defined by geographic regions for operations participating in NIKE Brand sales activity.\\n\\nThe breakdown of Revenues is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023 FISCAL 2022\\n\\n% CHANGE\\n\\n% CHANGE EXCLUDING CURRENCY (1) CHANGES FISCAL 2021\\n\\n% CHANGE\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n21,608 $ 13,418 7,248\\n\\n18,353 12,479 7,547\\n\\n18 % 8 % -4 %\\n\\n18 % $ 21 % 4 %\\n\\n17,179 11,456 8,290\\n\\n7 % 9 % -9 %\\n\\nAsia Pacific & Latin America Global Brand Divisions\\n\\n(3)\\n\\n(2)\\n\\n6,431 58\\n\\n5,955 102\\n\\n8 % -43 %\\n\\n17 % -43 %\\n\\n5,343 25\\n\\n11 % 308 %\\n\\nTOTAL NIKE BRAND Converse\\n\\n$\\n\\n48,763 $ 2,427\\n\\n44,436 2,346\\n\\n10 % 3 %\\n\\n16 % $ 8 %\\n\\n42,293 2,205\\n\\n5 % 6 %\\n\\n(4)\\n\\nCorporate TOTAL NIKE, INC. REVENUES\\n\\n$\\n\\n27\\n\\n51,217 $\\n\\n(72) 46,710\\n\\n— 10 %\\n\\n— 16 % $\\n\\n40 44,538\\n\\n— 5 %\\n\\n(1) The percent change excluding currency changes represents a non-GAAP financial measure. For further information, see \"Use of Non-GAAP Financial Measures\".\\n\\n(2) For additional information on the transition of our NIKE Brand businesses within our CASA territory to a third-party distributor, see Note 18 — Acquisitions and Divestitures of the Notes to Consolidated\\n\\nFinancial Statements contained in Item 8 of this Annual Report.\\n\\n(3) Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment.\\n\\n(4) Corporate revenues primarily consist of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse, but\\n\\nmanaged through our central foreign exchange risk management program.\\n\\nThe primary financial measure used by the Company to evaluate performance is Earnings Before Interest and Taxes (\"EBIT\"). As discussed in Note 15 — Operating Segments and Related Information in the accompanying Notes to the Consolidated Financial Statements, certain corporate costs are not included in EBIT.\\n\\nThe breakdown of EBIT is as follows:\\n\\n(Dollars in millions)\\n\\nFISCAL 2023\\n\\nFISCAL 2022\\n\\n% CHANGE\\n\\nFISCAL 2021\\n\\nNorth America Europe, Middle East & Africa Greater China\\n\\n$\\n\\n5,454 3,531 2,283\\n\\n$\\n\\n5,114 3,293 2,365\\n\\n7 % $ 7 % -3 %\\n\\n5,089 2,435 3,243\\n\\nAsia Pacific & Latin America Global Brand Divisions (1)'),\n",
+ " Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"NIKE, INC. CONSOLIDATED STATEMENTS OF INCOME\\n\\n(In millions, except per share data)\\n\\nRevenues Cost of sales\\n\\nGross profit\\n\\nDemand creation expense Operating overhead expense\\n\\nTotal selling and administrative expense\\n\\nInterest expense (income), net\\n\\nOther (income) expense, net Income before income taxes\\n\\nIncome tax expense NET INCOME\\n\\nEarnings per common share:\\n\\nBasic Diluted\\n\\nWeighted average common shares outstanding:\\n\\nBasic Diluted\\n\\nThe accompanying Notes to the Consolidated Financial Statements are an integral part of this statement.\\n\\n$\\n\\n$\\n\\n$ $\\n\\nYEAR ENDED MAY 31,\\n\\n2023\\n\\n2022\\n\\n2021\\n\\n51,217 $ 28,925\\n\\n46,710 $ 25,231\\n\\n44,538 24,576\\n\\n22,292 4,060 12,317\\n\\n21,479 3,850 10,954\\n\\n19,962 3,114 9,911\\n\\n16,377 (6)\\n\\n14,804 205\\n\\n13,025 262\\n\\n(280) 6,201\\n\\n(181) 6,651\\n\\n14 6,661\\n\\n1,131 5,070 $\\n\\n605 6,046 $\\n\\n934 5,727\\n\\n3.27 $ 3.23 $\\n\\n3.83 $ 3.75 $\\n\\n3.64 3.56\\n\\n1,551.6 1,569.8\\n\\n1,578.8 1,610.8\\n\\n1,573.0 1,609.4\\n\\n2023 FORM 10-K 55\\n\\nTable of Contents\\n\\nNIKE, INC. CONSOLIDATED STATEMENTS OF COMPREHENSIVE INCOME\\n\\nYEAR ENDED MAY 31,\\n\\n(Dollars in millions)\\n\\n2023\\n\\n2022\\n\\nNet income Other comprehensive income (loss), net of tax:\\n\\n$\\n\\n5,070 $\\n\\n6,046 $\\n\\nChange in net foreign currency translation adjustment\\n\\n267\\n\\n(522)\\n\\nChange in net gains (losses) on cash flow hedges Change in net gains (losses) on other\\n\\n(348) (6)\\n\\n1,214 6\\n\\nTotal other comprehensive income (loss), net of tax TOTAL COMPREHENSIVE INCOME\\n\\n$\\n\\n(87) 4,983 $\\n\\n698 6,744 $\\n\\nThe accompanying Notes to the Consolidated Financial Statements are an integral part of this statement.\\n\\n2023 FORM 10-K 56\\n\\n2021\\n\\n5,727\\n\\n496\\n\\n(825) 5\\n\\n(324) 5,403\\n\\nTable of Contents\\n\\nNIKE, INC. CONSOLIDATED BALANCE SHEETS\\n\\n(In millions)\\n\\nASSETS\\n\\nCurrent assets:\\n\\nCash and equivalents Short-term investments\\n\\nAccounts receivable, net Inventories Prepaid expenses and other current assets\\n\\nTotal current assets\\n\\nProperty, plant and equipment, net\\n\\nOperating lease right-of-use assets, net Identifiable intangible assets, net Goodwill\\n\\nDeferred income taxes and other assets\\n\\nTOTAL ASSETS\\n\\nLIABILITIES AND SHAREHOLDERS' EQUITY Current liabilities:\\n\\nCurrent portion of long-term debt Notes payable Accounts payable\\n\\nCurrent portion of operating lease liabilities Accrued liabilities Income taxes payable\\n\\nTotal current liabilities\\n\\nLong-term debt\\n\\nOperating lease liabilities Deferred income taxes and other liabilities Commitments and contingencies (Note 16)\\n\\nRedeemable preferred stock Shareholders' equity: Common stock at stated value:\"),\n",
+ " Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"Tax (expense) benefit Gain (loss) net of tax\\n\\n5 (14)\\n\\n(9) 22\\n\\nTotal net gain (loss) reclassified for the period\\n\\n$\\n\\n463 $\\n\\n30\\n\\n2023 FORM 10-K 82\\n\\nTable of Contents\\n\\nNOTE 14 — REVENUES\\n\\nDISAGGREGATION OF REVENUES The following tables present the Company's Revenues disaggregated by reportable operating segment, major product line and distribution channel:\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\\n\\nYEAR ENDED MAY 31, 2023 ASIA PACIFIC & LATIN (1)\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE\\n\\nAMERICA\\n\\nBRAND CONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by: Footwear\\n\\n$\\n\\n14,897 $\\n\\n8,260 $\\n\\n5,435 $\\n\\n4,543 $\\n\\n— $\\n\\n33,135 $\\n\\n2,155 $\\n\\n— $\\n\\n35,290\\n\\nApparel Equipment Other\\n\\n5,947 764 —\\n\\n4,566 592 —\\n\\n1,666 147 —\\n\\n1,664 224 —\\n\\n— — 58\\n\\n13,843 1,727 58\\n\\n90 28 154\\n\\n— — 27\\n\\n13,933 1,755 239\\n\\nTOTAL REVENUES\\n\\n$\\n\\n21,608 $\\n\\n13,418 $\\n\\n7,248 $\\n\\n6,431 $\\n\\n58 $\\n\\n48,763 $\\n\\n2,427 $\\n\\n27 $\\n\\n51,217\\n\\nRevenues by:\\n\\nSales to Wholesale Customers Sales through Direct to Consumer\\n\\n$\\n\\n11,273 $ 10,335\\n\\n8,522 $ 4,896\\n\\n3,866 $ 3,382\\n\\n3,736 $ 2,695\\n\\n— $ —\\n\\n27,397 $ 21,308\\n\\n1,299 $ 974\\n\\n— $ —\\n\\n28,696 22,282\\n\\nOther\\n\\nTOTAL REVENUES\\n\\n$\\n\\n—\\n\\n21,608 $\\n\\n—\\n\\n13,418 $\\n\\n— 7,248 $\\n\\n— 6,431 $\\n\\n58 58 $\\n\\n58\\n\\n48,763 $\\n\\n154 2,427 $\\n\\n27 27 $\\n\\n239 51,217\\n\\n(1) Refer to Note 18 — Acquisitions and Divestitures for additional information on the transition of the Company's NIKE Brand businesses in its CASA territory to third-party distributors.\\n\\nYEAR ENDED MAY 31, 2022\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\\n\\nASIA PACIFIC & LATIN AMERICA\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE\\n\\nBRAND CONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by: Footwear Apparel\\n\\n$\\n\\n12,228 $ 5,492\\n\\n7,388 $ 4,527\\n\\n5,416 $ 1,938\\n\\n4,111 $ 1,610\\n\\n— $ —\\n\\n29,143 $ 13,567\\n\\n2,094 $ 103\\n\\n— $ —\\n\\n31,237 13,670\\n\\nEquipment Other\\n\\n633 —\\n\\n564 —\\n\\n193 —\\n\\n234 —\\n\\n— 102\\n\\n1,624 102\\n\\n26 123\\n\\n— (72)\\n\\n1,650 153\\n\\nTOTAL REVENUES Revenues by:\\n\\n$\\n\\n18,353 $\\n\\n12,479 $\\n\\n7,547 $\\n\\n5,955 $\\n\\n102 $\\n\\n44,436 $\\n\\n2,346 $\\n\\n(72) $\\n\\n46,710\\n\\nSales to Wholesale Customers Sales through Direct to Consumer Other\\n\\n$\\n\\n9,621 $ 8,732 —\\n\\n8,377 $ 4,102 —\\n\\n4,081 $ 3,466 —\\n\\n3,529 $ 2,426 —\\n\\n— $ — 102\\n\\n25,608 $ 18,726 102\\n\\n1,292 $ 931 123\\n\\n— $ — (72)\\n\\n26,900 19,657 153\\n\\nTOTAL REVENUES\\n\\n$\\n\\n18,353 $\\n\\n12,479 $\\n\\n7,547 $\\n\\n5,955 $\\n\\n102 $\\n\\n44,436 $\\n\\n2,346 $\\n\\n(72) $\\n\\n46,710\\n\\n2023 FORM 10-K 83\\n\\nTable of Contents\\n\\nYEAR ENDED MAY 31, 2021\\n\\n(Dollars in millions)\\n\\nNORTH AMERICA\\n\\nEUROPE, MIDDLE EAST & AFRICA\\n\\nGREATER CHINA\"),\n",
+ " Document(metadata={'source': 'resources/nke-10k-2023.pdf'}, page_content=\"ASIA PACIFIC & LATIN AMERICA\\n\\n(1)\\n\\nGLOBAL BRAND DIVISIONS\\n\\nTOTAL NIKE BRAND\\n\\nCONVERSE CORPORATE\\n\\nTOTAL NIKE, INC.\\n\\nRevenues by:\\n\\nFootwear Apparel Equipment\\n\\n$\\n\\n11,644 $ 5,028 507\\n\\n6,970 $ 3,996 490\\n\\n5,748 $ 2,347 195\\n\\n3,659 $ 1,494 190\\n\\n— $ — —\\n\\n28,021 $ 12,865 1,382\\n\\n1,986 $ 104 29\\n\\n— $ — —\\n\\n30,007 12,969 1,411\\n\\nOther\\n\\nTOTAL REVENUES\\n\\n$\\n\\n—\\n\\n17,179 $\\n\\n—\\n\\n11,456 $\\n\\n— 8,290 $\\n\\n— 5,343 $\\n\\n25 25 $\\n\\n25\\n\\n42,293 $\\n\\n86 2,205 $\\n\\n40 40 $\\n\\n151 44,538\\n\\nRevenues by:\\n\\nSales to Wholesale Customers $\\n\\n10,186 $\\n\\n7,812 $\\n\\n4,513 $\\n\\n3,387 $\\n\\n— $\\n\\n25,898 $\\n\\n1,353 $\\n\\n— $\\n\\n27,251\\n\\nSales through Direct to Consumer Other\\n\\n6,993 —\\n\\n3,644 —\\n\\n3,777 —\\n\\n1,956 —\\n\\n— 25\\n\\n16,370 25\\n\\n766 86\\n\\n— 40\\n\\n17,136 151\\n\\nTOTAL REVENUES\\n\\n$\\n\\n17,179 $\\n\\n11,456 $\\n\\n8,290 $\\n\\n5,343 $\\n\\n25 $\\n\\n42,293 $\\n\\n2,205 $\\n\\n40 $\\n\\n44,538\\n\\n(1) Refer to Note 18 — Acquisitions and Divestitures for additional information on the transition of the Company's NIKE Brand business in Brazil to a third-party distributor.\\n\\nFor the fiscal years ended May 31, 2023, 2022 and 2021, Global Brand Divisions revenues include NIKE Brand licensing and other miscellaneous revenues that are not part of a geographic operating segment. Converse Other revenues were primarily attributable to licensing businesses. Corporate revenues primarily consisted of foreign currency hedge gains and losses related to revenues generated by entities within the NIKE Brand geographic operating segments and Converse but managed through the Company's central foreign exchange risk management program.\\n\\nAs of May 31, 2023 and 2022, the Company did not have any contract assets and had an immaterial amount of contract liabilities recorded in Accrued liabilities on the Consolidated Balance Sheets.\\n\\nSALES-RELATED RESERVES\\n\\nAs of May 31, 2023 and 2022, the Company's sales-related reserve balance, which includes returns, post-invoice sales discounts and miscellaneous claims, was $994 million and $1,015 million, respectively, recorded in Accrued liabilities on the Consolidated Balance Sheets. The estimated cost of inventory for expected product returns was $226 million and $194 million as of May 31, 2023 and 2022, respectively, and was recorded in Prepaid expenses and other current assets on the Consolidated Balance Sheets.\\n\\nNOTE 15 — OPERATING SEGMENTS AND RELATED INFORMATION\")],\n",
+ " 'answer': \"Nike's revenue last year was $51,217 million.\"}"
+ ]
+ },
+ "execution_count": 109,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain.chains import create_retrieval_chain\n",
+ "from langchain.chains.combine_documents import create_stuff_documents_chain\n",
+ "\n",
+ "question_answer_chain = create_stuff_documents_chain(llm, prompt)\n",
+ "rag_chain = create_retrieval_chain(rds.as_retriever(), question_answer_chain)\n",
+ "\n",
+ "rag_chain.invoke({\"input\": \"What was nike's revenue last year?\"})"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## (Optional) Creating a test set\n",
+ "\n",
+ "Now that our setup is complete and we have our RAG app to evaluate we need a test set to evaluate against. The ragas library provides a helpful class for generating a synthetic test set given our data as input that we will use here. The output of this generation is a set of `questions`, `contexts`, and `ground_truth`. \n",
+ "\n",
+ "The questions are generated by an LLM based on slices of context from the provided doc and the ground_truth is determined via a critic LLM. Note there is nothing special about this data itself and you can provide your own `questions` and `ground_truth` for evaluation purposes. When starting a project however, there is often a lack of quality human labeled data to be used for evaluation and a synthetic dataset is a valuable place to start if pre live user/process data (which should be incorporated as an ultimate goal).\n",
+ "\n",
+ "For more detail see [the docs](https://docs.ragas.io/en/stable/concepts/testset_generation.html)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 15,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NBVAL_SKIP\n",
+ "# source: https://docs.ragas.io/en/latest/getstarted/testset_generation.html\n",
+ "from ragas.testset.generator import TestsetGenerator\n",
+ "from ragas.testset.evolutions import simple, reasoning, multi_context\n",
+ "from ragas.run_config import RunConfig\n",
+ "from langchain_openai import ChatOpenAI, OpenAIEmbeddings\n",
+ "\n",
+ "run_config = RunConfig(\n",
+ " timeout=200,\n",
+ " max_wait=160,\n",
+ " max_retries=3,\n",
+ ")\n",
+ "\n",
+ "# generator with openai models\n",
+ "generator_llm = ChatOpenAI(model=\"gpt-3.5-turbo-16k\")\n",
+ "critic_llm = ChatOpenAI(model=\"gpt-4o-mini\")\n",
+ "embeddings = OpenAIEmbeddings()\n",
+ "\n",
+ "generator = TestsetGenerator.from_langchain(\n",
+ " generator_llm,\n",
+ " critic_llm,\n",
+ " embeddings,\n",
+ " run_config=run_config,\n",
+ ")\n",
+ "\n",
+ "testset = generator.generate_with_langchain_docs(\n",
+ " chunks,\n",
+ " test_size=10,\n",
+ " distributions={\n",
+ " simple: 0.5,\n",
+ " reasoning: 0.25,\n",
+ " multi_context: 0.25\n",
+ " },\n",
+ " run_config=run_config\n",
+ ")\n",
+ "\n",
+ "# save to csv since this can be a time consuming process\n",
+ "testset.to_pandas().to_csv(\"resources/new_testset.csv\", index=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Evaluation helper functions\n",
+ "\n",
+ "The following code takes a RetrievalQA chain, testset dataframe, and the metrics to be evaluated and returns a dataframe including the metrics calculated."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 110,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "from datasets import Dataset\n",
+ "from ragas import evaluate\n",
+ "from ragas.run_config import RunConfig\n",
+ "\n",
+ "def parse_contexts(source_docs):\n",
+ " return [doc.page_content for doc in source_docs]\n",
+ "\n",
+ "def create_evaluation_dataset(chain, testset):\n",
+ " res_set = {\n",
+ " \"question\": [],\n",
+ " \"answer\": [],\n",
+ " \"contexts\": [],\n",
+ " \"ground_truth\": []\n",
+ " }\n",
+ "\n",
+ " for _, row in testset.iterrows():\n",
+ " result = chain.invoke({\"input\": row[\"question\"]})\n",
+ "\n",
+ " res_set[\"question\"].append(row[\"question\"])\n",
+ " res_set[\"answer\"].append(result[\"answer\"])\n",
+ "\n",
+ " contexts = parse_contexts(result[\"context\"])\n",
+ "\n",
+ " if not len(contexts):\n",
+ " print(f\"no contexts found for question: {row['question']}\")\n",
+ " res_set[\"contexts\"].append(contexts)\n",
+ " res_set[\"ground_truth\"].append(str(row[\"ground_truth\"]))\n",
+ "\n",
+ " return Dataset.from_dict(res_set)\n",
+ "\n",
+ "def evaluate_dataset(eval_dataset, metrics, llm, embeddings):\n",
+ "\n",
+ " run_config = RunConfig(max_retries=1) # see ragas docs for more run_config options\n",
+ "\n",
+ " eval_result = evaluate(\n",
+ " eval_dataset,\n",
+ " metrics=metrics,\n",
+ " run_config=run_config,\n",
+ " llm=llm,\n",
+ " embeddings=embeddings\n",
+ " )\n",
+ "\n",
+ " eval_df = eval_result.to_pandas()\n",
+ " return eval_df"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Create the evaluation data\n",
+ "\n",
+ "Input: chain to be evaluated and a pregenerated test set\n", + "Output: dataset formatted for use with ragas evaluation function" + ] + }, + { + "cell_type": "code", + "execution_count": 111, + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " question \\\n",
+ "0 What are short-term investments and how are th... \n",
+ "1 What are some of the risks and uncertainties a... \n",
+ "2 What is NIKE's policy regarding securities ana... \n",
+ "3 What are the revenues for the Footwear and App... \n",
+ "4 How do master netting arrangements impact the ... \n",
+ "\n",
+ " contexts \\\n",
+ "0 [\"CASH AND EQUIVALENTS Cash and equivalents re... \n",
+ "1 ['Our NIKE Direct operations, including our re... \n",
+ "2 [\"Investors should also be aware that while NI... \n",
+ "3 ['(Dollars in millions, except per share data)... \n",
+ "4 [\"The Company records the assets and liabiliti... \n",
+ "\n",
+ " ground_truth evolution_type \\\n",
+ "0 Short-term investments are highly liquid inves... simple \n",
+ "1 Many factors unique to retail operations, some... simple \n",
+ "2 NIKE's policy is to not disclose any material ... simple \n",
+ "3 The revenues for the Footwear and Apparel cate... simple \n",
+ "4 The Company records the assets and liabilities... simple \n",
+ "\n",
+ " metadata episode_done \n",
+ "0 [{'source': 'resources/nke-10k-2023.pdf'}] True \n",
+ "1 [{'source': 'resources/nke-10k-2023.pdf'}] True \n",
+ "2 [{'source': 'resources/nke-10k-2023.pdf'}] True \n",
+ "3 [{'source': 'resources/nke-10k-2023.pdf'}] True \n",
+ "4 [{'source': 'resources/nke-10k-2023.pdf'}] True "
+ ]
+ },
+ "execution_count": 111,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "testset_df = pd.read_csv(\"resources/testset_15.csv\")\n",
+ "testset_df.head()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 112,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "eval_dataset = create_evaluation_dataset(rag_chain, testset_df)\n",
+ "eval_dataset.to_pandas().shape"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Evaluate generation metrics\n",
+ "Generation metrics quantify how well the RAG app did creating answers to the provided questions (i.e. the G in **R**etrival **A**ugments **G**eneration). We will calculate the generation metrics **faithfulness** and **answer relevancy** for this example.\n",
+ "\n",
+ "The ragas libary conveniently abstracts the calculation of these metrics so we don't have to write redundant code but please review the following definitions in order to build intuition around what these metrics actually measure.\n",
+ "\n",
+ "Note: the following examples are paraphrased from the [ragas docs](https://docs.ragas.io/en/stable/concepts/metrics/index.html)\n",
+ "\n",
+ "------\n",
+ "\n",
+ "### Faithfulness\n",
+ "\n",
+ "An answer to a question can be said to be \"faithful\" if the **claims** that are made in the answer **can be inferred** from the **context**.\n",
+ "\n",
+ "#### Mathematically:\n",
+ "\n",
+ "$$\n",
+ "Faithfullness\\ score = \\frac{Number\\ of\\ claims\\ in\\ the\\ generated\\ answer\\ that\\ can\\ be\\ inferred\\ from\\ the\\ given\\ context}{Total\\ number\\ of\\ claim\\ in\\ the\\ generated\\ answer}\n",
+ "$$\n",
+ "\n",
+ "#### Example process:\n",
+ "\n",
+ "> Question: Where and when was Einstein born?\n",
+ "> \n",
+ "> Context: Albert Einstein (born 14 March 1879) was a German-born theoretical physicist, widely held to be one of the greatest and most influential scientists of all time\n",
+ ">\n",
+ "> answer: Einstein was born in Germany on 20th March 1879.\n",
+ "\n",
+ "Step 1: Use LLM to break generated answer into individual statements.\n",
+ "- “Einstein was born in Germany.”\n",
+ "- “Einstein was born on 20th March 1879.”\n",
+ "\n",
+ "Step 2: For each statement use LLM to verify if it can be inferred from the context.\n",
+ "- “Einstein was born in Germany.” => yes. \n",
+ "- “Einstein was born on 20th March 1879.” => no.\n",
+ "\n",
+ "Step 3: plug into formula\n",
+ "\n",
+ "Number of claims inferred from context = 1\n",
+ "Total number of claims = 2\n",
+ "Faithfulness = 1/2\n",
+ "\n",
+ "### Answer Relevance\n",
+ "\n",
+ "An answer can be said to be relevant if it directly addresses the question (intuitively).\n",
+ "\n",
+ "#### Example process:\n",
+ "\n",
+ "1. Use an LLM to generate \"hypothetical\" questions to a given answer with the following prompt:\n",
+ "\n",
+ " > Generate a question for the given answer.\n",
+ " > answer: [answer]\n",
+ "\n",
+ "2. Embed the generated \"hypothetical\" questions as vectors.\n",
+ "3. Calculate the cosine similarity of the hypothetical questions and the original question, sum those similarities, and divide by n.\n",
+ "\n",
+ "With data:\n",
+ "\n",
+ "> Question: Where is France and what is it’s capital?\n",
+ "> \n",
+ "> answer: France is in western Europe.\n",
+ "\n",
+ "Step 1 - use LLM to create 'n' variants of question from the generated answer.\n",
+ "\n",
+ "- “In which part of Europe is France located?”\n",
+ "- “What is the geographical location of France within Europe?”\n",
+ "- “Can you identify the region of Europe where France is situated?”\n",
+ "\n",
+ "Step 2 - Calculate the mean cosine similarity between the generated questions and the actual question.\n",
+ "\n",
+ "## Now let's implement using our helper functions\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 114,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "dd9cabb4b0c448b08cad96d2ef3391a2",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ "Evaluating: 0%| | 0/15 [00:00\n",
+ "\n",
+ "| \n", + " | question | \n", + "contexts | \n", + "ground_truth | \n", + "evolution_type | \n", + "metadata | \n", + "episode_done | \n", + "
|---|---|---|---|---|---|---|
| 0 | \n", + "What are short-term investments and how are th... | \n", + "[\"CASH AND EQUIVALENTS Cash and equivalents re... | \n", + "Short-term investments are highly liquid inves... | \n", + "simple | \n", + "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", + "True | \n", + "
| 1 | \n", + "What are some of the risks and uncertainties a... | \n", + "['Our NIKE Direct operations, including our re... | \n", + "Many factors unique to retail operations, some... | \n", + "simple | \n", + "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", + "True | \n", + "
| 2 | \n", + "What is NIKE's policy regarding securities ana... | \n", + "[\"Investors should also be aware that while NI... | \n", + "NIKE's policy is to not disclose any material ... | \n", + "simple | \n", + "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", + "True | \n", + "
| 3 | \n", + "What are the revenues for the Footwear and App... | \n", + "['(Dollars in millions, except per share data)... | \n", + "The revenues for the Footwear and Apparel cate... | \n", + "simple | \n", + "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", + "True | \n", + "
| 4 | \n", + "How do master netting arrangements impact the ... | \n", + "[\"The Company records the assets and liabiliti... | \n", + "The Company records the assets and liabilities... | \n", + "simple | \n", + "[{'source': 'resources/nke-10k-2023.pdf'}] | \n", + "True | \n", + "
| \n", + " | faithfulness | \n", + "answer_relevancy | \n", + "
|---|---|---|
| count | \n", + "15.000000 | \n", + "15.000000 | \n", + "
| mean | \n", + "0.781229 | \n", + "0.938581 | \n", + "
| std | \n", + "0.362666 | \n", + "0.085342 | \n", + "
| min | \n", + "0.000000 | \n", + "0.736997 | \n", + "
| 25% | \n", + "0.652778 | \n", + "0.926596 | \n", + "
| 50% | \n", + "1.000000 | \n", + "0.975230 | \n", + "
| 75% | \n", + "1.000000 | \n", + "0.994168 | \n", + "
| max | \n", + "1.000000 | \n", + "1.000000 | \n", + "
| \n", + " | context_recall | \n", + "context_precision | \n", + "
|---|---|---|
| count | \n", + "15.000000 | \n", + "15.000000 | \n", + "
| mean | \n", + "0.966667 | \n", + "0.925926 | \n", + "
| std | \n", + "0.129099 | \n", + "0.145352 | \n", + "
| min | \n", + "0.500000 | \n", + "0.500000 | \n", + "
| 25% | \n", + "1.000000 | \n", + "0.916667 | \n", + "
| 50% | \n", + "1.000000 | \n", + "1.000000 | \n", + "
| 75% | \n", + "1.000000 | \n", + "1.000000 | \n", + "
| max | \n", + "1.000000 | \n", + "1.000000 | \n", + "
\n",
+ "\n",
+ "\n",
+ " \n",
+ "
\n",
+ "
"
+ ],
+ "text/plain": [
+ " context_recall context_precision faithfulness answer_relevancy\n",
+ "count 15.000000 15.000000 15.000000 15.000000\n",
+ "mean 0.966667 0.925926 0.781229 0.938581\n",
+ "std 0.129099 0.145352 0.362666 0.085342\n",
+ "min 0.500000 0.500000 0.000000 0.736997\n",
+ "25% 1.000000 0.916667 0.652778 0.926596\n",
+ "50% 1.000000 1.000000 1.000000 0.975230\n",
+ "75% 1.000000 1.000000 1.000000 0.994168\n",
+ "max 1.000000 1.000000 1.000000 1.000000"
+ ]
+ },
+ "execution_count": 121,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "metrics.describe()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Analysis\n",
+ "Overall our RAG app showed pretty good performance. All values indicated above 0.6, which from anecdotal experience, is a reasonable lower-bound for performance however obviously higher values are more ideal. It is worth noting that generation metrics can be a bit more hazy in terms of ideal ranges since the LLM evaluation cannot yet capture the way a response feels to a user. For these metrics it's important to make sure they are not severely low however blind optimization to the top can result in a very uncreative chat experience which may or may not be ideal for the intended use case.\n",
+ "\n",
+ "## Review\n",
+ "\n",
+ "- we initialized our RAG app with data from a 10k document\n",
+ "- generated a testset to evaluate \n",
+ "- calculated both retrieval and generation metrics\n",
+ "\n",
+ "## Next steps\n",
+ "\n",
+ "Now that we know how to measure our system we can quickly and easily experiment with different techniques with a baseline in place to improve our systems.\n",
+ "\n",
+ "## Cleanup"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 122,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from redisvl.index import SearchIndex\n",
+ "\n",
+ "idx = SearchIndex.from_existing(\n",
+ " index_name,\n",
+ " redis_url=REDIS_URL\n",
+ ")\n",
+ "\n",
+ "idx.delete()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.9"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
}
diff --git a/python-recipes/RAG/07_user_role_based_rag.ipynb b/python-recipes/RAG/07_user_role_based_rag.ipynb
index 34a2ee6e..f7b4466e 100644
--- a/python-recipes/RAG/07_user_role_based_rag.ipynb
+++ b/python-recipes/RAG/07_user_role_based_rag.ipynb
@@ -60,7 +60,7 @@
}
],
"source": [
- "%pip install -q 'redisvl>=0.3.8' openai langchain-community pypdf"
+ "%pip install -q \"redisvl>=0.4.1\" openai langchain-community pypdf"
]
},
{
@@ -1780,7 +1780,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.10"
+ "version": "3.11.9"
}
},
"nbformat": 4,
diff --git a/python-recipes/agents/00_langgraph_redis_agentic_rag.ipynb b/python-recipes/agents/00_langgraph_redis_agentic_rag.ipynb
index 8bc01444..f00a37ef 100644
--- a/python-recipes/agents/00_langgraph_redis_agentic_rag.ipynb
+++ b/python-recipes/agents/00_langgraph_redis_agentic_rag.ipynb
@@ -1,674 +1,674 @@
{
- "cells": [
- {
- "cell_type": "markdown",
- "id": "1VK8QKOVG2Ek",
- "metadata": {
- "id": "1VK8QKOVG2Ek"
- },
- "source": [
- "\n",
- "\n",
- "# Agentic RAG with LangGraph and Redis\n",
- "\n",
- "| \n", + " | context_recall | \n", + "context_precision | \n", + "faithfulness | \n", + "answer_relevancy | \n", + "
|---|---|---|---|---|
| count | \n", + "15.000000 | \n", + "15.000000 | \n", + "15.000000 | \n", + "15.000000 | \n", + "
| mean | \n", + "0.966667 | \n", + "0.925926 | \n", + "0.781229 | \n", + "0.938581 | \n", + "
| std | \n", + "0.129099 | \n", + "0.145352 | \n", + "0.362666 | \n", + "0.085342 | \n", + "
| min | \n", + "0.500000 | \n", + "0.500000 | \n", + "0.000000 | \n", + "0.736997 | \n", + "
| 25% | \n", + "1.000000 | \n", + "0.916667 | \n", + "0.652778 | \n", + "0.926596 | \n", + "
| 50% | \n", + "1.000000 | \n", + "1.000000 | \n", + "1.000000 | \n", + "0.975230 | \n", + "
| 75% | \n", + "1.000000 | \n", + "1.000000 | \n", + "1.000000 | \n", + "0.994168 | \n", + "
| max | \n", + "1.000000 | \n", + "1.000000 | \n", + "1.000000 | \n", + "1.000000 | \n", + "
\n",
- "\n",
- "This notebook demonstrates the implementation of a Retrieval Agent using LangGraph and LangChain components. It showcases a flexible question-answering system that combines document retrieval with language model generation. The system uses an LLM with access to a retriever tool, making decisions about when to retrieve information from an index. Redis is utilized as a vector store for efficient document retrieval and embedding storage. Key features include adaptive query rewriting, document relevance assessment, and multi-step processing. The notebook illustrates how LangGraph can be used to create a sophisticated workflow for handling complex queries, integrating retrieval, reasoning, and generation capabilities in a single system.\n",
- "\n",
- "[Retrieval Agents](https://python.langchain.com/docs/tutorials/qa_chat_history/#agents) are useful when we want to make decisions about whether to retrieve from an index.\n",
- "\n",
- "To implement a retrieval agent, we simply need to give an LLM access to a retriever tool.\n",
- "\n",
- "We can incorporate this into [LangGraph](https://langchain-ai.github.io/langgraph/).\n",
- "\n",
- ""
- ]
- },
- {
- "cell_type": "markdown",
- "id": "425fb020-e864-40ce-a31f-8da40c73d14b",
- "metadata": {
- "id": "425fb020-e864-40ce-a31f-8da40c73d14b"
- },
- "source": [
- "## Setup\n",
- "\n",
- "First, let's download the required packages and set our API keys:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "id": "969fb438",
- "metadata": {
- "id": "969fb438"
- },
- "outputs": [],
- "source": [
- "%pip install -q langchain-community tiktoken langchain-openai langchainhub langchain-redis langchain langgraph langchain-text-splitters bs4"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "id": "e4958a8c",
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "1VK8QKOVG2Ek",
+ "metadata": {
+ "id": "1VK8QKOVG2Ek"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "# Agentic RAG with LangGraph and Redis\n",
+ "\n",
+ "\n",
+ "\n",
+ "This notebook demonstrates the implementation of a Retrieval Agent using LangGraph and LangChain components. It showcases a flexible question-answering system that combines document retrieval with language model generation. The system uses an LLM with access to a retriever tool, making decisions about when to retrieve information from an index. Redis is utilized as a vector store for efficient document retrieval and embedding storage. Key features include adaptive query rewriting, document relevance assessment, and multi-step processing. The notebook illustrates how LangGraph can be used to create a sophisticated workflow for handling complex queries, integrating retrieval, reasoning, and generation capabilities in a single system.\n",
+ "\n",
+ "[Retrieval Agents](https://python.langchain.com/docs/tutorials/qa_chat_history/#agents) are useful when we want to make decisions about whether to retrieve from an index.\n",
+ "\n",
+ "To implement a retrieval agent, we simply need to give an LLM access to a retriever tool.\n",
+ "\n",
+ "We can incorporate this into [LangGraph](https://langchain-ai.github.io/langgraph/).\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "425fb020-e864-40ce-a31f-8da40c73d14b",
+ "metadata": {
+ "id": "425fb020-e864-40ce-a31f-8da40c73d14b"
+ },
+ "source": [
+ "## Setup\n",
+ "\n",
+ "First, let's download the required packages and set our API keys:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "969fb438",
+ "metadata": {
+ "id": "969fb438"
+ },
+ "outputs": [],
+ "source": [
+ "%pip install -q langchain-community tiktoken langchain-openai langchainhub \"langchain-redis>=0.2.0\" langchain langgraph langchain-text-splitters bs4"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "e4958a8c",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "e4958a8c",
+ "outputId": "276c5d89-a4d7-4c79-d307-b619a5489830"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "OPENAI_API_KEY:··········\n"
+ ]
+ }
+ ],
+ "source": [
+ "import getpass\n",
+ "import os\n",
+ "\n",
+ "\n",
+ "def _set_env(key: str):\n",
+ " if key not in os.environ:\n",
+ " os.environ[key] = getpass.getpass(f\"{key}:\")\n",
+ "\n",
+ "\n",
+ "_set_env(\"OPENAI_API_KEY\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "Po4K08Uoa5HJ",
+ "metadata": {
+ "id": "Po4K08Uoa5HJ"
+ },
+ "source": [
+ "### Setup Redis"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "VLy0onoAa7KI",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "VLy0onoAa7KI",
+ "outputId": "b346e76e-e87d-437f-c9fa-78647db77f4e"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb jammy main\n",
+ "Starting redis-stack-server, database path /var/lib/redis-stack\n"
+ ]
+ }
+ ],
+ "source": [
+ "# NBVAL_SKIP\n",
+ "%%sh\n",
+ "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
+ "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
+ "sudo apt-get update > /dev/null 2>&1\n",
+ "sudo apt-get install redis-stack-server > /dev/null 2>&1\n",
+ "redis-stack-server --daemonize yes"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "7c2KKPhOh4zM",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "7c2KKPhOh4zM",
+ "outputId": "0e314576-b34e-4881-ddf0-80d686810091"
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Connecting to Redis at: redis://localhost:6379\n"
+ ]
+ }
+ ],
+ "source": [
+ "# Use the environment variable if set, otherwise default to localhost\n",
+ "REDIS_URL = os.getenv(\"REDIS_URL\", \"redis://localhost:6379\")\n",
+ "print(f\"Connecting to Redis at: {REDIS_URL}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "c74e4532",
+ "metadata": {
+ "id": "c74e4532"
+ },
+ "source": [
+ "## Retriever\n",
+ "\n",
+ "First, we index 3 blog posts. For this we setup a retriever using Redis as a vector store."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "id": "e50c9efe-4abe-42fa-b35a-05eeeede9ec6",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "e50c9efe-4abe-42fa-b35a-05eeeede9ec6",
+ "outputId": "f3ab6120-eb1e-4de8-dcc6-0abb7fe9201b"
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "WARNING:langchain_community.utils.user_agent:USER_AGENT environment variable not set, consider setting it to identify your requests.\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "18:31:28 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
+ "18:31:28 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n",
+ "18:31:30 httpx INFO HTTP Request: POST https://api.openai.com/v1/embeddings \"HTTP/1.1 200 OK\"\n"
+ ]
+ }
+ ],
+ "source": [
+ "from langchain_community.document_loaders import WebBaseLoader\n",
+ "\n",
+ "from langchain_redis import RedisVectorStore\n",
+ "from langchain_openai import OpenAIEmbeddings\n",
+ "from langchain_text_splitters import RecursiveCharacterTextSplitter\n",
+ "\n",
+ "urls = [\n",
+ " \"https://lilianweng.github.io/posts/2023-06-23-agent/\",\n",
+ " \"https://lilianweng.github.io/posts/2023-03-15-prompt-engineering/\",\n",
+ " \"https://lilianweng.github.io/posts/2023-10-25-adv-attack-llm/\",\n",
+ "]\n",
+ "\n",
+ "docs = [WebBaseLoader(url).load() for url in urls]\n",
+ "docs_list = [item for sublist in docs for item in sublist]\n",
+ "\n",
+ "text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(\n",
+ " chunk_size=100, chunk_overlap=50\n",
+ ")\n",
+ "doc_splits = text_splitter.split_documents(docs_list)\n",
+ "\n",
+ "# Add to document chunks to Redis\n",
+ "vectorstore = RedisVectorStore.from_documents(\n",
+ " doc_splits,\n",
+ " OpenAIEmbeddings(),\n",
+ " redis_url=REDIS_URL,\n",
+ " index_name=\"rag-redis\"\n",
+ ")\n",
+ "# get RedisVectorStore as a retriever\n",
+ "retriever = vectorstore.as_retriever()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "225d2277-45b2-4ae8-a7d6-62b07fb4a002",
+ "metadata": {
+ "id": "225d2277-45b2-4ae8-a7d6-62b07fb4a002"
+ },
+ "source": [
+ "Then we create a retriever tool."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "0b97bdd8-d7e3-444d-ac96-5ef4725f9048",
+ "metadata": {
+ "id": "0b97bdd8-d7e3-444d-ac96-5ef4725f9048"
+ },
+ "outputs": [],
+ "source": [
+ "from langchain.tools.retriever import create_retriever_tool\n",
+ "\n",
+ "retriever_tool = create_retriever_tool(\n",
+ " retriever,\n",
+ " \"retrieve_blog_posts\",\n",
+ " \"Search and return information about Lilian Weng blog posts on LLM agents, prompt engineering, and adversarial attacks on LLMs.\",\n",
+ ")\n",
+ "\n",
+ "tools = [retriever_tool]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "fe6e8f78-1ef7-42ad-b2bf-835ed5850553",
+ "metadata": {
+ "id": "fe6e8f78-1ef7-42ad-b2bf-835ed5850553"
+ },
+ "source": [
+ "## Agent State\n",
+ "\n",
+ "We will define a graph.\n",
+ "\n",
+ "A `state` object that it passes around to each node.\n",
+ "\n",
+ "Our state will be a list of `messages`.\n",
+ "\n",
+ "Each node in our graph will append to it."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 7,
+ "id": "0e378706-47d5-425a-8ba0-57b9acffbd0c",
+ "metadata": {
+ "id": "0e378706-47d5-425a-8ba0-57b9acffbd0c"
+ },
+ "outputs": [],
+ "source": [
+ "from typing import Annotated, Sequence, TypedDict\n",
+ "\n",
+ "from langchain_core.messages import BaseMessage\n",
+ "\n",
+ "from langgraph.graph.message import add_messages\n",
+ "\n",
+ "\n",
+ "class AgentState(TypedDict):\n",
+ " # The add_messages function defines how an update should be processed\n",
+ " # Default is to replace. add_messages says \"append\"\n",
+ " messages: Annotated[Sequence[BaseMessage], add_messages]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "dc949d42-8a34-4231-bff0-b8198975e2ce",
+ "metadata": {
+ "id": "dc949d42-8a34-4231-bff0-b8198975e2ce"
+ },
+ "source": [
+ "## Nodes and Edges\n",
+ "\n",
+ "We can lay out an agentic RAG graph like this:\n",
+ "\n",
+ "* The state is a set of messages\n",
+ "* Each node will update (append to) state\n",
+ "* Conditional edges decide which node to visit next\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "278d1d83-dda6-4de4-bf8b-be9965c227fa",
+ "metadata": {
+ "id": "278d1d83-dda6-4de4-bf8b-be9965c227fa"
+ },
+ "outputs": [],
+ "source": [
+ "from typing import Annotated, Literal, Sequence, TypedDict\n",
+ "\n",
+ "from langchain_core.messages import BaseMessage, HumanMessage\n",
+ "from langchain_core.output_parsers import StrOutputParser\n",
+ "from langchain_core.prompts import PromptTemplate, ChatPromptTemplate\n",
+ "from langchain_openai import ChatOpenAI\n",
+ "# NOTE: you must use langchain-core >= 0.3 with Pydantic v2\n",
+ "from pydantic import BaseModel, Field\n",
+ "\n",
+ "\n",
+ "from langgraph.prebuilt import tools_condition\n",
+ "\n",
+ "### Edges\n",
+ "\n",
+ "\n",
+ "def grade_documents(state) -> Literal[\"generate\", \"rewrite\"]:\n",
+ " \"\"\"\n",
+ " Determines whether the retrieved documents are relevant to the question.\n",
+ "\n",
+ " Args:\n",
+ " state (messages): The current state\n",
+ "\n",
+ " Returns:\n",
+ " str: A decision for whether the documents are relevant or not\n",
+ " \"\"\"\n",
+ "\n",
+ " print(\"---CHECK RELEVANCE---\")\n",
+ "\n",
+ " # Data model\n",
+ " class grade(BaseModel):\n",
+ " \"\"\"Binary score for relevance check.\"\"\"\n",
+ "\n",
+ " binary_score: str = Field(description=\"Relevance score 'yes' or 'no'\")\n",
+ "\n",
+ " # LLM\n",
+ " model = ChatOpenAI(temperature=0, model=\"gpt-4-0125-preview\", streaming=True)\n",
+ "\n",
+ " # LLM with tool and validation\n",
+ " llm_with_tool = model.with_structured_output(grade)\n",
+ "\n",
+ " # Prompt\n",
+ " prompt = PromptTemplate(\n",
+ " template=\"\"\"You are a grader assessing relevance of a retrieved document to a user question. \\n\n",
+ " Here is the retrieved document: \\n\\n {context} \\n\\n\n",
+ " Here is the user question: {question} \\n\n",
+ " If the document contains keyword(s) or semantic meaning related to the user question, grade it as relevant. \\n\n",
+ " Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question.\"\"\",\n",
+ " input_variables=[\"context\", \"question\"],\n",
+ " )\n",
+ "\n",
+ " # Chain\n",
+ " chain = prompt | llm_with_tool\n",
+ "\n",
+ " messages = state[\"messages\"]\n",
+ " last_message = messages[-1]\n",
+ "\n",
+ " question = messages[0].content\n",
+ " docs = last_message.content\n",
+ "\n",
+ " scored_result = chain.invoke({\"question\": question, \"context\": docs})\n",
+ "\n",
+ " score = scored_result.binary_score\n",
+ "\n",
+ " if score == \"yes\":\n",
+ " print(\"---DECISION: DOCS RELEVANT---\")\n",
+ " return \"generate\"\n",
+ "\n",
+ " else:\n",
+ " print(\"---DECISION: DOCS NOT RELEVANT---\")\n",
+ " print(score)\n",
+ " return \"rewrite\"\n",
+ "\n",
+ "\n",
+ "### Nodes\n",
+ "\n",
+ "\n",
+ "def agent(state):\n",
+ " \"\"\"\n",
+ " Invokes the agent model to generate a response based on the current state. Given\n",
+ " the question, it will decide to retrieve using the retriever tool, or simply end.\n",
+ "\n",
+ " Args:\n",
+ " state (messages): The current state\n",
+ "\n",
+ " Returns:\n",
+ " dict: The updated state with the agent response appended to messages\n",
+ " \"\"\"\n",
+ " print(\"---CALL AGENT---\")\n",
+ " messages = state[\"messages\"]\n",
+ " model = ChatOpenAI(temperature=0, streaming=True, model=\"gpt-4-turbo\")\n",
+ " model = model.bind_tools(tools)\n",
+ " response = model.invoke(messages)\n",
+ " # We return a list, because this will get added to the existing list\n",
+ " return {\"messages\": [response]}\n",
+ "\n",
+ "\n",
+ "def rewrite(state):\n",
+ " \"\"\"\n",
+ " Transform the query to produce a better question.\n",
+ "\n",
+ " Args:\n",
+ " state (messages): The current state\n",
+ "\n",
+ " Returns:\n",
+ " dict: The updated state with re-phrased question\n",
+ " \"\"\"\n",
+ "\n",
+ " print(\"---TRANSFORM QUERY---\")\n",
+ " messages = state[\"messages\"]\n",
+ " question = messages[0].content\n",
+ "\n",
+ " msg = [\n",
+ " HumanMessage(\n",
+ " content=f\"\"\" \\n\n",
+ " Look at the input and try to reason about the underlying semantic intent / meaning. \\n\n",
+ " Here is the initial question:\n",
+ " \\n ------- \\n\n",
+ " {question}\n",
+ " \\n ------- \\n\n",
+ " Formulate an improved question: \"\"\",\n",
+ " )\n",
+ " ]\n",
+ "\n",
+ " # Grader\n",
+ " model = ChatOpenAI(temperature=0, model=\"gpt-4-0125-preview\", streaming=True)\n",
+ " response = model.invoke(msg)\n",
+ " return {\"messages\": [response]}\n",
+ "\n",
+ "\n",
+ "def generate(state):\n",
+ " \"\"\"\n",
+ " Generate answer\n",
+ "\n",
+ " Args:\n",
+ " state (messages): The current state\n",
+ "\n",
+ " Returns:\n",
+ " dict: The updated state with re-phrased question\n",
+ " \"\"\"\n",
+ " print(\"---GENERATE---\")\n",
+ " messages = state[\"messages\"]\n",
+ " question = messages[0].content\n",
+ " last_message = messages[-1]\n",
+ "\n",
+ " docs = last_message.content\n",
+ "\n",
+ " # Prompt\n",
+ " prompt = ChatPromptTemplate.from_messages(\n",
+ " [\n",
+ " (\n",
+ " \"system\",\n",
+ " \"You are an assistant for question-answering tasks. Use the following pieces of retrieved context to answer the question. If you don't know the answer, just say that you don't know. Use three sentences maximum and keep the answer concise.\",\n",
+ " ),\n",
+ " (\"system\", \"Context: {context}\"),\n",
+ " (\"human\", \"Question: {question} \"),\n",
+ " ]\n",
+ " )\n",
+ "\n",
+ " # LLM\n",
+ " llm = ChatOpenAI(model_name=\"gpt-3.5-turbo\", temperature=0, streaming=True)\n",
+ "\n",
+ " # Chain\n",
+ " rag_chain = prompt | llm | StrOutputParser()\n",
+ "\n",
+ " # Run\n",
+ " response = rag_chain.invoke({\"context\": docs, \"question\": question})\n",
+ " return {\"messages\": [response]}"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "955882ef-7467-48db-ae51-de441f2fc3a7",
+ "metadata": {
+ "id": "955882ef-7467-48db-ae51-de441f2fc3a7"
+ },
+ "source": [
+ "## Graph\n",
+ "\n",
+ "* Start with an agent, `call_model`\n",
+ "* Agent make a decision to call a function\n",
+ "* If so, then `action` to call tool (retriever)\n",
+ "* Then call agent with the tool output added to messages (`state`)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "id": "8718a37f-83c2-4f16-9850-e61e0f49c3d4",
+ "metadata": {
+ "id": "8718a37f-83c2-4f16-9850-e61e0f49c3d4"
+ },
+ "outputs": [],
+ "source": [
+ "from langgraph.graph import END, StateGraph, START\n",
+ "from langgraph.prebuilt import ToolNode\n",
+ "\n",
+ "# Define a new graph\n",
+ "workflow = StateGraph(AgentState)\n",
+ "\n",
+ "# Define the nodes we will cycle between\n",
+ "workflow.add_node(\"agent\", agent) # agent\n",
+ "retrieve = ToolNode([retriever_tool])\n",
+ "workflow.add_node(\"retrieve\", retrieve) # retrieval\n",
+ "workflow.add_node(\"rewrite\", rewrite) # Re-writing the question\n",
+ "workflow.add_node(\n",
+ " \"generate\", generate\n",
+ ") # Generating a response after we know the documents are relevant\n",
+ "# Call agent node to decide to retrieve or not\n",
+ "workflow.add_edge(START, \"agent\")\n",
+ "\n",
+ "# Decide whether to retrieve\n",
+ "workflow.add_conditional_edges(\n",
+ " \"agent\",\n",
+ " # Assess agent decision\n",
+ " tools_condition,\n",
+ " {\n",
+ " # Translate the condition outputs to nodes in our graph\n",
+ " \"tools\": \"retrieve\",\n",
+ " END: END,\n",
+ " },\n",
+ ")\n",
+ "\n",
+ "# Edges taken after the `action` node is called.\n",
+ "workflow.add_conditional_edges(\n",
+ " \"retrieve\",\n",
+ " # Assess agent decision\n",
+ " grade_documents,\n",
+ ")\n",
+ "workflow.add_edge(\"generate\", END)\n",
+ "workflow.add_edge(\"rewrite\", \"agent\")\n",
+ "\n",
+ "# Compile\n",
+ "graph = workflow.compile()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "7b5a1d35",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 473
+ },
+ "id": "7b5a1d35",
+ "outputId": "7b95dcbe-5a26-42b5-9708-8a1020564622"
+ },
+ "outputs": [
+ {
+ "data": {
+ "image/jpeg": "/9j/4AAQSkZJRgABAQAAAQABAAD/4gHYSUNDX1BST0ZJTEUAAQEAAAHIAAAAAAQwAABtbnRyUkdCIFhZWiAH4AABAAEAAAAAAABhY3NwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQAA9tYAAQAAAADTLQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAlkZXNjAAAA8AAAACRyWFlaAAABFAAAABRnWFlaAAABKAAAABRiWFlaAAABPAAAABR3dHB0AAABUAAAABRyVFJDAAABZAAAAChnVFJDAAABZAAAAChiVFJDAAABZAAAAChjcHJ0AAABjAAAADxtbHVjAAAAAAAAAAEAAAAMZW5VUwAAAAgAAAAcAHMAUgBHAEJYWVogAAAAAAAAb6IAADj1AAADkFhZWiAAAAAAAABimQAAt4UAABjaWFlaIAAAAAAAACSgAAAPhAAAts9YWVogAAAAAAAA9tYAAQAAAADTLXBhcmEAAAAAAAQAAAACZmYAAPKnAAANWQAAE9AAAApbAAAAAAAAAABtbHVjAAAAAAAAAAEAAAAMZW5VUwAAACAAAAAcAEcAbwBvAGcAbABlACAASQBuAGMALgAgADIAMAAxADb/2wBDAAMCAgMCAgMDAwMEAwMEBQgFBQQEBQoHBwYIDAoMDAsKCwsNDhIQDQ4RDgsLEBYQERMUFRUVDA8XGBYUGBIUFRT/2wBDAQMEBAUEBQkFBQkUDQsNFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBQUFBT/wAARCAHIAUADASIAAhEBAxEB/8QAHQABAAMAAwEBAQAAAAAAAAAAAAUGBwMECAIBCf/EAFcQAAEEAQIDAgcICw0GBQUAAAEAAgMEBQYRBxIhEzEUFSIyQVGUCBZCVVZh0dMXIzZUcXR1k7K01CYzNDdSU3KBkZWxs9IkNWJkocEJGEN2pESiwsPw/8QAGgEBAQADAQEAAAAAAAAAAAAAAAECAwQFBv/EADcRAQABAgEHCgUDBQEAAAAAAAABAhEDEiExUWFx0QQTFCMzQVKRocEFY6Kx8CJikhUyQlOB4f/aAAwDAQACEQMRAD8A/qmiIgIiICIiAiIgIiICIiAiKGzWYsx2osZi2Mlys7DIHzNLoa0e+3aS7EE9ejWAgvIIBAa9zcqaZrm0CWllZDGXyPbGxve5x2A/rUe7VGGaSHZeiCPQbLPpUdFoHFTyixlmOz9zcntsntKG79NmR7cjBt08lo+ffcld8aUwjQAMPQAHQAVWfQt1sGNMzP5+alzP331YX44oe0s+lPfVhfjih7Sz6U96uF+J6HszPoT3q4X4noezM+hOp2+i5j31YX44oe0s+lPfVhfjih7Sz6U96uF+J6HszPoT3q4X4noezM+hOp2+hmPfVhfjih7Sz6U99WF+OKHtLPpT3q4X4noezM+hPerhfieh7Mz6E6nb6GZ2KmZx99/JVvVrL/5MMzXH/oV3FB3NDacyEfJYwONmb6OapHuOu/Q7bg79dwuk7F3dINNjGS28ji2DeXFzPM0kbf5UD3HmJH824kEdG8u2xZGHVmonPt4/m9LQtKLhp3IchUhs1pGzQTMD45Gno5p6grmWiYtmlBERQEREBERAREQEREBERAREQEREBERAREQEREBERAREQFV9BbZChczb9nT5S1JKHeqFjjHC35gGNadh05nOPpJNoVY4bjsNJVqLtxLj5JaUgI22McjmA/gIAI9YIK6Kc2FVMa48s/vEL3LOiIudENrDWOG0Bpy7n9QZCPGYim0OnsygkN3cGtAABLiXOAAAJJIAG5WXa691RpjS+mdN5zGxXsvSy+ehwzyMbcZJX3I7Vxi7Ayc7WkFsZaC8nyd9iFbuOWIxGc4YZinnMLmM9jnmEvqafjc++1wmYWSwhpDuaNwbJ06+Qeju44NbPEPUHCjF5TMYjUOfh0xr2lkqHhuNEOavYiB7CZJKrQC6UF8g25WueGb8oJ6ht2qvdDaC0RXxk2cy9nHsyNRt+ESYq4XMgPdJM0REwjv37UN22O+2y7equOeh9GW8VVymbAs5am6/j4adWe265A0s3dEIWP5z5bTyt3JG7gNgSMY4rZjUWvNTyx2cRxCj0hkNPjxLjcBVnpOnyDnyslbfc0tdDs0Q8rZnNjLXOJ3O4TghpPNQar4H2clgMnTbhuHtnGW5LtKSMVLbJKkfZuLhs1xEcnL/ACmgkbjqg0bSHujMNq3jBqHQjKORry49tXwW0/G3A2w6SKSWTtCYQ2ANDAGl7hzknlJ7lrqw/C2Mhof3TGt33tPZqzjdW18T4vytCi+xUjdAyWOVs8jRtCQXNPlbAgrcEBERBWNLbYzUGoMK3YQQyR367Bv5Ec/PzN/OxzH5g4D0KzqsYZvheu9SXGg9nDXqY/cjYF7O1mdsfT0sM/sPzqzrox/777I+0LOkREXOgiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAq3kIZdNZazmK0D56FsNOQghY58rXtAa2djRvzHlAa5oG5axpb1byusiLZRXkTsnSsK1qHSOkuKeIqtzWJxOqcYx/bVxcgjtRB+xbzN3BG+xI3Hzquf+WzhPtt9jfS23q8UQf6VaL+h8XcuS3IWz4y7KS6SxjbD67pCRtu8MIa87bdXA9w9QXXOiZ/RqjPNHq7aI/wCMS2ZGFOiq2+OH/hmdXS3BrQeh8qMnp7R2DwmRDHRi3j8fFDLynvbzNaDsdh0VyVX95Nj5VZ789D9UnvJsfKrPfnofqk5vD8fpJaNa0Isr4i4/K6Xp4OWjqnMF9zNUaEvbSwkdlLM1j9vtY8rYnb5/QVbPeTY+VWe/PQ/VJzeH4/SS0a09lMXTzeNt47IVYbtC3E6CxWsMD45Y3AhzHNPQggkEH1qhR+5v4UwyNezhxpdj2kOa5uJgBBHcQeVT/vJsfKrPfnofqk95Nj5VZ789D9UnN4fj9JLRrQdX3OvC2jahs1+HemILELxJHLHioGuY4HcEEN6EEbq0ZnUgr2TjMaI7uce3dtbm8mEHukmI8xn/AFdts3crpnQjZgG289nLce2xYbxhDvwmIMP/AFU1iMJQwNTwbH1IqkO5cWxt25nHvc497ifST1KWwqM98qfTj+aTND4wGFjwONZVY8zSF75Zp3DZ0sr3Fz3n8JJ6egbDuCkURaaqpqmap0ygiIsQREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERBnvGcgY3Su5I/dPiu78Zb84WhLPeM+/i3Su233T4rztvvlvrWhICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiDPONI3xmlPKDf3UYrvH/MtWhrPONO3izSm/T91GK9G/8A9S1aGgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiruoNT2KN0Y7F047+S7MTSCeUxQwsJIaXvDXHckHZoB32O+w6qK8e6w+8MH7XN9Wumnk9dUXzRvmFsu6KkePdYfeGD9rm+rTx7rD7wwftc31az6LXrjzgs80e7M913a4M65xGlr2hZLtSC3RzlPKNyTWNtsikDns5DC7kIe1zd9z6D6dl6c4Pa8u8UOGen9V38HJpufLV/Cm42WftnRRlx7N3Pyt35mcj+4bc23XbdY/7oPgPa90XU07DqCnia7sNfbaZLXtS88sR27WAns+jXgN6+ggFaxBltWVYI4YcZgYYY2hjI47MrWtaBsAAIugA9CdFr1x5wWXpFSPHusPvDB+1zfVp491h94YP2ub6tOi16484LLuipbNR6prHtLOIxlmFvV0dO48SkenlD4w0n1Alo+cK1YvJ1szjq96pJ2tawwSMcQWnY+sHqD6CD1B6FasTBrw4vOjZnLO0iItCCIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgocB319qbf0R1B/VyOUyoav932p/6FT9BymV69f+O6n7QynSIiLWxEREBF0cvnMfgYIZsldgoxTTx1Y32JAwPlkcGRxt373OcQAB1JK7yAutwuJOjofmuXQNvQBalAXZXV4W/cdF+O3v1uZTF7Cd8fape5bURF5qCIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgodf7vtT/wBCp+g5TKhq/wB32p/6FT9BymV69f8Ajup+0MqtLzrx/uZTUur72E0tZ1K3NYXCeMbUmO1CcTRpte6TsnvDY3meUmN/kEcnK3qRuqjk+JF/U9ThjZ1pqfUGmtPZnRTMk27pp8sMlzMERl7XGFpd0Y7nZH5ri4gg7bL0Jq/g1o7Xuchy+dwrb1+ODwUyCxLE2aHmLuymYx4bMzck8kgc3qenUrN+JXud5bcWmqmjcJhfFuHqS044shnMrQmhjc8PDWTVpCXRgjzHg7dOUtA2XNMTpYszxF7idnZuHXD+4chDkWaRGdyMMmpJ8RbtWHWDGe0sNimlJY3lLohygGXr0aGq0eJ+IEGouFekNX6oyFQ5C9mWTvw2Xe6axTjgEsEc1gRxF8jduUyBjXbDcEFxK0XB+59xmV4eaYwvECV+rc5hRK6PMttWILMRe9x5I52yCblDS2Pq/dwYC7qrhjOFelsO/TLqeKbA7TYnGL5ZpP8AZ+2aWynq7yy4E7l/Mdzv39UimR5g1hUtai4f2MJl85mrsGneK1PD1Lj8lM2z4K6es5okla4Oe5gncGvcS5pa0g7gFevMJiYsDiamPhntWYq0YjbLesvsTPA9L5HkuefncSVXL/CDSGUwWosNbw0djG6gunJZKCSaQ9tZIjHag827HDsoyOQt2LQRseqndM6aoaQwdXEYuOWKjWDhG2exJYeN3Fx3kkc57upPeSsoiwlF1eFv3HRfjt79bmXaXV4W/cdF+O3v1uZZYvYTvj7VL3LaiIvNQREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQERQmX1RDSlt0sfF45zdeOGV+KqzxtmYyV5YyR/O4BjPJedz1Ijfyh5HKQr1f7vtT/wBCp+g5TKh7mntQYvN5PNQugzXhrww0Imis6KFgIi5HOcQ9/V3NzFoO425eXZ3z42z3yNyntVL69evFsSIqiY0RGeYjRFu+WUxdNIoTxtnvkZlfaqX16eNs98jMr7VS+vTI/dH8o4lk2iz3XnGOtwxq42xqfCXsRDkrsePqvmsVD2k7/Nb5Mx2HTq47NHpIVn8bZ75GZX2ql9emR+6P5RxLJtFCeNs98jMr7VS+vTxtnvkZlfaqX16ZH7o/lHEsm1W+HFTOVtLxS0L0FuKbN2ZJYMgzlEFXwiVsjIXRgHm5h2gL+bclzdwC0t7bbmo7m8UOl7FKV3QTX7VfsmfORHI9xA9QHX1q1adwrNPYWrj2Sun7IEulcNi97iXOdt6N3Enb51qxpinCmi8XmYnNMToidW80Q6NfWEcc1avlcfcw1m1ckp1mTsErJi0bteJIi5jWvb1aHlrj1HKCNlOV7EVuBk0ErJoZBzMkjcHNcPWCO9cir7NEY2lJSfixLg20/CDFXxjzBWc6brI58Dftcjufyw5zSQ7cg+U7m81isCKtw2NRYZkTLkFfO1YMe6Sa5UPY2prLT0Y2uQWbPb6e1Gzhttsd293FaqxmXtspRWBDkzUivPxtj7Xaihk81z4j5TeoLT06EEd4QS6IiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgLguX62PZG+1YirMkkZCx0zwwOe4hrGDfvcSQAO8k7KJzGoJ2z28Zh67bmbZUFmNtkSR1W7vLG88waQDuHnlG7tmHp3LnradhbkJrtyaTI2HTtsQizs6Om5sZj+0N28jcOfu7q49o4FxGwAdGKfKapZDJEyXCYmWOxFMyzGWXnnfkjfHs7aIbbvHMC7qzdrDuFN4/HwYypHXrsLY42tYC95e9wa0NBc5xLnHZoG7iSdupXZRAREQEREHhz3e/ufeJXGfV+mruPy2BpaWrWK+Mx9axbnbP4VYka100jWwlrRzco6OJ5WA7bkheteEmJ1RgeG2nsZrO1Svamp1W17lrHSPfDMWEta8Oe1riSwNLt2jyifwqK41BpxmlOYkD30YruG/Xwlq0NAREQEREBERAXRzWDoaixtjH5OpFdp2GGOSKVu4c0kH+rqAdx3EA94XeRBXbOEzGNFybDZUzPkFdsNDKjtK0LY+knI9oEodI3vL3PAc0EN6uDvqTV8WOndHmKVjDtkyDMfUnm5ZIrbnjeN7TGXcgcfI+2Bh5xtseZhdYEQfgIcAQQQeoIX6q/Fouljp4ZMM9+BYL0l+xXx7I2Q25JP33tWFpB5j5Zc3ldzdd+rt/mrm8rjPAK2bx/byzeEGXI4tu9SFsflMMjXO7RhezfYAPaHNILurOYLEi6mKy1HO42tkcZcr5HH2oxLBbqStlimYRuHMe0kOBHcQdl20BERAREQEREBERAXxNMyBhfI4MaPSV9qNz/APu1/wDSH+KDn8a1Pvhn9qeNan3wz+1YDrjjpjNC8UtPaPu0rsgylKe2+3Wo2bHZFr2NjaGxRO5uYudzO38jlHNtztKktUcdNDaMz8mGzGfjqX4RGbDRBLJHVEnmdvKxhZDzejtHN3B37kG2eNan3wz+1PGtT74Z/asW1lxu0ToDKjG53ONqXBCLEkcVeacQRE7CSZ0bHCJh2PlPLR07104uKJbxXzOCsTUI9OUdN1s42/ueYmSWdryX83L2YZCHDYes7kdwbo/L02NLnWWAD51FxS5LUNlkjO0xOLhnnilikaDNeYGcjHse1/2pnMXOB848rD5IJBzbQPFDTXE6KzNp6/JeZXbG6QSVJoAGv5ix7e1Y3na7lOzm7g7dFr+L/wB3V/6AQfmKxVTB42rj6FdlWlWjbFDDGNmsaBsAu2iICIiAiIgIiIM+4yuc3HaV5QSffPix037vCG79y0FZ5xq28WaU32H7qMV3/jLVoaAiIgIiICIiAiIgIiICIiCDyWl2T2rGQx9mXGZd1OSpFYY97oW8zucPfX5hHI4P3IcRzAOeA4B7t+GXU02BFg6ghZTpQisxuWjP2ieSTyXbs3LoQ1+3nEt2e0858oNsSICKCi007E2xLhZ20IZ7z7l+tI10rLBe3Z/Ju77U7mDX+T5JPOS0l5cuXB6jjyjYK9qE4vMvrixLibErHTxN5iwnySQ5vMCOYEg9PWgmEREBERAREQFG5/8A3a/+kP8AFSS6OZgks0XMjbzvJHQfhQeb+K8t/S/GDQerm4PLZvD1KGSx9rxNTfbmgkm8HdEXRs3dynsnDmA2B2323VE1BBmtJ4vjJpQ6MzmoMhrO5atYm7SpGWrOy1VjhayabzYeyc1wIkLfJG7d916r8TXP5h39oTxNc/mHf2hB5SxGNznBebXWLyelc3q+xqDGUGUbuJpOtRWZIqDKr4JnjpEA9jnAv2byyE777hcmh9A5fh/qDHYvU2nslqDG2+HFXCWX0Ie3hM9ft3TVXOBHKXMkDWEkBxOwO69Quxs77ba/ZtMrWdqY+0HOBvsDy777b79fWF2PE1z+Yd/aEGB+53sZ+tk81h21tRs0FRq1m4d2raRrXq8nliSs0kB0sTGhmz3Akb7czgN16gxf+7q/9AKt+Jrn8w7+0KzY+N0NKFjxyua0AhB2EREBERAREQEREGecaXcuM0p1Lf3UYodPxlq0NZ5xpJGN0ptt91GK7wPvlvrWhoCIiAiIgIiICIiAiIgIiICIiAo7NYWLM05ou2mpWXROiiv1CGWK/MWndjiCO9jDsQWnlAcHDopFEEPBm3VMg2jlXV6k9mw6HHu7X+GAR9oQAe6QNEhLNyS2Jzh0B5ZhcF2my/Vlge57A9paJIncr2EjbmaR3EegqPwdq0x82Nux2HTVGsZHesviJvs5G7zbRhoaeYkOHI0A9w5SCgl0REBERAREQEREFer0yOIF62aFRoOMrxNvNk3sP2lmJjc3fowbgg7dS53q6WFZPV4wcO28QL+Q99+jWxTYytA3Itz9czyubLO4xFnabBjecOB9JkcPQtYQEREBERAREQEREBERBnfGrbxZpTckfuoxXcP+ZatEWd8a9vFmlN/lTiv1lq0RAREQEREBERAREQEREBERAREQEREBQmqcPPfqsu42vQk1BRD5MdNkGu7Njy3ZzS5nlNa8eSSN9uh5XcoBm0QdPEZiln8bBfx1mO3TmBMc0R3B2JBHzEEEEHqCCD1C7ireIuNx2rMrhpskyeWyBk6dFtPsvB4DyskHaAcsm8oc8k+UDL13HKVZEBQuY1tp7T9oVsnnMdj7JHN2Nm0xj9vXyk77Lu5q47H4e9aYAXwQSStB9bWkj/BVHSVSOtgKUgHNPZiZPPM7q+aRzQXPcT1JJP8AV3dwXXg4VNVM116NixrlJfZS0d8qcR7bH9KfZS0d8qcR7bH9K5kW7msHVPnHBczh+ylo75U4j22P6U+ylo75U4j22P6VzInNYOqfOOBmfzq0X7l7S+J92dasWcrjDw2xsvjypPJajMMzi7miq7kkEsk84H4LP+IL+h/2UtHfKnEe2x/SuZE5rB1T5xwMzh+ylo75U4j22P6U+ylo75U4j22P6VzInNYOqfOOBmcP2UtHfKnEe2x/SpPDatwmopHx4vL0cjIxvO5lWwyRwbvtuQDvtv03XSVd13y09OW8swcl3FxuuV52jy2OYN9gfURu0juIcQdwVlGBhVzFNN4mdsT7QZpzNFREXmMRERAREQZ3xqO2M0p3fdTih1H/ADLVoizzjSXDGaU5d9/fRit9vV4S3daGgIiICIiAiIgIiICIiAiIgIiICIiAiIgrupbpxma05YfkbFWtLbdTfVir9rHZdJG4sD3d8fK5gId3bnY+d0sSruvbfi/Twt+H2sa2C7TkfNTh7Z7mCzHzRlv8l43Y494a9xHcrEgi9VfcxmPxOb9Aqvaa+5zFfikX6AVh1V9zGY/E5v0Cq9pr7nMV+KRfoBejg9jO/wBl7kkiLBtO+6ftXeFdjiRndJN0/pCOu8xTOyrZbNiwJxAyNsZja0Me8kCR727bblob5SszEI3lF54wvuqJNVOzWEp0MENStw1nJ43xRqavlKzzEBzMlkiYTE8czXAFjmuAdsTsV+aW486u07wE0HqfUmmI81lM3LiqFcUcmDNeNpjQJ3AwsbG8uO/ZblvXzwOqxyoHohFimW15qGpxW4b0tTYY6fiuxZOZxxmojNW5ooXktsRGuztWhgY9p5m8rnHoeXrXNP8Au0cHnc5hWtp4puCzN+KhTnh1FWmyTXSv5IpJqDfLjY5xbv5TnNDt3NGx2ZUD0cix/Ge6BfktNYCZunuTU+S1G/TU2C8N3NWeKR/hDzL2flMZDG6bfkG4LR033VNyvu2dPY7JXLDK+Jn01TvOoy2jqOqzJO5ZeyfNHjz5bow7cjdwc5o5g3YjdlQPSSrnEj+L/UX4hN+gVY1XOJH8X+ovxCb9ArpwO1o3x91jTDRkRF4yCIiAiIgzvjWN8ZpT/wB04n9ZatEWd8a/92aT/wDdOJ/WWrREBERAREQEREBERAREQEREBERAREQEREFc4iXPF+icxaORtYkQQGQ3aUPbTRAEElrPhH5vnVjVc4i3fF2hM9a8YWsV2NOR/h1KHtpoNh57GfCcPQFY0EXqr7mMx+JzfoFV7TX3OYr8Ui/QCsOqvuYzH4nN+gVXtNfc5ivxSL9AL0cHsZ3+y9ySWHUfc+X7XuZaXDXI5SvTzVVrZYcjTBmiisR2jYheA4NLmhwaCCBuN/wrcUVmIlGc6QwWuMlUylTW1bStKvYpmrGdOdu+Rz3Ah73Ola3lGxGzAHbH4RVDw3BnX32PtDaTy9jTjoNIZnD2KlynNOH2qlN55jIx0ezJSxrNmglpPNu4dF6CRSwzziDw1ta14g6Hy/aV/FOHjyUV6GR7myyNs1uyaIwGkHrvvuRsO7dVnhJw54g8N4cHpe1LpTJ6Rw+8EOUMcwyc1ZrSIWOj5RG17fIBeHkEN83c7raUS0XuMtxHBGHF8eMtr/wsOo2aYNfGfBhvSNZFYsgbbAuhgrs37/3zu36wHD/hXrvhhZbp3EP0rkdDsyclqC1kWT+Ma9aWYyyQcjW8j3AveGyF423G7TtstxRMmAVc4kfxf6i/EJv0CrGq3xJ3+x9qPbv8Xz7b/wBArowO1o3x91jTDR0Vbj1Nk6DGeONP2Yezxzrtq1i3i7XZK3zq8YAbPK8jq3aHyh06O2ae7jNWYfMWYatXIQOvy048gKEjuztNryHZkjoXbSMaSCPKaOoI7wQvGRLoiICIiDO+NWxx2kgSRvqnF9w/5hpWiLO+Muxg0ZGQSX6ox22x26h5d/8AitEQEREBERAREQEREBERAREQEREBERAREQVziNeGM0JnrbsrNghDTkeclXg7eStsP3xsfwiO/b0qxqu8RMh4q0Lnbgy0mCMFOSTxlDW8JfW2H74Ivhkd/L6VYkHTzNN2RxF6owgPngkiBPoLmkf91UNJXI7GBpwg8lmtCyCxA7o+GRrQHMcD1BB/tGxHQhXtQuY0Vp/UNgWMpg8bkZwOUS2qkcjwPVu4E7LqwcWmmmaK9C7HWRcP2K9GfJPCf3fF/pT7FejPknhP7vi/0rfzuDrnyjiZnMi4fsV6M+SeE/u+L/Sn2K9GfJPCf3fF/pTncHXPlHEzOZFXa/CnSI1vecdGVBAcdXDZ31ojTc7tZt2sj26SgbFzturXRj4KnPsV6M+SeE/u+L/SnO4OufKOJmcyLh+xXoz5J4T+74v9KfYr0Z8k8J/d8X+lOdwdc+UcTM5lXdd8t3T1vERnnvZSN1SvA0+W9zwQTt16AbuJ7gGklTn2K9GfJPCf3fF/pUphtKYTTrnOxWIo41728rnVKzIiRvvsS0Dpv1WUY+FRMVU3mY2RHvK5ozpVdLLYTH52lYp5GlXvVbELq80NiMPbJG7zmEHvB2G4+Zd1F5jFW7OjDFHZOHzGQw08lSOpDyS9vBXDD5LmQy8zA7bySQBuO/rsR+37Wp8W3L2IaVLORMbCcfTgkNaeQ9BMJHvJZv3ubtyj4J2842NEFdv66xmFdknZYWMRVoSwxPvXYSytJ2oHKWSdWkbnlJ3HKe/bcEz0ViKcyCORkhjdyP5HA8rvUfUeo6LkUHe0Thb8005otq2p7EVua1Re6rNNLF0jMkkRa54A6bOJBBIIIJCCr8Wz2mZ4bVvvjVMX/wBlO3N/+paIsg1th8o3ilw5p1s5JdeMzez3Z5SFkgr12VHV5IYuyEZAAtnldIXlpk6lzQGi+sz+YpujbktPyEzZF1SOTFWBZYyA/vdiXnEbmA9zmta/lPpc3ykFiRQ+L1diMzuK15nOLMlPspg6GQzM89gY8BxIHXoO7qOnVTCAiIgIiICIiAiIgIiICIiAiIgIiIK9xCuux2h85aZkbGIdDUkeL1Sv4RLBsPPbH8Mj1elWFVziLcGP0JnrJyNrECKnI/w6lD200Gw89jPhOHoCsaAiIgIiICIiCuxV+TiFanFK6O1xcLDdMu9U8sshEYZ6JBzkl3pBA9CsSruQrdjrrD3W1L8xlp2aj54pf9mhHNFIO0Z6XHkIa4d3lD4SsSAiIgIiICIiAiIgIi+ZJGQxvkke1kbAXOc47AAd5JQZ7BtmuP1mQAluntOtg59+naXrHO5v4Q2hET8z2rRFn/BuOTKYbK6snLjJqnISZODmBHLTDWxUwAerd4IopC30Olf3kknQEHVvYqllDXNynXtmtK2eAzxNf2UjfNe3ceS4egjqoetoyPFvqDF5PI42vDbktS1Wz9vHY5/Ojd2weWM38oNjLOU93QkGxIgrtGbU9F+Nr3oMfl2ySTi3fpudVMLACYS2B5fzE+a77YNjsQNiQ1jtdYy27GwW+2w2RvwSTxY/Js7GfljP2wHqWkt23OxPTyu7qrEviWJk8b45GNkjeC1zHDcOB7wQg+muD2hzSC0jcEelfqrkehqGNjaMI+bAdjQfj6sNB/LVrNJ3a5tY7w8zD1aSzu3b5vRfMtvUmDgLpaUWoq1bHNc59KRsF21badntZC/aINc3ygTK3YjbqDuAsqKKx+psfkb8uPZMYcjDDFPLTnaWSxsk80kHv6gjcbgEEd6lUBERAREQEREBERAREQV3iJd8XaFztrxjZxHZU5H+H04O3mg2Hnsj+ER6B6VYlXeIlw0NC52yMhaxRiqSP8NpQdvNBsPPZH8Jw9A9KsSAiIgIiICIiCB1njX3MS23XqWL+Qxkov06tayK755WA7R858nZ4LmEO8nyupHeJmtYZbrxzRndkjQ4bEHv+cdFyqrk1tC3HEijjtOW5dw2OKRr4700znPc4jdgZK6Tfchm0m5JeZfJC0IiICIiAiIgIiICz3iXNJq67W0BReebKR9tmpoz1q4zcteN/Q+dwMLO47ds8dYiFYdYavZpiCvBXrHJ5y850eOxcb+R1h4A3LnbHkibuC+QghoPc5xa13zonSb9MUJpb1luSz2QeLOTyIj7MTzbAbMbueSJgAYxm5Ia0blzi5zgsEUTIImRxsbHGwBrWMGwaB3AD0BfaIgIiICIiAiIgj89p/GapxM+My9CvksfPy9pWsxh7HFrg5p2PpDmtcD3ggEbEBdKxjsvj7ctjG3BeZZuQyS1Mk/ljrwbBsogcxhcDt5Ya/mBcC3dgdu2dRB0sRlo8zVdPHDYrlkr4XxWoXRSNcxxaeh7xuNw4btc0hzSQQT3VBZKlJDqrEZCri/CpJWy07lwWjF4PBymRrjH3SntGMaPSztHkEAvBnUBERAREQEREBERBXeIlvwHQudseH28X2VOR3htCHtp4dh57GfCcPQFYlXeIlvwHQudseH28X2VOR3htCHtp4dh57GfCcPQFYkBERAREQEREBfjmhzSDvsRt0OxVJtZXK6jv3WUMi/DY+pM+sJIIo3zTyN6Pd9sa5rWh3QAAk8pJPXZcHifO/LTMezUf2ddsclm36qoif8AvtErZ5Gg/wDEJkyHHfReg9MV7D9IRZoY69nM71uZKJ47Jh5Cxhhaxz+YF28jwyMu5T2jXe91551/7lrSfE/MVctqOxcu5atI2aO/DDVrT87SCC6SKFrnbEDziVpHifO/LTMezUf2dXovzI+rgW2r8ioPifO/LTMezUf2dPE+d+WmY9mo/s6dF+ZH1cC21fkVB8T535aZj2aj+zp4nzvy0zHs1H9nTovzI+rgW2r8q1qfWbMLdr4nH1TmdR2mdpBjIpAzlj5uUzTP2PZQtPe8gk7EMa92zTDeJ878tMx7NR/Z1GYPQVrTti/Ypaqy7bF+wbVueWKpLJO893M90BcWgeS1u+zGgNaGtAAdF/fH1cC21bdJaTlwslnJZS7411Dea0W7oYY4w1u/LFDGSezibudm7kkklznOJJsiq2mc1eZlpMLlJW3JhB4TXutYGGVgcGvD2joHNLm9W9CHDoNutpXLiYc4dWTJOYREWtBERARRWps373sNNcbD4TNzMhhg5uUSSveGMaTsdhzOG52Ow3Ox2VYfj9Q2fLl1bcrynq5lGpWZED6miSORwHq3cT85XTh4E4kZV4iNt/aJWy+IqD4nzvy0zHs1H9nTxPnflpmPZqP7OtvRfmR9XAttX5FQfE+d+WmY9mo/s6eJ878tMx7NR/Z06L8yPq4FtryV7q/3UXFfhDx5xOnKmldL5mATNu6asz07ZmeZY3wEEMsta947SRh8nbqCAOi9w6bdlX6dxbs62s3NmrEb4pAiAWOQdp2YcSQzm5ttyTtt1KyzU/BurrPUWnc7ms7kr+W09O6zjLMkFMGvI4AFwAgAd3AjmB2IBGxG6s/ifO/LTMezUf2dOi/Mj6uBbavyKg+J878tMx7NR/Z08T535aZj2aj+zp0X5kfVwLbV+RUHxPnflpmPZqP7Ov1uIzrXAnWeXdse416Wx/8Ajp0X5kfVwLbV9RVfTOavNysuFycrbdhsHhMFxrOQyx8wa4PaOgc0kdR0IcOg2VoXLiYc4dWTJoERFrRXeIlvwHQudseH28X2VOR3htCHtp4dh57GfCcPQFYlXeIlvwHQudseH28X2VOR3htCHtp4dh57GfCcPQFYkBERAREQEREGe6N/gmV/LGQ/WpFPqA0b/BMr+WMh+tSKfXsY3aVMqtMiIi1MRERARQ9rUraepIcQ/HZBzJKctx2TZBvTiDHNb2b5N+kh5tw3bqGuO/RfukdV4zXOmcbqDDTmzisjC2xWmcxzC+N3ceVwBG/qI3UEuiIqImp/GdjfyPc/zqqvKo1T+M7G/ke5/nVVeVo5Tpp3e8rPcIiLjQREQVLib9z9P8rY/wDW4l2l1eJv3P0/ytj/ANbiXaXp4fYU759l7hEX5zAuLdxzAbkb9dv/AOBRH6iIgIiICL5llbDE+R55WMBc4+oBRWkdV4zXOmcbqDDTmzisjC2xWmcxzC+N3ceVwBG/qI3UEuiIqImj/GdR/I9n/Orq8qjUf4zqP5Hs/wCdXV5WnlWmnd7ys9wiIuJFd4iW/AdC52x4fbxfZU5HeG0Ie2nh2HnsZ8Jw9AViVd4iW/AdC52x4fbxfZU5HeG0Ie2nh2HnsZ8Jw9AViQEREBERAREQZ7o3+CZX8sZD9akU+oDRv8Eyv5YyH61Ip9exjdpUyq0y856S0LjM5xu42Z+fFxZfOYrJ0ZMSy3u9lew3GQOa+Np6NeXcoLu/ZoG6zHghoJ2tsfobVbtdaVpaqtXY7GQkNSwM3cmYS61Snc+6Q/ma2RrmGLlDerWtAC9nU8Nj8fcvW6tGtWt3ntktzwwtZJYe1oY10jgN3kNa1oJ32AA7goyvw/0vU1HLqCDTeIhz0pJkysdCJtp5PQ7yhvMd/wAK5sli8i0J9Y6Vs2MFjG2osRwUvT5GZgG5ylOaTmghHrLMfJZ7t/K7MelSVbRker6nBzJagryCbXGrb+or1fnLCYpqNh0ELiDvyiuyGNzfSOYHoSvX4xtQOtOFWEOt/wAIIjG83khvl9PK8kAdfQNl15dP4ud+NfJjacj8Y7moufAwmqeQs3i6eQeQlvk7dCR3JkDA73DPS+J90hDhsdp6hWx17h5kKs9KGu0RzRi3VY1jm9xHKSNvnWVYjxVpb3GGkptKS4zCSZSxioNWZCs080UDpuzmktdk9kgbuCx+zmu5S8Bw717Vdhse/Lx5V1Gs7KRwOqsvGFvbthc4OdGH7cwaXNaS3fYloPoUdj9B6ZxL8o+jp3E035Xc5B1ejFGbm++/bbN+2ec7zt+8+tJpGRcAuGkei9bZe3i9VaXs4uTHRx2NPaUrSQ12SufzRWntfam5XFrZG7gN5h378q3pQumdFae0VXlg09gcZgYJnc8kWMpx1mvd6yGNAJ/CppZRFosImp/GdjfyPc/zqqvKo1T+M7G/ke5/nVVeVq5Tpp3e8rPcIiLjQREQVLib9z9P8rY/9biXaXV4m/c/T/K2P/W4l2l6eH2FO+fZe5lnuoHZBnArVLsbZdVnbHC6Qssiu+WATx9tEyQkbPkj5429dyXgDqV56yWIn0joTjHrfhxgbOjsJYxGOp1InsAtRFkrjdnZGx5fC1sUpO27SXMLh1AcvXOvdLv1ro/KYNlmGm67F2YmsUorkbeoPlQygseDtsQ4elZ5wc9zvBwt1LlM9Nfxlm5epMoOq4PBw4il2bXl/M6CNzg+Qk7c5PQbgDqtdUTMoyTE8MKmnsTqLMYDWWj30n6TyTrOI0rXnjORifAeznmElybmLHbbScvMedwLjuvzHadZw9ZwRzmj6Rg1LntO3m3ZGvc+TJyjEmxEJi4ntCJmNI3327hsOi9N4fhtpHT0V+PFaWwuMjyDHR3GU8dDELLXec2QNaOcHc7g796kWaZw8bsU5uKotdiWlmOIrMBpNLOzIh6faxyeT5O3k9O5MkeQ+AugW6kj4dauqa80rV1DZmhuXpK1awMzknhhdbqWHvuuEjthIHDstmlvM1rQAFL8PcNpbRHADXXEPJafOdycFzPsc7tXMmEBuzxmGOUHeFh73Fm2xc53evTFHh/pfGagnz1PTeIqZycky5OChEyzIT380obzHf5yu/S09isbjJsdUxlOrj5nSOkqQ12MieZHF0hcwDYlxc4u3HUuJPekUWHlDhFpSpg+Muf0WTpebE5vRUly9gNNOlkpdp27GDnEkr+d5ZK4F4DOZpG7fSqvjGYjTnuQeH/vYmxODbmchi62rr8bTyshcZGPdc7KSOQMMrWRvPO07OcNx1XsfB8PNK6ZfWfh9M4fEuqmQwOo0IoTF2mwk5OVo5eblbvt37DfuX1S0BpfHHKmppvEVTlt/GPY0YmeGb779ts37Z5zvO37z61MgZT7n/hu3RGqs9PjdU6auYqWpDHNp/S1eSGtXm5nFlgsfZm5HPZu08vKHBoPUjdboofTWjcBoupJV09g8bgqsjud8OMqR12Od6y1gAJUws4i0WETR/jOo/kez/nV1eVRqP8AGdR/I9n/ADq6vK18q007veVnuERFxIrvES2KGhc7ZN+3ixFUkf4bQh7WeHYeexnwnD0BWJV3iJb8A0NnLAv2sWY6kjvDaMPbTwdPPYz4Th6ArEgIiICIiAiIgz7R45a2WB23GXvnbf12ZCP+hCnlwZTSd6O/YuYO/BSdad2litcrumic/bbnZyvaWE7DfvB232BLnHp+IdYfGeD9hm+uXrTXh4k5eVEX38GU586TRRniHWHxng/YZvrk8Q6w+M8H7DN9cp1fjj14JZJoozxDrD4zwfsM31yeIdYfGeD9hm+uTq/HHrwLJNFGeIdYfGeD9hm+uTxDrD4zwfsM31ydX449eBZJoozxDrD4zwfsM31yNwOr+Yb5LCbenajN9cnV+OPXgWfFMb8TMeR3DEWwevdvNW2/wKvKgtO6afipprt62MhlJ2NjfO2Lso2MG5DI2bu5W7kk7ucST1JAaGzq48eumuqIp0RFiRERcyCIiCpcTRvp6me4DK48knoB/tcS7SmMtiq+bx09K20vgmGzuVxa4EHcOBHUEEAgjqCAVVn6c1VBsyDM4uxG3oJLVB4kI/4uSUNJ9ZAA69wXoYNdE4cUTNpiZ9bcGWmEiijPEOsPjPB+wzfXJ4h1h8Z4P2Gb65ber8cevBLJNFGeIdYfGeD9hm+uTxDrD4zwfsM31ydX449eBZJoqZqG9q7Aag0tizZws7s7clptlFSYCHkrTT8xHa9d+x5dunnbqweIdYfGeD9hm+uTq/HHrwLJNFGeIdYfGeD9hm+uTxDrD4zwfsM31ydX449eBZJoozxDrD4zwfsM31yDA6v3G+Swm3p2ozfXJ1fjj14FnxRG/EymR3NxFnf5t5oNv8D/AGK8qC07pp+JlmuXbQyGUna1kk7Y+yjYwbkMjZu7lbuSTu5xJPUkBoE6uPlFdNdUZOiIsSIiLmRXOIlzwDRGZseMrOI5K52vU4O2mgPQczWfCPzKxqucQrrcfpG9M7JWcR1ijF2pD20sZdK1o5W7HfcuA+YHf0KxoCIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgz3iQ4Ra94UuI3D89ZhB6dCcVed6R6mHu2/7LQln3GB5o+8nK85jix+p6XaHcgbWOemAdvW603/otBQEREBERAREQEREFc15fNDC1uXIWcZJPkqNdk9WATPJfaiaWcp7mvBLHO+C1xd6FY1X9TWntyunKUOQsUZbN4uc2CASCeOOKR7o3uPmNOzfK799h8LcWBAREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREFe4g6WdrTReXw0Uwq2bMB8GsO32hnaQ+KTp18mRrHdPUnD/Vg1xo/GZkwGnZnjLLdNx3dVsscY54Hf8AFHK17D87SrCs/wA3G7hrqC5qWvFzabyTmvzkLAS6pK1oaLzQPg8oa2Ubeaxkg25JOcNARfEM0diJksT2yxPaHMew7tcD1BB9IX2gIiICIiAiKMzmXkxtcR04I7+UlH+zUXWGQmXymhzt3dzGcwc4gOIHc1xIaQ6NWzLktbW+SbIQ1cZWFd8D4AyrYllLX87Xnq9zGsA6eSO1I3J3DbCujhcS3CY5lRtm1c2e+R092Yyyvc95e7dx7hu47NGzWjZrQGgAd5AREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREBEUfm8/jtOUTcyl2GjWB5e0meG8x9DR6ST6AOpWVNM1Tk0xeRUJGHhJK+aMOfoeV5fMwbuOEcSPLaAP4ISSXfzHf+9b9jK6V4o6U1vqPUeBwObrZTK6dmZXykNfmcK8jgSGc+3K4jlcHBpPK5pa7ZwIWd8VeKFXWOhs1gtN5fP6fyN6DsYc7jqA7WuC4cxYJXMIJbzN5hs5vNzNIcAV5Q9yNw31R7mbi3kMlZ3zGk8jSkqT+DjksAhwfE/snHl33Gx8roHFd8fDuVzn5uVs/o6izH7PuH+Jc3+Zh+tT7PuH+Jc3+Zh+tV/pvK/9clmnIsx+z7h/iXN/mYfrVFas48sm0tl48Fh8qzNvqStousRxNjbOWERlxEhIAdtvsE/pvK/9clli0Xxz0rxGp5uTTFiXOW8Nkn4u7j6jWmxFKJHMDyC4NEbg0vbISGkAjcOa5ot2IxElNz7N2aK/k5AWvttrtiIj5i5sTdtzyN3Owc5x6kkndfzZ9y/wE1Rwd4lU9aZbU2QxliJ/+00sJVbYbfhc4GSCZ0j2Dldt/JcQQHDZwBH9A8Lxi0xmJ44H25cZYkOzI8jC6EOO+wAefIJJ6Ac2/wAywr5ByqiMqrDmxaV2REXAgiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiICIiAiIgIiIInVOo62k8FayloF0cIAbG3zpHuIaxg+cuIH9a875bJ3NR5R2TykjZrrhs0N37OBv8iMHuHrPee8rQuPd15n01j9/tUks9tw37zGxrBv6/34n8IHqWar7f4NyamjB5+Y/VVfy0E5oERF9CwEWLat42ZiHVOdxuAq13Q4V4gk8IxV+463NyB7mNfXYWRAcwbu7mO+55QNie9X4jay1TnpMfg8fjMU5uCp5d0eahmdNHJKZQ6BzWub6WAc3Tl2PR2/k8nSsOZyYzyrW0WRY3i5nNdR6VpaWpY+tlMriBmrk2U55IKkPMI+VrWFrnuc/mA6jYN3PqXf8Ac9+EnReU8NEQuePsn2wgJMfP4U/m5d+u2++2/XZWjlFOJXFNOie/y4o05fj2NlY5j2h7HDYtcNwQv1F1C9cLNdy4PI1cDfmdLi7LuypyyuJdXlPmxbn/ANN3c0fBds0bhwDNsXk/K8wxtlzHujkYwyMe07Frm9WkfOCAV6mxV3xji6dvbl7eFkuw9HM0H/uvi/jXJqcKunGoi2Ve++O//rZpi7tIiL5tBERAREQEREBERAREQEREBERAREQEREBERAREQEREBERBl/HfEOlxOKzLG8wx1gsmP8mGUBpd/U8Rb+obn0LKV6hs1obtaWvYiZNBKwxyRyDdr2kbEEekELB9Y8NsnpGZ8tGvYyuF72PiBlsVx/JkaPKeB6Hjc7ecBtzO+v8AhHLaIo6PiTaY0cN9yYuyabXmUimexug9SSta4gSMdR5XfON7QOx+cL8fr7KNcQNA6lcAduYOobH/AOUrIzNY9+4F2AOHe10gDh6OoPUL68b0fv2v+db9K+l5uvxT6cGNpZ+7h1mn5nIZ7TeobGkTnWxz5HG26EVtzJgwN52EP5WP5QAer2kjfqrLS0WamusnqR14yuu42vjzXMW3KYnyO5+bfrv2ndsNtu/r0m/G9H79r/nW/Snjej9+1/zrfpUpwKaZvEbe/T+SWlmWK4H3tM0dLPwWqDjs1hcccVLckoCaG5XLuflfCXjlIcNwQ7pue/dSWmcdkeE+Hdi243L6xltW7N+W9j4asIa6WVzy1zZJ29fK+D0/B3K9+N6P37X/ADrfpTxvR+/a/wCdb9Kwp5NTRnoiYn815u4tKse//K7fcBqb8HNQ/alMad1DbzrpxZ09lMEIg0tORNciXffzeylk7tuu+3eNt13/ABvQ+/a/51v0rnx0zs3Y8GxMMmXsn/0qQ7Tbrt5TvNb+FxAWzJqo/VVVm224LaSahNmHRYuqN7WQeKsQ2J6u73fga3mcfmaV6lrV2VK8UEY2jiYGNHqAGwVE4a8NXaae7K5VzJsxI0sjjjO8dRh72tPwnn4Tv6h03LtAXxHxbllPKcSKMPPTT365lloiwiIvCQREQEREBERAREQEREBERAREQEREBERAREQEREBERAREQEREHVtYuleO9mnBYPrlia7/ABC6/vaxHxVS9nZ9CIs4rqjNErc97WI+KqXs7PoT3tYj4qpezs+hEV5yvXJeT3tYj4qpezs+hPe1iPiql7Oz6EROcr1yXkGm8QDuMXSB/F2fQu/FDHAwMjY2Ng7msGwCIsZqqq0yj7REWI//2Q==",
+ "text/plain": [
+ "