![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "rT9HzsnQ1uiz"
- },
- "source": [
- "## Environment Setup\n",
- "\n",
- "### Pull Github Materials\n",
- "Because you are likely running this notebook in **Google Colab**, we need to first\n",
- "pull the necessary dataset and materials directly from GitHub.\n",
- "\n",
- "**If you are running this notebook locally**, FYI you may not need to perform this\n",
- "step at all."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "AJJ2UW6M1ui0",
- "outputId": "0f5773b7-a292-4ee6-f4bd-20dc40ca2aba"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Cloning into 'temp_repo'...\n",
- "remote: Enumerating objects: 384, done.\u001b[K\n",
- "remote: Counting objects: 100% (247/247), done.\u001b[K\n",
- "remote: Compressing objects: 100% (159/159), done.\u001b[K\n",
- "remote: Total 384 (delta 135), reused 153 (delta 74), pack-reused 137 (from 1)\u001b[K\n",
- "Receiving objects: 100% (384/384), 64.50 MiB | 8.97 MiB/s, done.\n",
- "Resolving deltas: 100% (159/159), done.\n"
- ]
- }
- ],
- "source": [
- "# NBVAL_SKIP\n",
- "!git clone https://github.com/redis-developer/redis-ai-resources.git temp_repo\n",
- "!mv temp_repo/python-recipes/RAG/resources .\n",
- "!rm -rf temp_repo"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "Z67mf6T91ui2"
- },
- "source": [
- "### Install Python Dependencies"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "DgxBQFXQ1ui2",
- "outputId": "c3c399d6-e294-4a3a-a0a3-82d818509991"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/261.4 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m \u001b[32m256.0/261.4 kB\u001b[0m \u001b[31m21.8 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m261.4/261.4 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m96.1/96.1 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.4/2.4 MB\u001b[0m \u001b[31m55.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m298.0/298.0 kB\u001b[0m \u001b[31m18.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m53.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m408.7/408.7 kB\u001b[0m \u001b[31m27.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m3.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m49.5/49.5 kB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[?25h"
- ]
- }
- ],
- "source": [
- "%pip install -q redis \"redisvl>=0.4.1\" langchain-community pypdf sentence-transformers langchain openai pandas"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "peC8ThuVJkD7"
- },
- "source": [
- "### Install Redis Stack\n",
- "\n",
- "Later in this tutorial, Redis will be used to store, index, and query vector\n",
- "embeddings created from PDF document chunks. **We need to make sure we have a Redis\n",
- "instance available.**"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "zMKHJ7oWJkD8"
- },
- "source": [
- "#### For Colab\n",
- "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "c0d5lfNxJkD8",
- "outputId": "f96e72fa-b9f3-476f-bc9e-328bd30d1344"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb jammy main\n",
- "Starting redis-stack-server, database path /var/lib/redis-stack\n"
- ]
- }
- ],
- "source": [
- "# NBVAL_SKIP\n",
- "%%sh\n",
- "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
- "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
- "sudo apt-get update > /dev/null 2>&1\n",
- "sudo apt-get install redis-stack-server > /dev/null 2>&1\n",
- "redis-stack-server --daemonize yes"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "2arb8Ic0JkD8"
- },
- "source": [
- "#### For Alternative Environments\n",
- "There are many ways to get the necessary redis-stack instance running\n",
- "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
- "own version of Redis Enterprise running, that works too!\n",
- "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
- "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "DhP1w0R9JkD8"
- },
- "source": [
- "### Define the Redis Connection URL\n",
- "\n",
- "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {
- "id": "ggh5TzhkJkD9"
- },
- "outputs": [],
- "source": [
- "import os\n",
- "\n",
- "# Replace values below with your own if using Redis Cloud instance\n",
- "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
- "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n",
- "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n",
- "\n",
- "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n",
- "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\""
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "b3ErDmsIJkD9"
- },
- "source": [
- "## Simplified Vector Search with RedisVL"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "KrtWWU4I1ui3"
- },
- "source": [
- "### Dataset Preparation (PDF Documents)\n",
- "\n",
- "To best demonstrate Redis as a vector database layer, we will load a single\n",
- "financial (10k filings) doc and preprocess it using some helpers from LangChain:\n",
- "\n",
- "- `PyPDFLoader` is not the only document loader type that LangChain provides. Docs: https://python.langchain.com/docs/integrations/document_loaders/pypdfloader/\n",
- "- `RecursiveCharacterTextSplitter` is what we use to create smaller chunks of text from the doc. Docs: https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/recursive_text_splitter"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "uijl2qFH1ui3",
- "outputId": "a99b3fcb-7cfd-4dbd-f258-57779cfcae3c"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Listing available documents ... ['resources/nke-10k-2023.pdf', 'resources/amzn-10k-2023.pdf', 'resources/jnj-10k-2023.pdf', 'resources/aapl-10k-2023.pdf', 'resources/testset_15.csv', 'resources/retrieval_basic_rag_test.csv', 'resources/2022-chevy-colorado-ebrochure.pdf', 'resources/nvd-10k-2023.pdf', 'resources/testset.csv', 'resources/msft-10k-2023.pdf', 'resources/propositions.json', 'resources/generation_basic_rag_test.csv']\n"
- ]
- }
- ],
- "source": [
- "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
- "from langchain_community.document_loaders import PyPDFLoader\n",
- "\n",
- "# Load list of pdfs from a folder\n",
- "data_path = \"resources/\"\n",
- "docs = [os.path.join(data_path, file) for file in os.listdir(data_path)]\n",
- "\n",
- "print(\"Listing available documents ...\", docs)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "anya8hVnT6K_",
- "outputId": "a8430acc-2e6d-45fd-fc8b-601fbbd8289b"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "Done preprocessing. Created 211 chunks of the original pdf resources/nke-10k-2023.pdf\n"
- ]
- }
- ],
- "source": [
- "# pick out the Nike doc for this exercise\n",
- "doc = [doc for doc in docs if \"nke\" in doc][0]\n",
- "\n",
- "# set up the file loader/extractor and text splitter to create chunks\n",
- "text_splitter = RecursiveCharacterTextSplitter(\n",
- " chunk_size=2500, chunk_overlap=0\n",
- ")\n",
- "loader = PyPDFLoader(doc, headers = None)\n",
- "\n",
- "# extract, load, and make chunks\n",
- "chunks = loader.load_and_split(text_splitter)\n",
- "\n",
- "print(\"Done preprocessing. Created\", len(chunks), \"chunks of the original pdf\", doc)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "fDN4XopTJkD9"
- },
- "source": [
- "### Text embedding generation with RedisVL\n",
- "RedisVL has built-in extensions and utilities to aid the GenAI development process. In the following snipit we utilize the HFTextVectorizer redisvl in tandem with the **all-MiniLM-L6-v2** class to generate vector embeddings for the chunks created above. These embeddings capture the \"meaning\" of the text so that we can retrieve the relevant chunks later when a user's query is semantically related."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 661,
- "referenced_widgets": [
- "cbd44245af844dca8e568691cc1c15c5",
- "3109d0d320274ad0bb941608ee3df5e3",
- "6c902ce903bb4e25a127ec277e2b2c45",
- "954b76e059024b15be48fb5064ab2fb7",
- "160c4567015f4b1bba43dc7e1e4712fb",
- "712fcb54fabc430c9567240a2ddd4a76",
- "f96ce89375924097ab9f4cd130fd7b41",
- "58c687581a8d4d3a828686cd066a32b3",
- "df2305a9a6634dffbc08567f62047b27",
- "218e8977786b42e1b825a14d44164d82",
- "8bc8cb91c6274c08a72c91c91dddf4ef",
- "abee8aeb772f48dab4661dca40277788",
- "300b9716084a4a24bf479ae7200b87d1",
- "ff76433f165146f0b39d2488a33b318e",
- "98fe1e1e066541ec942a05ec416fa53f",
- "be9c6f9905fd440884261e09367fe659",
- "9d7bd9a50eea407eb60c41c1534f295d",
- "968f389c21cf469daee8284a7b14c251",
- "39f7677d9d8a4bdf8f4eb4756fae3ed2",
- "959248b437054a43a0393c71a603b35f",
- "6b3711002db148f790eab617f7f40eb4",
- "5a3363012166483d90abb10b476772bf",
- "92e02308d4d94725b73cc324d8cd9906",
- "6fe679c08e2b46dd8657160d974912e0",
- "61fc922ce98c4fefbebe7bb6a8ee9317",
- "2cc139350de742989b6e24d70e490a54",
- "995465a251f64f7a9c1e5541a7f28d4d",
- "56b8c445444b4d39b2c9fb199586ff93",
- "5f2ad751dab24f6aaae736c01e582c14",
- "54331fe70c934a7894903d5ca7a960ce",
- "6270fcf4772f40d59a6f6842060f36a4",
- "14e24b722ecf47a49ebe42e8c3492c1e",
- "b5e36e428e3541fd8a237d0f28a023e1",
- "6aa3f285fd8a4a84882b7bece1b639ac",
- "d20425f4a0594c319bc51ee60d773f79",
- "a046d9ff7e1d4577ab28315d681ac36b",
- "c9468d94408a4d36a20eae07624a6a09",
- "902551f09b44499b8c8dd88bbdf50a4a",
- "5477b553050e42c0b8ed7c2c8c17c025",
- "fcbac845d7c24db6a85e82f190e69a75",
- "82f4af2b827c4d98a762c2e7ebd03d6e",
- "146de95acc214f60b854553ab983b7ae",
- "a356517795234ab6abb3ffd71b05f296",
- "1757bba5dca64bf3b7d359cd2537e9c5",
- "59d890877f8b4f7aa436fa4b82e4cf8d",
- "9a0acbad43204038b8ca4edeeb0e0d61",
- "38518362236e470898cdbfb48ee0d381",
- "9aac56d1808d490797bbb175c5afb226",
- "2f848e63b87847d1a299c04052d567d6",
- "52395bed9f6d455897d8d489e7dcb0d3",
- "4e2332a6f482448597a9d4988fec7cf6",
- "ac55276fbd5a4404ba065a19849119c5",
- "fae66f22c38247ad85078f6ad2530ced",
- "a3fcad6db08c4f07adf4ee817afce77a",
- "557fb6c9f787412a8bff6f4798087bb7",
- "a4c7c73d90cf44acb43740b223be8101",
- "010e7ce97cfb43f195d1dd1811584ea2",
- "484f1fc0b5844726b3ac203440ddbdc8",
- "9368d437c3534a33b0010ea77be8a5e2",
- "50c576ca5f914c65aeb5b7c03f4b0fa2",
- "80bcb933a16c40788a3ad354e545acfe",
- "2bfc17a97664452787740dc202eae370",
- "600f4d36b66d40ecb8353db981d0f1f4",
- "1cb7ce33be9345e992769fb7cdeb0e75",
- "f1204ffea0da4058a3973e6d79a8d36c",
- "b91aa35f8bfb4cb29724a0cf864a3158",
- "b225fd0da4c24d97a502a2df731d1037",
- "9ed0c298163645a8a10f7704354b3d2c",
- "3a2d93764f7645258777f75d2a33b214",
- "4d21de5d79b74e7d9dc5ccfb36827358",
- "927cb59be15747418fba1a56d7e22e21",
- "4a5e1f7a57d446e980090aae0325b990",
- "33175a3341134f7ebba6232440e9a770",
- "d503a8e5ea4f4bc089c4ae3e95ce1af4",
- "73ffa18b349849fdb7264b748b4189e9",
- "316f2f8a79ad4b0aa140f149383b2eff",
- "1c9b5e2acf0141898ab2a0639a79d209",
- "dd6707fe0bae4aab842dac25bf31880d",
- "4682a7ebe86a4a60ab6b793718435302",
- "1617b257e66c409db6c4ca0d0944a933",
- "63825f6200a944bd8c66602a64eee67c",
- "6cad7dfb6dd4441fb569c5533ef044e8",
- "1a76918edd75460e8d572e59d3aa5413",
- "1b3112662eb2481087fb3af6e79a4480",
- "23127b47d99d406c9a53520a3697972b",
- "1cb27bb3b5354879b7f1a73a24df923d",
- "77f646bb598d471cacdf772d9799a8df",
- "66782c677c2040d0ae19e7c6da6186ce",
- "c24f6df83a0b46ecbad2be4583d3bb1b",
- "9101630e52a04193804e02341e38830a",
- "9c9441eac4fe46078709fbf9c84c4a4e",
- "e9ecac569557483d89b848e31b1a4f85",
- "a641f0330b134a48844212dd72dafa57",
- "9e2c06d967be46ecbb56e0e0268c9a65",
- "da39e3fbf61941dc9fc05d00fb44a468",
- "a516325f85594525aac760a5c0d1a0d2",
- "55529d65863a4a5fb25dca02f0e885e2",
- "532e6cc744b54e12a677f33af75318f0",
- "c9c3f643f9b0472ab9dce2649139bb6a",
- "26d0829f64b248ada2b0f46b746cd8b1",
- "448556b65d2f419ca6cd395ce6d11f3f",
- "c0cf7a81656c4fd98d2418fd6336c6ae",
- "5c88eed231d14f2da8961a4ac7837417",
- "b4ca94c7f8534b4e857c57a619a7f116",
- "c18a7f2b29e54916ba81510b2bb21902",
- "067c697db37d43d8b6fa3b155a794f00",
- "006473c1d4a247208c17d3258909adb0",
- "8375e9fcaa4a46d895dc074cfed92149",
- "56cb8feab6c047ca8afb2acfda4d35d1",
- "29ce854a35e94a47af82522cc9f8a92b",
- "8e394c924a00479ba046afb5eeacc5f3",
- "86148800470449979a8baeb58b5f5c88",
- "386648192f9e403680aa57d1444e4465",
- "c12d9b3dfbe045a3bfba0ecd790af191",
- "0dbce80382dc41429050a896f3203c4e",
- "90e4273246e44f7c95db4456a00755a3",
- "d57525fd237d4c519e52c76ee7208a30",
- "6db6a832f6b44c3eb82f93fd60fda7fb",
- "dfcbee09be344b2f8b55ef1c9ddfbd76",
- "0428e3d1575c4ac6b6dfca617d144b7d",
- "dc42c19d950943a88630242dd188c1a7",
- "3fb33de4563749d7827c735380453b58",
- "3d8d6ea4a4ef4493b8033bcc62476375",
- "e7693807a9154e7482b4611be6421a0d",
- "150b6eaa9bd64dce908775d230740038",
- "4b59623304314a35b030ff805e5bf699",
- "1bf348fa5757429790b9272f037fc93a",
- "470138741a50479bb930f00a060cc61e",
- "589f8fbac4e0492e81e35cc6424a75bc",
- "2d92057e09554dcdbe405aafc0f602db",
- "6eb2d7bb05f442519211928645384c3a",
- "d2206237f06a4419a7304a199dff2e8a",
- "40f12f8bb6a04034b8c7a95d984469f2",
- "98e4143c2bbb42cea2566686eff2fa6a",
- "981b3a05c8ae42d29ffb81156ebc1a7d",
- "b8513aac81224b139347dfe5011f1563",
- "09c487bb35b6439aaa298665873ee84b",
- "da636d6c421f49f48ef43db194faae5e",
- "958bab205e204f87bce793f79869a28b",
- "8e93910fca484d93ab2eddea9540d307",
- "0a6226f65d354c55b3370c6e87dcc246",
- "685026baa834438aa8060a9e681c3263",
- "fe189eed0a834221bd8adb0bdc44b4c8"
- ]
- },
- "id": "N3iQ2aLEJkD9",
- "outputId": "b0f0d2c1-41dc-4932-990b-53d2912af19e"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "True"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import warnings\n",
- "import pandas as pd\n",
- "from tqdm.auto import tqdm\n",
- "from redisvl.utils.vectorize import HFTextVectorizer\n",
- "\n",
- "warnings.filterwarnings(\"ignore\")\n",
- "\n",
- "hf = HFTextVectorizer(\"sentence-transformers/all-MiniLM-L6-v2\")\n",
- "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
- "\n",
- "# Embed each chunk content\n",
- "embeddings = hf.embed_many([chunk.page_content for chunk in chunks])\n",
- "\n",
- "# Check to make sure we've created enough embeddings, 1 per document chunk\n",
- "len(embeddings) == len(chunks)"
- ]
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "R2-i8jBl9GRH"
+ },
+ "source": [
+ "\n",
+ "\n",
+ "# RAG from scratch with the Redis Vector Library\n",
+ "\n",
+ "\n",
+ "In this recipe we will cover the basic of the Redis Vector Library and build a basic RAG app from scratch.\n",
+ "\n",
+ "## Let's Begin!\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "rT9HzsnQ1uiz"
+ },
+ "source": [
+ "## Environment Setup\n",
+ "\n",
+ "### Pull Github Materials\n",
+ "Because you are likely running this notebook in **Google Colab**, we need to first\n",
+ "pull the necessary dataset and materials directly from GitHub.\n",
+ "\n",
+ "**If you are running this notebook locally**, FYI you may not need to perform this\n",
+ "step at all."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
},
+ "id": "AJJ2UW6M1ui0",
+ "outputId": "0f5773b7-a292-4ee6-f4bd-20dc40ca2aba",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T04:41:18.607703Z",
+ "start_time": "2025-04-24T04:41:11.664107Z"
+ }
+ },
+ "source": [
+ "# NBVAL_SKIP\n",
+ "!git clone https://github.com/redis-developer/redis-ai-resources.git temp_repo\n",
+ "!mv temp_repo/python-recipes/RAG/resources .\n",
+ "!rm -rf temp_repo"
+ ],
+ "outputs": [
{
- "cell_type": "markdown",
- "metadata": {
- "id": "5baI0xDQ1ui-"
- },
- "source": [
- "### Define a schema and create an index\n",
- "\n",
- "Below we connect to Redis and create an index that contains a text field, tag field, and vector field."
- ]
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Cloning into 'temp_repo'...\r\n",
+ "remote: Enumerating objects: 679, done.\u001B[K\r\n",
+ "remote: Counting objects: 100% (330/330), done.\u001B[Kjects: 82% (271/330)\u001B[K\r\n",
+ "remote: Compressing objects: 100% (214/214), done.\u001B[K\r\n",
+ "remote: Total 679 (delta 227), reused 148 (delta 115), pack-reused 349 (from 2)\u001B[K\r\n",
+ "Receiving objects: 100% (679/679), 57.80 MiB | 11.09 MiB/s, done.\r\n",
+ "Resolving deltas: 100% (295/295), done.\r\n",
+ "mv: rename temp_repo/python-recipes/RAG/resources to ./resources: Directory not empty\r\n"
+ ]
+ }
+ ],
+ "execution_count": 8
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Z67mf6T91ui2"
+ },
+ "source": [
+ "### Install Python Dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
},
+ "id": "DgxBQFXQ1ui2",
+ "outputId": "c3c399d6-e294-4a3a-a0a3-82d818509991",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T04:41:20.572419Z",
+ "start_time": "2025-04-24T04:41:18.616143Z"
+ }
+ },
+ "source": [
+ "%pip install -q redis \"redisvl>=0.4.1\" langchain-community pypdf sentence-transformers langchain openai pandas"
+ ],
+ "outputs": [
{
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {
- "id": "zB1EW_9n1ui-"
- },
- "outputs": [],
- "source": [
- "from redisvl.index import SearchIndex\n",
- "\n",
- "\n",
- "index_name = \"redisvl\"\n",
- "\n",
- "\n",
- "schema = {\n",
- " \"index\": {\n",
- " \"name\": index_name,\n",
- " \"prefix\": \"chunk\"\n",
- " },\n",
- " \"fields\": [\n",
- " {\n",
- " \"name\": \"chunk_id\",\n",
- " \"type\": \"tag\",\n",
- " \"attrs\": {\n",
- " \"sortable\": True\n",
- " }\n",
- " },\n",
- " {\n",
- " \"name\": \"content\",\n",
- " \"type\": \"text\"\n",
- " },\n",
- " {\n",
- " \"name\": \"text_embedding\",\n",
- " \"type\": \"vector\",\n",
- " \"attrs\": {\n",
- " \"dims\": 384,\n",
- " \"distance_metric\": \"cosine\",\n",
- " \"algorithm\": \"hnsw\",\n",
- " \"datatype\": \"float32\"\n",
- " }\n",
- " }\n",
- " ]\n",
- "}"
- ]
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\r\n",
+ "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m A new release of pip is available: \u001B[0m\u001B[31;49m24.0\u001B[0m\u001B[39;49m -> \u001B[0m\u001B[32;49m25.0.1\u001B[0m\r\n",
+ "\u001B[1m[\u001B[0m\u001B[34;49mnotice\u001B[0m\u001B[1;39;49m]\u001B[0m\u001B[39;49m To update, run: \u001B[0m\u001B[32;49mpip install --upgrade pip\u001B[0m\r\n",
+ "Note: you may need to restart the kernel to use updated packages.\n"
+ ]
+ }
+ ],
+ "execution_count": 9
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "peC8ThuVJkD7"
+ },
+ "source": [
+ "### Install Redis Stack\n",
+ "\n",
+ "Later in this tutorial, Redis will be used to store, index, and query vector\n",
+ "embeddings created from PDF document chunks. **We need to make sure we have a Redis\n",
+ "instance available.**"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "zMKHJ7oWJkD8"
+ },
+ "source": [
+ "#### For Colab\n",
+ "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
},
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "id": "LKuQku2CJkD9"
- },
- "outputs": [],
- "source": [
- "# create an index from schema and the client\n",
- "index = SearchIndex.from_dict(schema, redis_url=REDIS_URL)\n",
- "index.create(overwrite=True, drop=True)"
- ]
+ "id": "c0d5lfNxJkD8",
+ "outputId": "f96e72fa-b9f3-476f-bc9e-328bd30d1344"
+ },
+ "source": [
+ "# NBVAL_SKIP\n",
+ "%%sh\n",
+ "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
+ "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
+ "sudo apt-get update > /dev/null 2>&1\n",
+ "sudo apt-get install redis-stack-server > /dev/null 2>&1\n",
+ "redis-stack-server --daemonize yes"
+ ],
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "2arb8Ic0JkD8"
+ },
+ "source": [
+ "#### For Alternative Environments\n",
+ "There are many ways to get the necessary redis-stack instance running\n",
+ "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
+ "own version of Redis Enterprise running, that works too!\n",
+ "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
+ "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "DhP1w0R9JkD8"
+ },
+ "source": [
+ "### Define the Redis Connection URL\n",
+ "\n",
+ "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "ggh5TzhkJkD9",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T16:46:45.583246Z",
+ "start_time": "2025-04-24T16:46:45.581177Z"
+ }
+ },
+ "source": [
+ "import os\n",
+ "\n",
+ "# Replace values below with your own if using Redis Cloud instance\n",
+ "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
+ "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n",
+ "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n",
+ "\n",
+ "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n",
+ "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\""
+ ],
+ "outputs": [],
+ "execution_count": 3
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "b3ErDmsIJkD9"
+ },
+ "source": [
+ "## Simplified Vector Search with RedisVL"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "KrtWWU4I1ui3"
+ },
+ "source": [
+ "### Dataset Preparation (PDF Documents)\n",
+ "\n",
+ "To best demonstrate Redis as a vector database layer, we will load a single\n",
+ "financial (10k filings) doc and preprocess it using some helpers from LangChain:\n",
+ "\n",
+ "- `PyPDFLoader` is not the only document loader type that LangChain provides. Docs: https://python.langchain.com/docs/integrations/document_loaders/pypdfloader/\n",
+ "- `RecursiveCharacterTextSplitter` is what we use to create smaller chunks of text from the doc. Docs: https://python.langchain.com/docs/how_to/recursive_text_splitter/"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
},
+ "id": "uijl2qFH1ui3",
+ "outputId": "a99b3fcb-7cfd-4dbd-f258-57779cfcae3c",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T16:46:46.043726Z",
+ "start_time": "2025-04-24T16:46:45.600472Z"
+ }
+ },
+ "source": [
+ "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
+ "from langchain_community.document_loaders import PyPDFLoader\n",
+ "\n",
+ "# Load list of pdfs from a folder\n",
+ "data_path = \"resources/\"\n",
+ "docs = [os.path.join(data_path, file) for file in os.listdir(data_path)]\n",
+ "\n",
+ "print(\"Listing available documents ...\", docs)"
+ ],
+ "outputs": [
{
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "L6GOqmeN1ui_",
- "outputId": "91a199e3-d087-4b15-9544-d59efa6033c5"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[32m14:47:36\u001b[0m \u001b[34m[RedisVL]\u001b[0m \u001b[1;30mINFO\u001b[0m Indices:\n",
- "\u001b[32m14:47:36\u001b[0m \u001b[34m[RedisVL]\u001b[0m \u001b[1;30mINFO\u001b[0m 1. redisvl\n",
- "\u001b[32m14:47:36\u001b[0m \u001b[34m[RedisVL]\u001b[0m \u001b[1;30mINFO\u001b[0m 2. movies\n"
- ]
- }
- ],
- "source": [
- "# use the RedisVL CLI tool to list all indices\n",
- "!rvl index listall"
- ]
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Listing available documents ... ['resources/nke-10k-2023.pdf', 'resources/amzn-10k-2023.pdf', 'resources/jnj-10k-2023.pdf', 'resources/aapl-10k-2023.pdf', 'resources/testset_15.csv', 'resources/retrieval_basic_rag_test.csv', 'resources/2022-chevy-colorado-ebrochure.pdf', 'resources/nvd-10k-2023.pdf', 'resources/testset.csv', 'resources/msft-10k-2023.pdf', 'resources/propositions.json', 'resources/generation_basic_rag_test.csv']\n"
+ ]
+ }
+ ],
+ "execution_count": 4
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
},
+ "id": "anya8hVnT6K_",
+ "outputId": "a8430acc-2e6d-45fd-fc8b-601fbbd8289b",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T16:46:50.509810Z",
+ "start_time": "2025-04-24T16:46:46.104219Z"
+ }
+ },
+ "source": [
+ "# pick out the Nike doc for this exercise\n",
+ "doc = [doc for doc in docs if \"nke\" in doc][0]\n",
+ "\n",
+ "# set up the file loader/extractor and text splitter to create chunks\n",
+ "text_splitter = RecursiveCharacterTextSplitter(\n",
+ " chunk_size=2500, chunk_overlap=0\n",
+ ")\n",
+ "loader = PyPDFLoader(doc, headers = None)\n",
+ "\n",
+ "# extract, load, and make chunks\n",
+ "chunks = loader.load_and_split(text_splitter)\n",
+ "\n",
+ "print(\"Done preprocessing. Created\", len(chunks), \"chunks of the original pdf\", doc)"
+ ],
+ "outputs": [
{
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "C70C-UWj1ujA",
- "outputId": "1fb7a2d6-ae6d-4536-b4b7-702620efd128"
- },
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "\n",
- "Index Information:\n",
- "╭──────────────┬────────────────┬────────────┬─────────────────┬────────────╮\n",
- "│ Index Name │ Storage Type │ Prefixes │ Index Options │ Indexing │\n",
- "├──────────────┼────────────────┼────────────┼─────────────────┼────────────┤\n",
- "│ redisvl │ HASH │ ['chunk'] │ [] │ 0 │\n",
- "╰──────────────┴────────────────┴────────────┴─────────────────┴────────────╯\n",
- "Index Fields:\n",
- "╭────────────────┬────────────────┬────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬─────────────────┬────────────────┬────────────────┬────────────────┬─────────────────┬────────────────╮\n",
- "│ Name │ Attribute │ Type │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │\n",
- "├────────────────┼────────────────┼────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────────────────┼────────────────┼────────────────┼────────────────┼─────────────────┼────────────────┤\n",
- "│ chunk_id │ chunk_id │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │ │ │ │ │\n",
- "│ content │ content │ TEXT │ WEIGHT │ 1 │ │ │ │ │ │ │ │ │ │ │\n",
- "│ text_embedding │ text_embedding │ VECTOR │ algorithm │ HNSW │ data_type │ FLOAT32 │ dim │ 384 │ distance_metric │ COSINE │ M │ 16 │ ef_construction │ 200 │\n",
- "╰────────────────┴────────────────┴────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴─────────────────┴────────────────┴────────────────┴────────────────┴─────────────────┴────────────────╯\n"
- ]
- }
- ],
- "source": [
- "# get info about the index\n",
- "!rvl index info -i redisvl"
- ]
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Done preprocessing. Created 211 chunks of the original pdf resources/nke-10k-2023.pdf\n"
+ ]
+ }
+ ],
+ "execution_count": 5
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "fDN4XopTJkD9"
+ },
+ "source": [
+ "### Text embedding generation with RedisVL\n",
+ "RedisVL has built-in extensions and utilities to aid the GenAI development process. In the following snipit we utilize the HFTextVectorizer redisvl in tandem with the **all-MiniLM-L6-v2** class to generate vector embeddings for the chunks created above. These embeddings capture the \"meaning\" of the text so that we can retrieve the relevant chunks later when a user's query is semantically related."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 661,
+ "referenced_widgets": [
+ "cbd44245af844dca8e568691cc1c15c5",
+ "3109d0d320274ad0bb941608ee3df5e3",
+ "6c902ce903bb4e25a127ec277e2b2c45",
+ "954b76e059024b15be48fb5064ab2fb7",
+ "160c4567015f4b1bba43dc7e1e4712fb",
+ "712fcb54fabc430c9567240a2ddd4a76",
+ "f96ce89375924097ab9f4cd130fd7b41",
+ "58c687581a8d4d3a828686cd066a32b3",
+ "df2305a9a6634dffbc08567f62047b27",
+ "218e8977786b42e1b825a14d44164d82",
+ "8bc8cb91c6274c08a72c91c91dddf4ef",
+ "abee8aeb772f48dab4661dca40277788",
+ "300b9716084a4a24bf479ae7200b87d1",
+ "ff76433f165146f0b39d2488a33b318e",
+ "98fe1e1e066541ec942a05ec416fa53f",
+ "be9c6f9905fd440884261e09367fe659",
+ "9d7bd9a50eea407eb60c41c1534f295d",
+ "968f389c21cf469daee8284a7b14c251",
+ "39f7677d9d8a4bdf8f4eb4756fae3ed2",
+ "959248b437054a43a0393c71a603b35f",
+ "6b3711002db148f790eab617f7f40eb4",
+ "5a3363012166483d90abb10b476772bf",
+ "92e02308d4d94725b73cc324d8cd9906",
+ "6fe679c08e2b46dd8657160d974912e0",
+ "61fc922ce98c4fefbebe7bb6a8ee9317",
+ "2cc139350de742989b6e24d70e490a54",
+ "995465a251f64f7a9c1e5541a7f28d4d",
+ "56b8c445444b4d39b2c9fb199586ff93",
+ "5f2ad751dab24f6aaae736c01e582c14",
+ "54331fe70c934a7894903d5ca7a960ce",
+ "6270fcf4772f40d59a6f6842060f36a4",
+ "14e24b722ecf47a49ebe42e8c3492c1e",
+ "b5e36e428e3541fd8a237d0f28a023e1",
+ "6aa3f285fd8a4a84882b7bece1b639ac",
+ "d20425f4a0594c319bc51ee60d773f79",
+ "a046d9ff7e1d4577ab28315d681ac36b",
+ "c9468d94408a4d36a20eae07624a6a09",
+ "902551f09b44499b8c8dd88bbdf50a4a",
+ "5477b553050e42c0b8ed7c2c8c17c025",
+ "fcbac845d7c24db6a85e82f190e69a75",
+ "82f4af2b827c4d98a762c2e7ebd03d6e",
+ "146de95acc214f60b854553ab983b7ae",
+ "a356517795234ab6abb3ffd71b05f296",
+ "1757bba5dca64bf3b7d359cd2537e9c5",
+ "59d890877f8b4f7aa436fa4b82e4cf8d",
+ "9a0acbad43204038b8ca4edeeb0e0d61",
+ "38518362236e470898cdbfb48ee0d381",
+ "9aac56d1808d490797bbb175c5afb226",
+ "2f848e63b87847d1a299c04052d567d6",
+ "52395bed9f6d455897d8d489e7dcb0d3",
+ "4e2332a6f482448597a9d4988fec7cf6",
+ "ac55276fbd5a4404ba065a19849119c5",
+ "fae66f22c38247ad85078f6ad2530ced",
+ "a3fcad6db08c4f07adf4ee817afce77a",
+ "557fb6c9f787412a8bff6f4798087bb7",
+ "a4c7c73d90cf44acb43740b223be8101",
+ "010e7ce97cfb43f195d1dd1811584ea2",
+ "484f1fc0b5844726b3ac203440ddbdc8",
+ "9368d437c3534a33b0010ea77be8a5e2",
+ "50c576ca5f914c65aeb5b7c03f4b0fa2",
+ "80bcb933a16c40788a3ad354e545acfe",
+ "2bfc17a97664452787740dc202eae370",
+ "600f4d36b66d40ecb8353db981d0f1f4",
+ "1cb7ce33be9345e992769fb7cdeb0e75",
+ "f1204ffea0da4058a3973e6d79a8d36c",
+ "b91aa35f8bfb4cb29724a0cf864a3158",
+ "b225fd0da4c24d97a502a2df731d1037",
+ "9ed0c298163645a8a10f7704354b3d2c",
+ "3a2d93764f7645258777f75d2a33b214",
+ "4d21de5d79b74e7d9dc5ccfb36827358",
+ "927cb59be15747418fba1a56d7e22e21",
+ "4a5e1f7a57d446e980090aae0325b990",
+ "33175a3341134f7ebba6232440e9a770",
+ "d503a8e5ea4f4bc089c4ae3e95ce1af4",
+ "73ffa18b349849fdb7264b748b4189e9",
+ "316f2f8a79ad4b0aa140f149383b2eff",
+ "1c9b5e2acf0141898ab2a0639a79d209",
+ "dd6707fe0bae4aab842dac25bf31880d",
+ "4682a7ebe86a4a60ab6b793718435302",
+ "1617b257e66c409db6c4ca0d0944a933",
+ "63825f6200a944bd8c66602a64eee67c",
+ "6cad7dfb6dd4441fb569c5533ef044e8",
+ "1a76918edd75460e8d572e59d3aa5413",
+ "1b3112662eb2481087fb3af6e79a4480",
+ "23127b47d99d406c9a53520a3697972b",
+ "1cb27bb3b5354879b7f1a73a24df923d",
+ "77f646bb598d471cacdf772d9799a8df",
+ "66782c677c2040d0ae19e7c6da6186ce",
+ "c24f6df83a0b46ecbad2be4583d3bb1b",
+ "9101630e52a04193804e02341e38830a",
+ "9c9441eac4fe46078709fbf9c84c4a4e",
+ "e9ecac569557483d89b848e31b1a4f85",
+ "a641f0330b134a48844212dd72dafa57",
+ "9e2c06d967be46ecbb56e0e0268c9a65",
+ "da39e3fbf61941dc9fc05d00fb44a468",
+ "a516325f85594525aac760a5c0d1a0d2",
+ "55529d65863a4a5fb25dca02f0e885e2",
+ "532e6cc744b54e12a677f33af75318f0",
+ "c9c3f643f9b0472ab9dce2649139bb6a",
+ "26d0829f64b248ada2b0f46b746cd8b1",
+ "448556b65d2f419ca6cd395ce6d11f3f",
+ "c0cf7a81656c4fd98d2418fd6336c6ae",
+ "5c88eed231d14f2da8961a4ac7837417",
+ "b4ca94c7f8534b4e857c57a619a7f116",
+ "c18a7f2b29e54916ba81510b2bb21902",
+ "067c697db37d43d8b6fa3b155a794f00",
+ "006473c1d4a247208c17d3258909adb0",
+ "8375e9fcaa4a46d895dc074cfed92149",
+ "56cb8feab6c047ca8afb2acfda4d35d1",
+ "29ce854a35e94a47af82522cc9f8a92b",
+ "8e394c924a00479ba046afb5eeacc5f3",
+ "86148800470449979a8baeb58b5f5c88",
+ "386648192f9e403680aa57d1444e4465",
+ "c12d9b3dfbe045a3bfba0ecd790af191",
+ "0dbce80382dc41429050a896f3203c4e",
+ "90e4273246e44f7c95db4456a00755a3",
+ "d57525fd237d4c519e52c76ee7208a30",
+ "6db6a832f6b44c3eb82f93fd60fda7fb",
+ "dfcbee09be344b2f8b55ef1c9ddfbd76",
+ "0428e3d1575c4ac6b6dfca617d144b7d",
+ "dc42c19d950943a88630242dd188c1a7",
+ "3fb33de4563749d7827c735380453b58",
+ "3d8d6ea4a4ef4493b8033bcc62476375",
+ "e7693807a9154e7482b4611be6421a0d",
+ "150b6eaa9bd64dce908775d230740038",
+ "4b59623304314a35b030ff805e5bf699",
+ "1bf348fa5757429790b9272f037fc93a",
+ "470138741a50479bb930f00a060cc61e",
+ "589f8fbac4e0492e81e35cc6424a75bc",
+ "2d92057e09554dcdbe405aafc0f602db",
+ "6eb2d7bb05f442519211928645384c3a",
+ "d2206237f06a4419a7304a199dff2e8a",
+ "40f12f8bb6a04034b8c7a95d984469f2",
+ "98e4143c2bbb42cea2566686eff2fa6a",
+ "981b3a05c8ae42d29ffb81156ebc1a7d",
+ "b8513aac81224b139347dfe5011f1563",
+ "09c487bb35b6439aaa298665873ee84b",
+ "da636d6c421f49f48ef43db194faae5e",
+ "958bab205e204f87bce793f79869a28b",
+ "8e93910fca484d93ab2eddea9540d307",
+ "0a6226f65d354c55b3370c6e87dcc246",
+ "685026baa834438aa8060a9e681c3263",
+ "fe189eed0a834221bd8adb0bdc44b4c8"
+ ]
},
+ "id": "N3iQ2aLEJkD9",
+ "outputId": "b0f0d2c1-41dc-4932-990b-53d2912af19e",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T16:46:55.588165Z",
+ "start_time": "2025-04-24T16:46:50.528240Z"
+ }
+ },
+ "source": [
+ "import warnings\n",
+ "import pandas as pd\n",
+ "from redisvl.utils.vectorize import HFTextVectorizer, BaseVectorizer\n",
+ "\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "\n",
+ "hf = HFTextVectorizer(\"sentence-transformers/all-MiniLM-L6-v2\")\n",
+ "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
+ "\n",
+ "# Embed each chunk content\n",
+ "embeddings = hf.embed_many([chunk.page_content for chunk in chunks])\n",
+ "\n",
+ "# Check to make sure we've created enough embeddings, 1 per document chunk\n",
+ "len(embeddings) == len(chunks)"
+ ],
+ "outputs": [
{
- "cell_type": "markdown",
- "metadata": {
- "id": "Qrj-jeGmBRTL"
- },
- "source": [
- "### Process and load dataset\n",
- "Below we use the RedisVL index to simply load the list of document chunks to Redis db."
+ "data": {
+ "text/plain": [
+ "True"
]
- },
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "execution_count": 6
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "5baI0xDQ1ui-"
+ },
+ "source": [
+ "### Define a schema and create an index\n",
+ "\n",
+ "Below we connect to Redis and create an index that contains a text field, tag field, and vector field."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "zB1EW_9n1ui-",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T16:46:55.611260Z",
+ "start_time": "2025-04-24T16:46:55.598846Z"
+ }
+ },
+ "source": [
+ "from redisvl.index import SearchIndex\n",
+ "\n",
+ "\n",
+ "index_name = \"redisvl\"\n",
+ "\n",
+ "\n",
+ "schema = {\n",
+ " \"index\": {\n",
+ " \"name\": index_name,\n",
+ " \"prefix\": \"chunk\"\n",
+ " },\n",
+ " \"fields\": [\n",
+ " {\n",
+ " \"name\": \"chunk_id\",\n",
+ " \"type\": \"tag\",\n",
+ " \"attrs\": {\n",
+ " \"sortable\": True\n",
+ " }\n",
+ " },\n",
+ " {\n",
+ " \"name\": \"content\",\n",
+ " \"type\": \"text\"\n",
+ " },\n",
+ " {\n",
+ " \"name\": \"text_embedding\",\n",
+ " \"type\": \"vector\",\n",
+ " \"attrs\": {\n",
+ " \"dims\": 384,\n",
+ " \"distance_metric\": \"cosine\",\n",
+ " \"algorithm\": \"hnsw\",\n",
+ " \"datatype\": \"float32\"\n",
+ " }\n",
+ " }\n",
+ " ]\n",
+ "}"
+ ],
+ "outputs": [],
+ "execution_count": 7
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "LKuQku2CJkD9",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T16:46:55.630056Z",
+ "start_time": "2025-04-24T16:46:55.620207Z"
+ }
+ },
+ "source": [
+ "# create an index from schema and the client\n",
+ "index = SearchIndex.from_dict(schema, redis_url=REDIS_URL)\n",
+ "index.create(overwrite=True, drop=True)"
+ ],
+ "outputs": [
{
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {
- "id": "Zsg09Keg1ujA"
- },
- "outputs": [],
- "source": [
- "# load expects an iterable of dictionaries\n",
- "from redisvl.redis.utils import array_to_buffer\n",
- "\n",
- "data = [\n",
- " {\n",
- " 'chunk_id': i,\n",
- " 'content': chunk.page_content,\n",
- " # For HASH -- must convert embeddings to bytes\n",
- " 'text_embedding': array_to_buffer(embeddings[i], dtype='float32')\n",
- " } for i, chunk in enumerate(chunks)\n",
- "]\n",
- "\n",
- "# RedisVL handles batching automatically\n",
- "keys = index.load(data, id_field=\"chunk_id\")"
- ]
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "09:46:55 redisvl.index.index INFO Index already exists, overwriting.\n"
+ ]
+ }
+ ],
+ "execution_count": 8
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
},
- {
- "cell_type": "markdown",
- "metadata": {
- "id": "-ZsFB-6Z1ujB"
- },
- "source": [
- "### Query the database\n",
- "Now we can use the RedisVL index to perform similarity search operations with Redis"
- ]
+ "id": "L6GOqmeN1ui_",
+ "outputId": "91a199e3-d087-4b15-9544-d59efa6033c5"
+ },
+ "source": [
+ "# use the RedisVL CLI tool to list all indices\n",
+ "!rvl index listall"
+ ],
+ "outputs": [],
+ "execution_count": null
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/"
},
+ "id": "C70C-UWj1ujA",
+ "outputId": "1fb7a2d6-ae6d-4536-b4b7-702620efd128",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T16:46:56.828176Z",
+ "start_time": "2025-04-24T16:46:56.283831Z"
+ }
+ },
+ "source": [
+ "# get info about the index\n",
+ "!rvl index info -i redisvl"
+ ],
+ "outputs": [
{
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 85,
- "referenced_widgets": [
- "c75d5ab2049146e580efab9da9bbcdb0",
- "9ce1fb951e79468baa9d1aebfa4c4fae",
- "e96d1546380146078c18ec78363f7dac",
- "a3c36bb0d3b74c8ea56bf03521465b81",
- "9f306cfd66dc441aba923d4e051911fc",
- "9e3289444cb142c29ad7d569be2e25b8",
- "c20443e17308425596679c0544dab528",
- "f0bdd8f4d7b84bd5a1c209c591ce8787",
- "126743b52b254e54aa4f65bcb9e65aea",
- "debae380e6d24fb8ae712a6dd2226152",
- "aacb6f8ca39846d89e1e4e96656e3a36"
- ]
- },
- "id": "BkFv-_iC1ujB",
- "outputId": "c398d356-6bb7-43a9-ca95-cb7f167d1f38"
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- "'*=>[KNN 3 @text_embedding $vector AS vector_distance] RETURN 3 chunk_id content vector_distance SORTBY vector_distance ASC DIALECT 2 LIMIT 0 3'"
- ]
- },
- "execution_count": 11,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "from redisvl.query import VectorQuery\n",
- "\n",
- "query = \"Nike profit margins and company performance\"\n",
- "\n",
- "query_embedding = hf.embed(query)\n",
- "\n",
- "vector_query = VectorQuery(\n",
- " vector=query_embedding,\n",
- " vector_field_name=\"text_embedding\",\n",
- " num_results=3,\n",
- " return_fields=[\"chunk_id\", \"content\"],\n",
- " return_score=True\n",
- ")\n",
- "\n",
- "# show the raw redis query\n",
- "str(vector_query)"
- ]
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\r\n",
+ "\r\n",
+ "Index Information:\r\n",
+ "╭──────────────┬────────────────┬────────────┬─────────────────┬────────────╮\r\n",
+ "│ Index Name │ Storage Type │ Prefixes │ Index Options │ Indexing │\r\n",
+ "├──────────────┼────────────────┼────────────┼─────────────────┼────────────┤\r\n",
+ "│ redisvl │ HASH │ ['chunk'] │ [] │ 0 │\r\n",
+ "╰──────────────┴────────────────┴────────────┴─────────────────┴────────────╯\r\n",
+ "Index Fields:\r\n",
+ "╭────────────────┬────────────────┬────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬─────────────────┬────────────────┬────────────────┬────────────────┬─────────────────┬────────────────╮\r\n",
+ "│ Name │ Attribute │ Type │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │\r\n",
+ "├────────────────┼────────────────┼────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────────────────┼────────────────┼────────────────┼────────────────┼─────────────────┼────────────────┤\r\n",
+ "│ chunk_id │ chunk_id │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │ │ │ │ │\r\n",
+ "│ content │ content │ TEXT │ WEIGHT │ 1 │ │ │ │ │ │ │ │ │ │ │\r\n",
+ "│ text_embedding │ text_embedding │ VECTOR │ algorithm │ HNSW │ data_type │ FLOAT32 │ dim │ 384 │ distance_metric │ COSINE │ M │ 16 │ ef_construction │ 200 │\r\n",
+ "╰────────────────┴────────────────┴────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴─────────────────┴────────────────┴────────────────┴────────────────┴─────────────────┴────────────────╯\r\n"
+ ]
+ }
+ ],
+ "execution_count": 10
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "Qrj-jeGmBRTL"
+ },
+ "source": [
+ "### Process and load dataset\n",
+ "Below we use the RedisVL index to simply load the list of document chunks to Redis db."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "id": "Zsg09Keg1ujA",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T16:46:56.895623Z",
+ "start_time": "2025-04-24T16:46:56.836700Z"
+ }
+ },
+ "source": [
+ "# load expects an iterable of dictionaries\n",
+ "from redisvl.redis.utils import array_to_buffer\n",
+ "\n",
+ "data = [\n",
+ " {\n",
+ " 'chunk_id': i,\n",
+ " 'content': chunk.page_content,\n",
+ " # For HASH -- must convert embeddings to bytes\n",
+ " 'text_embedding': array_to_buffer(embeddings[i], dtype='float32')\n",
+ " } for i, chunk in enumerate(chunks)\n",
+ "]\n",
+ "\n",
+ "# RedisVL handles batching automatically\n",
+ "keys = index.load(data, id_field=\"chunk_id\")"
+ ],
+ "outputs": [],
+ "execution_count": 11
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "id": "-ZsFB-6Z1ujB"
+ },
+ "source": [
+ "### Query the database\n",
+ "Now we can use the RedisVL index to perform similarity search operations with Redis"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "metadata": {
+ "colab": {
+ "base_uri": "https://localhost:8080/",
+ "height": 85,
+ "referenced_widgets": [
+ "c75d5ab2049146e580efab9da9bbcdb0",
+ "9ce1fb951e79468baa9d1aebfa4c4fae",
+ "e96d1546380146078c18ec78363f7dac",
+ "a3c36bb0d3b74c8ea56bf03521465b81",
+ "9f306cfd66dc441aba923d4e051911fc",
+ "9e3289444cb142c29ad7d569be2e25b8",
+ "c20443e17308425596679c0544dab528",
+ "f0bdd8f4d7b84bd5a1c209c591ce8787",
+ "126743b52b254e54aa4f65bcb9e65aea",
+ "debae380e6d24fb8ae712a6dd2226152",
+ "aacb6f8ca39846d89e1e4e96656e3a36"
+ ]
},
+ "id": "BkFv-_iC1ujB",
+ "outputId": "c398d356-6bb7-43a9-ca95-cb7f167d1f38",
+ "ExecuteTime": {
+ "end_time": "2025-04-24T16:46:56.991529Z",
+ "start_time": "2025-04-24T16:46:56.903370Z"
+ }
+ },
+ "source": [
+ "from redisvl.query import VectorQuery\n",
+ "\n",
+ "query = \"Nike profit margins and company performance\"\n",
+ "\n",
+ "query_embedding = hf.embed(query)\n",
+ "\n",
+ "vector_query = VectorQuery(\n",
+ " vector=query_embedding,\n",
+ " vector_field_name=\"text_embedding\",\n",
+ " num_results=3,\n",
+ " return_fields=[\"chunk_id\", \"content\"],\n",
+ " return_score=True\n",
+ ")\n",
+ "\n",
+ "# show the raw redis query\n",
+ "str(vector_query)"
+ ],
+ "outputs": [
{
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/",
- "height": 143
- },
- "id": "5reL5qTW1ujC",
- "outputId": "dd58f191-54f5-4226-c4e1-70207d58f2dc"
- },
- "outputs": [
- {
- "data": {
- "text/html": [
- "| \n", - " | id | \n", - "vector_distance | \n", - "chunk_id | \n", - "content | \n", - "
|---|---|---|---|---|
| 0 | \n", - "chunk:87 | \n", - "0.334264576435 | \n", - "87 | \n", - "Asia Pacific & Latin America 1,932 1,896 2 % 1... | \n", - "
| 1 | \n", - "chunk:79 | \n", - "0.350993335247 | \n", - "79 | \n", - "Table of Contents\\nCONSOLIDA TED OPERA TING RE... | \n", - "
| 2 | \n", - "chunk:86 | \n", - "0.371814072132 | \n", - "86 | \n", - "Table of Contents\\nOPERA TING SEGMENTS\\nAs dis... | \n", - "
| \n", + " | id | \n", + "vector_distance | \n", + "chunk_id | \n", + "content | \n", + "
|---|---|---|---|---|
| 0 | \n", + "chunk:88 | \n", + "0.337694525719 | \n", + "88 | \n", + "Asia Pacific & Latin America 1,932 1,896 2 % 1... | \n", + "
| 1 | \n", + "chunk:80 | \n", + "0.34205275774 | \n", + "80 | \n", + "Table of Contents\\nCONSOLIDATED OPERATING RESU... | \n", + "
| 2 | \n", + "chunk:87 | \n", + "0.357761025429 | \n", + "87 | \n", + "Table of Contents\\nOPERATING SEGMENTS\\nAs disc... | \n", + "
| \n", - " | id | \n", - "payload | \n", - "vector_distance | \n", - "chunk_id | \n", - "
|---|---|---|---|---|
| 0 | \n", - "chunk:79 | \n", - "None | \n", - "0.350993335247 | \n", - "79 | \n", - "
| 1 | \n", - "chunk:82 | \n", - "None | \n", - "0.378765702248 | \n", - "82 | \n", - "
| 2 | \n", - "chunk:86 | \n", - "None | \n", - "0.371814072132 | \n", - "86 | \n", - "
| 3 | \n", - "chunk:87 | \n", - "None | \n", - "0.334264576435 | \n", - "87 | \n", - "
| \n", + " | id | \n", + "payload | \n", + "vector_distance | \n", + "chunk_id | \n", + "
|---|---|---|---|---|
| 0 | \n", + "chunk:80 | \n", + "None | \n", + "0.34205275774 | \n", + "80 | \n", + "
| 1 | \n", + "chunk:83 | \n", + "None | \n", + "0.378765881062 | \n", + "83 | \n", + "
| 2 | \n", + "chunk:87 | \n", + "None | \n", + "0.357761025429 | \n", + "87 | \n", + "
| 3 | \n", + "chunk:88 | \n", + "None | \n", + "0.337694525719 | \n", + "88 | \n", + "
| \n", - " | id | \n", - "vector_distance | \n", - "content | \n", - "
|---|---|---|---|
| 0 | \n", - "chunk:82 | \n", - "0.378765702248 | \n", - "Table of Contents\\nGROSS MARGIN\\nFISCAL 2023 C... | \n", - "
| 1 | \n", - "chunk:128 | \n", - "0.418757259846 | \n", - "Table of Contents\\nNIKE, INC.\\nCONSOLIDATED ST... | \n", - "
| 2 | \n", - "chunk:72 | \n", - "0.466709017754 | \n", - "Table of Contents\\nITEM 7. MANAGEM ENT'S DISCU... | \n", - "
| 3 | \n", - "chunk:62 | \n", - "0.493393957615 | \n", - "existing businesses, such as our NIKE Direct o... | \n", - "
| \n", + " | id | \n", + "vector_distance | \n", + "content | \n", + "
|---|---|---|---|
| 0 | \n", + "chunk:83 | \n", + "0.378765881062 | \n", + "Table of Contents\\nGROSS MARGIN\\nFISCAL 2023 C... | \n", + "
| 1 | \n", + "chunk:129 | \n", + "0.418757200241 | \n", + "Table of Contents\\nNIKE, INC.\\nCONSOLIDATED ST... | \n", + "
| 2 | \n", + "chunk:73 | \n", + "0.465415120125 | \n", + "Table of Contents\\nITEM 7. MANAGEMENT'S DISCUS... | \n", + "
| 3 | \n", + "chunk:63 | \n", + "0.49339401722 | \n", + "existing businesses, such as our NIKE Direct o... | \n", + "
| \n", - " | id | \n", - "vector_distance | \n", - "content | \n", - "
|---|---|---|---|
| 0 | \n", - "chunk:87 | \n", - "0.334264576435 | \n", - "Asia Pacific & Latin America 1,932 1,896 2 % 1... | \n", - "
| 1 | \n", - "chunk:79 | \n", - "0.350993335247 | \n", - "Table of Contents\\nCONSOLIDA TED OPERA TING RE... | \n", - "
| 2 | \n", - "chunk:86 | \n", - "0.371814072132 | \n", - "Table of Contents\\nOPERA TING SEGMENTS\\nAs dis... | \n", - "
| 3 | \n", - "chunk:82 | \n", - "0.378765702248 | \n", - "Table of Contents\\nGROSS MARGIN\\nFISCAL 2023 C... | \n", - "
| \n", + " | id | \n", + "vector_distance | \n", + "content | \n", + "
|---|---|---|---|
| 0 | \n", + "chunk:88 | \n", + "0.337694525719 | \n", + "Asia Pacific & Latin America 1,932 1,896 2 % 1... | \n", + "
| 1 | \n", + "chunk:80 | \n", + "0.34205275774 | \n", + "Table of Contents\\nCONSOLIDATED OPERATING RESU... | \n", + "
| 2 | \n", + "chunk:87 | \n", + "0.357761025429 | \n", + "Table of Contents\\nOPERATING SEGMENTS\\nAs disc... | \n", + "
| 3 | \n", + "chunk:83 | \n", + "0.378765881062 | \n", + "Table of Contents\\nGROSS MARGIN\\nFISCAL 2023 C... | \n", + "
| \n", - " | id | \n", - "vector_distance | \n", - "content | \n", - "
|---|---|---|---|
| 0 | \n", - "chunk:87 | \n", - "0.334264576435 | \n", - "Asia Pacific & Latin America 1,932 1,896 2 % 1... | \n", - "
| 1 | \n", - "chunk:79 | \n", - "0.350993335247 | \n", - "Table of Contents\\nCONSOLIDA TED OPERA TING RE... | \n", - "
| 2 | \n", - "chunk:86 | \n", - "0.371814072132 | \n", - "Table of Contents\\nOPERA TING SEGMENTS\\nAs dis... | \n", - "
| 3 | \n", - "chunk:82 | \n", - "0.378765702248 | \n", - "Table of Contents\\nGROSS MARGIN\\nFISCAL 2023 C... | \n", - "
| \n", + " | id | \n", + "vector_distance | \n", + "content | \n", + "
|---|---|---|---|
| 0 | \n", + "chunk:88 | \n", + "0.337694525719 | \n", + "Asia Pacific & Latin America 1,932 1,896 2 % 1... | \n", + "
| 1 | \n", + "chunk:80 | \n", + "0.34205275774 | \n", + "Table of Contents\\nCONSOLIDATED OPERATING RESU... | \n", + "
| 2 | \n", + "chunk:87 | \n", + "0.357761025429 | \n", + "Table of Contents\\nOPERATING SEGMENTS\\nAs disc... | \n", + "
| 3 | \n", + "chunk:83 | \n", + "0.378765881062 | \n", + "Table of Contents\\nGROSS MARGIN\\nFISCAL 2023 C... | \n", + "
\n"
]
@@ -22,9 +23,9 @@
"source": [
"## Prepare data\n",
"\n",
- "In this examples we will load a list of movie objects with the following attributes: `title`, `rating`, `description`, and `genre`.\n",
+ "In this examples we will load a list of movies with the following attributes: `title`, `rating`, `description`, and `genre`.\n",
"\n",
- "For the vector part of our vector search we will embed the description so that user's can search for movies that best match what they're looking for.\n",
+ "We will embed the movie description so that user's can search for movies that best match the kind of movie that they're looking for.\n",
"\n",
"**If you are running this notebook locally**, FYI you may not need to perform this step at all."
]
@@ -38,7 +39,7 @@
"base_uri": "https://localhost:8080/"
},
"id": "b966a9b5",
- "outputId": "61565924-8e01-4411-fac7-82346bb10e87"
+ "outputId": "8fb1aed9-94a3-47b2-af50-4eac9b08d7f1"
},
"outputs": [
{
@@ -46,12 +47,12 @@
"output_type": "stream",
"text": [
"Cloning into 'temp_repo'...\n",
- "remote: Enumerating objects: 384, done.\u001b[K\n",
- "remote: Counting objects: 100% (247/247), done.\u001b[K\n",
- "remote: Compressing objects: 100% (159/159), done.\u001b[K\n",
- "remote: Total 384 (delta 135), reused 153 (delta 74), pack-reused 137 (from 1)\u001b[K\n",
- "Receiving objects: 100% (384/384), 64.50 MiB | 15.56 MiB/s, done.\n",
- "Resolving deltas: 100% (159/159), done.\n"
+ "remote: Enumerating objects: 669, done.\u001b[K\n",
+ "remote: Counting objects: 100% (320/320), done.\u001b[K\n",
+ "remote: Compressing objects: 100% (207/207), done.\u001b[K\n",
+ "remote: Total 669 (delta 219), reused 141 (delta 112), pack-reused 349 (from 2)\u001b[K\n",
+ "Receiving objects: 100% (669/669), 57.77 MiB | 20.61 MiB/s, done.\n",
+ "Resolving deltas: 100% (287/287), done.\n"
]
}
],
@@ -74,30 +75,14 @@
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": null,
"id": "c620286e",
"metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "c620286e",
- "outputId": "d69d35a0-29b2-4a9c-aa13-acf27d85a414"
+ "id": "c620286e"
},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/261.4 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m261.4/261.4 kB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[?25h\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/96.1 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m96.1/96.1 kB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m5.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
- "\u001b[?25h"
- ]
- }
- ],
+ "outputs": [],
"source": [
- "%pip install -q redis \"redisvl>=0.4.1\" numpy sentence-transformers pandas"
+ "%pip install -q \"redisvl==0.5.2\" sentence-transformers pandas nltk"
]
},
{
@@ -119,25 +104,12 @@
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": null,
"id": "2cb85a99",
"metadata": {
- "colab": {
- "base_uri": "https://localhost:8080/"
- },
- "id": "2cb85a99",
- "outputId": "70660a1f-9d1c-408b-f7a5-5981054fabc3"
+ "id": "2cb85a99"
},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb jammy main\n",
- "Starting redis-stack-server, database path /var/lib/redis-stack\n"
- ]
- }
- ],
+ "outputs": [],
"source": [
"# NBVAL_SKIP\n",
"%%sh\n",
@@ -177,7 +149,7 @@
},
{
"cell_type": "code",
- "execution_count": 2,
+ "execution_count": 1,
"id": "aefda1d1",
"metadata": {
"id": "aefda1d1"
@@ -185,6 +157,9 @@
"outputs": [],
"source": [
"import os\n",
+ "import warnings\n",
+ "\n",
+ "warnings.filterwarnings('ignore')\n",
"\n",
"# Replace values below with your own if using Redis Cloud instance\n",
"REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
@@ -207,322 +182,102 @@
},
{
"cell_type": "code",
- "execution_count": 5,
+ "execution_count": 48,
"id": "370c1fcc",
"metadata": {
- "id": "370c1fcc"
+ "colab": {
+ "base_uri": "https://localhost:8080/"
+ },
+ "id": "370c1fcc",
+ "outputId": "2b5297c6-83b7-468f-b2ac-c47acf13ba2e"
},
- "outputs": [],
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "True"
+ ]
+ },
+ "execution_count": 48,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
"source": [
"from redis import Redis\n",
"\n",
- "client = Redis.from_url(REDIS_URL)"
- ]
- },
- {
- "cell_type": "markdown",
- "id": "jCXiuk9ZTN_K",
- "metadata": {
- "id": "jCXiuk9ZTN_K"
- },
- "source": [
- "### Load Data"
+ "client = Redis.from_url(REDIS_URL)\n",
+ "client.ping()"
]
},
{
"cell_type": "code",
- "execution_count": 5,
- "id": "8d561462",
+ "execution_count": 4,
+ "id": "H4w8c3Bevzq4",
"metadata": {
"colab": {
- "base_uri": "https://localhost:8080/",
- "height": 206
+ "base_uri": "https://localhost:8080/"
},
- "id": "8d561462",
- "outputId": "04daf079-cd07-4369-b6ac-5c192b75163c"
+ "id": "H4w8c3Bevzq4",
+ "outputId": "a4d3b9a4-adda-436e-9aef-b4b0120720ab"
},
"outputs": [
{
"data": {
- "text/html": [
- "| \n", - " | title | \n", - "genre | \n", - "rating | \n", - "description | \n", - "
|---|---|---|---|---|
| 0 | \n", - "Explosive Pursuit | \n", - "action | \n", - "7 | \n", - "A daring cop chases a notorious criminal acros... | \n", - "
| 1 | \n", - "Skyfall | \n", - "action | \n", - "8 | \n", - "James Bond returns to track down a dangerous n... | \n", - "
| 2 | \n", - "Fast & Furious 9 | \n", - "action | \n", - "6 | \n", - "Dom and his crew face off against a high-tech ... | \n", - "
| 3 | \n", - "Black Widow | \n", - "action | \n", - "7 | \n", - "Natasha Romanoff confronts her dark past and f... | \n", - "
| 4 | \n", - "John Wick | \n", - "action | \n", - "8 | \n", - "A retired hitman seeks vengeance against those... | \n", - "
| \n", + " | title | \n", + "genre | \n", + "rating | \n", + "description | \n", + "vector | \n", + "
|---|---|---|---|---|---|
| 0 | \n", + "Explosive Pursuit | \n", + "action | \n", + "7 | \n", + "A daring cop chases a notorious criminal acros... | \n", + "b'\\x9bf|=\\na\\n;\\xbf\\x91\\xb7;\\x19\\xcb~\\xbd\\xd9d... | \n", + "
| 1 | \n", + "Skyfall | \n", + "action | \n", + "8 | \n", + "James Bond returns to track down a dangerous n... | \n", + "b'\\x9aD\\x9e\\xbd0\\x9b\\x89\\xbc\\xc3\\x16\\x95\\xbc\\x... | \n", + "
| 2 | \n", + "Fast & Furious 9 | \n", + "action | \n", + "6 | \n", + "Dom and his crew face off against a high-tech ... | \n", + "b'*\\xa5\\xc7\\xbc\\xf6,\\xa2=?\\x19H\\xbcK\\xc6t\\xbd\\... | \n", + "
| 3 | \n", + "Black Widow | \n", + "action | \n", + "7 | \n", + "Natasha Romanoff confronts her dark past and f... | \n", + "b'u\\xeb\\x85\\xbd\\x0e\\xcdo\\xbd&\\xe8\\xc2\\xbb6\\xcf... | \n", + "
| 4 | \n", + "John Wick | \n", + "action | \n", + "8 | \n", + "A retired hitman seeks vengeance against those... | \n", + "b'\\xaf<x\\xbb\\xfb.\\xc5=B\\x86:;\\xce\\xd0\\x94<\\xf9... | \n", + "
| \n", + " | id | \n", + "vector_distance | \n", + "title | \n", + "genre | \n", + "
|---|---|---|---|---|
| 0 | \n", + "movies:01JSHDN7Q4GG029M45HQY8Q5T2 | \n", + "0.64973795414 | \n", + "Fast & Furious 9 | \n", + "action | \n", + "
| 1 | \n", + "movies:01JSHDN7Q40QYH6Q6TD7ES4TSG | \n", + "0.763235092163 | \n", + "Mad Max: Fury Road | \n", + "action | \n", + "
| 2 | \n", + "movies:01JSHDN7Q4AS7C9VT582PWK14J | \n", + "0.792449712753 | \n", + "The Lego Movie | \n", + "comedy | \n", + "
| \n", + " | id | \n", + "vector_distance | \n", + "title | \n", + "genre | \n", + "
|---|---|---|---|---|
| 0 | \n", + "movies:01JSHDN7Q4GG029M45HQY8Q5T2 | \n", + "0.64973795414 | \n", + "Fast & Furious 9 | \n", + "action | \n", + "
| 1 | \n", + "movies:01JSHDN7Q40QYH6Q6TD7ES4TSG | \n", + "0.763235092163 | \n", + "Mad Max: Fury Road | \n", + "action | \n", + "
| 2 | \n", + "movies:01JSHDN7Q4AHFG0J8D7Q8QS1BG | \n", + "0.796153008938 | \n", + "Explosive Pursuit | \n", + "action | \n", + "