diff --git a/python-recipes/recommendation-systems/00_content_filtering.ipynb b/python-recipes/recommendation-systems/00_content_filtering.ipynb new file mode 100644 index 00000000..e01eba2c --- /dev/null +++ b/python-recipes/recommendation-systems/00_content_filtering.ipynb @@ -0,0 +1,1122 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "lzVKwB8Vy4oO" + }, + "source": [ + "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", + "\n", + "# Recommendation Systems: Content Filtering with RedisVL\n", + "\n", + "Recommendation systems are a common application of machine learning and serve many industries from e-commerce to music streaming platforms. However, there are many different architechtures that can be used.\n", + "\n", + "In this notebook we'll demonstrate how to build a **[content filtering](https://en.wikipedia.org/wiki/Recommender_system#:~:text=of%20hybrid%20systems.-,Content%2Dbased%20filtering,-%5Bedit%5D)**\n", + "recommender from scratch using `redisvl` and an IMDB movies dataset.\n", + "\n", + "## What is content filtering?\n", + "Content filtering recommender systems are built on the premise that a person will want to be recommended things that are similar to things they already like.\n", + "\n", + "In the case of movies, if a person watches and enjoys a nature documentary we should recommend other nature documentaries. Or if they like classic black & white horror films we should recommend more of those.\n", + "\n", + "The question we need to answer is, 'what does it mean for movies to be similar?'. There are exact matching strategies, like using a movie's labelled genre like 'Horror', or 'Sci Fi', but that can lock people in to only a few genres. Or what if it's not the genre that a person likes, but certain story arcs that are common among many genres?\n", + "\n", + "For our content filtering recommender we'll measure similarity between movies as semantic similarity of their descriptions and keywords.\n", + "\n", + "## Let's Begin!\n", + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_iw74d6SzA5i" + }, + "source": [ + "## Environment Setup\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WVau8LK81pJN" + }, + "source": [ + "### Install Python Dependencies" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "id": "HSWpCEdOzHyb" + }, + "outputs": [], + "source": [ + "# NBVAL_SKIP\n", + "!pip install -q redis redisvl sentence_transformers pandas requests" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "bEYZp90izwmZ" + }, + "source": [ + "### Install Redis Stack\n", + "\n", + "Later in this tutorial, Redis will be used to store, index, and query vector\n", + "embeddings. **We need to make sure we have a Redis instance available.**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "y04FGPfO0FNp" + }, + "source": [ + "#### Redis in Colab\n", + "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "SJD6uOnHz0Oq", + "outputId": "5b79aef9-fe3a-413c-c2b2-c73d370f3f46" + }, + "outputs": [], + "source": [ + "# NBVAL_SKIP\n", + "%%sh\n", + "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", + "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", + "sudo apt-get update > /dev/null 2>&1\n", + "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", + "redis-stack-server --daemonize yes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lX6eei__z9AK" + }, + "source": [ + "#### Other ways to get Redis\n", + "There are many ways to get the necessary redis-stack instance running\n", + "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.io/try-free/). Or, if you have your\n", + "own version of Redis Enterprise running, that works too!\n", + "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", + "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "skN74dq-1YqO" + }, + "source": [ + "### Define the Redis Connection URL\n", + "\n", + "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "eKDuyN0ky4oP" + }, + "outputs": [], + "source": [ + "import ast\n", + "import os\n", + "import pandas as pd\n", + "import pickle\n", + "import requests\n", + "import warnings\n", + "warnings.filterwarnings('ignore')\n", + "\n", + "# Replace values below with your own if using Redis Cloud instance\n", + "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", + "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", + "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", + "\n", + "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", + "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "aRqr7Um3y4oP" + }, + "source": [ + "## Prepare The Dataset" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "naSiAmZBy4oP" + }, + "source": [ + "Start by downloading the movies data and doing a quick inspection of it." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 293 + }, + "id": "vVyYhgoly4oP", + "outputId": "51bfe86a-95fd-416c-a629-ab43cda9531d" + }, + "outputs": [ + { + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "summary": "{\n \"name\": \"df\",\n \"rows\": 23922,\n \"fields\": [\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 23922,\n \"samples\": [\n \"The Graduate\",\n \"Ayngaran\",\n \"Acting Ka Bhoot\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"runtime\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 1526,\n \"samples\": [\n \"\\u20b93,500,000,000 (estimated)\",\n \"57 minutes\",\n \"$21,471,047\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"rating\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1.9521543600532218,\n \"min\": 0.0,\n \"max\": 9.9,\n \"num_unique_values\": 91,\n \"samples\": [\n 4.6,\n 0.0,\n 2.1\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"rating_count\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 107222,\n \"min\": 0,\n \"max\": 2600000,\n \"num_unique_values\": 1681,\n \"samples\": [\n 783000,\n 959,\n 3100\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"genres\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 741,\n \"samples\": [\n \"['Adventure', 'Comedy', 'Romance']\",\n \"['Adventure', 'Comedy', 'Film-Noir']\",\n \"['Adventure', 'Comedy', 'History']\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"overview\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 23485,\n \"samples\": [\n \"A young cavalry doctor, against orders, treats very sick Indians who are forced to stay on unhealthy land, which could lead to a war.\",\n \"An ex-policeman/school janitor (Billy Blanks) shows a new student (Kenn Scott) how to defend himself from a martial-arts bully.\",\n \"A socially-criticized, financially-cornered girl becomes an outlaw to dodge the situation.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"keywords\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 21132,\n \"samples\": [\n \"['dream', 'husband wife relationship', 'african american', 'uncle nephew relationship', 'teenage boy', 'teen angst', 'cynicism', 'midlife crisis', 'unrequited love', 'regret']\",\n \"['bare chested male', 'lion wrestling', 'man lion relationship', 'male underwear', 'briefs', 'blood', 'experiment', 'human animal relationship', 'home invasion', 'jungle']\",\n \"['thailand', 'evil child', 'tsunami', 'jungle', 'island', 'burma', 'boat', 'disembowelment', 'feral child', 'rape']\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"director\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 11405,\n \"samples\": [\n \"Franco Rossi\",\n \"Jamil Dehlavi\",\n \"Andrea Berloff\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"cast\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 23736,\n \"samples\": [\n \"['Leo McCarey', 'Mildred Cram', 'Cary Grant', 'Deborah Kerr', 'Richard Denning']\",\n \"['K\\u00f4sei Amano', 'Nozomi Band\\u00f4', 'Shigeaki Kubo', 'Shintar\\u00f4 Akiyama', 'K\\u00f4sei Amano']\",\n \"['Robert Sabaroff', 'Jim Brown', 'Diahann Carroll', 'Ernest Borgnine', 'Gordon Flemyng']\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"writer\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 15276,\n \"samples\": [\n \"Cris Loveless\",\n \"Anand Gandhi\",\n \"Mike Flanagan\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"year\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 134,\n \"samples\": [\n \"(XXXIII)\",\n \"1975\",\n \"2013\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"path\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 23922,\n \"samples\": [\n \"/title/tt0061722/\",\n \"/title/tt7023644/\",\n \"/title/tt17320574/\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", + "type": "dataframe", + "variable_name": "df" + }, + "text/html": [ + "\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
titleruntimeratingrating_countgenresoverviewkeywordsdirectorcastwriteryearpath
0The Story of the Kelly Gang1 hour 10 minutes6.0772['Action', 'Adventure', 'Biography']Story of Ned Kelly, an infamous 19th-century A...['ned kelly', 'australia', 'historic figure', ...Charles Tait['Elizabeth Tait', 'John Tait', 'Nicholas Brie...Charles Tait1906/title/tt0000574/
1Fantômas - À l'ombre de la guillotinenot-released6.92300['Crime', 'Drama']Inspector Juve is tasked to investigate and ca...['silent film', 'france', 'hotel', 'duchess', ...Louis Feuillade['Louis Feuillade', 'Pierre Souvestre', 'René ...Marcel Allain1913/title/tt0002844/
2Cabiria2 hours 28 minutes7.13500['Adventure', 'Drama', 'History']Cabiria is a Roman child when her home is dest...['carthage', 'slave', 'moloch', '3rd century b...Giovanni Pastrone['Titus Livius', 'Giovanni Pastrone', 'Italia ...Gabriele D'Annunzio1914/title/tt0003740/
3The Life of General Villanot-released6.765['Action', 'Adventure', 'Biography']The life and career of Panccho Villa from youn...['chihuahua mexico', 'chihuahua', 'sonora mexi...Christy Cabanne['Frank E. Woods', 'Raoul Walsh', 'Eagle Eye',...Raoul Walsh1914/title/tt0004223/
4The Patchwork Girl of Oznot-released5.4484['Adventure', 'Family', 'Fantasy']Ojo and Unc Nunkie are out of food, so they de...['silent film', 'journey', 'magic wand', 'wiza...J. Farrell MacDonald['Violet MacMillan', 'Frank Moore', 'Raymond R...L. Frank Baum1914/title/tt0004457/
\n", + "
\n", + "
\n", + "\n", + "
\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "
\n", + "\n", + "\n", + "
\n", + " \n", + "\n", + "\n", + "\n", + " \n", + "
\n", + "\n", + "
\n", + "
\n" + ], + "text/plain": [ + " title runtime rating \\\n", + "0 The Story of the Kelly Gang 1 hour 10 minutes 6.0 \n", + "1 Fantômas - À l'ombre de la guillotine not-released 6.9 \n", + "2 Cabiria 2 hours 28 minutes 7.1 \n", + "3 The Life of General Villa not-released 6.7 \n", + "4 The Patchwork Girl of Oz not-released 5.4 \n", + "\n", + " rating_count genres \\\n", + "0 772 ['Action', 'Adventure', 'Biography'] \n", + "1 2300 ['Crime', 'Drama'] \n", + "2 3500 ['Adventure', 'Drama', 'History'] \n", + "3 65 ['Action', 'Adventure', 'Biography'] \n", + "4 484 ['Adventure', 'Family', 'Fantasy'] \n", + "\n", + " overview \\\n", + "0 Story of Ned Kelly, an infamous 19th-century A... \n", + "1 Inspector Juve is tasked to investigate and ca... \n", + "2 Cabiria is a Roman child when her home is dest... \n", + "3 The life and career of Panccho Villa from youn... \n", + "4 Ojo and Unc Nunkie are out of food, so they de... \n", + "\n", + " keywords director \\\n", + "0 ['ned kelly', 'australia', 'historic figure', ... Charles Tait \n", + "1 ['silent film', 'france', 'hotel', 'duchess', ... Louis Feuillade \n", + "2 ['carthage', 'slave', 'moloch', '3rd century b... Giovanni Pastrone \n", + "3 ['chihuahua mexico', 'chihuahua', 'sonora mexi... Christy Cabanne \n", + "4 ['silent film', 'journey', 'magic wand', 'wiza... J. Farrell MacDonald \n", + "\n", + " cast writer \\\n", + "0 ['Elizabeth Tait', 'John Tait', 'Nicholas Brie... Charles Tait \n", + "1 ['Louis Feuillade', 'Pierre Souvestre', 'René ... Marcel Allain \n", + "2 ['Titus Livius', 'Giovanni Pastrone', 'Italia ... Gabriele D'Annunzio \n", + "3 ['Frank E. Woods', 'Raoul Walsh', 'Eagle Eye',... Raoul Walsh \n", + "4 ['Violet MacMillan', 'Frank Moore', 'Raymond R... L. Frank Baum \n", + "\n", + " year path \n", + "0 1906 /title/tt0000574/ \n", + "1 1913 /title/tt0002844/ \n", + "2 1914 /title/tt0003740/ \n", + "3 1914 /title/tt0004223/ \n", + "4 1914 /title/tt0004457/ " + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "try:\n", + " df = pd.read_csv(\"datasets/content_filtering/25k_imdb_movie_dataset.csv\")\n", + "except:\n", + " import requests\n", + " # download the file\n", + " url = 'https://redis-ai-resources.s3.us-east-2.amazonaws.com/recommenders/datasets/content-filtering/25k_imdb_movie_dataset.csv'\n", + " r = requests.get(url)\n", + "\n", + " #save the file as a csv\n", + " if not os.path.exists('./datasets/content_filtering'):\n", + " os.makedirs('./datasets/content_filtering')\n", + " with open('./datasets/content_filtering/25k_imdb_movie_dataset.csv', 'wb') as f:\n", + " f.write(r.content)\n", + " df = pd.read_csv(\"datasets/content_filtering/25k_imdb_movie_dataset.csv\")\n", + "\n", + "df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EwLWwLBSy4oQ" + }, + "source": [ + "As with any machine learning task, the first step is to clean our data.\n", + "\n", + "We'll drop some columns that we don't plan to use, and fill missing values with some reasonable defaults.\n", + "\n", + "Lastly, we'll do a quick check to make sure we've filled in all the null and missing values." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 366 + }, + "id": "3MGGS677y4oQ", + "outputId": "34db9d84-4557-49ca-f227-24e0c95e6268" + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
0
title0
rating0
rating_count0
genres0
overview0
keywords0
director0
cast0
year0
\n", + "

" + ], + "text/plain": [ + "title 0\n", + "rating 0\n", + "rating_count 0\n", + "genres 0\n", + "overview 0\n", + "keywords 0\n", + "director 0\n", + "cast 0\n", + "year 0\n", + "dtype: int64" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "roman_numerals = ['(I)','(II)','(III)','(IV)', '(V)', '(VI)', '(VII)', '(VIII)', '(IX)', '(XI)', '(XII)', '(XVI)', '(XIV)', '(XXXIII)', '(XVIII)', '(XIX)', '(XXVII)']\n", + "\n", + "def replace_year(x):\n", + " if x in roman_numerals:\n", + " return 1998 # the average year of the dataset\n", + " else:\n", + " return x\n", + "\n", + "df.drop(columns=['runtime', 'writer', 'path'], inplace=True)\n", + "df['year'] = df['year'].apply(replace_year) # replace roman numerals with average year\n", + "df['genres'] = df['genres'].apply(ast.literal_eval) # convert string representation of list to list\n", + "df['keywords'] = df['keywords'].apply(ast.literal_eval) # convert string representation of list to list\n", + "df['cast'] = df['cast'].apply(ast.literal_eval) # convert string representation of list to list\n", + "df = df[~df['overview'].isnull()] # drop rows with missing overviews\n", + "df = df[~df['overview'].isin(['none'])] # drop rows with 'none' as the overview\n", + "\n", + "# make sure we've filled all missing values\n", + "df.isnull().sum()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wvP1wPsky4oQ" + }, + "source": [ + "## Generate Vector Embeddings" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "cvrNi2mEy4oQ" + }, + "source": [ + "Since we movie similarity as semantic similarity of movie descriptions we need a way to generate semantic vector embeddings of these descriptions.\n", + "\n", + "RedisVL supports many different embedding generators. For this example we'll use a HuggingFace model that is rated well for semantic similarity use cases.\n", + "\n", + "RedisVL also supports complex query logic, beyond just vector similarity. To showcase this we'll generate an embedding from each movies' `overview` text and list of `plot keywords`,\n", + "and use the remaining fields like, `genres`, `year`, and `rating` as filterable fields to target our vector queries to.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 53 + }, + "id": "ha6fiptly4oQ", + "outputId": "db080921-c6ee-4b79-cf26-094409916ca2" + }, + "outputs": [ + { + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "string" + }, + "text/plain": [ + "'The Story of the Kelly Gang. Story of Ned Kelly, an infamous 19th-century Australian outlaw. ned kelly, australia, historic figure, australian western, first of its kind, directorial debut, australian history, 19th century, victoria australia, australian'" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# add a column to the dataframe with all the text we want to embed\n", + "df[\"full_text\"] = df[\"title\"] + \". \" + df[\"overview\"] + \" \" + df['keywords'].apply(lambda x: ', '.join(x))\n", + "df[\"full_text\"][0]" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "cT-RsO5Uy4oQ" + }, + "outputs": [], + "source": [ + "# NBVAL_SKIP\n", + "# # this step will take a while, but only needs to be done once for your entire dataset\n", + "# currently taking 10 minutes to run, so we've gone ahead and saved the vectors to a file for you\n", + "# if you don't want to wait, you can skip the cell and load the vectors from the file in the next cell\n", + "from redisvl.utils.vectorize import HFTextVectorizer\n", + "\n", + "vectorizer = HFTextVectorizer(model='sentence-transformers/paraphrase-MiniLM-L6-v2')\n", + "\n", + "df['embedding'] = df['full_text'].apply(lambda x: vectorizer.embed(x, as_buffer=False))\n", + "pickle.dump(df['embedding'], open('datasets/content_filtering/text_embeddings.pkl', 'wb'))" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "Dyxs5dyWy4oQ" + }, + "outputs": [], + "source": [ + "try:\n", + " with open('datasets/content_filtering/text_embeddings.pkl', 'rb') as vector_file:\n", + " df['embedding'] = pickle.load(vector_file)\n", + "except:\n", + " embeddings_url = 'https://redis-ai-resources.s3.us-east-2.amazonaws.com/recommenders/datasets/content-filtering/text_embeddings.pkl'\n", + " r = requests.get(embeddings_url)\n", + " with open('./datasets/content_filtering/text_embeddings.pkl', 'wb') as f:\n", + " f.write(r.content)\n", + " with open('datasets/content_filtering/text_embeddings.pkl', 'rb') as vector_file:\n", + " df['embedding'] = pickle.load(vector_file)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Fvc8MxVxy4oQ" + }, + "source": [ + "## Define our Search Schema\n", + "Our data is now ready to be loaded into Redis. The last step is to define our search index schema that specifies each of our data fields and the size and type of our embedding vectors.\n", + "\n", + "We'll load this from the accompanying `content_filtering_schema.yaml` file." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xIyHzviuy4oQ" + }, + "source": [ + "This schema defines what each entry will look like within Redis. It will need to specify the name of each field, like `title`, `rating`, and `rating-count`, as well as the type of each field, like `text` or `numeric`.\n", + "\n", + "The vector component of each entry similarly needs its dimension (dims), distance metric, algorithm and datatype (dtype) attributes specified." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "id": "fzfELmSjy4oR" + }, + "outputs": [], + "source": [ + "from redis import Redis\n", + "from redisvl.schema import IndexSchema\n", + "from redisvl.index import SearchIndex\n", + "\n", + "# define a redis client\n", + "client = Redis.from_url(REDIS_URL)\n", + "\n", + "# define our movie schema\n", + "movie_schema = IndexSchema.from_dict(\n", + " {\n", + " 'index': {\n", + " 'name': 'movies_recommendation',\n", + " 'prefix': 'movie',\n", + " 'storage_type': 'json'\n", + " },\n", + " 'fields': [\n", + " {'name': 'title', 'type': 'text'},\n", + " {'name': 'rating', 'type': 'numeric'},\n", + " {'name': 'rating_count', 'type': 'numeric'},\n", + " {'name': 'genres', 'type': 'tag'},\n", + " {'name': 'overview', 'type': 'text'},\n", + " {'name': 'keywords', 'type': 'tag'},\n", + " {'name': 'cast', 'type': 'tag'},\n", + " {'name': 'writer', 'type': 'text'},\n", + " {'name': 'year', 'type': 'numeric'},\n", + " {'name': 'full_text', 'type': 'text'},\n", + " {\n", + " 'name': 'embedding',\n", + " 'type': 'vector',\n", + " 'attrs': {\n", + " 'dims': 384,\n", + " 'algorithm': 'flat',\n", + " 'datatype': 'float32',\n", + " 'distance_metric': 'cosine'\n", + " }\n", + " }\n", + " ]\n", + "})\n", + "\n", + "index = SearchIndex(movie_schema, redis_client=client)\n", + "index.create(overwrite=True, drop=True)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "L9-aPwzby4oR" + }, + "source": [ + "## Load products into vector DB\n", + "Now that we have all our data cleaned and a defined schema we can load the data into RedisVL.\n", + "\n", + "We need to convert this data into a format that RedisVL can understand, which is a list of dictionaries.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "id": "Z45nA5Zoy4oR" + }, + "outputs": [], + "source": [ + "data = df.to_dict(orient='records')\n", + "keys = index.load(data)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IL8n0CxAy4oR" + }, + "source": [ + "## Querying to get recommendations\n", + "\n", + "We now have a working content filtering recommender system, all we need a starting point, so let's say we want to find movies similar to the movie with the title \"20,000 Leagues Under the Sea\"\n", + "\n", + "We can use the title to find the movie in the dataset and then use the vector to find similar movies." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "8tmwZ9fUy4oR", + "outputId": "8624ae3c-9384-42fd-f8c8-e87f959c2bba" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "{'id': 'movie:345589922cb348a098930568d5e7d02a', 'vector_distance': '0.584869861603', 'title': 'The Odyssey', 'overview': 'The aquatic adventure of the highly influential and fearlessly ambitious pioneer, innovator, filmmaker, researcher, and conservationist, Jacques-Yves Cousteau, covers roughly thirty years of an inarguably rich in achievements life.'}\n", + "{'id': 'movie:5147986e894d43879f4d90d6ed85dfd0', 'vector_distance': '0.633292078972', 'title': 'The Inventor', 'overview': 'Inventing flying contraptions, war machines and studying cadavers, Leonardo da Vinci tackles the meaning of life itself with the help of French princess Marguerite de Nevarre.'}\n", + "{'id': 'movie:da53156795ab4026b51e9dde88b02fa6', 'vector_distance': '0.658123493195', 'title': 'Ruin', 'overview': 'The film follows a nameless ex-Nazi captain who navigates the ruins of post-WWII Germany determined to atone for his crimes during the war by hunting down the surviving members of his former SS Death Squad.'}\n", + "{'id': 'movie:3e14e33c09944a70810aa7e24a2f78ef', 'vector_distance': '0.688094377518', 'title': 'The Raven', 'overview': 'A man with incredible powers is sought by the government and military.'}\n", + "{'id': 'movie:2a4c39f73e6b49e8b32ea1ce456e5833', 'vector_distance': '0.694671332836', 'title': 'Get the Girl', 'overview': 'Sebastain \"Bash\" Danye, a legendary gun for hire hangs up his weapon to retire peacefully with his \\'it\\'s complicated\\' partner Renee. Their quiet lives are soon interrupted when they find an unconscious woman on their property, Maddie. While nursing her back to health, some bad me... Read all'}\n" + ] + } + ], + "source": [ + "from redisvl.query import RangeQuery\n", + "\n", + "query_vector = df[df['title'] == '20,000 Leagues Under the Sea']['embedding'].values[0] # one good match\n", + "\n", + "query = RangeQuery(\n", + " vector=query_vector,\n", + " vector_field_name='embedding',\n", + " num_results=5,\n", + " distance_threshold=0.7,\n", + " return_fields=['title', 'overview', 'vector_distance']\n", + ")\n", + "\n", + "results = index.query(query)\n", + "for r in results:\n", + " print(r)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "pe6YzwT4y4oR" + }, + "source": [ + "## Generating user recommendations\n", + "This systems works, but we can make it even better.\n", + "\n", + "Production recommender systems often have fields that can be configured. Users can specify if they want to see a romantic comedy or a horror film, or only see new releases.\n", + "\n", + "Let's go ahead and add this functionality by using the tags we've defined in our schema." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "id": "wcRNJ4evy4oR" + }, + "outputs": [], + "source": [ + "from redisvl.query.filter import Tag, Num, Text\n", + "\n", + "def make_filter(genres=None, release_year=None, keywords=None):\n", + " flexible_filter = (\n", + " (Num(\"year\") > release_year) & # only show movies released after this year\n", + " (Tag(\"genres\") == genres) & # only show movies that match at least one in list of genres\n", + " (Text(\"full_text\") % keywords) # only show movies that contain at least one of the keywords\n", + " )\n", + " return flexible_filter\n", + "\n", + "def get_recommendations(movie_vector, num_results=5, distance=0.6, filter=None):\n", + " query = RangeQuery(\n", + " vector=movie_vector,\n", + " vector_field_name='embedding',\n", + " num_results=num_results,\n", + " distance_threshold=distance,\n", + " return_fields=['title', 'overview', 'genres'],\n", + " filter_expression=filter,\n", + " )\n", + "\n", + " recommendations = index.query(query)\n", + " return recommendations" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ulR_eyqCy4oR" + }, + "source": [ + "As a final demonstration we'll find movies similar to the classic horror film 'Nosferatu'.\n", + "The process has 3 steps:\n", + "- fetch the vector embedding of our film Nosferatu\n", + "- optionally define any hard filters we want. Here we'll specify we want horror movies made on or after 1990\n", + "- perform the vector range query to find similar movies that meet our filter criteria" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "TOb-p4p3y4oR", + "outputId": "e20dd31d-31fe-4dfc-e586-2e62f8e097b2" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "- Wolfman:\n", + "\t A man becomes afflicted by an ancient curse after he is bitten by a werewolf.\n", + "\t Genres: [\"Horror\"]\n", + "- Off Season:\n", + "\t Tenn's relentless search for his father takes him back to his childhood town only to find a community gripped by fear. As he travels deeper into the bitter winter wilderness of the town he uncovers a dreadful secret buried long ago.\n", + "\t Genres: [\"Horror\",\"Mystery\",\"Thriller\"]\n", + "- Pieces:\n", + "\t The co-eds of a Boston college campus are targeted by a mysterious killer who is creating a human jigsaw puzzle from their body parts.\n", + "\t Genres: [\"Horror\",\"Mystery\",\"Thriller\"]\n", + "- Cursed:\n", + "\t A prominent psychiatrist at a state run hospital wrestles with madness and a dark supernatural force as he and a female police detective race to stop an escaped patient from butchering five people held hostage in a remote mansion.\n", + "\t Genres: [\"Horror\",\"Thriller\"]\n", + "- The Home:\n", + "\t The Home unfolds after a young man is nearly killed during an accident that leaves him physically and emotionally scarred. To recuperate, he is taken to a secluded nursing home where the elderly residents appear to be suffering from delusions. But after witnessing a violent attac... Read all\n", + "\t Genres: [\"Action\",\"Fantasy\",\"Horror\"]\n" + ] + } + ], + "source": [ + "movie_vector = df[df['title'] == 'Nosferatu']['embedding'].values[0]\n", + "\n", + "filter = make_filter(genres=['Horror'], release_year=1990)\n", + "\n", + "recs = get_recommendations(movie_vector, distance=0.8, filter=filter)\n", + "\n", + "for rec in recs:\n", + " print(f\"- {rec['title']}:\\n\\t {rec['overview']}\\n\\t Genres: {rec['genres']}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FVQYQ26Sy4oR" + }, + "source": [ + "### Now you have a working content filtering recommender system with Redis.\n", + "Don't forget to clean up once you're done." + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "Iv-SSFgUy4oR", + "outputId": "33bb43f1-60e5-4d22-a283-2e2f7a87a612" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Deleted 143 keys\n" + ] + } + ], + "source": [ + "# clean up your index\n", + "while remaining := index.clear():\n", + " print(f\"Deleted {remaining} keys\")" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "redis-ai-res", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.10" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/python-recipes/recommendation-systems/01_collaborative_filtering.ipynb b/python-recipes/recommendation-systems/01_collaborative_filtering.ipynb new file mode 100644 index 00000000..1c800990 --- /dev/null +++ b/python-recipes/recommendation-systems/01_collaborative_filtering.ipynb @@ -0,0 +1,3154 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "0yt_EsBYy4BW" + }, + "source": [ + "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", + "\n", + "# Recommendation Systems: Collaborative Filtering with RedisVL\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Bs_UYLAcy4BY" + }, + "source": [ + "Recommendation systems are a common application of machine learning and serve many industries from e-commerce to music streaming platforms. However, there are many different architectures that can be followed to build a recommendation system.\n", + "\n", + "In a previous example notebook we demonstrated how to do **[content filtering with RedisVL](content_filtering.ipynb)**. We encourage you to start there before diving into this notebook.\n", + "\n", + "In this notebook we'll demonstrate how to build a **[collaborative filtering](https://en.wikipedia.org/wiki/Collaborative_filtering)**\n", + "recommendation system using `redisvl` and the large IMDB movies dataset.\n", + "\n", + "To generate vectors we'll use the popular Python package [Surprise](https://surpriselib.com/)\n", + "\n", + "\n", + "## Let's Begin\n", + "\"Open" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "fgm4G4iL5uk5" + }, + "source": [ + "## Environment Setup" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "-2w9rumN5xAl" + }, + "source": [ + "### Install Python Dependencies" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "5Be3atBDy4BY", + "outputId": "0cdf686f-b56f-4fe6-ce67-d2c073a5a63d" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/154.4 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[91m╸\u001b[0m \u001b[32m153.6/154.4 kB\u001b[0m \u001b[31m5.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m154.4/154.4 kB\u001b[0m \u001b[31m3.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25h Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n", + " Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n", + " Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m261.4/261.4 kB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m95.6/95.6 kB\u001b[0m \u001b[31m7.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m6.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25h Building wheel for scikit-surprise (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n" + ] + } + ], + "source": [ + "# NBVAL_SKIP\n", + "!pip install redis redisvl pandas requests scikit-surprise --quiet" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "TIyVITyg5zrY" + }, + "source": [ + "### Install Redis Stack\n", + "\n", + "Later in this tutorial, Redis will be used to store, index, and query vector\n", + "embeddings. **We need to make sure we have a Redis instance available.**" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "1XC8mwRQ51TB" + }, + "source": [ + "#### Redis in Colab\n", + "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "6md30bfs53wE", + "outputId": "52f2ae3e-c883-40e0-f48c-401c0727e99c" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb jammy main\n", + "Starting redis-stack-server, database path /var/lib/redis-stack\n" + ] + } + ], + "source": [ + "# NBVAL_SKIP\n", + "%%sh\n", + "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n", + "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n", + "sudo apt-get update > /dev/null 2>&1\n", + "sudo apt-get install redis-stack-server > /dev/null 2>&1\n", + "redis-stack-server --daemonize yes" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UeVEmWAS51VX" + }, + "source": [ + "#### Other ways to get Redis\n", + "There are many ways to get the necessary redis-stack instance running\n", + "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.io/try-free/). Or, if you have your\n", + "own version of Redis Enterprise running, that works too!\n", + "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n", + "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "xe3zxMKN6CWe" + }, + "source": [ + "### Define the Redis Connection URL\n", + "\n", + "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "id": "oKPmgBW1y4BY" + }, + "outputs": [], + "source": [ + "import os\n", + "import requests\n", + "import pandas as pd\n", + "import numpy as np\n", + "import warnings\n", + "warnings.filterwarnings('ignore')\n", + "\n", + "from surprise import SVD\n", + "from surprise import Dataset, Reader\n", + "from surprise.model_selection import train_test_split\n", + "\n", + "\n", + "# Replace values below with your own if using Redis Cloud instance\n", + "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", + "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", + "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", + "\n", + "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", + "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "phSPoyHwy4BZ" + }, + "source": [ + "## Prepare The Dataset\n", + "\n", + "To build a collaborative filtering example using the Surprise library and the Movies dataset, we need to first load the data, format it according to the requirements of Surprise, and then apply a collaborative filtering algorithm like SVD." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "oPL_ynlay4BZ" + }, + "outputs": [], + "source": [ + "def fetch_dataframe(file_name):\n", + " try:\n", + " df = pd.read_csv('datasets/collaborative_filtering/' + file_name)\n", + " except:\n", + " url = 'https://redis-ai-resources.s3.us-east-2.amazonaws.com/recommenders/datasets/collaborative-filtering/'\n", + " r = requests.get(url + file_name)\n", + " if not os.path.exists('datasets/collaborative_filtering'):\n", + " os.makedirs('datasets/collaborative_filtering')\n", + " with open('datasets/collaborative_filtering/' + file_name, 'wb') as f:\n", + " f.write(r.content)\n", + " df = pd.read_csv('datasets/collaborative_filtering/' + file_name)\n", + " return df\n" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "id": "d6tt04l3y4BZ" + }, + "outputs": [], + "source": [ + "ratings_df = fetch_dataframe('ratings_small.csv') # for a larger example use 'ratings.csv' instead\n", + "\n", + "# only keep the columns we need: userId, movieId, rating\n", + "ratings_df = ratings_df[['userId', 'movieId', 'rating']]\n", + "\n", + "reader = Reader(rating_scale=(0.0, 5.0))\n", + "\n", + "ratings_data = Dataset.load_from_df(ratings_df, reader)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "JfoGb1Pgy4BZ" + }, + "source": [ + "# What is Collaborative Filtering" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "e6Ok7gnBy4BZ" + }, + "source": [ + "A lot is going to happen in the code cell below. We split our full data into train and test sets. We defined the collaborative filtering algorithm to use, which in this case is the Singular Value Decomposition (SVD) algorithm. lastly, we fit our model to our data.\n", + "\n", + "It's worth going into more detail why we chose this algorithm and what it is computing in the `svd.fit(train_set)` method we're calling.\n", + "First, let's think about what data it's receiving - our ratings data. This only contains the userIds, movieIds, and the user's ratings of their watched movies on a scale of 1 to 5.\n", + "\n", + "We can put this data into a matrix with rows being users and columns being movies\n", + "\n", + "| RATINGS| movie_1 | movie_2 | movie_3 | movie_4 | movie_5 | movie_6 | ....... |\n", + "| ----- | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n", + "| user_1 | 4 | 1 | | 4 | | 5 | |\n", + "| user_2 | | 5 | 5 | 2 | 1 | | |\n", + "| user_3 | | | | | 1 | | |\n", + "| user_4 | 4 | 1 | | 4 | | ? | |\n", + "| user_5 | | 4 | 5 | 2 | | | |\n", + "| ...... | | | | | | | |\n", + "\n", + "Our empty cells aren't zero's, they're missing ratings, so `user_1` has never rated `movie_3`. They may like it or hate it." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Hrcaomgby4BZ" + }, + "source": [ + "Unlike Content Filtering, here we're only considering the ratings that users assign. We don't know the plot or genre or release year of any of these films. We don't even know the title.\n", + "But we can still build a recommender by assuming that users have similar tastes to each other. As an intuitive example, we can see that `user_1` and `user_4` have very similar ratings on several movies, so we will assume that `user_4` will rate `movie_6` highly, just as `user_1` did. This is the idea behind collaborative filtering." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "4lhR94bky4BZ" + }, + "source": [ + "That's the intuition, but what about the math? Since we only have this matrix to work with, what we want to do is decompose it into two constituent matrices.\n", + "Lets call our ratings matrix `[R]`. We want to find two other matrices, a user matrix `[U]`, and a movies matrix `[M]` that fit the equation:\n", + "\n", + "`[U] * [M] = [R]`\n", + "\n", + "`[U]` will look like:\n", + "|user_1_feature_1 | user_1_feature_2 | user_1_feature_3 | user_1_feature_4 | ... | user_1_feature_k |\n", + "| ----- | --------- | --------- | --------- | --- | --------- |\n", + "|user_2_feature_1 | user_2_feature_2 | user_2_feature_3 | user_2_feature_4 | ... | user_2_feature_k |\n", + "|user_3_feature_1 | user_3_feature_2 | user_3_feature_3 | user_3_feature_4 | ... | user_3_feature_k |\n", + "| ... | . | . | . | ... | . |\n", + "|user_N_feature_1 | user_N_feature_2 | user_N_feature_3 | user_N_feature_4 | ... | user_N_feature_k |\n", + "\n", + "`[M]` will look like:\n", + "\n", + "| movie_1_feature_1 | movie_2_feature_1 | movie_3_feature_1 | ... | movie_M_feature_1 |\n", + "| --- | --- | --- | --- | --- |\n", + "| movie_1_feature_2 | movie_2_feature_2 | movie_3_feature_2 | ... | movie_M_feature_2 |\n", + "| movie_1_feature_3 | movie_2_feature_3 | movie_3_feature_3 | ... | movie_M_feature_3 |\n", + "| movie_1_feature_4 | movie_2_feature_4 | movie_3_feature_4 | ... | movie_M_feature_4 |\n", + "| ... | . | . | ... | . |\n", + "| movie_1_feature_k | movie_2_feature_k | movie_3_feature_k | ... | movie_M_feature_k |\n", + "\n", + "\n", + "these features are called the latent features (or latent factors) and are the values we're trying to find when we call the `svd.fit(training_data)` method. The algorithm that computes these features from our ratings matrix is the SVD algorithm. The number of users and movies is set by our data. The size of the latent feature vectors `k` is a parameter we choose. We'll keep it at the default 100 for this notebook." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "79Jr0aq7y4BZ", + "outputId": "3dc28253-6919-41dd-9bc6-dc3436d8f093" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# split the data into training and testing sets (80% train, 20% test)\n", + "train_set, test_set = train_test_split(ratings_data, test_size=0.2)\n", + "\n", + "# use SVD (Singular Value Decomposition) for collaborative filtering\n", + "svd = SVD(n_factors=100, biased=False) # we'll set biased to False so that predictions are of the form \"rating_prediction = user_vector dot item_vector\"\n", + "\n", + "# train the algorithm on the train_set\n", + "svd.fit(train_set)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "Nzc1RaQiy4Ba" + }, + "source": [ + "## Extracting The User and Movie Vectors" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "IE-7Noq1y4Ba" + }, + "source": [ + "Now that the SVD algorithm has computed our `[U]` and `[M]` matrices - which are both really just lists of vectors - we can load them into our Redis instance.\n", + "\n", + "The Surprise SVD model stores user and movie vectors in two attributes:\n", + "\n", + "`svd.pu`: user features matrix (a matrix where each row corresponds to the latent features of a user).\n", + "`svd.qi`: item features matrix (a matrix where each row corresponds to the latent features of an item/movie).\n", + "\n", + "It's worth noting that the matrix `svd.qi` is the transpose of the matrix `[M]` we defined above. This way each row corresponds to one movie." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "qiBeC3RTy4Ba", + "outputId": "1a75d9d0-5de7-4f61-8984-110af1e553bf" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "we have 671 users with feature vectors of size 100\n", + "we have 8416 movies with feature vectors of size 100\n" + ] + } + ], + "source": [ + "user_vectors = svd.pu # user latent features (matrix)\n", + "movie_vectors = svd.qi # movie latent features (matrix)\n", + "\n", + "print(f'we have {user_vectors.shape[0]} users with feature vectors of size {user_vectors.shape[1]}')\n", + "print(f'we have {movie_vectors.shape[0]} movies with feature vectors of size {movie_vectors.shape[1]}')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2OUMFAAUy4Ba" + }, + "source": [ + "# Predicting User Ratings\n", + "The great thing about collaborative filtering is that using our user and movie vectors we can predict the rating any user will give to any movie in our dataset.\n", + "And unlike content filtering, there is no assumption that all the movies a user will be recommended are similar to each other. A user can be recommended dark horror films and light-hearted animations.\n", + "\n", + "Looking back at our SVD algorithm the equation is [User_features] * [Movie_features].transpose = [Ratings]\n", + "So to get a prediction of what a user will rate a movie they haven't seen yet we just need to take the dot product of that user's feature vector and a movie's feature vector." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "1nZCx8BEy4Ba", + "outputId": "ca596038-b682-4fd9-eac3-6ebeab25bc37" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "the predicted rating of user 347 on movie 5515 is 1.3424785244492834\n" + ] + } + ], + "source": [ + "# surprise casts userId and movieId to inner ids, so we have to use their mapping to know which rows to use\n", + "inner_uid = train_set.to_inner_uid(347) # userId\n", + "inner_iid = train_set.to_inner_iid(5515) # movieId\n", + "\n", + "# predict one user's rating of one film\n", + "predicted_rating = np.dot(user_vectors[inner_uid], movie_vectors[inner_iid])\n", + "print(f'the predicted rating of user {347} on movie {5515} is {predicted_rating}')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "EpWDhSr8y4Ba" + }, + "source": [ + "## Adding Movie Data\n", + "while our collaborative filtering algorithm was trained solely on user's ratings of movies, and doesn't require any data about the movies themselves - like the title, genres, or release year - we'll want that information stored as metadata.\n", + "\n", + "We can grab this data from our `movies_metadata.csv` file, clean it, and join it to our user ratings via the `movieId` column" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 707 + }, + "id": "iI3JW5Epy4Ba", + "outputId": "86221795-5001-4d54-ef67-87570f8f03f3" + }, + "outputs": [ + { + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "type": "dataframe", + "variable_name": "movies_df" + }, + "text/html": [ + "\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
belongs_to_collectionbudgetgenreshomepageidimdb_idoriginal_languageoriginal_titleoverviewpopularity...release_daterevenueruntimespoken_languagesstatustaglinetitlevideovote_averagevote_count
0{'id': 10194, 'name': 'Toy Story Collection', ...30000000[{'id': 16, 'name': 'Animation'}, {'id': 35, '...http://toystory.disney.com/toy-story862tt0114709enToy StoryLed by Woody, Andy's toys live happily in his ...21.946943...1995-10-3037355403381.0[{'iso_639_1': 'en', 'name': 'English'}]ReleasedNaNToy StoryFalse7.75415
1NaN65000000[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...NaN8844tt0113497enJumanjiWhen siblings Judy and Peter discover an encha...17.015539...1995-12-15262797249104.0[{'iso_639_1': 'en', 'name': 'English'}, {'iso...ReleasedRoll the dice and unleash the excitement!JumanjiFalse6.92413
2{'id': 119050, 'name': 'Grumpy Old Men Collect...0[{'id': 10749, 'name': 'Romance'}, {'id': 35, ...NaN15602tt0113228enGrumpier Old MenA family wedding reignites the ancient feud be...11.712900...1995-12-220101.0[{'iso_639_1': 'en', 'name': 'English'}]ReleasedStill Yelling. Still Fighting. Still Ready for...Grumpier Old MenFalse6.592
3NaN16000000[{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam...NaN31357tt0114885enWaiting to ExhaleCheated on, mistreated and stepped on, the wom...3.859495...1995-12-2281452156127.0[{'iso_639_1': 'en', 'name': 'English'}]ReleasedFriends are the people who let you be yourself...Waiting to ExhaleFalse6.134
4{'id': 96871, 'name': 'Father of the Bride Col...0[{'id': 35, 'name': 'Comedy'}]NaN11862tt0113041enFather of the Bride Part IIJust when George Banks has recovered from his ...8.387519...1995-02-1076578911106.0[{'iso_639_1': 'en', 'name': 'English'}]ReleasedJust When His World Is Back To Normal... He's ...Father of the Bride Part IIFalse5.7173
\n", + "

5 rows × 23 columns

\n", + "
\n", + "
\n", + "\n", + "
\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "
\n", + "\n", + "\n", + "
\n", + " \n", + "\n", + "\n", + "\n", + " \n", + "
\n", + "\n", + "
\n", + "
\n" + ], + "text/plain": [ + " belongs_to_collection budget \\\n", + "0 {'id': 10194, 'name': 'Toy Story Collection', ... 30000000 \n", + "1 NaN 65000000 \n", + "2 {'id': 119050, 'name': 'Grumpy Old Men Collect... 0 \n", + "3 NaN 16000000 \n", + "4 {'id': 96871, 'name': 'Father of the Bride Col... 0 \n", + "\n", + " genres \\\n", + "0 [{'id': 16, 'name': 'Animation'}, {'id': 35, '... \n", + "1 [{'id': 12, 'name': 'Adventure'}, {'id': 14, '... \n", + "2 [{'id': 10749, 'name': 'Romance'}, {'id': 35, ... \n", + "3 [{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam... \n", + "4 [{'id': 35, 'name': 'Comedy'}] \n", + "\n", + " homepage id imdb_id original_language \\\n", + "0 http://toystory.disney.com/toy-story 862 tt0114709 en \n", + "1 NaN 8844 tt0113497 en \n", + "2 NaN 15602 tt0113228 en \n", + "3 NaN 31357 tt0114885 en \n", + "4 NaN 11862 tt0113041 en \n", + "\n", + " original_title \\\n", + "0 Toy Story \n", + "1 Jumanji \n", + "2 Grumpier Old Men \n", + "3 Waiting to Exhale \n", + "4 Father of the Bride Part II \n", + "\n", + " overview popularity ... \\\n", + "0 Led by Woody, Andy's toys live happily in his ... 21.946943 ... \n", + "1 When siblings Judy and Peter discover an encha... 17.015539 ... \n", + "2 A family wedding reignites the ancient feud be... 11.712900 ... \n", + "3 Cheated on, mistreated and stepped on, the wom... 3.859495 ... \n", + "4 Just when George Banks has recovered from his ... 8.387519 ... \n", + "\n", + " release_date revenue runtime \\\n", + "0 1995-10-30 373554033 81.0 \n", + "1 1995-12-15 262797249 104.0 \n", + "2 1995-12-22 0 101.0 \n", + "3 1995-12-22 81452156 127.0 \n", + "4 1995-02-10 76578911 106.0 \n", + "\n", + " spoken_languages status \\\n", + "0 [{'iso_639_1': 'en', 'name': 'English'}] Released \n", + "1 [{'iso_639_1': 'en', 'name': 'English'}, {'iso... Released \n", + "2 [{'iso_639_1': 'en', 'name': 'English'}] Released \n", + "3 [{'iso_639_1': 'en', 'name': 'English'}] Released \n", + "4 [{'iso_639_1': 'en', 'name': 'English'}] Released \n", + "\n", + " tagline \\\n", + "0 NaN \n", + "1 Roll the dice and unleash the excitement! \n", + "2 Still Yelling. Still Fighting. Still Ready for... \n", + "3 Friends are the people who let you be yourself... \n", + "4 Just When His World Is Back To Normal... He's ... \n", + "\n", + " title video vote_average vote_count \n", + "0 Toy Story False 7.7 5415 \n", + "1 Jumanji False 6.9 2413 \n", + "2 Grumpier Old Men False 6.5 92 \n", + "3 Waiting to Exhale False 6.1 34 \n", + "4 Father of the Bride Part II False 5.7 173 \n", + "\n", + "[5 rows x 23 columns]" + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "movies_df = fetch_dataframe('movies_metadata.csv')\n", + "movies_df.head()" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 554 + }, + "id": "-rzS_ZKty4Ba", + "outputId": "9d0130cb-0b64-438d-b343-41433c668d52" + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
0
budget0
genres0
id0
imdb_id0
original_language0
overview0
popularity0
release_date0
revenue0
runtime0
status0
tagline0
title0
vote_average0
vote_count0
\n", + "

" + ], + "text/plain": [ + "budget 0\n", + "genres 0\n", + "id 0\n", + "imdb_id 0\n", + "original_language 0\n", + "overview 0\n", + "popularity 0\n", + "release_date 0\n", + "revenue 0\n", + "runtime 0\n", + "status 0\n", + "tagline 0\n", + "title 0\n", + "vote_average 0\n", + "vote_count 0\n", + "dtype: int64" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "\n", + "import datetime\n", + "\n", + "movies_df.drop(columns=['homepage', 'production_countries', 'production_companies', 'spoken_languages', 'video', 'original_title', 'video', 'poster_path', 'belongs_to_collection'], inplace=True)\n", + "\n", + "# drop rows that have missing values\n", + "movies_df.dropna(subset=['imdb_id'], inplace=True)\n", + "\n", + "movies_df['original_language'] = movies_df['original_language'].fillna('unknown')\n", + "movies_df['overview'] = movies_df['overview'].fillna('')\n", + "movies_df['popularity'] = movies_df['popularity'].fillna(0)\n", + "movies_df['release_date'] = movies_df['release_date'].fillna('1900-01-01').apply(lambda x: datetime.datetime.strptime(x, \"%Y-%m-%d\").timestamp())\n", + "movies_df['revenue'] = movies_df['revenue'].fillna(0)\n", + "movies_df['runtime'] = movies_df['runtime'].fillna(0)\n", + "movies_df['status'] = movies_df['status'].fillna('unknown')\n", + "movies_df['tagline'] = movies_df['tagline'].fillna('')\n", + "movies_df['title'] = movies_df['title'].fillna('')\n", + "movies_df['vote_average'] = movies_df['vote_average'].fillna(0)\n", + "movies_df['vote_count'] = movies_df['vote_count'].fillna(0)\n", + "movies_df['genres'] = movies_df['genres'].apply(lambda x: [g['name'] for g in eval(x)] if x != '' else []) # convert to a list of genre names\n", + "movies_df['imdb_id'] = movies_df['imdb_id'].apply(lambda x: x[2:] if str(x).startswith('tt') else x).astype(int) # remove leading 'tt' from imdb_id\n", + "\n", + "# make sure we've filled all missing values\n", + "movies_df.isnull().sum()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "RRvIphk4y4Ba" + }, + "source": [ + "We'll have to map these movies to their ratings, which we'll do so with the `links.csv` file that matches `movieId`, `imdbId`, and `tmdbId`.\n", + "Let's do that now." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "metadata": { + "id": "5-rMcu06y4Ba" + }, + "outputs": [], + "source": [ + "links_df = fetch_dataframe('links_small.csv') # for a larger example use 'links.csv' instead\n", + "\n", + "movies_df = movies_df.merge(links_df, left_on='imdb_id', right_on='imdbId', how='inner')" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "P99FGXl8y4Ba" + }, + "source": [ + "We'll want to move our SVD user vectors and movie vectors and their corresponding userId and movieId into 2 dataframes for later processing." + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 623 + }, + "id": "oXzaoT-7y4Ba", + "outputId": "cdb6f5f1-2546-4dc8-dafb-ca761560fa67" + }, + "outputs": [ + { + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "summary": "{\n \"name\": \"movies_df\",\n \"rows\": 8376,\n \"fields\": [\n {\n \"column\": \"budget\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 34480356,\n \"min\": 0,\n \"max\": 380000000,\n \"num_unique_values\": 569,\n \"samples\": [\n 3900000,\n 34000000,\n 1020000\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"genres\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"id\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 59272,\n \"min\": 2,\n \"max\": 416437,\n \"num_unique_values\": 8370,\n \"samples\": [\n 57103,\n 19994,\n 28476\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"imdb_id\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 707596,\n \"min\": 417,\n \"max\": 5794766,\n \"num_unique_values\": 8370,\n \"samples\": [\n 52619,\n 1131734,\n 64806\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"original_language\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 40,\n \"samples\": [\n \"no\",\n \"cs\",\n \"vi\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"overview\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 8355,\n \"samples\": [\n \"Americans Jeff and Tommy, hunting in Scotland, stumble upon a village - Brigadoon. They soon learn that the town appears once every 100 years in order to preserve its peace and special beauty. The citizens go to bed at night and when they wake up, it's 100 years later. Tommy falls in love with a beautiful young woman, Fiona, and is torn between staying or going back to his hectic life in New York.\",\n \"Samba migrated to France 10 years ago from Senegal, and has since been plugging away at various lowly jobs. Alice is a senior executive who has recently undergone a burnout. Both struggle to get out of their dead-end lives. Samba's willing to do whatever it takes to get working papers, while Alice tries to get her life back on track until fate draws them together.\",\n \"In the future, medical technology has advanced to the point where people can buy artificial organs to extend their lives. But if they default on payments, an organization known as the Union sends agents to repossess the organs. Remy is one of the best agents in the business, but when he becomes the recipient of an artificial heart, he finds himself in the same dire straits as his many victims.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"popularity\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 9.65651267841049,\n \"min\": 4e-06,\n \"max\": 547.488298,\n \"num_unique_values\": 8370,\n \"samples\": [\n 11.519662,\n 11.23915,\n 1.567121\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"release_date\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 601949685.0164723,\n \"min\": -2208988800.0,\n \"max\": 1473897600.0,\n \"num_unique_values\": 5636,\n \"samples\": [\n 484963200.0,\n 873676800.0,\n 1076284800.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"revenue\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 134749919,\n \"min\": 0,\n \"max\": 2787965087,\n \"num_unique_values\": 4325,\n \"samples\": [\n 41610884,\n 115101622,\n 15520023\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"runtime\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 31.141679628227166,\n \"min\": 0.0,\n \"max\": 1140.0,\n \"num_unique_values\": 226,\n \"samples\": [\n 78.0,\n 720.0,\n 155.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"status\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Rumored\",\n \"In Production\",\n \"unknown\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"tagline\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 6563,\n \"samples\": [\n \"She's the hottest thing on the beach. She's also his best friend's daughter!\",\n \"It's a comedy. And a drama. Just like life.\",\n \"You're all going to die.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 8126,\n \"samples\": [\n \"The Deep End of the Ocean\",\n \"Red River\",\n \"The End of Evangelion\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"vote_average\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1.0010720538980293,\n \"min\": 0.0,\n \"max\": 10.0,\n \"num_unique_values\": 72,\n \"samples\": [\n 5.7,\n 3.4,\n 7.3\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"vote_count\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1029,\n \"min\": 0,\n \"max\": 14075,\n \"num_unique_values\": 1719,\n \"samples\": [\n 358,\n 3576,\n 1045\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"movieId\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 8370,\n \"samples\": [\n \"34608\",\n \"71205\",\n \"8015\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"imdbId\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 707596,\n \"min\": 417,\n \"max\": 5794766,\n \"num_unique_values\": 8370,\n \"samples\": [\n 52619,\n 1131734,\n 64806\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"tmdbId\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 59097.38021323738,\n \"min\": 2.0,\n \"max\": 416437.0,\n \"num_unique_values\": 8370,\n \"samples\": [\n 57103.0,\n 19994.0,\n 28476.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"movie_vector\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", + "type": "dataframe", + "variable_name": "movies_df" + }, + "text/html": [ + "\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
budgetgenresidimdb_idoriginal_languageoverviewpopularityrelease_daterevenueruntimestatustaglinetitlevote_averagevote_countmovieIdimdbIdtmdbIdmovie_vector
030000000[Animation, Comedy, Family]862114709enLed by Woody, Andy's toys live happily in his ...21.946943815011200.037355403381.0ReleasedToy Story7.754151114709862.0[-0.49346923675184573, -0.16727664416375823, 0...
165000000[Adventure, Fantasy, Family]8844113497enWhen siblings Judy and Peter discover an encha...17.015539818985600.0262797249104.0ReleasedRoll the dice and unleash the excitement!Jumanji6.9241321134978844.0[-0.10770150143386471, -0.007231946857062053, ...
20[Romance, Comedy]15602113228enA family wedding reignites the ancient feud be...11.712900819590400.00101.0ReleasedStill Yelling. Still Fighting. Still Ready for...Grumpier Old Men6.592311322815602.0[-0.3496835380235819, 0.024321376508654746, -0...
316000000[Comedy, Drama, Romance]31357114885enCheated on, mistreated and stepped on, the wom...3.859495819590400.081452156127.0ReleasedFriends are the people who let you be yourself...Waiting to Exhale6.134411488531357.0[0.011927156686291933, -0.08701536485937247, 0...
40[Comedy]11862113041enJust when George Banks has recovered from his ...8.387519792374400.076578911106.0ReleasedJust When His World Is Back To Normal... He's ...Father of the Bride Part II5.7173511304111862.0[-0.07110798835836052, 0.18216637030291705, -0...
\n", + "
\n", + "
\n", + "\n", + "
\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "
\n", + "\n", + "\n", + "
\n", + " \n", + "\n", + "\n", + "\n", + " \n", + "
\n", + "\n", + "
\n", + "
\n" + ], + "text/plain": [ + " budget genres id imdb_id original_language \\\n", + "0 30000000 [Animation, Comedy, Family] 862 114709 en \n", + "1 65000000 [Adventure, Fantasy, Family] 8844 113497 en \n", + "2 0 [Romance, Comedy] 15602 113228 en \n", + "3 16000000 [Comedy, Drama, Romance] 31357 114885 en \n", + "4 0 [Comedy] 11862 113041 en \n", + "\n", + " overview popularity \\\n", + "0 Led by Woody, Andy's toys live happily in his ... 21.946943 \n", + "1 When siblings Judy and Peter discover an encha... 17.015539 \n", + "2 A family wedding reignites the ancient feud be... 11.712900 \n", + "3 Cheated on, mistreated and stepped on, the wom... 3.859495 \n", + "4 Just when George Banks has recovered from his ... 8.387519 \n", + "\n", + " release_date revenue runtime status \\\n", + "0 815011200.0 373554033 81.0 Released \n", + "1 818985600.0 262797249 104.0 Released \n", + "2 819590400.0 0 101.0 Released \n", + "3 819590400.0 81452156 127.0 Released \n", + "4 792374400.0 76578911 106.0 Released \n", + "\n", + " tagline \\\n", + "0 \n", + "1 Roll the dice and unleash the excitement! \n", + "2 Still Yelling. Still Fighting. Still Ready for... \n", + "3 Friends are the people who let you be yourself... \n", + "4 Just When His World Is Back To Normal... He's ... \n", + "\n", + " title vote_average vote_count movieId imdbId \\\n", + "0 Toy Story 7.7 5415 1 114709 \n", + "1 Jumanji 6.9 2413 2 113497 \n", + "2 Grumpier Old Men 6.5 92 3 113228 \n", + "3 Waiting to Exhale 6.1 34 4 114885 \n", + "4 Father of the Bride Part II 5.7 173 5 113041 \n", + "\n", + " tmdbId movie_vector \n", + "0 862.0 [-0.49346923675184573, -0.16727664416375823, 0... \n", + "1 8844.0 [-0.10770150143386471, -0.007231946857062053, ... \n", + "2 15602.0 [-0.3496835380235819, 0.024321376508654746, -0... \n", + "3 31357.0 [0.011927156686291933, -0.08701536485937247, 0... \n", + "4 11862.0 [-0.07110798835836052, 0.18216637030291705, -0... " + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# build a dataframe out of the user vectors and their userIds\n", + "user_vectors_and_ids = {train_set.to_raw_uid(inner_id): user_vectors[inner_id].tolist() for inner_id in train_set.all_users()}\n", + "user_vector_df = pd.Series(user_vectors_and_ids).to_frame('user_vector')\n", + "\n", + "# now do the same for the movie vectors and their movieIds\n", + "movie_vectors_and_ids = {train_set.to_raw_iid(inner_id): movie_vectors[inner_id].tolist() for inner_id in train_set.all_items()}\n", + "movie_vector_df = pd.Series(movie_vectors_and_ids).to_frame('movie_vector')\n", + "\n", + "# merge the movie vector series with the movies dataframe using the movieId and id fields\n", + "movies_df = movies_df.merge(movie_vector_df, left_on='movieId', right_index=True, how='inner')\n", + "movies_df['movieId'] = movies_df['movieId'].apply(lambda x: str(x)) # need to cast to a string as this is a tag field in our search schema\n", + "movies_df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2X0TKN-iy4Ba" + }, + "source": [ + "## Redis Handles the Scale\n", + "\n", + "Especially for large datasets like the 45,000 movie catalog we're dealing with, you'll want Redis to do the heavy lifting of vector search.\n", + "All that's needed is to define the search index and load our data we've cleaned and merged with our vectors.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "id": "Tr4jRdydy4Ba" + }, + "outputs": [], + "source": [ + "from redis import Redis\n", + "from redisvl.schema import IndexSchema\n", + "from redisvl.index import SearchIndex\n", + "\n", + "client = Redis.from_url(REDIS_URL)\n", + "\n", + "movie_schema = IndexSchema.from_dict({\n", + " 'index': {\n", + " 'name': 'movies',\n", + " 'prefix': 'movie',\n", + " 'storage_type': 'json'\n", + " },\n", + " 'fields': [\n", + " {'name': 'movieId','type': 'tag'},\n", + " {'name': 'genres', 'type': 'tag'},\n", + " {'name': 'original_language', 'type': 'tag'},\n", + " {'name': 'overview', 'type': 'text'},\n", + " {'name': 'popularity', 'type': 'numeric'},\n", + " {'name': 'release_date', 'type': 'numeric'},\n", + " {'name': 'revenue', 'type': 'numeric'},\n", + " {'name': 'runtime', 'type': 'numeric'},\n", + " {'name': 'status', 'type': 'tag'},\n", + " {'name': 'tagline', 'type': 'text'},\n", + " {'name': 'title', 'type': 'text'},\n", + " {'name': 'vote_average', 'type': 'numeric'},\n", + " {'name': 'vote_count', 'type': 'numeric'},\n", + " {\n", + " 'name': 'movie_vector',\n", + " 'type': 'vector',\n", + " 'attrs': {\n", + " 'dims': 100,\n", + " 'algorithm': 'flat',\n", + " 'datatype': 'float32',\n", + " 'distance_metric': 'ip'\n", + " }\n", + " }\n", + " ]\n", + "})\n", + "\n", + "\n", + "movie_index = SearchIndex(movie_schema, redis_client=client)\n", + "movie_index.create(overwrite=True, drop=True)\n", + "\n", + "movie_keys = movie_index.load(movies_df.to_dict(orient='records'))" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 710 + }, + "id": "OfM7T2Qby4Bb", + "outputId": "13744db3-124c-42ac-a33a-dc91fb9b7618" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "number of movies 8376\n", + "size of movie df 8376\n", + "unique movie ids 8370\n", + "unique movie titles 8126\n", + "unique movies rated 9065\n" + ] + }, + { + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "summary": "{\n \"name\": \"movies_df\",\n \"rows\": 8376,\n \"fields\": [\n {\n \"column\": \"budget\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 34480356,\n \"min\": 0,\n \"max\": 380000000,\n \"num_unique_values\": 569,\n \"samples\": [\n 3900000,\n 34000000,\n 1020000\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"genres\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"id\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 59272,\n \"min\": 2,\n \"max\": 416437,\n \"num_unique_values\": 8370,\n \"samples\": [\n 57103,\n 19994,\n 28476\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"imdb_id\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 707596,\n \"min\": 417,\n \"max\": 5794766,\n \"num_unique_values\": 8370,\n \"samples\": [\n 52619,\n 1131734,\n 64806\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"original_language\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 40,\n \"samples\": [\n \"no\",\n \"cs\",\n \"vi\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"overview\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 8355,\n \"samples\": [\n \"Americans Jeff and Tommy, hunting in Scotland, stumble upon a village - Brigadoon. They soon learn that the town appears once every 100 years in order to preserve its peace and special beauty. The citizens go to bed at night and when they wake up, it's 100 years later. Tommy falls in love with a beautiful young woman, Fiona, and is torn between staying or going back to his hectic life in New York.\",\n \"Samba migrated to France 10 years ago from Senegal, and has since been plugging away at various lowly jobs. Alice is a senior executive who has recently undergone a burnout. Both struggle to get out of their dead-end lives. Samba's willing to do whatever it takes to get working papers, while Alice tries to get her life back on track until fate draws them together.\",\n \"In the future, medical technology has advanced to the point where people can buy artificial organs to extend their lives. But if they default on payments, an organization known as the Union sends agents to repossess the organs. Remy is one of the best agents in the business, but when he becomes the recipient of an artificial heart, he finds himself in the same dire straits as his many victims.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"popularity\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 9.65651267841049,\n \"min\": 4e-06,\n \"max\": 547.488298,\n \"num_unique_values\": 8370,\n \"samples\": [\n 11.519662,\n 11.23915,\n 1.567121\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"release_date\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 601949685.0164723,\n \"min\": -2208988800.0,\n \"max\": 1473897600.0,\n \"num_unique_values\": 5636,\n \"samples\": [\n 484963200.0,\n 873676800.0,\n 1076284800.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"revenue\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 134749919,\n \"min\": 0,\n \"max\": 2787965087,\n \"num_unique_values\": 4325,\n \"samples\": [\n 41610884,\n 115101622,\n 15520023\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"runtime\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 31.141679628227166,\n \"min\": 0.0,\n \"max\": 1140.0,\n \"num_unique_values\": 226,\n \"samples\": [\n 78.0,\n 720.0,\n 155.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"status\",\n \"properties\": {\n \"dtype\": \"category\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Rumored\",\n \"In Production\",\n \"unknown\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"tagline\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 6563,\n \"samples\": [\n \"She's the hottest thing on the beach. She's also his best friend's daughter!\",\n \"It's a comedy. And a drama. Just like life.\",\n \"You're all going to die.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"title\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 8126,\n \"samples\": [\n \"The Deep End of the Ocean\",\n \"Red River\",\n \"The End of Evangelion\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"vote_average\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1.0010720538980293,\n \"min\": 0.0,\n \"max\": 10.0,\n \"num_unique_values\": 72,\n \"samples\": [\n 5.7,\n 3.4,\n 7.3\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"vote_count\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 1029,\n \"min\": 0,\n \"max\": 14075,\n \"num_unique_values\": 1719,\n \"samples\": [\n 358,\n 3576,\n 1045\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"movieId\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 8370,\n \"samples\": [\n \"34608\",\n \"71205\",\n \"8015\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"imdbId\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 707596,\n \"min\": 417,\n \"max\": 5794766,\n \"num_unique_values\": 8370,\n \"samples\": [\n 52619,\n 1131734,\n 64806\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"tmdbId\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 59097.38021323738,\n \"min\": 2.0,\n \"max\": 416437.0,\n \"num_unique_values\": 8370,\n \"samples\": [\n 57103.0,\n 19994.0,\n 28476.0\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"movie_vector\",\n \"properties\": {\n \"dtype\": \"object\",\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", + "type": "dataframe", + "variable_name": "movies_df" + }, + "text/html": [ + "\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
budgetgenresidimdb_idoriginal_languageoverviewpopularityrelease_daterevenueruntimestatustaglinetitlevote_averagevote_countmovieIdimdbIdtmdbIdmovie_vector
030000000[Animation, Comedy, Family]862114709enLed by Woody, Andy's toys live happily in his ...21.946943815011200.037355403381.0ReleasedToy Story7.754151114709862.0[-0.49346923675184573, -0.16727664416375823, 0...
165000000[Adventure, Fantasy, Family]8844113497enWhen siblings Judy and Peter discover an encha...17.015539818985600.0262797249104.0ReleasedRoll the dice and unleash the excitement!Jumanji6.9241321134978844.0[-0.10770150143386471, -0.007231946857062053, ...
20[Romance, Comedy]15602113228enA family wedding reignites the ancient feud be...11.712900819590400.00101.0ReleasedStill Yelling. Still Fighting. Still Ready for...Grumpier Old Men6.592311322815602.0[-0.3496835380235819, 0.024321376508654746, -0...
316000000[Comedy, Drama, Romance]31357114885enCheated on, mistreated and stepped on, the wom...3.859495819590400.081452156127.0ReleasedFriends are the people who let you be yourself...Waiting to Exhale6.134411488531357.0[0.011927156686291933, -0.08701536485937247, 0...
40[Comedy]11862113041enJust when George Banks has recovered from his ...8.387519792374400.076578911106.0ReleasedJust When His World Is Back To Normal... He's ...Father of the Bride Part II5.7173511304111862.0[-0.07110798835836052, 0.18216637030291705, -0...
\n", + "
\n", + "
\n", + "\n", + "
\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "
\n", + "\n", + "\n", + "
\n", + " \n", + "\n", + "\n", + "\n", + " \n", + "
\n", + "\n", + "
\n", + "
\n" + ], + "text/plain": [ + " budget genres id imdb_id original_language \\\n", + "0 30000000 [Animation, Comedy, Family] 862 114709 en \n", + "1 65000000 [Adventure, Fantasy, Family] 8844 113497 en \n", + "2 0 [Romance, Comedy] 15602 113228 en \n", + "3 16000000 [Comedy, Drama, Romance] 31357 114885 en \n", + "4 0 [Comedy] 11862 113041 en \n", + "\n", + " overview popularity \\\n", + "0 Led by Woody, Andy's toys live happily in his ... 21.946943 \n", + "1 When siblings Judy and Peter discover an encha... 17.015539 \n", + "2 A family wedding reignites the ancient feud be... 11.712900 \n", + "3 Cheated on, mistreated and stepped on, the wom... 3.859495 \n", + "4 Just when George Banks has recovered from his ... 8.387519 \n", + "\n", + " release_date revenue runtime status \\\n", + "0 815011200.0 373554033 81.0 Released \n", + "1 818985600.0 262797249 104.0 Released \n", + "2 819590400.0 0 101.0 Released \n", + "3 819590400.0 81452156 127.0 Released \n", + "4 792374400.0 76578911 106.0 Released \n", + "\n", + " tagline \\\n", + "0 \n", + "1 Roll the dice and unleash the excitement! \n", + "2 Still Yelling. Still Fighting. Still Ready for... \n", + "3 Friends are the people who let you be yourself... \n", + "4 Just When His World Is Back To Normal... He's ... \n", + "\n", + " title vote_average vote_count movieId imdbId \\\n", + "0 Toy Story 7.7 5415 1 114709 \n", + "1 Jumanji 6.9 2413 2 113497 \n", + "2 Grumpier Old Men 6.5 92 3 113228 \n", + "3 Waiting to Exhale 6.1 34 4 114885 \n", + "4 Father of the Bride Part II 5.7 173 5 113041 \n", + "\n", + " tmdbId movie_vector \n", + "0 862.0 [-0.49346923675184573, -0.16727664416375823, 0... \n", + "1 8844.0 [-0.10770150143386471, -0.007231946857062053, ... \n", + "2 15602.0 [-0.3496835380235819, 0.024321376508654746, -0... \n", + "3 31357.0 [0.011927156686291933, -0.08701536485937247, 0... \n", + "4 11862.0 [-0.07110798835836052, 0.18216637030291705, -0... " + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# sanity check we merged all dataframes properly and have the right sizes of movies, users, vectors, ids, etc.\n", + "number_of_movies = len(movies_df.to_dict(orient='records'))\n", + "size_of_movie_df = movies_df.shape[0]\n", + "\n", + "print('number of movies', number_of_movies)\n", + "print('size of movie df', size_of_movie_df)\n", + "\n", + "unique_movie_ids = movies_df['id'].nunique()\n", + "print('unique movie ids', unique_movie_ids)\n", + "\n", + "unique_movie_titles = movies_df['title'].nunique()\n", + "print('unique movie titles', unique_movie_titles)\n", + "\n", + "unique_movies_rated = ratings_df['movieId'].nunique()\n", + "print('unique movies rated', unique_movies_rated)\n", + "movies_df.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "KkOHmwzly4Bb" + }, + "source": [ + "For a complete solution we'll store the user vectors and their watched list in Redis also. We won't be searching over these user vectors so no need to define an index for them. A direct JSON look up will suffice." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "id": "9NDkXLl7y4Bb" + }, + "outputs": [], + "source": [ + "from redis.commands.json.path import Path\n", + "\n", + "# use a Redis pipeline to store user data\n", + "with client.pipeline(transaction=False) as pipe:\n", + " for user_id, user_vector in user_vectors_and_ids.items():\n", + " user_key = f\"user:{user_id}\"\n", + " watched_list_ids = ratings_df[ratings_df['userId'] == user_id]['movieId'].tolist()\n", + "\n", + " user_data = {\n", + " \"user_vector\": user_vector,\n", + " \"watched_list_ids\": watched_list_ids\n", + " }\n", + " pipe.json().set(user_key, Path.root_path(), user_data)\n", + "\n", + " pipe.execute()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "nZsmIZoVy4Bb" + }, + "source": [ + "Unlike in content filtering, where we want to compute vector similarity between items and we use cosine distance between items vectors to do so, in collaborative filtering we instead try to compute the predicted rating a user will give to a movie by taking the inner product of the user and movie vector.\n", + "\n", + "This is why in our `collaborative_filtering_schema.yaml` we use `ip` (inner product) as our distance metric.\n", + "\n", + "It's also why we'll use our user vector as the query vector when we do a query. Let's pick a random user and their corresponding user vector to see what this looks like." + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "-kDk_tAYy4Bb", + "outputId": "77f39d26-8f3e-47c7-a98d-8a9d7207df3f" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "vector distance: -3.65109921,\t predicted rating: 4.65109921,\t title: Pulp Fiction, \n", + "vector distance: -3.53361845,\t predicted rating: 4.53361845,\t title: Cinema Paradiso, \n", + "vector distance: -3.52137375,\t predicted rating: 4.52137375,\t title: The Lord of the Rings: The Fellowship of the Ring, \n", + "vector distance: -3.47318125,\t predicted rating: 4.47318125,\t title: Star Wars, \n", + "vector distance: -3.44219446,\t predicted rating: 4.44219446,\t title: The African Queen, \n", + "vector distance: -3.42338753,\t predicted rating: 4.42338753,\t title: Raging Bull, \n", + "vector distance: -3.41860008,\t predicted rating: 4.41860008,\t title: Singin' in the Rain, \n", + "vector distance: -3.39997673,\t predicted rating: 4.39997673,\t title: Raiders of the Lost Ark, \n", + "vector distance: -3.37291861,\t predicted rating: 4.37291861,\t title: Band of Brothers, \n", + "vector distance: -3.34529161,\t predicted rating: 4.34529161,\t title: The Godfather, \n", + "vector distance: -3.33239079,\t predicted rating: 4.33239079,\t title: 12 Angry Men, \n", + "vector distance: -3.33077049,\t predicted rating: 4.33077049,\t title: The Philadelphia Story, \n" + ] + } + ], + "source": [ + "from redisvl.query import RangeQuery\n", + "\n", + "user_vector = client.json().get(f\"user:{352}\")[\"user_vector\"]\n", + "\n", + "# the distance metric 'ip' inner product is computing \"score = 1 - u * v\" and returning the minimum, which corresponds to the max of \"u * v\"\n", + "# this is what we want. The predicted rating on a scale of 0 to 5 is then -(score - 1) == -score + 1\n", + "query = RangeQuery(\n", + " vector=user_vector,\n", + " vector_field_name='movie_vector',\n", + " num_results=12,\n", + " return_score=True,\n", + " return_fields=['title', 'genres']\n", + ")\n", + "\n", + "results = movie_index.query(query)\n", + "\n", + "for r in results:\n", + " # compute our predicted rating on a scale of 0 to 5 from our vector distance\n", + " r['predicted_rating'] = - float(r['vector_distance']) + 1.\n", + " print(f\"vector distance: {float(r['vector_distance']):.08f},\\t predicted rating: {r['predicted_rating']:.08f},\\t title: {r['title']}, \")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "u2idXuAxy4Bb" + }, + "source": [ + "## Adding All the Bells & Whistles\n", + "Vector search handles the bulk of our collaborative filtering recommendation system and is a great approach to generating personalized recommendations that are unique to each user.\n", + "\n", + "To up our RecSys game even further we can leverage RedisVL Filter logic to give more control to what users are shown. Why have only one feed of recommended movies when you can have several, each with its own theme and personalized to each user." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "metadata": { + "id": "9KOo7e39y4Bb" + }, + "outputs": [], + "source": [ + "\n", + "from redisvl.query.filter import Tag, Num, Text\n", + "\n", + "def get_recommendations(user_id, filters=None, num_results=10):\n", + " user_vector = client.json().get(f\"user:{user_id}\")[\"user_vector\"]\n", + " query = RangeQuery(\n", + " vector=user_vector,\n", + " vector_field_name='movie_vector',\n", + " num_results=num_results,\n", + " filter_expression=filters,\n", + " return_fields=['title', 'overview', 'genres']\n", + " )\n", + "\n", + " results = movie_index.query(query)\n", + "\n", + " return [(r['title'], r['overview'], r['genres'], r['vector_distance']) for r in results]\n", + "\n", + "top_picks_for_you = get_recommendations(user_id=42) # general SVD results, no filter\n", + "\n", + "block_buster_filter = Num('revenue') > 30_000_000\n", + "block_buster_hits = get_recommendations(user_id=42, filters=block_buster_filter)\n", + "\n", + "classics_filter = Num('release_date') < datetime.datetime(1990, 1, 1).timestamp()\n", + "classics = get_recommendations(user_id=42, filters=classics_filter)\n", + "\n", + "popular_filter = (Num('popularity') > 50) & (Num('vote_average') > 7)\n", + "Whats_popular = get_recommendations(user_id=42, filters=popular_filter)\n", + "\n", + "indie_filter = (Num('revenue') < 1_000_000) & (Num('popularity') > 10)\n", + "indie_hits = get_recommendations(user_id=42, filters=indie_filter)\n", + "\n", + "fruity = Text('title') % 'apple|orange|peach|banana|grape|pineapple'\n", + "fruity_films = get_recommendations(user_id=42, filters=fruity)\n" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 363 + }, + "id": "knhZkvCqy4Bb", + "outputId": "05c3ae2b-2b5c-471e-9d2a-31fe970cd9de" + }, + "outputs": [ + { + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "summary": "{\n \"name\": \"all_recommendations\",\n \"rows\": 10,\n \"fields\": [\n {\n \"column\": \"top picks\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"The Usual Suspects\",\n \"The Shawshank Redemption\",\n \"Band of Brothers\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"block busters\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"The Empire Strikes Back\",\n \"Raiders of the Lost Ark\",\n \"The Godfather: Part II\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"classics\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"The Godfather\",\n \"Raiders of the Lost Ark\",\n \"The Philadelphia Story\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"what's popular\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"Whiplash\",\n \"The Dark Knight\",\n \"Blade Runner\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"indie hits\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"Thirteen Conversations About One Thing\",\n \"Yojimbo\",\n \"My Neighbor Totoro\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"fruity films\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 10,\n \"samples\": [\n \"The Apple Dumpling Gang\",\n \"What's Eating Gilbert Grape\",\n \"A Clockwork Orange\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", + "type": "dataframe", + "variable_name": "all_recommendations" + }, + "text/html": [ + "\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
top picksblock bustersclassicswhat's popularindie hitsfruity films
0The African QueenThe Lord of the Rings: The Fellowship of the RingThe African QueenThe Shawshank RedemptionShineThe Grapes of Wrath
1The Shawshank RedemptionRaiders of the Lost ArkRaiders of the Lost ArkThe Dark KnightYojimboWhat's Eating Gilbert Grape
2The Lord of the Rings: The Fellowship of the RingStar WarsCinema ParadisoFight ClubThe ProfessionalPineapple Express
3Raiders of the Lost ArkIn the Name of the FatherStar WarsPulp FictionSeven SamuraiJames and the Giant Peach
4Lock, Stock and Two Smoking BarrelsThe Dark KnightThe Godfather: Part IIThe AvengersThe PostmanBananas
5Band of BrothersThe Godfather: Part IIThe Philadelphia StoryBlade RunnerMy Neighbor TotoroA Clockwork Orange
6Cinema ParadisoDie HardDie HardGone GirlThe Meaning of LifeOrange County
7Star WarsGood Will HuntingThe Empire Strikes BackGuardians of the GalaxyCastle in the SkyAdam's Apples
8The Usual SuspectsThe Empire Strikes BackThe GodfatherWhiplashThirteen Conversations About One ThingThe Apple Dumpling Gang
9In the Name of the FatherBraveheartIndiana Jones and the Last CrusadeDeadpoolAguirre: The Wrath of GodHerbie Goes Bananas
\n", + "
\n", + "
\n", + "\n", + "
\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "
\n", + "\n", + "\n", + "
\n", + " \n", + "\n", + "\n", + "\n", + " \n", + "
\n", + "\n", + "
\n", + "
\n" + ], + "text/plain": [ + " top picks \\\n", + "0 The African Queen \n", + "1 The Shawshank Redemption \n", + "2 The Lord of the Rings: The Fellowship of the Ring \n", + "3 Raiders of the Lost Ark \n", + "4 Lock, Stock and Two Smoking Barrels \n", + "5 Band of Brothers \n", + "6 Cinema Paradiso \n", + "7 Star Wars \n", + "8 The Usual Suspects \n", + "9 In the Name of the Father \n", + "\n", + " block busters \\\n", + "0 The Lord of the Rings: The Fellowship of the Ring \n", + "1 Raiders of the Lost Ark \n", + "2 Star Wars \n", + "3 In the Name of the Father \n", + "4 The Dark Knight \n", + "5 The Godfather: Part II \n", + "6 Die Hard \n", + "7 Good Will Hunting \n", + "8 The Empire Strikes Back \n", + "9 Braveheart \n", + "\n", + " classics what's popular \\\n", + "0 The African Queen The Shawshank Redemption \n", + "1 Raiders of the Lost Ark The Dark Knight \n", + "2 Cinema Paradiso Fight Club \n", + "3 Star Wars Pulp Fiction \n", + "4 The Godfather: Part II The Avengers \n", + "5 The Philadelphia Story Blade Runner \n", + "6 Die Hard Gone Girl \n", + "7 The Empire Strikes Back Guardians of the Galaxy \n", + "8 The Godfather Whiplash \n", + "9 Indiana Jones and the Last Crusade Deadpool \n", + "\n", + " indie hits fruity films \n", + "0 Shine The Grapes of Wrath \n", + "1 Yojimbo What's Eating Gilbert Grape \n", + "2 The Professional Pineapple Express \n", + "3 Seven Samurai James and the Giant Peach \n", + "4 The Postman Bananas \n", + "5 My Neighbor Totoro A Clockwork Orange \n", + "6 The Meaning of Life Orange County \n", + "7 Castle in the Sky Adam's Apples \n", + "8 Thirteen Conversations About One Thing The Apple Dumpling Gang \n", + "9 Aguirre: The Wrath of God Herbie Goes Bananas " + ] + }, + "execution_count": 20, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# put all these titles into a single pandas dataframe, where each column is one category\n", + "all_recommendations = pd.DataFrame(columns=[\"top picks\", \"block busters\", \"classics\", \"what's popular\", \"indie hits\", \"fruity films\"])\n", + "all_recommendations[\"top picks\"] = [m[0] for m in top_picks_for_you]\n", + "all_recommendations[\"block busters\"] = [m[0] for m in block_buster_hits]\n", + "all_recommendations[\"classics\"] = [m[0] for m in classics]\n", + "all_recommendations[\"what's popular\"] = [m[0] for m in Whats_popular]\n", + "all_recommendations[\"indie hits\"] = [m[0] for m in indie_hits]\n", + "all_recommendations[\"fruity films\"] = [m[0] for m in fruity_films]\n", + "\n", + "all_recommendations.head(10)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_Pt0foXTy4Bb" + }, + "source": [ + "## Keeping Things Fresh\n", + "You've probably noticed that a few movies get repeated in these lists. That's not surprising as all our results are personalized and things like `popularity` and `user_rating` and `revenue` are likely highly correlated. And it's more than likely that at least some of the recommendations we're expecting to be highly rated by a given user are ones they've already watched and rated highly.\n", + "\n", + "We need a way to filter out movies that a user has already seen, and movies that we've already recommended to them before.\n", + "We could use a Tag filter on our queries to filter out movies by their id, but this gets cumbersome quickly.\n", + "Luckily Redis offers an easy answer to keeping recommendations new and interesting, and that answer is Bloom Filters." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "metadata": { + "id": "7LIDqFiGy4Bb" + }, + "outputs": [], + "source": [ + "# rewrite the get_recommendations() function to use a bloom filter and apply it before we return results\n", + "def get_unique_recommendations(user_id, filters=None, num_results=10):\n", + " user_data = client.json().get(f\"user:{user_id}\")\n", + " user_vector = user_data[\"user_vector\"]\n", + " watched_movies = user_data[\"watched_list_ids\"]\n", + "\n", + " # use a Bloom Filter to filter out movies that the user has already watched\n", + " client.bf().insert('user_watched_list', [f\"{user_id}:{movie_id}\" for movie_id in watched_movies])\n", + "\n", + " query = RangeQuery(\n", + " vector=user_vector,\n", + " vector_field_name='movie_vector',\n", + " num_results=num_results * 5, # fetch more results to account for watched movies\n", + " filter_expression=filters,\n", + " return_fields=['title', 'overview', 'genres', 'movieId'],\n", + " )\n", + "\n", + " results = movie_index.query(query)\n", + "\n", + " matches = client.bf().mexists(\"user_watched_list\", *[f\"{user_id}:{r['movieId']}\" for r in results])\n", + "\n", + " recommendations = [\n", + " (r['title'], r['overview'], r['genres'], r['vector_distance'], r['movieId'])\n", + " for i, r in enumerate(results) if matches[i] == 0\n", + " ][:num_results]\n", + "\n", + " # add these recommendations to the bloom filter so they don't appear again\n", + " client.bf().insert('user_watched_list', [f\"{user_id}:{r[4]}\" for r in recommendations])\n", + " return recommendations\n", + "\n", + "# example usage\n", + "# create a bloom filter for all our users\n", + "try:\n", + " client.bf().create(f\"user_watched_list\", 0.01, 10000)\n", + "except Exception as e:\n", + " client.delete(\"user_watched_list\")\n", + " client.bf().create(f\"user_watched_list\", 0.01, 10000)\n", + "\n", + "user_id = 42\n", + "\n", + "top_picks_for_you = get_unique_recommendations(user_id=user_id, num_results=5) # general SVD results, no filter\n", + "block_buster_hits = get_unique_recommendations(user_id=user_id, filters=block_buster_filter, num_results=5)\n", + "classics = get_unique_recommendations(user_id=user_id, filters=classics_filter, num_results=5)\n", + "whats_popular = get_unique_recommendations(user_id=user_id, filters=popular_filter, num_results=5)\n", + "indie_hits = get_unique_recommendations(user_id=user_id, filters=indie_filter, num_results=5)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 206 + }, + "id": "0S940BAiy4Bb", + "outputId": "88430699-eb2d-4034-fa1e-6cbe592f814f", + "vscode": { + "languageId": "ruby" + } + }, + "outputs": [ + { + "data": { + "application/vnd.google.colaboratory.intrinsic+json": { + "summary": "{\n \"name\": \"all_recommendations\",\n \"rows\": 5,\n \"fields\": [\n {\n \"column\": \"top picks\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Lock, Stock and Two Smoking Barrels\",\n \"In the Name of the Father\",\n \"Cinema Paradiso\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"block busters\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"The Godfather\",\n \"One Flew Over the Cuckoo's Nest\",\n \"Hachi: A Dog's Tale\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"classics\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"The Princess Bride\",\n \"Raging Bull\",\n \"Roger & Me\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"what's popular\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Blade Runner\",\n \"Deadpool\",\n \"Gone Girl\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"indie hits\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 5,\n \"samples\": [\n \"Yojimbo\",\n \"The Postman\",\n \"The Professional\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}", + "type": "dataframe", + "variable_name": "all_recommendations" + }, + "text/html": [ + "\n", + "
\n", + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
top picksblock bustersclassicswhat's popularindie hits
0The African QueenThe Godfather: Part IIThe Philadelphia StoryFight ClubShine
1Lock, Stock and Two Smoking BarrelsThe GodfatherThe Princess BrideBlade RunnerYojimbo
2Cinema ParadisoHachi: A Dog's TaleRoger & MeGone GirlThe Professional
3The Usual SuspectsThe Silence of the LambsDead Poets SocietyWhiplashSeven Samurai
4In the Name of the FatherOne Flew Over the Cuckoo's NestRaging BullDeadpoolThe Postman
\n", + "
\n", + "
\n", + "\n", + "
\n", + " \n", + "\n", + " \n", + "\n", + " \n", + "
\n", + "\n", + "\n", + "
\n", + " \n", + "\n", + "\n", + "\n", + " \n", + "
\n", + "\n", + "
\n", + "
\n" + ], + "text/plain": [ + " top picks block busters \\\n", + "0 The African Queen The Godfather: Part II \n", + "1 Lock, Stock and Two Smoking Barrels The Godfather \n", + "2 Cinema Paradiso Hachi: A Dog's Tale \n", + "3 The Usual Suspects The Silence of the Lambs \n", + "4 In the Name of the Father One Flew Over the Cuckoo's Nest \n", + "\n", + " classics what's popular indie hits \n", + "0 The Philadelphia Story Fight Club Shine \n", + "1 The Princess Bride Blade Runner Yojimbo \n", + "2 Roger & Me Gone Girl The Professional \n", + "3 Dead Poets Society Whiplash Seven Samurai \n", + "4 Raging Bull Deadpool The Postman " + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# put all these titles into a single pandas dataframe , where each column is one category\n", + "all_recommendations = pd.DataFrame(columns=[\"top picks\", \"block busters\", \"classics\", \"what's popular\", \"indie hits\"])\n", + "all_recommendations[\"top picks\"] = [m[0] for m in top_picks_for_you]\n", + "all_recommendations[\"block busters\"] = [m[0] for m in block_buster_hits]\n", + "all_recommendations[\"classics\"] = [m[0] for m in classics]\n", + "all_recommendations[\"what's popular\"] = [m[0] for m in whats_popular]\n", + "all_recommendations[\"indie hits\"] = [m[0] for m in indie_hits]\n", + "\n", + "all_recommendations.head()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "0krqUx4uy4Bb" + }, + "source": [ + "## Conclusion\n", + "That's it! That's all it takes to build a highly scalable, personalized, customizable collaborative filtering recommendation system with Redis and RedisVL.\n" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "-Q70JEKJy4Bb", + "outputId": "76536b6e-2d1f-4e1e-b39a-9d08d081533a" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Deleted 4376 keys\n", + "Deleted 2000 keys\n", + "Deleted 1000 keys\n", + "Deleted 500 keys\n", + "Deleted 500 keys\n" + ] + }, + { + "data": { + "text/plain": [ + "671" + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# clean up your index\n", + "while remaining := movie_index.clear():\n", + " print(f\"Deleted {remaining} keys\")\n", + "\n", + "client.delete(\"user_watched_list\")\n", + "client.delete(*[f\"user:{user_id}\" for user_id in user_vectors_and_ids.keys()])" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": "redis-ai-res", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/python-recipes/recommendation-systems/collaborative_filtering.ipynb b/python-recipes/recommendation-systems/collaborative_filtering.ipynb deleted file mode 100644 index e96054d3..00000000 --- a/python-recipes/recommendation-systems/collaborative_filtering.ipynb +++ /dev/null @@ -1,1706 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", - "\n", - "# Collaborative Filtering in RedisVL\n", - "\n", - "\"Open" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Recommendation systems are a common application of machine learning and serve many industries from e-commerce to music streaming platforms.\n", - "\n", - "There are many different architectures that can be followed to build a recommendation system. In a previous example notebook we demonstrated how to do [content filtering with RedisVL](content_filtering.ipynb). We encourage you to start there before diving into this notebook.\n", - "\n", - "In this notebook we'll demonstrate how to build a [collaborative filtering](https://en.wikipedia.org/wiki/Collaborative_filtering)\n", - "recommendation system and use the large IMDB movies dataset as our example data.\n", - "\n", - "To generate our vectors we'll use the popular Python package [Surprise](https://surpriselib.com/)" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "metadata": {}, - "outputs": [], - "source": [ - "# NBVAL_SKIP\n", - "!pip install scikit-surprise --quiet" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import requests\n", - "import pandas as pd\n", - "import numpy as np\n", - "\n", - "from surprise import SVD\n", - "from surprise import Dataset, Reader\n", - "from surprise.model_selection import train_test_split\n", - "\n", - "\n", - "# Replace values below with your own if using Redis Cloud instance\n", - "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", - "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", - "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", - "\n", - "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", - "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "To build a collaborative filtering example using the Surprise library and the Movies dataset, we need to first load the data, format it according to the requirements of Surprise, and then apply a collaborative filtering algorithm like SVD." - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "metadata": {}, - "outputs": [], - "source": [ - "def fetch_dataframe(file_name):\n", - " try:\n", - " df = pd.read_csv('datasets/collaborative_filtering/' + file_name)\n", - " except:\n", - " url = 'https://redis-ai-resources.s3.us-east-2.amazonaws.com/recommenders/datasets/collaborative-filtering/'\n", - " r = requests.get(url + file_name)\n", - " if not os.path.exists('datasets/collaborative_filtering'):\n", - " os.makedirs('datasets/collaborative_filtering')\n", - " with open('datasets/collaborative_filtering/' + file_name, 'wb') as f:\n", - " f.write(r.content)\n", - " df = pd.read_csv('datasets/collaborative_filtering/' + file_name)\n", - " return df\n" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "metadata": {}, - "outputs": [], - "source": [ - "ratings_df = fetch_dataframe('ratings_small.csv') # for a larger example use 'ratings.csv' instead\n", - "\n", - "# only keep the columns we need: userId, movieId, rating\n", - "ratings_df = ratings_df[['userId', 'movieId', 'rating']]\n", - "\n", - "reader = Reader(rating_scale=(0.0, 5.0))\n", - "\n", - "ratings_data = Dataset.load_from_df(ratings_df, reader)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# What is Collaborative Filtering" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "A lot is going to happen in the code cell below. We split our full data into train and test sets. We defined the collaborative filtering algorithm to use, which in this case is the Singular Value Decomposition (SVD) algorithm. lastly, we fit our model to our data.\n", - "\n", - "It's worth going into more detail why we chose this algorithm and what it is computing in the `svd.fit(train_set)` method we're calling.\n", - "First, let's think about what data it's receiving - our ratings data. This only contains the userIds, movieIds, and the user's ratings of their watched movies on a scale of 1 to 5.\n", - "\n", - "We can put this data into a matrix with rows being users and columns being movies\n", - "\n", - "| RATINGS| movie_1 | movie_2 | movie_3 | movie_4 | movie_5 | movie_6 | ....... |\n", - "| ----- | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: | :-----: |\n", - "| user_1 | 4 | 1 | | 4 | | 5 | |\n", - "| user_2 | | 5 | 5 | 2 | 1 | | |\n", - "| user_3 | | | | | 1 | | |\n", - "| user_4 | 4 | 1 | | 4 | | ? | |\n", - "| user_5 | | 4 | 5 | 2 | | | |\n", - "| ...... | | | | | | | |\n", - "\n", - "Our empty cells aren't zero's, they're missing ratings, so `user_1` has never rated `movie_3`. They may like it or hate it." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Unlike Content Filtering, here we're only considering the ratings that users assign. We don't know the plot or genre or release year of any of these films. We don't even know the title.\n", - "But we can still build a recommender by assuming that users have similar tastes to each other. As an intuitive example, we can see that `user_1` and `user_4` have very similar ratings on several movies, so we will assume that `user_4` will rate `movie_6` highly, just as `user_1` did. This is the idea behind collaborative filtering." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "That's the intuition, but what about the math? Since we only have this matrix to work with, what we want to do is decompose it into two constituent matrices.\n", - "Lets call our ratings matrix `[R]`. We want to find two other matrices, a user matrix `[U]`, and a movies matrix `[M]` that fit the equation:\n", - "\n", - "`[U] * [M] = [R]`\n", - "\n", - "`[U]` will look like:\n", - "|user_1_feature_1 | user_1_feature_2 | user_1_feature_3 | user_1_feature_4 | ... | user_1_feature_k |\n", - "| ----- | --------- | --------- | --------- | --- | --------- |\n", - "|user_2_feature_1 | user_2_feature_2 | user_2_feature_3 | user_2_feature_4 | ... | user_2_feature_k |\n", - "|user_3_feature_1 | user_3_feature_2 | user_3_feature_3 | user_3_feature_4 | ... | user_3_feature_k |\n", - "| ... | . | . | . | ... | . |\n", - "|user_N_feature_1 | user_N_feature_2 | user_N_feature_3 | user_N_feature_4 | ... | user_N_feature_k |\n", - "\n", - "`[M]` will look like:\n", - "\n", - "| movie_1_feature_1 | movie_2_feature_1 | movie_3_feature_1 | ... | movie_M_feature_1 |\n", - "| --- | --- | --- | --- | --- |\n", - "| movie_1_feature_2 | movie_2_feature_2 | movie_3_feature_2 | ... | movie_M_feature_2 |\n", - "| movie_1_feature_3 | movie_2_feature_3 | movie_3_feature_3 | ... | movie_M_feature_3 |\n", - "| movie_1_feature_4 | movie_2_feature_4 | movie_3_feature_4 | ... | movie_M_feature_4 |\n", - "| ... | . | . | ... | . |\n", - "| movie_1_feature_k | movie_2_feature_k | movie_3_feature_k | ... | movie_M_feature_k |\n", - "\n", - "\n", - "these features are called the latent features (or latent factors) and are the values we're trying to find when we call the `svd.fit(training_data)` method. The algorithm that computes these features from our ratings matrix is the SVD algorithm. The number of users and movies is set by our data. The size of the latent feature vectors `k` is a parameter we choose. We'll keep it at the default 100 for this notebook." - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# split the data into training and testing sets (80% train, 20% test)\n", - "train_set, test_set = train_test_split(ratings_data, test_size=0.2)\n", - "\n", - "# use SVD (Singular Value Decomposition) for collaborative filtering\n", - "svd = SVD(n_factors=100, biased=False) # we'll set biased to False so that predictions are of the form \"rating_prediction = user_vector dot item_vector\"\n", - "\n", - "# train the algorithm on the train_set\n", - "svd.fit(train_set)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Extracting The User and Movie Vectors" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Now that the SVD algorithm has computed our `[U]` and `[M]` matrices - which are both really just lists of vectors - we can load them into our Redis instance.\n", - "\n", - "The Surprise SVD model stores user and movie vectors in two attributes:\n", - "\n", - "`svd.pu`: user features matrix (a matrix where each row corresponds to the latent features of a user).\n", - "`svd.qi`: item features matrix (a matrix where each row corresponds to the latent features of an item/movie).\n", - "\n", - "It's worth noting that the matrix `svd.qi` is the transpose of the matrix `[M]` we defined above. This way each row corresponds to one movie." - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "we have 671 users with feature vectors of size 100\n", - "we have 8397 movies with feature vectors of size 100\n" - ] - } - ], - "source": [ - "user_vectors = svd.pu # user latent features (matrix)\n", - "movie_vectors = svd.qi # movie latent features (matrix)\n", - "\n", - "print(f'we have {user_vectors.shape[0]} users with feature vectors of size {user_vectors.shape[1]}')\n", - "print(f'we have {movie_vectors.shape[0]} movies with feature vectors of size {movie_vectors.shape[1]}')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "# Predicting User Ratings\n", - "The great thing about collaborative filtering is that using our user and movie vectors we can predict the rating any user will give to any movie in our dataset.\n", - "And unlike content filtering, there is no assumption that all the movies a user will be recommended are similar to each other. A user can be recommended dark horror films and light-hearted animations.\n", - "\n", - "Looking back at our SVD algorithm the equation is [User_features] * [Movie_features].transpose = [Ratings]\n", - "So to get a prediction of what a user will rate a movie they haven't seen yet we just need to take the dot product of that user's feature vector and a movie's feature vector." - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "the predicted rating of user 347 on movie 5515 is 1.1069607933289707\n" - ] - } - ], - "source": [ - "# surprise casts userId and movieId to inner ids, so we have to use their mapping to know which rows to use\n", - "inner_uid = train_set.to_inner_uid(347) # userId\n", - "inner_iid = train_set.to_inner_iid(5515) # movieId\n", - "\n", - "# predict one user's rating of one film\n", - "predicted_rating = np.dot(user_vectors[inner_uid], movie_vectors[inner_iid])\n", - "print(f'the predicted rating of user {347} on movie {5515} is {predicted_rating}')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Adding Movie Data\n", - "while our collaborative filtering algorithm was trained solely on user's ratings of movies, and doesn't require any data about the movies themselves - like the title, genres, or release year - we'll want that information stored as metadata.\n", - "\n", - "We can grab this data from our `movies_metadata.csv` file, clean it, and join it to our user ratings via the `movieId` column" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
belongs_to_collectionbudgetgenreshomepageidimdb_idoriginal_languageoriginal_titleoverviewpopularity...release_daterevenueruntimespoken_languagesstatustaglinetitlevideovote_averagevote_count
0{'id': 10194, 'name': 'Toy Story Collection', ...30000000[{'id': 16, 'name': 'Animation'}, {'id': 35, '...http://toystory.disney.com/toy-story862tt0114709enToy StoryLed by Woody, Andy's toys live happily in his ...21.946943...1995-10-3037355403381.0[{'iso_639_1': 'en', 'name': 'English'}]ReleasedNaNToy StoryFalse7.75415
1NaN65000000[{'id': 12, 'name': 'Adventure'}, {'id': 14, '...NaN8844tt0113497enJumanjiWhen siblings Judy and Peter discover an encha...17.015539...1995-12-15262797249104.0[{'iso_639_1': 'en', 'name': 'English'}, {'iso...ReleasedRoll the dice and unleash the excitement!JumanjiFalse6.92413
2{'id': 119050, 'name': 'Grumpy Old Men Collect...0[{'id': 10749, 'name': 'Romance'}, {'id': 35, ...NaN15602tt0113228enGrumpier Old MenA family wedding reignites the ancient feud be...11.712900...1995-12-220101.0[{'iso_639_1': 'en', 'name': 'English'}]ReleasedStill Yelling. Still Fighting. Still Ready for...Grumpier Old MenFalse6.592
3NaN16000000[{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam...NaN31357tt0114885enWaiting to ExhaleCheated on, mistreated and stepped on, the wom...3.859495...1995-12-2281452156127.0[{'iso_639_1': 'en', 'name': 'English'}]ReleasedFriends are the people who let you be yourself...Waiting to ExhaleFalse6.134
4{'id': 96871, 'name': 'Father of the Bride Col...0[{'id': 35, 'name': 'Comedy'}]NaN11862tt0113041enFather of the Bride Part IIJust when George Banks has recovered from his ...8.387519...1995-02-1076578911106.0[{'iso_639_1': 'en', 'name': 'English'}]ReleasedJust When His World Is Back To Normal... He's ...Father of the Bride Part IIFalse5.7173
\n", - "

5 rows × 23 columns

\n", - "
" - ], - "text/plain": [ - " belongs_to_collection budget \\\n", - "0 {'id': 10194, 'name': 'Toy Story Collection', ... 30000000 \n", - "1 NaN 65000000 \n", - "2 {'id': 119050, 'name': 'Grumpy Old Men Collect... 0 \n", - "3 NaN 16000000 \n", - "4 {'id': 96871, 'name': 'Father of the Bride Col... 0 \n", - "\n", - " genres \\\n", - "0 [{'id': 16, 'name': 'Animation'}, {'id': 35, '... \n", - "1 [{'id': 12, 'name': 'Adventure'}, {'id': 14, '... \n", - "2 [{'id': 10749, 'name': 'Romance'}, {'id': 35, ... \n", - "3 [{'id': 35, 'name': 'Comedy'}, {'id': 18, 'nam... \n", - "4 [{'id': 35, 'name': 'Comedy'}] \n", - "\n", - " homepage id imdb_id original_language \\\n", - "0 http://toystory.disney.com/toy-story 862 tt0114709 en \n", - "1 NaN 8844 tt0113497 en \n", - "2 NaN 15602 tt0113228 en \n", - "3 NaN 31357 tt0114885 en \n", - "4 NaN 11862 tt0113041 en \n", - "\n", - " original_title \\\n", - "0 Toy Story \n", - "1 Jumanji \n", - "2 Grumpier Old Men \n", - "3 Waiting to Exhale \n", - "4 Father of the Bride Part II \n", - "\n", - " overview popularity ... \\\n", - "0 Led by Woody, Andy's toys live happily in his ... 21.946943 ... \n", - "1 When siblings Judy and Peter discover an encha... 17.015539 ... \n", - "2 A family wedding reignites the ancient feud be... 11.712900 ... \n", - "3 Cheated on, mistreated and stepped on, the wom... 3.859495 ... \n", - "4 Just when George Banks has recovered from his ... 8.387519 ... \n", - "\n", - " release_date revenue runtime \\\n", - "0 1995-10-30 373554033 81.0 \n", - "1 1995-12-15 262797249 104.0 \n", - "2 1995-12-22 0 101.0 \n", - "3 1995-12-22 81452156 127.0 \n", - "4 1995-02-10 76578911 106.0 \n", - "\n", - " spoken_languages status \\\n", - "0 [{'iso_639_1': 'en', 'name': 'English'}] Released \n", - "1 [{'iso_639_1': 'en', 'name': 'English'}, {'iso... Released \n", - "2 [{'iso_639_1': 'en', 'name': 'English'}] Released \n", - "3 [{'iso_639_1': 'en', 'name': 'English'}] Released \n", - "4 [{'iso_639_1': 'en', 'name': 'English'}] Released \n", - "\n", - " tagline \\\n", - "0 NaN \n", - "1 Roll the dice and unleash the excitement! \n", - "2 Still Yelling. Still Fighting. Still Ready for... \n", - "3 Friends are the people who let you be yourself... \n", - "4 Just When His World Is Back To Normal... He's ... \n", - "\n", - " title video vote_average vote_count \n", - "0 Toy Story False 7.7 5415 \n", - "1 Jumanji False 6.9 2413 \n", - "2 Grumpier Old Men False 6.5 92 \n", - "3 Waiting to Exhale False 6.1 34 \n", - "4 Father of the Bride Part II False 5.7 173 \n", - "\n", - "[5 rows x 23 columns]" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "movies_df = fetch_dataframe('movies_metadata.csv')\n", - "movies_df.head()" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "budget 0\n", - "genres 0\n", - "id 0\n", - "imdb_id 0\n", - "original_language 0\n", - "overview 0\n", - "popularity 0\n", - "release_date 0\n", - "revenue 0\n", - "runtime 0\n", - "status 0\n", - "tagline 0\n", - "title 0\n", - "vote_average 0\n", - "vote_count 0\n", - "dtype: int64" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "\n", - "import datetime\n", - "movies_df.drop(columns=['homepage', 'production_countries', 'production_companies', 'spoken_languages', 'video', 'original_title', 'video', 'poster_path', 'belongs_to_collection'], inplace=True)\n", - "\n", - "# drop rows that have missing values\n", - "movies_df.dropna(subset=['imdb_id'], inplace=True)\n", - "\n", - "movies_df['original_language'] = movies_df['original_language'].fillna('unknown')\n", - "movies_df['overview'] = movies_df['overview'].fillna('')\n", - "movies_df['popularity'] = movies_df['popularity'].fillna(0)\n", - "movies_df['release_date'] = movies_df['release_date'].fillna('1900-01-01').apply(lambda x: datetime.datetime.strptime(x, \"%Y-%m-%d\").timestamp())\n", - "movies_df['revenue'] = movies_df['revenue'].fillna(0)\n", - "movies_df['runtime'] = movies_df['runtime'].fillna(0)\n", - "movies_df['status'] = movies_df['status'].fillna('unknown')\n", - "movies_df['tagline'] = movies_df['tagline'].fillna('')\n", - "movies_df['title'] = movies_df['title'].fillna('')\n", - "movies_df['vote_average'] = movies_df['vote_average'].fillna(0)\n", - "movies_df['vote_count'] = movies_df['vote_count'].fillna(0)\n", - "movies_df['genres'] = movies_df['genres'].apply(lambda x: [g['name'] for g in eval(x)] if x != '' else []) # convert to a list of genre names\n", - "movies_df['imdb_id'] = movies_df['imdb_id'].apply(lambda x: x[2:] if str(x).startswith('tt') else x).astype(int) # remove leading 'tt' from imdb_id\n", - "\n", - "# make sure we've filled all missing values\n", - "movies_df.isnull().sum()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We'll have to map these movies to their ratings, which we'll do so with the `links.csv` file that matches `movieId`, `imdbId`, and `tmdbId`.\n", - "Let's do that now." - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "metadata": {}, - "outputs": [], - "source": [ - "links_df = fetch_dataframe('links_small.csv') # for a larger example use 'links.csv' instead\n", - "\n", - "movies_df = movies_df.merge(links_df, left_on='imdb_id', right_on='imdbId', how='inner')" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "We'll want to move our SVD user vectors and movie vectors and their corresponding userId and movieId into 2 dataframes for later processing." - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
budgetgenresidimdb_idoriginal_languageoverviewpopularityrelease_daterevenueruntimestatustaglinetitlevote_averagevote_countmovieIdimdbIdtmdbIdmovie_vector
030000000[Animation, Comedy, Family]862114709enLed by Woody, Andy's toys live happily in his ...21.946943815040000.037355403381.0ReleasedToy Story7.754151114709862.0[0.12184447241197785, -0.16994406060791697, 0....
165000000[Adventure, Fantasy, Family]8844113497enWhen siblings Judy and Peter discover an encha...17.015539819014400.0262797249104.0ReleasedRoll the dice and unleash the excitement!Jumanji6.9241321134978844.0[0.14683581574270926, -0.06365576587872183, 0....
20[Romance, Comedy]15602113228enA family wedding reignites the ancient feud be...11.712900819619200.00101.0ReleasedStill Yelling. Still Fighting. Still Ready for...Grumpier Old Men6.592311322815602.0[0.16698051985699827, -0.02406109383254372, 0....
316000000[Comedy, Drama, Romance]31357114885enCheated on, mistreated and stepped on, the wom...3.859495819619200.081452156127.0ReleasedFriends are the people who let you be yourself...Waiting to Exhale6.134411488531357.0[-0.10740791019437969, 0.09007945525146789, 0....
40[Comedy]11862113041enJust when George Banks has recovered from his ...8.387519792403200.076578911106.0ReleasedJust When His World Is Back To Normal... He's ...Father of the Bride Part II5.7173511304111862.0[0.11311012532803581, 0.025998675845395405, 0....
\n", - "
" - ], - "text/plain": [ - " budget genres id imdb_id original_language \\\n", - "0 30000000 [Animation, Comedy, Family] 862 114709 en \n", - "1 65000000 [Adventure, Fantasy, Family] 8844 113497 en \n", - "2 0 [Romance, Comedy] 15602 113228 en \n", - "3 16000000 [Comedy, Drama, Romance] 31357 114885 en \n", - "4 0 [Comedy] 11862 113041 en \n", - "\n", - " overview popularity \\\n", - "0 Led by Woody, Andy's toys live happily in his ... 21.946943 \n", - "1 When siblings Judy and Peter discover an encha... 17.015539 \n", - "2 A family wedding reignites the ancient feud be... 11.712900 \n", - "3 Cheated on, mistreated and stepped on, the wom... 3.859495 \n", - "4 Just when George Banks has recovered from his ... 8.387519 \n", - "\n", - " release_date revenue runtime status \\\n", - "0 815040000.0 373554033 81.0 Released \n", - "1 819014400.0 262797249 104.0 Released \n", - "2 819619200.0 0 101.0 Released \n", - "3 819619200.0 81452156 127.0 Released \n", - "4 792403200.0 76578911 106.0 Released \n", - "\n", - " tagline \\\n", - "0 \n", - "1 Roll the dice and unleash the excitement! \n", - "2 Still Yelling. Still Fighting. Still Ready for... \n", - "3 Friends are the people who let you be yourself... \n", - "4 Just When His World Is Back To Normal... He's ... \n", - "\n", - " title vote_average vote_count movieId imdbId \\\n", - "0 Toy Story 7.7 5415 1 114709 \n", - "1 Jumanji 6.9 2413 2 113497 \n", - "2 Grumpier Old Men 6.5 92 3 113228 \n", - "3 Waiting to Exhale 6.1 34 4 114885 \n", - "4 Father of the Bride Part II 5.7 173 5 113041 \n", - "\n", - " tmdbId movie_vector \n", - "0 862.0 [0.12184447241197785, -0.16994406060791697, 0.... \n", - "1 8844.0 [0.14683581574270926, -0.06365576587872183, 0.... \n", - "2 15602.0 [0.16698051985699827, -0.02406109383254372, 0.... \n", - "3 31357.0 [-0.10740791019437969, 0.09007945525146789, 0.... \n", - "4 11862.0 [0.11311012532803581, 0.025998675845395405, 0.... " - ] - }, - "execution_count": 11, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# build a dataframe out of the user vectors and their userIds\n", - "user_vectors_and_ids = {train_set.to_raw_uid(inner_id): user_vectors[inner_id].tolist() for inner_id in train_set.all_users()}\n", - "user_vector_df = pd.Series(user_vectors_and_ids).to_frame('user_vector')\n", - "\n", - "# now do the same for the movie vectors and their movieIds\n", - "movie_vectors_and_ids = {train_set.to_raw_iid(inner_id): movie_vectors[inner_id].tolist() for inner_id in train_set.all_items()}\n", - "movie_vector_df = pd.Series(movie_vectors_and_ids).to_frame('movie_vector')\n", - "\n", - "# merge the movie vector series with the movies dataframe using the movieId and id fields\n", - "movies_df = movies_df.merge(movie_vector_df, left_on='movieId', right_index=True, how='inner')\n", - "movies_df['movieId'] = movies_df['movieId'].apply(lambda x: str(x)) # need to cast to a string as this is a tag field in our search schema\n", - "movies_df.head()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## RedisVL Handles the Scale\n", - "\n", - "Especially for large datasets like the 45,000 movie catalog we're dealing with, you'll want Redis to do the heavy lifting of vector search.\n", - "All that's needed is to define the search index and load our data we've cleaned and merged with our vectors.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "12:05:35 redisvl.index.index INFO Index already exists, overwriting.\n" - ] - } - ], - "source": [ - "from redis import Redis\n", - "from redisvl.schema import IndexSchema\n", - "from redisvl.index import SearchIndex\n", - "\n", - "client = Redis.from_url(REDIS_URL)\n", - "\n", - "movie_schema = IndexSchema.from_yaml(\"collaborative_filtering_schema.yaml\")\n", - "\n", - "movie_index = SearchIndex(movie_schema, redis_client=client)\n", - "movie_index.create(overwrite=True, drop=True)\n", - "\n", - "movie_keys = movie_index.load(movies_df.to_dict(orient='records'))" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "number of movies 8358\n", - "size of movie df 8358\n", - "unique movie ids 8352\n", - "unique movie titles 8115\n", - "unique movies rated 9065\n" - ] - }, - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
budgetgenresidimdb_idoriginal_languageoverviewpopularityrelease_daterevenueruntimestatustaglinetitlevote_averagevote_countmovieIdimdbIdtmdbIdmovie_vector
030000000[Animation, Comedy, Family]862114709enLed by Woody, Andy's toys live happily in his ...21.946943815040000.037355403381.0ReleasedToy Story7.754151114709862.0[0.12184447241197785, -0.16994406060791697, 0....
165000000[Adventure, Fantasy, Family]8844113497enWhen siblings Judy and Peter discover an encha...17.015539819014400.0262797249104.0ReleasedRoll the dice and unleash the excitement!Jumanji6.9241321134978844.0[0.14683581574270926, -0.06365576587872183, 0....
20[Romance, Comedy]15602113228enA family wedding reignites the ancient feud be...11.712900819619200.00101.0ReleasedStill Yelling. Still Fighting. Still Ready for...Grumpier Old Men6.592311322815602.0[0.16698051985699827, -0.02406109383254372, 0....
316000000[Comedy, Drama, Romance]31357114885enCheated on, mistreated and stepped on, the wom...3.859495819619200.081452156127.0ReleasedFriends are the people who let you be yourself...Waiting to Exhale6.134411488531357.0[-0.10740791019437969, 0.09007945525146789, 0....
40[Comedy]11862113041enJust when George Banks has recovered from his ...8.387519792403200.076578911106.0ReleasedJust When His World Is Back To Normal... He's ...Father of the Bride Part II5.7173511304111862.0[0.11311012532803581, 0.025998675845395405, 0....
\n", - "
" - ], - "text/plain": [ - " budget genres id imdb_id original_language \\\n", - "0 30000000 [Animation, Comedy, Family] 862 114709 en \n", - "1 65000000 [Adventure, Fantasy, Family] 8844 113497 en \n", - "2 0 [Romance, Comedy] 15602 113228 en \n", - "3 16000000 [Comedy, Drama, Romance] 31357 114885 en \n", - "4 0 [Comedy] 11862 113041 en \n", - "\n", - " overview popularity \\\n", - "0 Led by Woody, Andy's toys live happily in his ... 21.946943 \n", - "1 When siblings Judy and Peter discover an encha... 17.015539 \n", - "2 A family wedding reignites the ancient feud be... 11.712900 \n", - "3 Cheated on, mistreated and stepped on, the wom... 3.859495 \n", - "4 Just when George Banks has recovered from his ... 8.387519 \n", - "\n", - " release_date revenue runtime status \\\n", - "0 815040000.0 373554033 81.0 Released \n", - "1 819014400.0 262797249 104.0 Released \n", - "2 819619200.0 0 101.0 Released \n", - "3 819619200.0 81452156 127.0 Released \n", - "4 792403200.0 76578911 106.0 Released \n", - "\n", - " tagline \\\n", - "0 \n", - "1 Roll the dice and unleash the excitement! \n", - "2 Still Yelling. Still Fighting. Still Ready for... \n", - "3 Friends are the people who let you be yourself... \n", - "4 Just When His World Is Back To Normal... He's ... \n", - "\n", - " title vote_average vote_count movieId imdbId \\\n", - "0 Toy Story 7.7 5415 1 114709 \n", - "1 Jumanji 6.9 2413 2 113497 \n", - "2 Grumpier Old Men 6.5 92 3 113228 \n", - "3 Waiting to Exhale 6.1 34 4 114885 \n", - "4 Father of the Bride Part II 5.7 173 5 113041 \n", - "\n", - " tmdbId movie_vector \n", - "0 862.0 [0.12184447241197785, -0.16994406060791697, 0.... \n", - "1 8844.0 [0.14683581574270926, -0.06365576587872183, 0.... \n", - "2 15602.0 [0.16698051985699827, -0.02406109383254372, 0.... \n", - "3 31357.0 [-0.10740791019437969, 0.09007945525146789, 0.... \n", - "4 11862.0 [0.11311012532803581, 0.025998675845395405, 0.... " - ] - }, - "execution_count": 13, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# sanity check we merged all dataframes properly and have the right sizes of movies, users, vectors, ids, etc.\n", - "number_of_movies = len(movies_df.to_dict(orient='records'))\n", - "size_of_movie_df = movies_df.shape[0]\n", - "\n", - "print('number of movies', number_of_movies)\n", - "print('size of movie df', size_of_movie_df)\n", - "\n", - "unique_movie_ids = movies_df['id'].nunique()\n", - "print('unique movie ids', unique_movie_ids)\n", - "\n", - "unique_movie_titles = movies_df['title'].nunique()\n", - "print('unique movie titles', unique_movie_titles)\n", - "\n", - "unique_movies_rated = ratings_df['movieId'].nunique()\n", - "print('unique movies rated', unique_movies_rated)\n", - "movies_df.head()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "For a complete solution we'll store the user vectors and their watched list in Redis also. We won't be searching over these user vectors so no need to define an index for them. A direct JSON look up will suffice." - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [], - "source": [ - "from redis.commands.json.path import Path\n", - "\n", - "# use a Redis pipeline to store user data and verify it in a single transaction\n", - "with client.pipeline() as pipe:\n", - " for user_id, user_vector in user_vectors_and_ids.items():\n", - " user_key = f\"user:{user_id}\"\n", - " watched_list_ids = ratings_df[ratings_df['userId'] == user_id]['movieId'].tolist()\n", - "\n", - " user_data = {\n", - " \"user_vector\": user_vector,\n", - " \"watched_list_ids\": watched_list_ids\n", - " }\n", - " pipe.json().set(user_key, Path.root_path(), user_data)\n", - " pipe.execute()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Unlike in content filtering, where we want to compute vector similarity between items and we use cosine distance between items vectors to do so, in collaborative filtering we instead try to compute the predicted rating a user will give to a movie by taking the inner product of the user and movie vector.\n", - "\n", - "This is why in our `collaborative_filtering_schema.yaml` we use `ip` (inner product) as our distance metric.\n", - "\n", - "It's also why we'll use our user vector as the query vector when we do a query. Let's pick a random user and their corresponding user vector to see what this looks like." - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "vector distance: -3.63527393,\t predicted rating: 4.63527393,\t title: Fight Club, \n", - "vector distance: -3.60445881,\t predicted rating: 4.60445881,\t title: All About Eve, \n", - "vector distance: -3.60197020,\t predicted rating: 4.60197020,\t title: Lock, Stock and Two Smoking Barrels, \n", - "vector distance: -3.59518766,\t predicted rating: 4.59518766,\t title: Midnight in Paris, \n", - "vector distance: -3.58543396,\t predicted rating: 4.58543396,\t title: It Happened One Night, \n", - "vector distance: -3.54092789,\t predicted rating: 4.54092789,\t title: Anne Frank Remembered, \n", - "vector distance: -3.51044893,\t predicted rating: 4.51044893,\t title: Pulp Fiction, \n", - "vector distance: -3.50941706,\t predicted rating: 4.50941706,\t title: Raging Bull, \n", - "vector distance: -3.49180365,\t predicted rating: 4.49180365,\t title: Cool Hand Luke, \n", - "vector distance: -3.47437143,\t predicted rating: 4.47437143,\t title: Rear Window, \n", - "vector distance: -3.41378117,\t predicted rating: 4.41378117,\t title: The Usual Suspects, \n", - "vector distance: -3.40533876,\t predicted rating: 4.40533876,\t title: Princess Mononoke, \n" - ] - } - ], - "source": [ - "from redisvl.query import RangeQuery\n", - "\n", - "user_vector = client.json().get(f\"user:{352}\")[\"user_vector\"]\n", - "\n", - "# the distance metric 'ip' inner product is computing \"score = 1 - u * v\" and returning the minimum, which corresponds to the max of \"u * v\"\n", - "# this is what we want. The predicted rating on a scale of 0 to 5 is then -(score - 1) == -score + 1\n", - "query = RangeQuery(vector=user_vector,\n", - " vector_field_name='movie_vector',\n", - " num_results=12,\n", - " return_score=True,\n", - " return_fields=['title', 'genres']\n", - " )\n", - "\n", - "results = movie_index.query(query)\n", - "\n", - "for r in results:\n", - " # compute our predicted rating on a scale of 0 to 5 from our vector distance\n", - " r['predicted_rating'] = - float(r['vector_distance']) + 1.\n", - " print(f\"vector distance: {float(r['vector_distance']):.08f},\\t predicted rating: {r['predicted_rating']:.08f},\\t title: {r['title']}, \")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Adding All the Bells & Whistles\n", - "Vector search handles the bulk of our collaborative filtering recommendation system and is a great approach to generating personalized recommendations that are unique to each user.\n", - "\n", - "To up our RecSys game even further we can leverage RedisVL Filter logic to give more control to what users are shown. Why have only one feed of recommended movies when you can have several, each with its own theme and personalized to each user." - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "metadata": {}, - "outputs": [], - "source": [ - "\n", - "from redisvl.query.filter import Tag, Num, Text\n", - "\n", - "def get_recommendations(user_id, filters=None, num_results=10):\n", - " user_vector = client.json().get(f\"user:{user_id}\")[\"user_vector\"]\n", - " query = RangeQuery(vector=user_vector,\n", - " vector_field_name='movie_vector',\n", - " num_results=num_results,\n", - " filter_expression=filters,\n", - " return_fields=['title', 'overview', 'genres'])\n", - "\n", - " results = movie_index.query(query)\n", - "\n", - " return [(r['title'], r['overview'], r['genres'], r['vector_distance']) for r in results]\n", - "\n", - "Top_picks_for_you = get_recommendations(user_id=42) # general SVD results, no filter\n", - "\n", - "block_buster_filter = Num('revenue') > 30_000_000\n", - "block_buster_hits = get_recommendations(user_id=42, filters=block_buster_filter)\n", - "\n", - "classics_filter = Num('release_date') < datetime.datetime(1990, 1, 1).timestamp()\n", - "classics = get_recommendations(user_id=42, filters=classics_filter)\n", - "\n", - "popular_filter = (Num('popularity') > 50) & (Num('vote_average') > 7)\n", - "Whats_popular = get_recommendations(user_id=42, filters=popular_filter)\n", - "\n", - "indie_filter = (Num('revenue') < 1_000_000) & (Num('popularity') > 10)\n", - "indie_hits = get_recommendations(user_id=42, filters=indie_filter)\n", - "\n", - "fruity = Text('title') % 'apple|orange|peach|banana|grape|pineapple'\n", - "fruity_films = get_recommendations(user_id=42, filters=fruity)\n" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
top picksblock bustersclassicswhat's popularindie hitsfruity films
0The Shawshank RedemptionForrest GumpCinema ParadisoThe Shawshank RedemptionCastle in the SkyWhat's Eating Gilbert Grape
1Forrest GumpThe Silence of the LambsThe African QueenPulp FictionMy Neighbor TotoroA Clockwork Orange
2Cinema ParadisoPulp FictionRaiders of the Lost ArkThe Dark KnightAll Quiet on the Western FrontThe Grapes of Wrath
3Lock, Stock and Two Smoking BarrelsRaiders of the Lost ArkThe Empire Strikes BackFight ClubArmy of DarknessPineapple Express
4The African QueenThe Empire Strikes BackIndiana Jones and the Last CrusadeWhiplashAll About EveJames and the Giant Peach
5The Silence of the LambsIndiana Jones and the Last CrusadeStar WarsBlade RunnerThe ProfessionalBananas
6Pulp FictionSchindler's ListThe Manchurian CandidateThe AvengersShineOrange County
7Raiders of the Lost ArkThe Lord of the Rings: The Return of the KingThe Godfather: Part IIGuardians of the GalaxyYojimboHerbie Goes Bananas
8The Empire Strikes BackThe Lord of the Rings: The Two TowersCastle in the SkyGone GirlBelle de JourThe Apple Dumpling Gang
9Indiana Jones and the Last CrusadeTerminator 2: Judgment DayBack to the FutureBig Hero 6Local HeroAdam's Apples
\n", - "
" - ], - "text/plain": [ - " top picks \\\n", - "0 The Shawshank Redemption \n", - "1 Forrest Gump \n", - "2 Cinema Paradiso \n", - "3 Lock, Stock and Two Smoking Barrels \n", - "4 The African Queen \n", - "5 The Silence of the Lambs \n", - "6 Pulp Fiction \n", - "7 Raiders of the Lost Ark \n", - "8 The Empire Strikes Back \n", - "9 Indiana Jones and the Last Crusade \n", - "\n", - " block busters \\\n", - "0 Forrest Gump \n", - "1 The Silence of the Lambs \n", - "2 Pulp Fiction \n", - "3 Raiders of the Lost Ark \n", - "4 The Empire Strikes Back \n", - "5 Indiana Jones and the Last Crusade \n", - "6 Schindler's List \n", - "7 The Lord of the Rings: The Return of the King \n", - "8 The Lord of the Rings: The Two Towers \n", - "9 Terminator 2: Judgment Day \n", - "\n", - " classics what's popular \\\n", - "0 Cinema Paradiso The Shawshank Redemption \n", - "1 The African Queen Pulp Fiction \n", - "2 Raiders of the Lost Ark The Dark Knight \n", - "3 The Empire Strikes Back Fight Club \n", - "4 Indiana Jones and the Last Crusade Whiplash \n", - "5 Star Wars Blade Runner \n", - "6 The Manchurian Candidate The Avengers \n", - "7 The Godfather: Part II Guardians of the Galaxy \n", - "8 Castle in the Sky Gone Girl \n", - "9 Back to the Future Big Hero 6 \n", - "\n", - " indie hits fruity films \n", - "0 Castle in the Sky What's Eating Gilbert Grape \n", - "1 My Neighbor Totoro A Clockwork Orange \n", - "2 All Quiet on the Western Front The Grapes of Wrath \n", - "3 Army of Darkness Pineapple Express \n", - "4 All About Eve James and the Giant Peach \n", - "5 The Professional Bananas \n", - "6 Shine Orange County \n", - "7 Yojimbo Herbie Goes Bananas \n", - "8 Belle de Jour The Apple Dumpling Gang \n", - "9 Local Hero Adam's Apples " - ] - }, - "execution_count": 17, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# put all these titles into a single pandas dataframe, where each column is one category\n", - "all_recommendations = pd.DataFrame(columns=[\"top picks\", \"block busters\", \"classics\", \"what's popular\", \"indie hits\", \"fruity films\"])\n", - "all_recommendations[\"top picks\"] = [m[0] for m in Top_picks_for_you]\n", - "all_recommendations[\"block busters\"] = [m[0] for m in block_buster_hits]\n", - "all_recommendations[\"classics\"] = [m[0] for m in classics]\n", - "all_recommendations[\"what's popular\"] = [m[0] for m in Whats_popular]\n", - "all_recommendations[\"indie hits\"] = [m[0] for m in indie_hits]\n", - "all_recommendations[\"fruity films\"] = [m[0] for m in fruity_films]\n", - "\n", - "all_recommendations.head(10)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Keeping Things Fresh\n", - "You've probably noticed that a few movies get repeated in these lists. That's not surprising as all our results are personalized and things like `popularity` and `user_rating` and `revenue` are likely highly correlated. And it's more than likely that at least some of the recommendations we're expecting to be highly rated by a given user are ones they've already watched and rated highly.\n", - "\n", - "We need a way to filter out movies that a user has already seen, and movies that we've already recommended to them before.\n", - "We could use a Tag filter on our queries to filter out movies by their id, but this gets cumbersome quickly.\n", - "Luckily Redis offers an easy answer to keeping recommendations new and interesting, and that answer is Bloom Filters." - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [], - "source": [ - "# rewrite the get_recommendations() function to use a bloom filter and apply it before we return results\n", - "def get_unique_recommendations(user_id, filters=None, num_results=10):\n", - " user_data = client.json().get(f\"user:{user_id}\")\n", - " user_vector = user_data[\"user_vector\"]\n", - " watched_movies = user_data[\"watched_list_ids\"]\n", - "\n", - " # use a Bloom Filter to filter out movies that the user has already watched\n", - " client.bf().insert('user_watched_list', [f\"{user_id}:{movie_id}\" for movie_id in watched_movies])\n", - "\n", - " query = RangeQuery(vector=user_vector,\n", - " vector_field_name='movie_vector',\n", - " num_results=num_results * 5, # fetch more results to account for watched movies\n", - " filter_expression=filters,\n", - " return_fields=['title', 'overview', 'genres', 'movieId'],\n", - " )\n", - " results = movie_index.query(query)\n", - "\n", - " matches = client.bf().mexists(\"user_watched_list\", *[f\"{user_id}:{r['movieId']}\" for r in results])\n", - "\n", - " recommendations = [\n", - " (r['title'], r['overview'], r['genres'], r['vector_distance'], r['movieId'])\n", - " for i, r in enumerate(results) if matches[i] == 0\n", - " ][:num_results]\n", - "\n", - " # add these recommendations to the bloom filter so they don't appear again\n", - " client.bf().insert('user_watched_list', [f\"{user_id}:{r[4]}\" for r in recommendations])\n", - " return recommendations\n", - "\n", - "# example usage\n", - "# create a bloom filter for all our users\n", - "try:\n", - " client.bf().create(f\"user_watched_list\", 0.01, 10000)\n", - "except Exception as e:\n", - " client.delete(\"user_watched_list\")\n", - " client.bf().create(f\"user_watched_list\", 0.01, 10000)\n", - "\n", - "user_id = 42\n", - "\n", - "top_picks_for_you = get_unique_recommendations(user_id=user_id, num_results=5) # general SVD results, no filter\n", - "block_buster_hits = get_unique_recommendations(user_id=user_id, filters=block_buster_filter, num_results=5)\n", - "classics = get_unique_recommendations(user_id=user_id, filters=classics_filter, num_results=5)\n", - "whats_popular = get_unique_recommendations(user_id=user_id, filters=popular_filter, num_results=5)\n", - "indie_hits = get_unique_recommendations(user_id=user_id, filters=indie_filter, num_results=5)" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "metadata": { - "vscode": { - "languageId": "ruby" - } - }, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
top picksblock bustersclassicswhat's popularindie hits
0Cinema ParadisoThe Manchurian CandidateCastle in the SkyFight ClubAll Quiet on the Western Front
1Lock, Stock and Two Smoking BarrelsToy Story12 Angry MenWhiplashArmy of Darkness
2The African QueenThe Godfather: Part IIMy Neighbor TotoroBlade RunnerAll About Eve
3The Silence of the LambsBack to the FutureIt Happened One NightGone GirlThe Professional
4Eat Drink Man WomanThe GodfatherStand by MeBig Hero 6Shine
\n", - "
" - ], - "text/plain": [ - " top picks block busters \\\n", - "0 Cinema Paradiso The Manchurian Candidate \n", - "1 Lock, Stock and Two Smoking Barrels Toy Story \n", - "2 The African Queen The Godfather: Part II \n", - "3 The Silence of the Lambs Back to the Future \n", - "4 Eat Drink Man Woman The Godfather \n", - "\n", - " classics what's popular indie hits \n", - "0 Castle in the Sky Fight Club All Quiet on the Western Front \n", - "1 12 Angry Men Whiplash Army of Darkness \n", - "2 My Neighbor Totoro Blade Runner All About Eve \n", - "3 It Happened One Night Gone Girl The Professional \n", - "4 Stand by Me Big Hero 6 Shine " - ] - }, - "execution_count": 19, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# put all these titles into a single pandas dataframe , where each column is one category\n", - "all_recommendations = pd.DataFrame(columns=[\"top picks\", \"block busters\", \"classics\", \"what's popular\", \"indie hits\"])\n", - "all_recommendations[\"top picks\"] = [m[0] for m in top_picks_for_you]\n", - "all_recommendations[\"block busters\"] = [m[0] for m in block_buster_hits]\n", - "all_recommendations[\"classics\"] = [m[0] for m in classics]\n", - "all_recommendations[\"what's popular\"] = [m[0] for m in whats_popular]\n", - "all_recommendations[\"indie hits\"] = [m[0] for m in indie_hits]\n", - "\n", - "all_recommendations.head()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Conclusion\n", - "That's it! That's all it takes to build a highly scalable, personalized, customizable collaborative filtering recommendation system with Redis and RedisVL.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Deleted 4358 keys\n", - "Deleted 2000 keys\n", - "Deleted 1000 keys\n", - "Deleted 500 keys\n", - "Deleted 500 keys\n" - ] - }, - { - "data": { - "text/plain": [ - "671" - ] - }, - "execution_count": 20, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# clean up your index\n", - "while remaining := movie_index.clear():\n", - " print(f\"Deleted {remaining} keys\")\n", - "\n", - "client.delete(\"user_watched_list\")\n", - "client.delete(*[f\"user:{user_id}\" for user_id in user_vectors_and_ids.keys()])" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "redis-ai-res", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.9" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/python-recipes/recommendation-systems/collaborative_filtering_schema.yaml b/python-recipes/recommendation-systems/collaborative_filtering_schema.yaml deleted file mode 100644 index af58d793..00000000 --- a/python-recipes/recommendation-systems/collaborative_filtering_schema.yaml +++ /dev/null @@ -1,40 +0,0 @@ -index: - name: movies - prefix: movie - storage_type: json - -fields: - - name: movieId - type: tag - - name: genres - type: tag - - name: original_language - type: tag - - name: overview - type: text - - name: popularity - type: numeric - - name: release_date - type: numeric - - name: revenue - type: numeric - - name: runtime - type: numeric - - name: status - type: tag - - name: tagline - type: text - - name: title - type: text - - name: vote_average - type: numeric - - name: vote_count - type: numeric - - - name: movie_vector - type: vector - attrs: - dims: 100 - distance_metric: ip - algorithm: flat - datatype: float32 \ No newline at end of file diff --git a/python-recipes/recommendation-systems/content_filtering.ipynb b/python-recipes/recommendation-systems/content_filtering.ipynb deleted file mode 100644 index 3d541e96..00000000 --- a/python-recipes/recommendation-systems/content_filtering.ipynb +++ /dev/null @@ -1,639 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120)\n", - "\n", - "# Content Filtering in RedisVL\n", - "\n", - "In this recipe you'll learn how to build a content filtering recommender system (RecSys) from scratch using RedisVl and an IMDB movies dataset.\n", - "\n", - "## Let's Begin!\n", - "\"Open" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Recommendation systems are a common application of machine learning and serve many industries from e-commerce to music streaming platforms.\n", - "\n", - "There are many different architechtures that can be followed to build a recommender system. \n", - "\n", - "In this notebook we'll demonstrate how to build a [content filtering](https://en.wikipedia.org/wiki/Recommender_system#:~:text=of%20hybrid%20systems.-,Content%2Dbased%20filtering,-%5Bedit%5D)\n", - "recommender and use the movies dataset as our example data." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Content Filtering recommender systems are built on the premise that a person will want to be recommended things that are similar to things they already like.\n", - "\n", - "In the case of movies, if a person watches and enjoys a nature documentary we should recommend other nature documentaries. Or if they like classic black & white horror films we should recommend more of those.\n", - "\n", - "The question we need to answer is, 'what does it mean for movies to be similar?'. There are exact matching strategies, like using a movie's labelled genre like 'Horror', or 'Sci Fi', but that can lock people in to only a few genres. Or what if it's not the genre that a person likes, but certain story arcs that are common among many genres?\n", - "\n", - "For our content filtering recommender we'll measure similarity between movies as semantic similarity of their descriptions and keywords." - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "metadata": {}, - "outputs": [], - "source": [ - "## IMPORTS\n", - "import pandas as pd\n", - "import ast\n", - "import os\n", - "import pickle\n", - "import requests\n", - "\n", - "# Replace values below with your own if using Redis Cloud instance\n", - "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n", - "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n", - "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n", - "\n", - "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n", - "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\"" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Prepare The Dataset" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Start by downloading the movies data and doing a quick inspection of it." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
titleruntimeratingrating_countgenresoverviewkeywordsdirectorcastwriteryearpath
0The Story of the Kelly Gang1 hour 10 minutes6.0772['Action', 'Adventure', 'Biography']Story of Ned Kelly, an infamous 19th-century A...['ned kelly', 'australia', 'historic figure', ...Charles Tait['Elizabeth Tait', 'John Tait', 'Nicholas Brie...Charles Tait1906/title/tt0000574/
1Fantômas - À l'ombre de la guillotinenot-released6.92300['Crime', 'Drama']Inspector Juve is tasked to investigate and ca...['silent film', 'france', 'hotel', 'duchess', ...Louis Feuillade['Louis Feuillade', 'Pierre Souvestre', 'René ...Marcel Allain1913/title/tt0002844/
2Cabiria2 hours 28 minutes7.13500['Adventure', 'Drama', 'History']Cabiria is a Roman child when her home is dest...['carthage', 'slave', 'moloch', '3rd century b...Giovanni Pastrone['Titus Livius', 'Giovanni Pastrone', 'Italia ...Gabriele D'Annunzio1914/title/tt0003740/
3The Life of General Villanot-released6.765['Action', 'Adventure', 'Biography']The life and career of Panccho Villa from youn...['chihuahua mexico', 'chihuahua', 'sonora mexi...Christy Cabanne['Frank E. Woods', 'Raoul Walsh', 'Eagle Eye',...Raoul Walsh1914/title/tt0004223/
4The Patchwork Girl of Oznot-released5.4484['Adventure', 'Family', 'Fantasy']Ojo and Unc Nunkie are out of food, so they de...['silent film', 'journey', 'magic wand', 'wiza...J. Farrell MacDonald['Violet MacMillan', 'Frank Moore', 'Raymond R...L. Frank Baum1914/title/tt0004457/
\n", - "
" - ], - "text/plain": [ - " title runtime rating \\\n", - "0 The Story of the Kelly Gang 1 hour 10 minutes 6.0 \n", - "1 Fantômas - À l'ombre de la guillotine not-released 6.9 \n", - "2 Cabiria 2 hours 28 minutes 7.1 \n", - "3 The Life of General Villa not-released 6.7 \n", - "4 The Patchwork Girl of Oz not-released 5.4 \n", - "\n", - " rating_count genres \\\n", - "0 772 ['Action', 'Adventure', 'Biography'] \n", - "1 2300 ['Crime', 'Drama'] \n", - "2 3500 ['Adventure', 'Drama', 'History'] \n", - "3 65 ['Action', 'Adventure', 'Biography'] \n", - "4 484 ['Adventure', 'Family', 'Fantasy'] \n", - "\n", - " overview \\\n", - "0 Story of Ned Kelly, an infamous 19th-century A... \n", - "1 Inspector Juve is tasked to investigate and ca... \n", - "2 Cabiria is a Roman child when her home is dest... \n", - "3 The life and career of Panccho Villa from youn... \n", - "4 Ojo and Unc Nunkie are out of food, so they de... \n", - "\n", - " keywords director \\\n", - "0 ['ned kelly', 'australia', 'historic figure', ... Charles Tait \n", - "1 ['silent film', 'france', 'hotel', 'duchess', ... Louis Feuillade \n", - "2 ['carthage', 'slave', 'moloch', '3rd century b... Giovanni Pastrone \n", - "3 ['chihuahua mexico', 'chihuahua', 'sonora mexi... Christy Cabanne \n", - "4 ['silent film', 'journey', 'magic wand', 'wiza... J. Farrell MacDonald \n", - "\n", - " cast writer \\\n", - "0 ['Elizabeth Tait', 'John Tait', 'Nicholas Brie... Charles Tait \n", - "1 ['Louis Feuillade', 'Pierre Souvestre', 'René ... Marcel Allain \n", - "2 ['Titus Livius', 'Giovanni Pastrone', 'Italia ... Gabriele D'Annunzio \n", - "3 ['Frank E. Woods', 'Raoul Walsh', 'Eagle Eye',... Raoul Walsh \n", - "4 ['Violet MacMillan', 'Frank Moore', 'Raymond R... L. Frank Baum \n", - "\n", - " year path \n", - "0 1906 /title/tt0000574/ \n", - "1 1913 /title/tt0002844/ \n", - "2 1914 /title/tt0003740/ \n", - "3 1914 /title/tt0004223/ \n", - "4 1914 /title/tt0004457/ " - ] - }, - "execution_count": 12, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "try:\n", - " df = pd.read_csv(\"datasets/content_filtering/25k_imdb_movie_dataset.csv\")\n", - "except:\n", - " import requests\n", - " # download the file\n", - " url = 'https://redis-ai-resources.s3.us-east-2.amazonaws.com/recommenders/datasets/content-filtering/25k_imdb_movie_dataset.csv'\n", - " r = requests.get(url)\n", - "\n", - " #save the file as a csv\n", - " if not os.path.exists('./datasets/content_filtering'):\n", - " os.makedirs('./datasets/content_filtering')\n", - " with open('./datasets/content_filtering/25k_imdb_movie_dataset.csv', 'wb') as f:\n", - " f.write(r.content)\n", - " df = pd.read_csv(\"datasets/content_filtering/25k_imdb_movie_dataset.csv\")\n", - "\n", - "df.head()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As with any machine learning task, the first step is to clean our data.\n", - "\n", - "We'll drop some columns that we don't plan to use, and fill missing values with some reasonable defaults.\n", - "\n", - "Lastly, we'll do a quick check to make sure we've filled in all the null and missing values." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "title 0\n", - "rating 0\n", - "rating_count 0\n", - "genres 0\n", - "overview 0\n", - "keywords 0\n", - "director 0\n", - "cast 0\n", - "year 0\n", - "dtype: int64" - ] - }, - "execution_count": 13, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "roman_numerals = ['(I)','(II)','(III)','(IV)', '(V)', '(VI)', '(VII)', '(VIII)', '(IX)', '(XI)', '(XII)', '(XVI)', '(XIV)', '(XXXIII)', '(XVIII)', '(XIX)', '(XXVII)']\n", - "\n", - "def replace_year(x):\n", - " if x in roman_numerals:\n", - " return 1998 # the average year of the dataset\n", - " else:\n", - " return x\n", - "\n", - "df.drop(columns=['runtime', 'writer', 'path'], inplace=True)\n", - "df['year'] = df['year'].apply(replace_year) # replace roman numerals with average year\n", - "df['genres'] = df['genres'].apply(ast.literal_eval) # convert string representation of list to list\n", - "df['keywords'] = df['keywords'].apply(ast.literal_eval) # convert string representation of list to list\n", - "df['cast'] = df['cast'].apply(ast.literal_eval) # convert string representation of list to list\n", - "df = df[~df['overview'].isnull()] # drop rows with missing overviews\n", - "df = df[~df['overview'].isin(['none'])] # drop rows with 'none' as the overview\n", - "\n", - "# make sure we've filled all missing values\n", - "df.isnull().sum()" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Generate Vector Embeddings" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "Since we movie similarity as semantic similarity of movie descriptions we need a way to generate semantic vector embeddings of these descriptions.\n", - "\n", - "RedisVL supports many different embedding generators. For this example we'll use a HuggingFace model that is rated well for semantic similarity use cases.\n", - "\n", - "RedisVL also supports complex query logic, beyond just vector similarity. To showcase this we'll generate an embedding from each movies' `overview` text and list of `plot keywords`,\n", - "and use the remaining fields like, `genres`, `year`, and `rating` as filterable fields to target our vector queries to.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'The Story of the Kelly Gang. Story of Ned Kelly, an infamous 19th-century Australian outlaw. ned kelly, australia, historic figure, australian western, first of its kind, directorial debut, australian history, 19th century, victoria australia, australian'" - ] - }, - "execution_count": 14, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "# add a column to the dataframe with all the text we want to embed\n", - "df[\"full_text\"] = df[\"title\"] + \". \" + df[\"overview\"] + \" \" + df['keywords'].apply(lambda x: ', '.join(x))\n", - "df[\"full_text\"][0]" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "metadata": {}, - "outputs": [], - "source": [ - "# NBVAL_SKIP\n", - "# # this step will take a while, but only needs to be done once for your entire dataset\n", - "# currently taking 10 minutes to run, so we've gone ahead and saved the vectors to a file for you\n", - "# if you don't want to wait, you can skip the cell and load the vectors from the file in the next cell\n", - "from redisvl.utils.vectorize import HFTextVectorizer\n", - "\n", - "vectorizer = HFTextVectorizer(model = 'sentence-transformers/paraphrase-MiniLM-L6-v2')\n", - "\n", - "df['embedding'] = df['full_text'].apply(lambda x: vectorizer.embed(x, as_buffer=False))\n", - "pickle.dump(df['embedding'], open('datasets/content_filtering/text_embeddings.pkl', 'wb'))" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "metadata": {}, - "outputs": [], - "source": [ - "try:\n", - " with open('datasets/content_filtering/text_embeddings.pkl', 'rb') as vector_file:\n", - " df['embedding'] = pickle.load(vector_file)\n", - "except:\n", - " embeddings_url = 'https://redis-ai-resources.s3.us-east-2.amazonaws.com/recommenders/datasets/content-filtering/text_embeddings.pkl'\n", - " r = requests.get(embeddings_url)\n", - " with open('./datasets/content_filtering/text_embeddings.pkl', 'wb') as f:\n", - " f.write(r.content)\n", - " with open('datasets/content_filtering/text_embeddings.pkl', 'rb') as vector_file:\n", - " df['embedding'] = pickle.load(vector_file)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Define our Search Schema\n", - "Our data is now ready to be loaded into Redis. The last step is to define our search index schema that specifies each of our data fields and the size and type of our embedding vectors.\n", - "\n", - "We'll load this from the accompanying `content_filtering_schema.yaml` file." - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "This schema defines what each entry will look like within Redis. It will need to specify the name of each field, like `title`, `rating`, and `rating-count`, as well as the type of each field, like `text` or `numeric`.\n", - "\n", - "The vector component of each entry similarly needs its dimension (dims), distance metric, algorithm and datatype (dtype) attributes specified." - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "09:44:30 redisvl.index.index INFO Index already exists, overwriting.\n" - ] - } - ], - "source": [ - "from redis import Redis\n", - "\n", - "client = Redis.from_url(REDIS_URL)\n", - "from redisvl.schema import IndexSchema\n", - "from redisvl.index import SearchIndex\n", - "\n", - "movie_schema = IndexSchema.from_yaml(\"content_filtering_schema.yaml\")\n", - "\n", - "index = SearchIndex(movie_schema, redis_client=client)\n", - "index.create(overwrite=True, drop=True)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Load products into vector DB\n", - "Now that we have all our data cleaned and a defined schema we can load the data into RedisVL.\n", - "\n", - "We need to convert this data into a format that RedisVL can understand, which is a list of dictionaries.\n" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "metadata": {}, - "outputs": [], - "source": [ - "data = df.to_dict(orient='records')\n", - "keys = index.load(data)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Querying to get recommendations\n", - "\n", - "We now have a working content filtering recommender system, all we need a starting point, so let's say we want to find movies similar to the movie with the title \"20,000 Leagues Under the Sea\"\n", - "\n", - "We can use the title to find the movie in the dataset and then use the vector to find similar movies." - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "{'id': 'movie:b64fc099d6af440a891e1dd8314e5af7', 'vector_distance': '0.584870040417', 'title': 'The Odyssey', 'overview': 'The aquatic adventure of the highly influential and fearlessly ambitious pioneer, innovator, filmmaker, researcher, and conservationist, Jacques-Yves Cousteau, covers roughly thirty years of an inarguably rich in achievements life.'}\n", - "{'id': 'movie:2fbd7803b51a4bf9a8fb1aa79244ad64', 'vector_distance': '0.63329231739', 'title': 'The Inventor', 'overview': 'Inventing flying contraptions, war machines and studying cadavers, Leonardo da Vinci tackles the meaning of life itself with the help of French princess Marguerite de Nevarre.'}\n", - "{'id': 'movie:224a785ca7ea4006bbcdac8aad5bf1bc', 'vector_distance': '0.658123672009', 'title': 'Ruin', 'overview': 'The film follows a nameless ex-Nazi captain who navigates the ruins of post-WWII Germany determined to atone for his crimes during the war by hunting down the surviving members of his former SS Death Squad.'}\n", - "{'id': 'movie:b7d0af286515427fbbf3866bc7e9b739', 'vector_distance': '0.688094437122', 'title': 'The Raven', 'overview': 'A man with incredible powers is sought by the government and military.'}\n", - "{'id': 'movie:9aceb1903b584648b3a5b5d1bdd383b2', 'vector_distance': '0.694671392441', 'title': 'Get the Girl', 'overview': 'Sebastain \"Bash\" Danye, a legendary gun for hire hangs up his weapon to retire peacefully with his \\'it\\'s complicated\\' partner Renee. Their quiet lives are soon interrupted when they find an unconscious woman on their property, Maddie. While nursing her back to health, some bad me... Read all'}\n" - ] - } - ], - "source": [ - "from redisvl.query import RangeQuery\n", - "\n", - "query_vector = df[df['title'] == '20,000 Leagues Under the Sea']['embedding'].values[0] # one good match\n", - "\n", - "query = RangeQuery(vector=query_vector,\n", - " vector_field_name='embedding',\n", - " num_results=5,\n", - " distance_threshold=0.7,\n", - " return_fields = ['title', 'overview', 'vector_distance'])\n", - "\n", - "results = index.query(query)\n", - "for r in results:\n", - " print(r)" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "## Generating user recommendations\n", - "This systems works, but we can make it even better.\n", - "\n", - "Production recommender systems often have fields that can be configured. Users can specify if they want to see a romantic comedy or a horror film, or only see new releases.\n", - "\n", - "Let's go ahead and add this functionality by using the tags we've defined in our schema." - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "metadata": {}, - "outputs": [], - "source": [ - "from redisvl.query.filter import Tag, Num, Text\n", - "\n", - "def make_filter(genres=None, release_year=None, keywords=None):\n", - " flexible_filter = (\n", - " (Num(\"year\") > release_year) & # only show movies released after this year\n", - " (Tag(\"genres\") == genres) & # only show movies that match at least one in list of genres\n", - " (Text(\"full_text\") % keywords) # only show movies that contain at least one of the keywords\n", - " )\n", - " return flexible_filter\n", - "\n", - "def get_recommendations(movie_vector, num_results=5, distance=0.6, filter=None):\n", - " query = RangeQuery(vector=movie_vector,\n", - " vector_field_name='embedding',\n", - " num_results=num_results,\n", - " distance_threshold=distance,\n", - " return_fields = ['title', 'overview', 'genres'],\n", - " filter_expression=filter,\n", - " )\n", - "\n", - " recommendations = index.query(query)\n", - " return recommendations" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "As a final demonstration we'll find movies similar to the classic horror film 'Nosferatu'.\n", - "The process has 3 steps:\n", - "- fetch the vector embedding of our film Nosferatu\n", - "- optionally define any hard filters we want. Here we'll specify we want horror movies made on or after 1990\n", - "- perform the vector range query to find similar movies that meet our filter criteria" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "- Wolfman:\n", - "\t A man becomes afflicted by an ancient curse after he is bitten by a werewolf.\n", - "\t Genres: [\"Horror\"]\n", - "- Off Season:\n", - "\t Tenn's relentless search for his father takes him back to his childhood town only to find a community gripped by fear. As he travels deeper into the bitter winter wilderness of the town he uncovers a dreadful secret buried long ago.\n", - "\t Genres: [\"Horror\",\"Mystery\",\"Thriller\"]\n", - "- Pieces:\n", - "\t The co-eds of a Boston college campus are targeted by a mysterious killer who is creating a human jigsaw puzzle from their body parts.\n", - "\t Genres: [\"Horror\",\"Mystery\",\"Thriller\"]\n", - "- Cursed:\n", - "\t A prominent psychiatrist at a state run hospital wrestles with madness and a dark supernatural force as he and a female police detective race to stop an escaped patient from butchering five people held hostage in a remote mansion.\n", - "\t Genres: [\"Horror\",\"Thriller\"]\n", - "- The Home:\n", - "\t The Home unfolds after a young man is nearly killed during an accident that leaves him physically and emotionally scarred. To recuperate, he is taken to a secluded nursing home where the elderly residents appear to be suffering from delusions. But after witnessing a violent attac... Read all\n", - "\t Genres: [\"Action\",\"Fantasy\",\"Horror\"]\n" - ] - } - ], - "source": [ - "movie_vector = df[df['title'] == 'Nosferatu']['embedding'].values[0]\n", - "\n", - "filter = make_filter(genres=['Horror'], release_year=1990)\n", - "\n", - "recs = get_recommendations(movie_vector, distance=0.8, filter=filter)\n", - "for rec in recs:\n", - " print(f\"- {rec['title']}:\\n\\t {rec['overview']}\\n\\t Genres: {rec['genres']}\")" - ] - }, - { - "cell_type": "markdown", - "metadata": {}, - "source": [ - "### Now you have a working content filtering recommender system with Redis.\n", - "Don't forget to clean up once you're done." - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "metadata": {}, - "outputs": [], - "source": [ - "# clean up your index\n", - "while remaining := index.clear():\n", - " print(f\"Deleted {remaining} keys\")" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "redis-ai-res", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.9" - } - }, - "nbformat": 4, - "nbformat_minor": 2 -} diff --git a/python-recipes/recommendation-systems/content_filtering_schema.yaml b/python-recipes/recommendation-systems/content_filtering_schema.yaml deleted file mode 100644 index 615e12dc..00000000 --- a/python-recipes/recommendation-systems/content_filtering_schema.yaml +++ /dev/null @@ -1,34 +0,0 @@ -index: - name: movies_recommendation - prefix: movie - storage_type: json - -fields: - - name: title - type: text - - name: rating - type: numeric - - name: rating_count - type: numeric - - name: genres - type: tag - - name: overview - type: text - - name: keywords - type: tag - - name: cast - type: tag - - name: writer - type: text - - name: year - type: numeric - - name: full_text - type: text - - - name: embedding - type: vector - attrs: - dims: 384 - distance_metric: cosine - algorithm: flat - dtype: float32 \ No newline at end of file