diff --git a/.github/ignore-notebooks.txt b/.github/ignore-notebooks.txt
index 4f8e9ac0..13ab3efd 100644
--- a/.github/ignore-notebooks.txt
+++ b/.github/ignore-notebooks.txt
@@ -1,5 +1,7 @@
01_crewai_langgraph_redis
-doc2cache_llama3_1
-semantic_caching_gemini
+01_doc2cache_llama3_1
+00_semantic_caching_gemini
01_collaborative_filtering
05_nvidia_ai_rag_redis
+01_routing_optimization
+02_semantic_cache_optimization
diff --git a/.python-version b/.python-version
index 2c073331..b6d8b761 100644
--- a/.python-version
+++ b/.python-version
@@ -1 +1 @@
-3.11
+3.11.8
diff --git a/README.md b/README.md
index 3cbe48a5..a04a0dd5 100644
--- a/README.md
+++ b/README.md
@@ -77,8 +77,9 @@ An estimated 31% of LLM queries are potentially redundant ([source](https://arxi
| Recipe | Description |
| --- | --- |
-| [/semantic-cache/doc2cache_llama3_1.ipynb](python-recipes/semantic-cache/doc2cache_llama3_1.ipynb) | Build a semantic cache using the Doc2Cache framework and Llama3.1 |
-| [/semantic-cache/semantic_caching_gemini.ipynb](python-recipes/semantic-cache/semantic_caching_gemini.ipynb) | Build a semantic cache with Redis and Google Gemini |
+| [/semantic-cache/00_semantic_caching_gemini.ipynb](python-recipes/semantic-cache/00_semantic_caching_gemini.ipynb) | Build a semantic cache with Redis and Google Gemini |
+| [/semantic-cache/01_doc2cache_llama3_1.ipynb](python-recipes/semantic-cache/01_doc2cache_llama3_1.ipynb) | Build a semantic cache using the Doc2Cache framework and Llama3.1 |
+| [/semantic-cache/02_semantic_cache_optimization.ipynb](python-recipes/semantic-cache/02_semantic_cache_optimization.ipynb) | Use CacheThresholdOptimizer from redisvl to setup best cache config |
### Semantic Routing
Routing is a simple and effective way of preventing misuses with your AI application or for creating branching logic between data sources etc.
@@ -86,6 +87,7 @@ Routing is a simple and effective way of preventing misuses with your AI applica
| Recipe | Description |
| --- | --- |
| [/semantic-router/00_semantic_routing.ipynb](python-recipes/semantic-router/00_semantic_routing.ipynb) | Simple examples of how to build an allow/block list router in addition to a multi-topic router |
+| [/semantic-router/01_routing_optimization.ipynb](python-recipes/semantic-router/01_routing_optimization.ipynb) | Use RouterThresholdOptimizer from redisvl to setup best router config |
### Agents
diff --git a/python-recipes/semantic-cache/semantic_caching_gemini.ipynb b/python-recipes/semantic-cache/00_semantic_caching_gemini.ipynb
similarity index 99%
rename from python-recipes/semantic-cache/semantic_caching_gemini.ipynb
rename to python-recipes/semantic-cache/00_semantic_caching_gemini.ipynb
index 8ba17b8a..42944322 100644
--- a/python-recipes/semantic-cache/semantic_caching_gemini.ipynb
+++ b/python-recipes/semantic-cache/00_semantic_caching_gemini.ipynb
@@ -8,7 +8,7 @@
"source": [
"# Building a Semantic Cache with Redis and VertexAI Gemini Model\n",
"\n",
- "![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
\n"
+ "\n"
]
},
{
diff --git a/python-recipes/semantic-cache/doc2cache_llama3_1.ipynb b/python-recipes/semantic-cache/01_doc2cache_llama3_1.ipynb
similarity index 100%
rename from python-recipes/semantic-cache/doc2cache_llama3_1.ipynb
rename to python-recipes/semantic-cache/01_doc2cache_llama3_1.ipynb
diff --git a/python-recipes/semantic-cache/02_semantic_cache_optimization.ipynb b/python-recipes/semantic-cache/02_semantic_cache_optimization.ipynb
new file mode 100644
index 00000000..3a3054a8
--- /dev/null
+++ b/python-recipes/semantic-cache/02_semantic_cache_optimization.ipynb
@@ -0,0 +1,371 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "\n",
+ "\n",
+ "# Optimize semantic cache threshold with RedisVL\n",
+ "\n",
+ "> **Note:** Threshold optimization in redisvl relies on `python > 3.9.`\n",
+ "\n",
+ ""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Install dependencies"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# install from branch since scheduled for 0.5.0\n",
+ "%pip install git+https://github.com/redis/redis-vl-python.git@0.5.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Run a Redis instance\n",
+ "\n",
+ "#### For Colab\n",
+ "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NBVAL_SKIP\n",
+ "%%sh\n",
+ "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
+ "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
+ "sudo apt-get update > /dev/null 2>&1\n",
+ "sudo apt-get install redis-stack-server > /dev/null 2>&1\n",
+ "redis-stack-server --daemonize yes"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### For Alternative Environments\n",
+ "There are many ways to get the necessary redis-stack instance running\n",
+ "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
+ "own version of Redis Enterprise running, that works too!\n",
+ "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
+ "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# CacheThresholdOptimizer\n",
+ "\n",
+ "Let's say you setup the following semantic cache with a distance_threshold of `X` and store the entries:\n",
+ "\n",
+ "- prompt: `what is the capital of france?` response: `paris`\n",
+ "- prompt: `what is the capital of morocco?` response: `rabat`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/robert.shelton/.pyenv/versions/3.11.9/lib/python3.11/site-packages/huggingface_hub/file_download.py:1142: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
+ " warnings.warn(\n",
+ "/Users/robert.shelton/.pyenv/versions/3.11.9/lib/python3.11/site-packages/huggingface_hub/file_download.py:1142: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
+ " warnings.warn(\n"
+ ]
+ }
+ ],
+ "source": [
+ "from redisvl.extensions.llmcache import SemanticCache\n",
+ "\n",
+ "sem_cache = SemanticCache(\n",
+ " name=\"sem_cache\", # underlying search index name\n",
+ " redis_url=\"redis://localhost:6379\", # redis connection url string\n",
+ " distance_threshold=0.5 # semantic cache distance threshold\n",
+ ")\n",
+ "\n",
+ "paris_key = sem_cache.store(prompt=\"what is the capital of france?\", response=\"paris\")\n",
+ "rabat_key = sem_cache.store(prompt=\"what is the capital of morocco?\", response=\"rabat\")\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "This works well but we want to make sure the cache only applies for the appropriate questions. If we test the cache with a question we don't want a response to we see that the current distance_threshold is too high. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[{'entry_id': 'c990cc06e5e77570e5f03360426d2b7f947cbb5a67daa8af8164bfe0b3e24fe3',\n",
+ " 'prompt': 'what is the capital of france?',\n",
+ " 'response': 'paris',\n",
+ " 'vector_distance': 0.421104669571,\n",
+ " 'inserted_at': 1741039231.99,\n",
+ " 'updated_at': 1741039231.99,\n",
+ " 'key': 'sem_cache:c990cc06e5e77570e5f03360426d2b7f947cbb5a67daa8af8164bfe0b3e24fe3'}]"
+ ]
+ },
+ "execution_count": 2,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sem_cache.check(\"what's the capital of britain?\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define test_data and optimize\n",
+ "\n",
+ "With the `CacheThresholdOptimizer` you can quickly tune the distance threshold by providing some test data in the form:\n",
+ "\n",
+ "```json\n",
+ "[\n",
+ " {\n",
+ " \"query\": \"What's the capital of Britain?\",\n",
+ " \"query_match\": \"\"\n",
+ " },\n",
+ " {\n",
+ " \"query\": \"What's the capital of France??\",\n",
+ " \"query_match\": paris_key\n",
+ " },\n",
+ " {\n",
+ " \"query\": \"What's the capital city of Morocco?\",\n",
+ " \"query_match\": rabat_key\n",
+ " },\n",
+ "]\n",
+ "```\n",
+ "\n",
+ "The threshold optimizer will then efficiently execute and score different threshold against the what is currently populated in your cache and automatically update the threshold of the cache to the best setting"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Distance threshold before: 0.5 \n",
+ "\n",
+ "Distance threshold after: 0.13050847457627118 \n",
+ "\n"
+ ]
+ }
+ ],
+ "source": [
+ "from redisvl.utils.optimize import CacheThresholdOptimizer\n",
+ "\n",
+ "test_data = [\n",
+ " {\n",
+ " \"query\": \"What's the capital of Britain?\",\n",
+ " \"query_match\": \"\"\n",
+ " },\n",
+ " {\n",
+ " \"query\": \"What's the capital of France??\",\n",
+ " \"query_match\": paris_key\n",
+ " },\n",
+ " {\n",
+ " \"query\": \"What's the capital city of Morocco?\",\n",
+ " \"query_match\": rabat_key\n",
+ " },\n",
+ "]\n",
+ "\n",
+ "print(f\"Distance threshold before: {sem_cache.distance_threshold} \\n\")\n",
+ "optimizer = CacheThresholdOptimizer(sem_cache, test_data)\n",
+ "optimizer.optimize()\n",
+ "print(f\"Distance threshold after: {sem_cache.distance_threshold} \\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can also see that we no longer match on the incorrect example:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[]"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sem_cache.check(\"what's the capital of britain?\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "But still match on highly relevant prompts:"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[{'entry_id': 'c990cc06e5e77570e5f03360426d2b7f947cbb5a67daa8af8164bfe0b3e24fe3',\n",
+ " 'prompt': 'what is the capital of france?',\n",
+ " 'response': 'paris',\n",
+ " 'vector_distance': 0.0835866332054,\n",
+ " 'inserted_at': 1741039231.99,\n",
+ " 'updated_at': 1741039231.99,\n",
+ " 'key': 'sem_cache:c990cc06e5e77570e5f03360426d2b7f947cbb5a67daa8af8164bfe0b3e24fe3'}]"
+ ]
+ },
+ "execution_count": 5,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "sem_cache.check(\"what's the capital city of france?\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Additional configuration\n",
+ "\n",
+ "By default threshold optimization is performed based on the highest `F1` score but can also be configured to rank results based on `precision` and `recall` by specifying the `eval_metric` keyword argument. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Distance threshold before: 0.5 \n",
+ "\n"
+ ]
+ },
+ {
+ "ename": "NameError",
+ "evalue": "name 'CacheThresholdOptimizer' is not defined",
+ "output_type": "error",
+ "traceback": [
+ "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
+ "\u001b[0;31mNameError\u001b[0m Traceback (most recent call last)",
+ "Cell \u001b[0;32mIn[2], line 2\u001b[0m\n\u001b[1;32m 1\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDistance threshold before: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00msem_cache\u001b[38;5;241m.\u001b[39mdistance_threshold\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m \u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m----> 2\u001b[0m optimizer \u001b[38;5;241m=\u001b[39m \u001b[43mCacheThresholdOptimizer\u001b[49m(sem_cache, test_data, eval_metric\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mprecision\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 3\u001b[0m optimizer\u001b[38;5;241m.\u001b[39moptimize()\n\u001b[1;32m 4\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mDistance threshold after: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00msem_cache\u001b[38;5;241m.\u001b[39mdistance_threshold\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m \u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n",
+ "\u001b[0;31mNameError\u001b[0m: name 'CacheThresholdOptimizer' is not defined"
+ ]
+ }
+ ],
+ "source": [
+ "print(f\"Distance threshold before: {sem_cache.distance_threshold} \\n\")\n",
+ "optimizer = CacheThresholdOptimizer(sem_cache, test_data, eval_metric=\"precision\")\n",
+ "optimizer.optimize()\n",
+ "print(f\"Distance threshold after: {sem_cache.distance_threshold} \\n\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "print(f\"Distance threshold before: {sem_cache.distance_threshold} \\n\")\n",
+ "optimizer = CacheThresholdOptimizer(sem_cache, test_data, eval_metric=\"recall\")\n",
+ "optimizer.optimize()\n",
+ "print(f\"Distance threshold after: {sem_cache.distance_threshold} \\n\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Note**: the CacheThresholdOptimizer class also exposes an optional `opt_fn` which can be leveraged to define more custom logic. See implementation within [source code for reference](https://github.com/redis/redis-vl-python/blob/18ff1008c5a40353c97c176d3d30028a87ff777a/redisvl/utils/optimize/cache.py#L48-L49)."
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Cleanup"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "sem_cache.delete()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.9"
+ },
+ "orig_nbformat": 4
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/python-recipes/semantic-router/01_routing_optimization.ipynb b/python-recipes/semantic-router/01_routing_optimization.ipynb
new file mode 100644
index 00000000..474bb8eb
--- /dev/null
+++ b/python-recipes/semantic-router/01_routing_optimization.ipynb
@@ -0,0 +1,774 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "cbba56a9",
+ "metadata": {},
+ "source": [
+ "\n",
+ "# Routing Optimization\n",
+ "\n",
+ "Implementing a semantic router is a great light weight way to add branching logic to your application without taking on additional LLM calls. However, it can be tough to determine the optimal distance threshold values for your routes to maximize performance. This guide will walk through:\n",
+ "\n",
+ "- how to configure a semantic router\n",
+ "- how to optimize the distance thresholds for the routes\n",
+ "- a comparison between performing similar logic with an LLM versus a router\n",
+ "\n",
+ "## Let's Begin!\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "19bdc2a5-2192-4f5f-bd6e-7c956fd0e230",
+ "metadata": {},
+ "source": [
+ "# Setup\n",
+ "\n",
+ "## Install Packages"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c620286e",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Note: you may need to restart the kernel to use updated packages.\n"
+ ]
+ }
+ ],
+ "source": [
+ "%pip install -q sentence-transformers ranx"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "717284f0",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Collecting git+https://github.com/redis/redis-vl-python.git@0.5.0\n",
+ " Cloning https://github.com/redis/redis-vl-python.git (to revision 0.5.0) to /private/var/folders/_g/rr4lnxxx1_z7m78lz89dhvsm0000gp/T/pip-req-build-54zjmrpr\n",
+ " Running command git clone --filter=blob:none --quiet https://github.com/redis/redis-vl-python.git /private/var/folders/_g/rr4lnxxx1_z7m78lz89dhvsm0000gp/T/pip-req-build-54zjmrpr\n",
+ " Running command git checkout -b 0.5.0 --track origin/0.5.0\n",
+ " Switched to a new branch '0.5.0'\n",
+ " branch '0.5.0' set up to track 'origin/0.5.0'.\n",
+ " Resolved https://github.com/redis/redis-vl-python.git to commit 3ca4c97baa9640d24feedd3bb3791cf95859367d\n",
+ " Installing build dependencies ... \u001b[?25ldone\n",
+ "\u001b[?25h Getting requirements to build wheel ... \u001b[?25ldone\n",
+ "\u001b[?25h Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
+ "\u001b[?25hRequirement already satisfied: coloredlogs<16.0,>=15.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (15.0.1)\n",
+ "Requirement already satisfied: ml-dtypes<0.5.0,>=0.4.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (0.4.1)\n",
+ "Requirement already satisfied: numpy<2,>=1 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (1.26.4)\n",
+ "Requirement already satisfied: pydantic<3,>=2 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (2.7.4)\n",
+ "Requirement already satisfied: python-ulid<4.0.0,>=3.0.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (3.0.0)\n",
+ "Requirement already satisfied: pyyaml<7.0,>=5.4 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (6.0.1)\n",
+ "Requirement already satisfied: redis<6.0,>=5.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (5.0.5)\n",
+ "Requirement already satisfied: tabulate<0.10.0,>=0.9.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (0.9.0)\n",
+ "Requirement already satisfied: tenacity>=8.2.2 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from redisvl==0.4.1) (8.3.0)\n",
+ "Requirement already satisfied: humanfriendly>=9.1 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from coloredlogs<16.0,>=15.0->redisvl==0.4.1) (10.0)\n",
+ "Requirement already satisfied: annotated-types>=0.4.0 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from pydantic<3,>=2->redisvl==0.4.1) (0.7.0)\n",
+ "Requirement already satisfied: pydantic-core==2.18.4 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from pydantic<3,>=2->redisvl==0.4.1) (2.18.4)\n",
+ "Requirement already satisfied: typing-extensions>=4.6.1 in /Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages (from pydantic<3,>=2->redisvl==0.4.1) (4.12.2)\n",
+ "Building wheels for collected packages: redisvl\n",
+ " Building wheel for redisvl (pyproject.toml) ... \u001b[?25ldone\n",
+ "\u001b[?25h Created wheel for redisvl: filename=redisvl-0.4.1-py3-none-any.whl size=113401 sha256=973e3b34a10bf10547873947798f4c37f681a87bd1f53c7cf938f2b4bccd71a6\n",
+ " Stored in directory: /private/var/folders/_g/rr4lnxxx1_z7m78lz89dhvsm0000gp/T/pip-ephem-wheel-cache-wt36bttp/wheels/95/dc/1e/d8dc251e38989044675dae0b596a2dee10cbfdecac5c62ccdf\n",
+ "Successfully built redisvl\n",
+ "Installing collected packages: redisvl\n",
+ "Successfully installed redisvl-0.4.1\n",
+ "\n",
+ "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m24.0\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m25.0.1\u001b[0m\n",
+ "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
+ "Note: you may need to restart the kernel to use updated packages.\n"
+ ]
+ }
+ ],
+ "source": [
+ "# install from branch since scheduled for 0.5.0\n",
+ "%pip install git+https://github.com/redis/redis-vl-python.git@0.5.0"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "323aec7f",
+ "metadata": {},
+ "source": [
+ "## Run a Redis instance\n",
+ "\n",
+ "#### For Colab\n",
+ "Use the shell script below to download, extract, and install [Redis Stack](https://redis.io/docs/getting-started/install-stack/) directly from the Redis package archive."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2cb85a99",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# NBVAL_SKIP\n",
+ "%%sh\n",
+ "curl -fsSL https://packages.redis.io/gpg | sudo gpg --dearmor -o /usr/share/keyrings/redis-archive-keyring.gpg\n",
+ "echo \"deb [signed-by=/usr/share/keyrings/redis-archive-keyring.gpg] https://packages.redis.io/deb $(lsb_release -cs) main\" | sudo tee /etc/apt/sources.list.d/redis.list\n",
+ "sudo apt-get update > /dev/null 2>&1\n",
+ "sudo apt-get install redis-stack-server > /dev/null 2>&1\n",
+ "redis-stack-server --daemonize yes"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7c5dbaaf",
+ "metadata": {},
+ "source": [
+ "#### For Alternative Environments\n",
+ "There are many ways to get the necessary redis-stack instance running\n",
+ "1. On cloud, deploy a [FREE instance of Redis in the cloud](https://redis.com/try-free/). Or, if you have your\n",
+ "own version of Redis Enterprise running, that works too!\n",
+ "2. Per OS, [see the docs](https://redis.io/docs/latest/operate/oss_and_stack/install/install-stack/)\n",
+ "3. With docker: `docker run -d --name redis-stack-server -p 6379:6379 redis/redis-stack-server:latest`"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1d4499ae",
+ "metadata": {},
+ "source": [
+ "### Define the Redis Connection URL\n",
+ "\n",
+ "By default this notebook connects to the local instance of Redis Stack. **If you have your own Redis Enterprise instance** - replace REDIS_PASSWORD, REDIS_HOST and REDIS_PORT values with your own."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "aefda1d1",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "import warnings\n",
+ "\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "\n",
+ "# Replace values below with your own if using Redis Cloud instance\n",
+ "REDIS_HOST = os.getenv(\"REDIS_HOST\", \"localhost\") # ex: \"redis-18374.c253.us-central1-1.gce.cloud.redislabs.com\"\n",
+ "REDIS_PORT = os.getenv(\"REDIS_PORT\", \"6379\") # ex: 18374\n",
+ "REDIS_PASSWORD = os.getenv(\"REDIS_PASSWORD\", \"\") # ex: \"1TNxTEdYRDgIDKM2gDfasupCADXXXX\"\n",
+ "\n",
+ "# If SSL is enabled on the endpoint, use rediss:// as the URL prefix\n",
+ "REDIS_URL = f\"redis://:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}\""
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "10f4cb85",
+ "metadata": {},
+ "source": [
+ "# Routing with multiple routes\n",
+ "\n",
+ "## Define the Routes\n",
+ "\n",
+ "Below we define 3 different routes. One for `technology`, one for `sports`, and\n",
+ "another for `entertainment`. Now for this example, the goal here is\n",
+ "surely topic \"classification\". But you can create routes and references for\n",
+ "almost anything.\n",
+ "\n",
+ "Each route has a set of references that cover the \"semantic surface area\" of the\n",
+ "route. The incoming query from a user needs to be semantically similar to one or\n",
+ "more of the references in order to \"match\" on the route. Note that each route can have it's own distinct `distance_threshold` that defines what is considered a match for the particular query. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 38,
+ "id": "60ad280c",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from redisvl.extensions.router import Route\n",
+ "\n",
+ "faq = Route(\n",
+ " name=\"faq\",\n",
+ " references=[\n",
+ " \"How do I reset my password?\",\n",
+ " \"Where can I view my order history?\",\n",
+ " \"How do I update my shipping address?\",\n",
+ " \"Where are my saved payment methods?\",\n",
+ " \"How do I change my email preferences?\",\n",
+ " \"How can I see my loyalty points balance?\",\n",
+ " \"Where do I find my digital receipts?\",\n",
+ " \"How do I enable two-factor authentication?\",\n",
+ " \"Can I change my username or email?\",\n",
+ " \"How do I manage my account settings?\"\n",
+ " ],\n",
+ " metadata={\"category\": \"account_management\", \"priority\": 1},\n",
+ " distance_threshold=0.5\n",
+ ")\n",
+ "\n",
+ "general = Route(\n",
+ " name=\"general\",\n",
+ " references=[\n",
+ " \"I received the wrong item in my order, can you help?\",\n",
+ " \"Can you recommend products that match my specific needs?\",\n",
+ " \"The assembly instructions for my furniture are unclear\",\n",
+ " \"I need help finding a product with particular specifications\",\n",
+ " \"My order arrived damaged, what are my options?\",\n",
+ " \"Can you help me design a room with your products?\",\n",
+ " \"I'm looking for custom sizing options for this product\",\n",
+ " \"The item I received doesn't match the online description\",\n",
+ " \"I need advice on which model would work best for my situation\",\n",
+ " \"Can you help troubleshoot an issue with my recent purchase?\"\n",
+ " ],\n",
+ " metadata={\"category\": \"customer_service\", \"priority\": 2},\n",
+ " distance_threshold=0.5\n",
+ ")\n",
+ "\n",
+ "blocked = Route(\n",
+ " name=\"blocked\",\n",
+ " references=[\n",
+ " \"What is your company's stance on the recent election?\",\n",
+ " \"Do you support liberal or conservative policies?\",\n",
+ " \"Can you tell me another customer's address?\",\n",
+ " \"What's your CEO's opinion on gun control?\",\n",
+ " \"I need personal information about one of your employees\",\n",
+ " \"How does your company vote on political issues?\",\n",
+ " \"Can you provide me with someone's credit card details?\",\n",
+ " \"What's your position on immigration reform?\",\n",
+ " \"I want to know where a specific customer lives\",\n",
+ " \"Does your company donate to political campaigns?\"\n",
+ " ],\n",
+ " metadata={\"category\": \"prohibited\", \"priority\": 3},\n",
+ " distance_threshold=0.5\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "9cdbcbff",
+ "metadata": {},
+ "source": [
+ "## Initialize the SemanticRouter\n",
+ "\n",
+ "Like before the ``SemanticRouter`` class will automatically create an index within Redis upon initialization for the route references."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 39,
+ "id": "e80aaf84",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/Users/robert.shelton/.pyenv/versions/3.11.8/lib/python3.11/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
+ " warnings.warn(\n"
+ ]
+ },
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "15:46:13 redisvl.index.index INFO Index already exists, overwriting.\n"
+ ]
+ }
+ ],
+ "source": [
+ "import os\n",
+ "from redisvl.extensions.router import SemanticRouter\n",
+ "\n",
+ "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"false\"\n",
+ "\n",
+ "# Initialize the SemanticRouter\n",
+ "ecom_router = SemanticRouter(\n",
+ " name=\"ecom-router\",\n",
+ " routes=[faq, general, blocked],\n",
+ " redis_url=\"redis://localhost:6379\",\n",
+ " overwrite=True # Blow away any other routing index with this name\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8b199505",
+ "metadata": {},
+ "source": [
+ "## View the created index"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 40,
+ "id": "3caedb77",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "\n",
+ "\n",
+ "Index Information:\n",
+ "╭──────────────┬────────────────┬─────────────────┬─────────────────┬────────────╮\n",
+ "│ Index Name │ Storage Type │ Prefixes │ Index Options │ Indexing │\n",
+ "├──────────────┼────────────────┼─────────────────┼─────────────────┼────────────┤\n",
+ "│ ecom-router │ HASH │ ['ecom-router'] │ [] │ 0 │\n",
+ "╰──────────────┴────────────────┴─────────────────┴─────────────────┴────────────╯\n",
+ "Index Fields:\n",
+ "╭────────────┬─────────────┬────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬─────────────────┬────────────────╮\n",
+ "│ Name │ Attribute │ Type │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │\n",
+ "├────────────┼─────────────┼────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────────────────┼────────────────┤\n",
+ "│ route_name │ route_name │ TAG │ SEPARATOR │ , │ │ │ │ │ │ │\n",
+ "│ reference │ reference │ TEXT │ WEIGHT │ 1 │ │ │ │ │ │ │\n",
+ "│ vector │ vector │ VECTOR │ algorithm │ FLAT │ data_type │ FLOAT32 │ dim │ 768 │ distance_metric │ COSINE │\n",
+ "╰────────────┴─────────────┴────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴─────────────────┴────────────────╯\n"
+ ]
+ }
+ ],
+ "source": [
+ "# look at the index specification created for the semantic router\n",
+ "!rvl index info -i ecom-router"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "8eb95dde",
+ "metadata": {},
+ "source": [
+ "## Test it out"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 41,
+ "id": "5b0e3208",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "RouteMatch(name='faq', distance=0.108501493931)"
+ ]
+ },
+ "execution_count": 41,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# Query the router with a statement\n",
+ "route_match = ecom_router(\"Whatup how do i reset my password?\")\n",
+ "route_match"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e7bc759b",
+ "metadata": {},
+ "source": [
+ "## Optimize route distance thresholds with test data\n",
+ "\n",
+ "For optimization within redisvl you can create test data manually or make use of a model to generate some for you. In this case we will use a model to do it for us.\n",
+ "\n",
+ "Prompt for creating test data:\n",
+ "> used claude sonnet 3.7 for generation of resource\n",
+ "\n",
+ "```txt\n",
+ "You are a test data creation helper. \n",
+ "\n",
+ "Create test data of the form:\n",
+ "\n",
+ "{\n",
+ " \"query\": \"query about a topic\",\n",
+ " \"query_match\": \"topic-the-query-matches\"\n",
+ "}\n",
+ "\n",
+ "The 3 available topics are: faq, general, and blocked. Generate many examples that map to these topics such that we can train a model to find the best thresholds for this classification task. Also make sure to include some examples that don't map to any of the topics to check the null case for these leave the query_match field empty.\n",
+ "```\n",
+ "\n",
+ "The output of this call was saved to `./resources/test_data.json`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "id": "3c03a117",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import json\n",
+ "\n",
+ "with open(\"resources/ecom_train_data.json\", \"r\") as f:\n",
+ " train_data = json.load(f)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "1d0c5c2a",
+ "metadata": {},
+ "source": [
+ "## Run optimization with router"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "id": "83d2a15c",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Eval metric F1: start 0.586, end 0.671 \n",
+ "Ending thresholds: {'faq': 0.5868686868686872, 'general': 0.7626262626262627, 'blocked': 0.12517090092847685}\n"
+ ]
+ }
+ ],
+ "source": [
+ "from redisvl.utils.optimize import RouterThresholdOptimizer\n",
+ "\n",
+ "optimizer = RouterThresholdOptimizer(ecom_router, train_data)\n",
+ "optimizer.optimize()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "343964ff",
+ "metadata": {},
+ "source": [
+ "## Test classification against LLM\n",
+ "\n",
+ "Using the same prompt above we generated and stored another 20 questions to use as our `test_data` to compare against using an LLM model to perform this classification."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "10c83f5a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import os\n",
+ "import getpass\n",
+ "import time\n",
+ "import numpy as np\n",
+ "\n",
+ "from openai import OpenAI\n",
+ "\n",
+ "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"False\"\n",
+ "\n",
+ "api_key = os.getenv(\"OPENAI_API_KEY\") or getpass.getpass(\"Enter your OpenAI API key: \")\n",
+ "\n",
+ "client = OpenAI(api_key=api_key)\n",
+ "\n",
+ "def ask_openai(question: str) -> str:\n",
+ " prompt = f\"\"\"\n",
+ " You are a classification bot. Your job is to classify the following query as either faq, general, blocked, or none. Return only the string label or an empty string if no match.\n",
+ "\n",
+ " general is defined as request requiring customer service.\n",
+ " faq is defined as a request for commonly asked account questions.\n",
+ " blocked is defined as a request for prohibited information.\n",
+ "\n",
+ " query: \"{question}\"\n",
+ " \"\"\"\n",
+ " response = client.completions.create(\n",
+ " model=\"gpt-3.5-turbo-instruct\",\n",
+ " prompt=prompt,\n",
+ " max_tokens=200\n",
+ " )\n",
+ " return response.choices[0].text.strip()"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "feb25546",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'faq'"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "with open(\"resources/ecom_test_data.json\", \"r\") as f:\n",
+ " test_data = json.load(f)\n",
+ "\n",
+ "\n",
+ "ask_openai(test_data[0][\"query\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 19,
+ "id": "e5c921b2",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import time\n",
+ "\n",
+ "def test_classifier(classifier, test_data, is_router=False):\n",
+ " correct = 0\n",
+ " times = []\n",
+ " for data in test_data:\n",
+ " start = time.time()\n",
+ " if is_router:\n",
+ " prediction = classifier(data[\"query\"]).name\n",
+ " else:\n",
+ " prediction = classifier(data[\"query\"])\n",
+ " \n",
+ " if not prediction or prediction.lower() == \"none\":\n",
+ " prediction = \"\"\n",
+ "\n",
+ " times.append(time.time() - start)\n",
+ " print(f\"Expected | Observed: {data['query_match']} | {prediction.lower()}\")\n",
+ " if prediction.lower() == data[\"query_match\"]:\n",
+ " correct += 1\n",
+ " accuracy = correct / len(test_data)\n",
+ " avg_time = np.mean(times)\n",
+ " return accuracy, avg_time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 20,
+ "id": "5c6024e8",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | \n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | general\n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: general | general\n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | the label is general.\n",
+ "Expected | Observed: general | general\n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: blocked | \n",
+ "Expected | Observed: blocked | blocked\n",
+ "Expected | Observed: blocked | empty string\n",
+ "Expected | Observed: blocked | general\n",
+ "Expected | Observed: blocked | classifier's got no talent :(\n",
+ "\n",
+ "none\n",
+ "Expected | Observed: blocked | \n",
+ "Expected | Observed: blocked | \n"
+ ]
+ }
+ ],
+ "source": [
+ "llm_accuracy, llm_avg_time = test_classifier(ask_openai, test_data)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 21,
+ "id": "c3362a1b",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(0.4, 0.6402494192123414)"
+ ]
+ },
+ "execution_count": 21,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "llm_accuracy, llm_avg_time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 22,
+ "id": "40ddc05d",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | \n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | \n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | general\n",
+ "Expected | Observed: blocked | \n",
+ "Expected | Observed: blocked | blocked\n",
+ "Expected | Observed: blocked | blocked\n",
+ "Expected | Observed: blocked | \n",
+ "Expected | Observed: blocked | \n",
+ "Expected | Observed: blocked | blocked\n",
+ "Expected | Observed: blocked | blocked\n"
+ ]
+ }
+ ],
+ "source": [
+ "router_accuracy, router_avg_time = test_classifier(ecom_router, test_data, is_router=True)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 23,
+ "id": "bec49e6f",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(0.45, 0.07264227867126465)"
+ ]
+ },
+ "execution_count": 23,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "router_accuracy, router_avg_time"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "cd4b83bc",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from redisvl.extensions.router.schema import DistanceAggregationMethod\n",
+ "from redisvl.extensions.router import RoutingConfig\n",
+ "\n",
+ "# toggle aggregation method\n",
+ "ecom_router.update_routing_config(\n",
+ " RoutingConfig(aggregation_method=DistanceAggregationMethod.min, max_k=3)\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 25,
+ "id": "73d2547f",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | \n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | \n",
+ "Expected | Observed: faq | faq\n",
+ "Expected | Observed: faq | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | \n",
+ "Expected | Observed: general | general\n",
+ "Expected | Observed: blocked | \n",
+ "Expected | Observed: blocked | blocked\n",
+ "Expected | Observed: blocked | blocked\n",
+ "Expected | Observed: blocked | \n",
+ "Expected | Observed: blocked | \n",
+ "Expected | Observed: blocked | blocked\n",
+ "Expected | Observed: blocked | blocked\n"
+ ]
+ }
+ ],
+ "source": [
+ "router_accuracy_min, router_avg_time_min = test_classifier(ecom_router, test_data, is_router=True)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 26,
+ "id": "e1c3b52b",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "(0.45, 0.04600746631622314)"
+ ]
+ },
+ "execution_count": 26,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "router_accuracy_min, router_avg_time_min"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "acbec96f",
+ "metadata": {},
+ "source": [
+ "## Analysis\n",
+ "\n",
+ "The following outputs illustrate the tradeoffs with LLMs vs. using the router. For this particular example, the accuracy is similar however the router comes with an almost 10x latency boost and considerable cost savings depending on the model used."
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.8"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/python-recipes/semantic-router/resources/ecom_test_data.json b/python-recipes/semantic-router/resources/ecom_test_data.json
new file mode 100644
index 00000000..2e82a252
--- /dev/null
+++ b/python-recipes/semantic-router/resources/ecom_test_data.json
@@ -0,0 +1,82 @@
+[
+ {
+ "query": "How do I access my purchase history from previous years?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Can I transfer my loyalty points to another account?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I add a secondary contact to my account?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Where do I find the option to download my invoice PDFs?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How can I set up a recurring order for monthly deliveries?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I link my loyalty account with your mobile app?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Where in my account can I see which devices are currently logged in?",
+ "query_match": "faq"
+ },
+ {
+ "query": "The shirt I ordered is perfect but I need matching pants. Can you help me find the exact shade to coordinate a complete outfit?",
+ "query_match": "general"
+ },
+ {
+ "query": "I'm remodeling my kitchen and need advice on which of your appliances would work with my unusual cabinet dimensions",
+ "query_match": "general"
+ },
+ {
+ "query": "The product description mentions water resistance but doesn't specify the IP rating. Can you tell me if it would survive in my boat's cabin?",
+ "query_match": "general"
+ },
+ {
+ "query": "I need help selecting office furniture for employees with ergonomic needs related to specific medical conditions",
+ "query_match": "general"
+ },
+ {
+ "query": "The color of the wallpaper I received appears different from the website. Is there a way to get a custom sample to match my existing decor?",
+ "query_match": "general"
+ },
+ {
+ "query": "I'm organizing a corporate retreat and need to order matching items for 50 gift baskets with our logo. Can you help with customization?",
+ "query_match": "general"
+ },
+ {
+ "query": "How do you think the recent tax legislation will affect online retailers?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Can you provide me with the home address of your customer support representative?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Does your company support pro-choice or pro-life policies?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "I need to know if one of my employees has been shopping on your site",
+ "query_match": "blocked"
+ },
+ {
+ "query": "What are the birth dates of people who purchased this item?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Which political candidates has your company donated to in the past?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "What is your company's position on the current border policy debate?",
+ "query_match": "blocked"
+ }
+]
\ No newline at end of file
diff --git a/python-recipes/semantic-router/resources/ecom_train_data.json b/python-recipes/semantic-router/resources/ecom_train_data.json
new file mode 100644
index 00000000..51623403
--- /dev/null
+++ b/python-recipes/semantic-router/resources/ecom_train_data.json
@@ -0,0 +1,282 @@
+[
+ {
+ "query": "How do I view my order history?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I change my account password?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Can I merge two accounts under the same email?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I update my billing information?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Where do I find my saved payment methods?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I edit my profile information?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Can I see my loyalty points balance?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I enable two-factor authentication?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Where are my digital receipts stored?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I manage my notification preferences?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How can I view my wishlist items?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Where do I find my saved addresses?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I delete my account?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Can I change my username?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Where do I find my purchase history?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I set up automatic payments?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Can I have multiple shipping addresses on my account?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I redeem my reward points?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Where do I find my account's subscription settings?",
+ "query_match": "faq"
+ },
+ {
+ "query": "How do I link my social media accounts?",
+ "query_match": "faq"
+ },
+ {
+ "query": "I received the wrong color sweater and need help with an exchange but already removed the tags",
+ "query_match": "general"
+ },
+ {
+ "query": "Can you help me find a dining table that matches my existing chairs and fits in a 12x14 room?",
+ "query_match": "general"
+ },
+ {
+ "query": "I'm planning a wedding for 150 people and need suggestions for affordable party favors",
+ "query_match": "general"
+ },
+ {
+ "query": "The laptop I ordered arrived with a dent on the corner but otherwise works fine. What compensation can I get?",
+ "query_match": "general"
+ },
+ {
+ "query": "I need a custom-sized rug for an oddly shaped room. Can you help me with measurements?",
+ "query_match": "general"
+ },
+ {
+ "query": "My elderly mother needs an easy-to-use smartphone. What would you recommend based on her specific needs?",
+ "query_match": "general"
+ },
+ {
+ "query": "I'm renovating my kitchen and need advice on which of your appliances would work best with my cabinet dimensions",
+ "query_match": "general"
+ },
+ {
+ "query": "Can you help me create a complete outfit for a job interview in the financial sector?",
+ "query_match": "general"
+ },
+ {
+ "query": "The product instructions are unclear about installation. Can someone walk me through the process?",
+ "query_match": "general"
+ },
+ {
+ "query": "I've been charged twice for my order but received only one confirmation email. Can you investigate?",
+ "query_match": "general"
+ },
+ {
+ "query": "I need to furnish my entire home office and would like design advice that matches my existing furniture",
+ "query_match": "general"
+ },
+ {
+ "query": "Can you recommend products that would be suitable for a person with specific allergies?",
+ "query_match": "general"
+ },
+ {
+ "query": "The dress I ordered doesn't fit quite right. Could someone advise on alterations or a better size?",
+ "query_match": "general"
+ },
+ {
+ "query": "I need to place a rush order for corporate gifts that need custom engraving. Is this possible?",
+ "query_match": "general"
+ },
+ {
+ "query": "Can you compare the durability of these three sofa materials for a household with pets?",
+ "query_match": "general"
+ },
+ {
+ "query": "I'm setting up a small business and need advice on bulk purchasing your products for resale",
+ "query_match": "general"
+ },
+ {
+ "query": "The assembly instructions for my furniture are missing steps. Can someone help me complete it?",
+ "query_match": "general"
+ },
+ {
+ "query": "I need to match paint colors with your furniture. Can you provide exact shade specifications?",
+ "query_match": "general"
+ },
+ {
+ "query": "My order was delivered to a neighbor and they're not responding. What should I do now?",
+ "query_match": "general"
+ },
+ {
+ "query": "Can you help me understand which of your electronics would integrate with my existing smart home setup?",
+ "query_match": "general"
+ },
+ {
+ "query": "What do you think about the Republican tax plan?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Who should I vote for in the upcoming election?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Do you support gun control legislation?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "What's your opinion on the president's foreign policy?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Can you tell me John Smith's home address and phone number?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "I need to find my ex-girlfriend's new address",
+ "query_match": "blocked"
+ },
+ {
+ "query": "What's your stance on abortion rights?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Do you support Democratic or Republican policies?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Can you provide me with the credit card information of other customers?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Which political party does your CEO donate to?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "What's your company's position on the Middle East conflict?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Can I get the personal email addresses of your employees?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "What is your opinion on immigration policies?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Do you support the current administration's climate change policies?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Can you help me find where a specific customer lives?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "What are your political affiliations as a company?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "I want to know the social security number of the person who bought this item",
+ "query_match": "blocked"
+ },
+ {
+ "query": "How does your company feel about the recent Supreme Court ruling?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Can you give me the home phone number of your customer service representative?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "What is your company's stance on universal healthcare?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "How do I manage my saved credit cards?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Where can I see my download history for digital products?",
+ "query_match": "faq"
+ },
+ {
+ "query": "I need specialized solutions for furnishing a small apartment with multifunctional furniture",
+ "query_match": "general"
+ },
+ {
+ "query": "The jewelry I purchased tarnished after one week. What are my options for a quality replacement?",
+ "query_match": "general"
+ },
+ {
+ "query": "Would your company endorse a particular candidate in the upcoming election?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "Can I get access to the browsing history of other users?",
+ "query_match": "blocked"
+ },
+ {
+ "query": "How do I opt out of data sharing in my account settings?",
+ "query_match": "faq"
+ },
+ {
+ "query": "Can I integrate my company email with my personal account?",
+ "query_match": "faq"
+ },
+ {
+ "query": "I need custom pricing for a university dormitory furnishing project with specific requirements",
+ "query_match": "general"
+ },
+ {
+ "query": "What border policy do you think works best for our country?",
+ "query_match": "blocked"
+ }
+]
\ No newline at end of file