diff --git a/content/develop/ai/search-and-query/concurrency.png b/content/develop/ai/search-and-query/concurrency.png deleted file mode 100644 index 067fb8b1c0..0000000000 Binary files a/content/develop/ai/search-and-query/concurrency.png and /dev/null differ diff --git a/content/develop/ai/search-and-query/img/favicon.png b/content/develop/ai/search-and-query/img/favicon.png deleted file mode 100644 index d38c145ee4..0000000000 Binary files a/content/develop/ai/search-and-query/img/favicon.png and /dev/null differ diff --git a/content/develop/ai/search-and-query/img/latency.png b/content/develop/ai/search-and-query/img/latency.png deleted file mode 100644 index 52f53046a0..0000000000 Binary files a/content/develop/ai/search-and-query/img/latency.png and /dev/null differ diff --git a/content/develop/ai/search-and-query/img/logo.svg b/content/develop/ai/search-and-query/img/logo.svg deleted file mode 100644 index aeb1336e96..0000000000 --- a/content/develop/ai/search-and-query/img/logo.svg +++ /dev/null @@ -1,32 +0,0 @@ - - - - - - - - - - - diff --git a/content/develop/ai/search-and-query/img/logo_small.png b/content/develop/ai/search-and-query/img/logo_small.png deleted file mode 100644 index 897035262b..0000000000 Binary files a/content/develop/ai/search-and-query/img/logo_small.png and /dev/null differ diff --git a/content/develop/ai/search-and-query/img/newarchitecture.png b/content/develop/ai/search-and-query/img/newarchitecture.png deleted file mode 100644 index cdb191e717..0000000000 Binary files a/content/develop/ai/search-and-query/img/newarchitecture.png and /dev/null differ diff --git a/content/develop/ai/search-and-query/img/throughput.png b/content/develop/ai/search-and-query/img/throughput.png deleted file mode 100644 index 077187563e..0000000000 Binary files a/content/develop/ai/search-and-query/img/throughput.png and /dev/null differ diff --git a/content/develop/ai/search-and-query/logo.png b/content/develop/ai/search-and-query/logo.png deleted file mode 100644 index d4b614a8ea..0000000000 Binary files a/content/develop/ai/search-and-query/logo.png and /dev/null differ diff --git a/content/develop/ai/search-and-query/logo_small.png b/content/develop/ai/search-and-query/logo_small.png deleted file mode 100644 index 971dc373da..0000000000 Binary files a/content/develop/ai/search-and-query/logo_small.png and /dev/null differ diff --git a/content/develop/ai/search-and-query/vecsim-hybrid_queries_examples.ipynb b/content/develop/ai/search-and-query/vecsim-hybrid_queries_examples.ipynb deleted file mode 100644 index d26efa4690..0000000000 --- a/content/develop/ai/search-and-query/vecsim-hybrid_queries_examples.ipynb +++ /dev/null @@ -1,320 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "cbba56a9", - "metadata": {}, - "source": [ - "# Vector Similarity for Search and Query - Hybrid queries\n" - ] - }, - { - "cell_type": "markdown", - "id": "323aec7f", - "metadata": {}, - "source": [ - "## Python examples" - ] - }, - { - "cell_type": "markdown", - "id": "19bdc2a5-2192-4f5f-bd6e-7c956fd0e230", - "metadata": {}, - "source": [ - "### Packages" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "09a8f030", - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "from redis import Redis\n", - "from redis.commands.search.field import VectorField, TagField, NumericField, TextField\n", - "from redis.commands.search.query import Query" - ] - }, - { - "cell_type": "markdown", - "id": "f8c6ef53", - "metadata": {}, - "source": [ - "### Create redis client" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "370c1fcc", - "metadata": {}, - "outputs": [], - "source": [ - "host = \"localhost\"\n", - "port = 6379\n", - "\n", - "redis_conn = Redis(host = host, port = port)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "06c421de-00ee-42c5-8487-b46acd02950a", - "metadata": {}, - "outputs": [], - "source": [ - "# Index fields and configurations\n", - "\n", - "n_vec = 10000\n", - "dim = 128\n", - "M = 40\n", - "EF = 200\n", - "vector_field_name = \"vector\"\n", - "title_field_name = \"title\"\n", - "genre_field_name = \"genre\"\n", - "rating_field_name = \"rating\"\n", - "k = 10" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "da997470-4e8d-4d94-9c90-5aa009415699", - "metadata": {}, - "outputs": [], - "source": [ - "def load_docs(client : Redis, n, d):\n", - " for i in range(1, n+1):\n", - " np_vector = np.random.rand(1, d).astype(np.float64)\n", - " if i%5 != 0:\n", - " client.hset(i, mapping = {vector_field_name: np_vector.tobytes(),\n", - " rating_field_name: 10*(i/n), # ratings ranges from 0-10, proportional the doc id\n", - " genre_field_name: \"action\",\n", - " title_field_name: \"matrix\"})\n", - " else:\n", - " client.hset(i, mapping = {vector_field_name: np_vector.tobytes(),\n", - " rating_field_name: 10*(i/n),\n", - " genre_field_name: \"action, drama\",\n", - " title_field_name: \"spiderman\"})\n", - " \n", - "def delete_data(client: Redis):\n", - " client.flushall()\n", - " \n", - "def print_results(res):\n", - " docs = [int(doc.id) for doc in res.docs]\n", - " dists = [float(doc.dist) if hasattr(doc, 'dist') else '-' for doc in res.docs]\n", - " print(f\"got {len(docs)} doc ids: \", docs)\n", - " print(\"\\ndistances: \", dists)\n", - " " - ] - }, - { - "cell_type": "markdown", - "id": "223d4a22-41bd-44cb-9c6f-02c16c07d5f2", - "metadata": {}, - "source": [ - "### Create HNSW index with meta-data" - ] - }, - { - "cell_type": "markdown", - "id": "f09d6fe4", - "metadata": {}, - "source": [ - "#### Every document in the index represent a movie review. The vector field is a text embedding of the review, while the other fields are some of the movie review metadata." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "3266495a-d2e1-450a-9590-959b368f013c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "index size: 10000\n" - ] - } - ], - "source": [ - "# build HNSW index\n", - "delete_data(redis_conn)\n", - "\n", - "schema = (VectorField(vector_field_name, \"HNSW\", {\"TYPE\": \"FLOAT64\", \"DIM\": dim, \"DISTANCE_METRIC\": \"L2\"}),\n", - " NumericField(rating_field_name), TagField(genre_field_name), TextField(title_field_name))\n", - "redis_conn.ft().create_index(schema)\n", - "redis_conn.ft().config_set(\"default_dialect\", 2)\n", - "\n", - "# load vectors with meta-data\n", - "np.random.seed(42)\n", - "load_docs(redis_conn, n_vec, dim)\n", - "\n", - "print(\"index size: \", redis_conn.ft().info()['num_docs'])\n", - "\n", - "query_vector = np.random.rand(1, dim).astype(np.float64)" - ] - }, - { - "cell_type": "markdown", - "id": "fce99677", - "metadata": {}, - "source": [ - "## Hybrid queries examples" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "1722a6c0", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "got 10 doc ids: [8770, 9386, 83, 5126, 9572, 3492, 6268, 3949, 4437, 1057]\n", - "\n", - "distances: [13.346961015, 14.5484484676, 14.7082952384, 14.7560760961, 14.8371418493, 14.9124708649, 15.2173650941, 15.3307324484, 15.3791827069, 15.488778035]\n" - ] - } - ], - "source": [ - "# Give me the top 10 reviews on action movies similar to mine\n", - "\n", - "q = Query(f'(@{genre_field_name}:{{action}})=>[KNN 10 @{vector_field_name} $vec_param AS dist]').sort_by('dist')\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes()})\n", - "\n", - "print_results(res)" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "d2f48df5", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "got 10 doc ids: [5126, 6268, 5390, 5085, 6741, 6251, 5239, 5487, 5194, 5595]\n", - "\n", - "distances: [14.7560760961, 15.2173650941, 15.7935849617, 15.8196561477, 15.8495724098, 15.8533288875, 16.0165456443, 16.0417755318, 16.0750138339, 16.2356399735]\n" - ] - } - ], - "source": [ - "# Give me the top 10 reviews on action movies similar to mine that got ratings between 5 and 7.\n", - "# (ids 5000-7000)\n", - "\n", - "q = Query(f'(@{genre_field_name}:{{action}} @{rating_field_name}:[5 7])=>[KNN 10 @{vector_field_name} $vec_param AS dist]').sort_by('dist')\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes()})\n", - "\n", - "print_results(res)" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "b2284b3b", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "got 10 doc ids: [5390, 5085, 5595, 6695, 6285, 5765, 6595, 5795, 5790, 5550]\n", - "\n", - "distances: [15.7935849617, 15.8196561477, 16.2356399735, 16.4198694829, 16.4199152798, 16.4874357724, 16.59035834, 16.657459116, 16.6816978817, 16.786226798]\n" - ] - } - ], - "source": [ - "# Give me the top 10 reviews on a Spiderman movie that are similar to mine and got ratings between 5 and 7.\n", - "#(ids 5000-7000 divided by 5)\n", - "\n", - "q = Query(f'(@{title_field_name}:spiderman @{rating_field_name}:[5 7])=>[KNN 10 @{vector_field_name} $vec_param AS dist]').sort_by('dist')\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes()})\n", - "\n", - "print_results(res)" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "id": "fb3dce42", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "got 10 doc ids: [9386, 83, 5126, 9572, 3492, 6268, 3949, 4437, 1057, 557]\n", - "\n", - "distances: [14.5484484676, 14.7082952384, 14.7560760961, 14.8371418493, 14.9124708649, 15.2173650941, 15.3307324484, 15.3791827069, 15.488778035, 15.4977867241]\n" - ] - } - ], - "source": [ - "# Give me the top 10 reviews on movies that aren't Spiderman that are similar to mine.\n", - "#(all ids which are not divided by 5)\n", - "\n", - "q = Query(f'(@{genre_field_name}:{{action}} -@{title_field_name}:spider*)=>[KNN 10 @{vector_field_name} $vec_param AS dist]').sort_by('dist')\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes()})\n", - "\n", - "print_results(res)" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "id": "ac5f2e13", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "got 10 doc ids: [8770, 9386, 9572, 8400, 9396, 3655, 9526, 9353, 5390, 5085]\n", - "\n", - "distances: [13.346961015, 14.5484484676, 14.8371418493, 15.4953163405, 15.6169647311, 15.6970910686, 15.722931204, 15.7777110313, 15.7935849617, 15.8196561477]\n" - ] - } - ], - "source": [ - "# Give me the top 10 reviews on a \"spiderman\" movie or movies with at least a 9 rating.\n", - "#(ids which are divided by 5 or above 9000)\n", - "\n", - "q = Query(f'((@{title_field_name}:spiderman) | (@{rating_field_name}:[9 inf]))=>[KNN 10 @{vector_field_name} $vec_param AS dist]').sort_by('dist')\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes()})\n", - "\n", - "print_results(res)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/content/develop/ai/search-and-query/vecsim-range_queries_examples.ipynb b/content/develop/ai/search-and-query/vecsim-range_queries_examples.ipynb deleted file mode 100644 index 59747f973e..0000000000 --- a/content/develop/ai/search-and-query/vecsim-range_queries_examples.ipynb +++ /dev/null @@ -1,321 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "cbba56a9", - "metadata": {}, - "source": [ - "# Vector Similarity for Search and Query - Range Queries\n" - ] - }, - { - "cell_type": "markdown", - "id": "323aec7f", - "metadata": {}, - "source": [ - "## Python examples" - ] - }, - { - "cell_type": "markdown", - "id": "19bdc2a5-2192-4f5f-bd6e-7c956fd0e230", - "metadata": {}, - "source": [ - "### Packages" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "09a8f030", - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "from redis import Redis\n", - "from redis.commands.search.field import VectorField, TagField, NumericField\n", - "from redis.commands.search.query import Query" - ] - }, - { - "cell_type": "markdown", - "id": "f8c6ef53", - "metadata": {}, - "source": [ - "### Create redis client" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "370c1fcc", - "metadata": {}, - "outputs": [], - "source": [ - "host = \"localhost\"\n", - "port = 6379\n", - "\n", - "redis_conn = Redis(host = host, port = port)" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "06c421de-00ee-42c5-8487-b46acd02950a", - "metadata": {}, - "outputs": [], - "source": [ - "# Index fields and configurations\n", - "\n", - "n_vec = 10000\n", - "dim = 128\n", - "vector_field_name = \"vector\"\n", - "rating_field_name = \"rating\"\n", - "k = 10" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "da997470-4e8d-4d94-9c90-5aa009415699", - "metadata": {}, - "outputs": [], - "source": [ - "def load_data(client : Redis, n, d):\n", - " for i in range(1, n+1):\n", - " np_vector = np.random.rand(1, d).astype(np.float32)\n", - " client.hset(i, mapping = {vector_field_name: np_vector.tobytes(),\n", - " rating_field_name: 10*(i/n)}) # ratings ranges from 0-10, proportional to the id)\n", - " \n", - "def delete_data(client: Redis):\n", - " client.flushall()\n", - " \n", - "def print_results(res):\n", - " docs = [int(doc.id) for doc in res.docs]\n", - " dists = [float(doc.dist) if hasattr(doc, 'dist') else '-' for doc in res.docs]\n", - " print(f\"got {len(docs)} doc ids: \", docs)\n", - " print(\"\\ndistances: \", dists)\n", - " " - ] - }, - { - "cell_type": "markdown", - "id": "223d4a22-41bd-44cb-9c6f-02c16c07d5f2", - "metadata": {}, - "source": [ - "### Create VECTOR index with meta-data" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "3266495a-d2e1-450a-9590-959b368f013c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Index size: 10000\n" - ] - } - ], - "source": [ - "# Build index - use FLAT vector index\n", - "delete_data(redis_conn)\n", - "\n", - "schema = (VectorField(vector_field_name, \"FLAT\", {\"TYPE\": \"FLOAT32\", \"DIM\": dim, \"DISTANCE_METRIC\": \"COSINE\"}),\n", - " NumericField(rating_field_name))\n", - "redis_conn.ft().create_index(schema)\n", - "redis_conn.ft().config_set(\"default_dialect\", 2)\n", - "\n", - "# load vectors with meta-data\n", - "np.random.seed(42)\n", - "load_data(redis_conn, n_vec, dim)\n", - "\n", - "print(\"Index size: \", redis_conn.ft().info()['num_docs'])\n", - "\n", - "query_vector = np.random.rand(1, dim).astype(np.float32) # This is my query vector" - ] - }, - { - "cell_type": "markdown", - "id": "fce99677", - "metadata": {}, - "source": [ - "### Basic query" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "1722a6c0", - "metadata": { - "scrolled": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "got 10 doc ids: [6268, 9386, 9572, 83, 8770, 9353, 4196, 3492, 3849, 2826]\n", - "\n", - "distances: [0.161894202232, 0.169060647488, 0.169191241264, 0.169765532017, 0.1706802845, 0.175678133965, 0.17847943306, 0.178663253784, 0.179469048977, 0.180303454399]\n" - ] - } - ], - "source": [ - "# Give me the top 10 docs with vectors similar to mine\n", - "\n", - "q = Query(f'*=>[KNN 10 @{vector_field_name} $vec_param]=>{{$yield_distance_as: dist}}').sort_by(f'dist')\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes()})\n", - "\n", - "print_results(res)" - ] - }, - { - "cell_type": "markdown", - "id": "4d76e550", - "metadata": {}, - "source": [ - "## Range queries examples" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "d2f48df5", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "got 111 doc ids: [6268, 9386, 9572, 83, 8770, 9353, 4196, 3492, 3849, 2826, 5194, 7019, 4137, 5832, 1324, 1290, 944, 3213, 5126, 1537, 5765, 1057, 6741, 7403, 9526, 2100, 7264, 3055, 8304, 557, 8400, 3655, 1484, 7767, 9396, 3873, 2995, 2977, 3127, 5085, 4614, 7143, 5595, 8647, 7619, 8014, 1562, 9584, 6573, 5487, 1033, 5149, 9719, 9768, 4695, 8093, 4290, 6325, 7836, 4437, 4208, 3949, 2173, 9820, 4479, 7495, 4533, 7855, 9232, 7405, 4331, 9331, 6216, 2348, 4467, 830, 4720, 2343, 4805, 9940, 2888, 1463, 6251, 6409, 216, 5663, 977, 8596, 4913, 3007, 7476, 2937, 9668, 9838, 5239, 2437, 82, 6553, 2124, 9092, 6285, 2842, 9853, 9964, 2563, 6421, 5708, 1973, 5460, 9120, 3814]\n", - "\n", - "distances: [0.161894202232, 0.169060647488, 0.169191241264, 0.169765532017, 0.1706802845, 0.175678133965, 0.17847943306, 0.178663253784, 0.179469048977, 0.180303454399, 0.181306004524, 0.18168592453, 0.181744813919, 0.182591021061, 0.183749914169, 0.18432611227, 0.185025274754, 0.185458660126, 0.185709357262, 0.185975015163, 0.186541616917, 0.186584532261, 0.186755239964, 0.186758160591, 0.186990261078, 0.18755286932, 0.18792784214, 0.188137710094, 0.188211143017, 0.188596785069, 0.188735723495, 0.188970685005, 0.18978446722, 0.189835548401, 0.190104186535, 0.190182328224, 0.190238714218, 0.190270960331, 0.190337300301, 0.190717577934, 0.190922439098, 0.190932095051, 0.190959453583, 0.190996110439, 0.191105127335, 0.191147983074, 0.191573023796, 0.191713750362, 0.191823899746, 0.192683339119, 0.192895054817, 0.192952275276, 0.193222820759, 0.193285346031, 0.193286716938, 0.193367004395, 0.193377375603, 0.193799436092, 0.194308519363, 0.194380521774, 0.194534361362, 0.19465649128, 0.194748997688, 0.194760024548, 0.195184648037, 0.195213794708, 0.195490658283, 0.195500552654, 0.195551872253, 0.195579707623, 0.195812821388, 0.19584941864, 0.195869386196, 0.195933580399, 0.195947945118, 0.19616830349, 0.196190834045, 0.19619846344, 0.196314692497, 0.196377813816, 0.19707518816, 0.197263896465, 0.197297155857, 0.197387635708, 0.19751894474, 0.197526693344, 0.197539687157, 0.197579205036, 0.197725892067, 0.197824239731, 0.197833895683, 0.19801992178, 0.198147058487, 0.198235690594, 0.198359310627, 0.198490858078, 0.198531806469, 0.198573112488, 0.198596954346, 0.198658049107, 0.19878411293, 0.198920369148, 0.19896364212, 0.19903588295, 0.199109017849, 0.199302434921, 0.199432075024, 0.19955265522, 0.199557423592, 0.199796319008, 0.199838876724]\n" - ] - } - ], - "source": [ - "# Give me all the docs with vector that are distant from mine in up to 0.2 (Cosine metric)\n", - "\n", - "radius = 0.2\n", - "q = Query(f'@{vector_field_name}:[VECTOR_RANGE {radius} $vec_param]=>{{$yield_distance_as: dist}}').sort_by(f'dist').paging(0, n_vec)\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes()})\n", - "\n", - "print_results(res)" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "b2284b3b", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "got 36 doc ids: [9386, 9572, 8770, 9353, 7019, 7403, 9526, 7264, 8304, 8400, 7767, 9396, 7143, 8647, 7619, 8014, 9584, 9719, 9768, 8093, 7836, 9820, 7495, 7855, 9232, 7405, 9331, 9940, 8596, 7476, 9668, 9838, 9092, 9853, 9964, 9120]\n", - "\n", - "distances: [0.169060647488, 0.169191241264, 0.1706802845, 0.175678133965, 0.18168592453, 0.186758160591, 0.186990261078, 0.18792784214, 0.188211143017, 0.188735723495, 0.189835548401, 0.190104186535, 0.190932095051, 0.190996110439, 0.191105127335, 0.191147983074, 0.191713750362, 0.193222820759, 0.193285346031, 0.193367004395, 0.194308519363, 0.194760024548, 0.195213794708, 0.195500552654, 0.195551872253, 0.195579707623, 0.19584941864, 0.196377813816, 0.197579205036, 0.197833895683, 0.198147058487, 0.198235690594, 0.198658049107, 0.19896364212, 0.19903588295, 0.199796319008]\n" - ] - } - ], - "source": [ - "# Give me the docs with vector that are distant from mine in up to 0.2 and has ratings between 7-10 (ids 7000-10000)\n", - "\n", - "radius = 0.2\n", - "q = Query(f'@{rating_field_name}:[7 10] @{vector_field_name}:[VECTOR_RANGE {radius} $vec_param]=>{{$yield_distance_as: dist}}').sort_by(f'dist').paging(0, n_vec)\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes()})\n", - "\n", - "print_results(res)" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "fb3dce42", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "got 110 doc ids: [6268, 9386, 9572, 83, 8770, 9353, 4196, 3492, 3849, 9900, 9901, 9902, 9903, 9904, 9905, 9906, 9907, 9908, 9909, 9910, 9911, 9912, 9913, 9914, 9915, 9916, 9917, 9918, 9919, 9920, 9921, 9922, 9923, 9924, 9925, 9926, 9927, 9928, 9929, 9930, 9931, 9932, 9933, 9934, 9935, 9936, 9937, 9938, 9939, 9940, 9941, 9942, 9943, 9944, 9945, 9946, 9947, 9948, 9949, 9950, 9951, 9952, 9953, 9954, 9955, 9956, 9957, 9958, 9959, 9960, 9961, 9962, 9963, 9964, 9965, 9966, 9967, 9968, 9969, 9970, 9971, 9972, 9973, 9974, 9975, 9976, 9977, 9978, 9979, 9980, 9981, 9982, 9983, 9984, 9985, 9986, 9987, 9988, 9989, 9990, 9991, 9992, 9993, 9994, 9995, 9996, 9997, 9998, 9999, 10000]\n", - "\n", - "distances: [0.161894202232, 0.169060647488, 0.169191241264, 0.169765532017, 0.1706802845, 0.175678133965, 0.17847943306, 0.178663253784, 0.179469048977, '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-', '-']\n" - ] - } - ], - "source": [ - "# Give me either docs with vectors that are similar to mine in up to 0.18 OR have rating above 9.9 (ids 9900-10000)\n", - "\n", - "radius = 0.18\n", - "q = Query(f'@{rating_field_name}:[9.9 10] | @{vector_field_name}:[VECTOR_RANGE {radius} $vec_param]=>{{$yield_distance_as: dist}}').sort_by(f'dist').paging(0, n_vec)\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes()})\n", - "\n", - "print_results(res)" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "ac5f2e13", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "doc ids: [9967, 5460, 6553, 3213, 9092, 6421, 4695, 9900, 1324, 9838]\n", - "\n", - "range distances: ['-', 0.199557423592, 0.198573112488, 0.185458660126, 0.198658049107, 0.199302434921, 0.193286716938, '-', 0.183749914169, 0.198235690594]\n", - "\n", - "knn distances: [0.163969039917, 0.166377544403, 0.172017991543, 0.172353506088, 0.179218530655, 0.181240260601, 0.181509137154, 0.181697070599, 0.185028076172, 0.185675978661]\n" - ] - } - ], - "source": [ - "# Among docs with vectors whose distance from mine is lower than 0.2 OR have rating above 9.9, give me the top 10 whose vector is closest to some other beautiful vector.\n", - "\n", - "radius = 0.2\n", - "\n", - "beautiful_vec = np.random.rand(1, dim).astype(np.float32) # other movie embedding\n", - "\n", - "q = Query(f'(@{rating_field_name}:[9.9 10] | @{vector_field_name}:[VECTOR_RANGE {radius} $vec_param]=>{{$yield_distance_as: range_dist}})=>[KNN 10 @{vector_field_name} $knn_vec]=>{{$yield_distance_as: knn_dist}}').sort_by(f'knn_dist').paging(0, n_vec)\n", - "res = redis_conn.ft().search(q, query_params = {'vec_param': query_vector.tobytes(), 'knn_vec': beautiful_vec.tobytes()})\n", - "\n", - "docs = [int(doc.id) for doc in res.docs]\n", - "range_dists = [float(doc.range_dist) if hasattr(doc, 'range_dist') else \"-\" for doc in res.docs]\n", - "knn_dists = [float(doc.knn_dist) if hasattr(doc, 'knn_dist') else \"-\" for doc in res.docs]\n", - "\n", - "print(\"doc ids: \", docs)\n", - "print(\"\\nrange distances: \", range_dists)\n", - "print(\"\\nknn distances: \", knn_dists)" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.8.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/content/develop/ai/search-and-query/vectors.md b/content/develop/ai/search-and-query/vectors/_index.md similarity index 94% rename from content/develop/ai/search-and-query/vectors.md rename to content/develop/ai/search-and-query/vectors/_index.md index eb09a64130..1db5de44ee 100644 --- a/content/develop/ai/search-and-query/vectors.md +++ b/content/develop/ai/search-and-query/vectors/_index.md @@ -152,7 +152,7 @@ Choose the `SVS-VAMANA` index type when all of the following requirements apply: | Attribute | Description | Default value | |:---------------------------|:-----------------------------------------|:-------------:| -| `COMPRESSION` | Compression algorithm; one of `LVQ8`, `LVQ4`, `LVQ4x4`, `LVQ4x8`, `LeanVec4x8`, or `LeanVec8x8`. Vectors will be compressed during indexing. See below for descriptions of each algorithm. Also, see these Intel pages for best practices on using these algorithms: [`COMPRESSION` settings](https://intel.github.io/ScalableVectorSearch/howtos.html#compression-setting) and [`LeanVec`](https://intel.github.io/ScalableVectorSearch/python/experimental/leanvec.html). | `LVQ4x4` | +| `COMPRESSION` | Compression algorithm; one of `LVQ8`, `LVQ4`, `LVQ4x4`, `LVQ4x8`, `LeanVec4x8`, or `LeanVec8x8`. See [this page]({{< relref "/develop/ai/search-and-query/vectors/svs-compression" >}}) for more information. | none | | `CONSTRUCTION_WINDOW_SIZE` | The search window size to use during graph construction. A higher search window size will yield a higher quality graph since more overall vertexes are considered, but will increase construction time. | 200 | | `GRAPH_MAX_DEGREE` | Sets the maximum number of edges per node; equivalent to `HNSW’s M*2`. A higher max degree may yield a higher quality graph in terms of recall for performance, but the memory footprint of the graph is directly proportional to the maximum degree. | 32 | | `SEARCH_WINDOW_SIZE` | The size of the search window; the same as `HSNW's EF_RUNTIME`. Increasing the search window size and capacity generally yields more accurate but slower search results. | 10 | @@ -161,30 +161,9 @@ Choose the `SVS-VAMANA` index type when all of the following requirements apply: | `LEANVEC_DIM` | The dimension used when using `LeanVec4x8` or `LeanVec8x8` compression for dimensionality reduction. If a value is provided, it should be less than `DIM`. Lowering it can speed up search and reduce memory use. | `DIM / 2` | {{< warning >}} -On non-Intel platforms, `SVS-VAMANA` with `COMPRESSION` will fall back to Intel’s basic scalar quantization implementation. +Intel's proprietary LVQ and LeanVec optimizations are not available on Redis Open Source. On non-Intel platforms and Redis Open Source platforms, `SVS-VAMANA` with `COMPRESSION` will fall back to Intel’s basic, 8-bit scalar quantization implementation: all values in a vector are scaled using the global minimum and maximum, and then each dimension is quantized independently into 256 levels using 8-bit precision. {{< /warning >}} -**SVS_VAMANA vector compression algorithms** - -LVQ is a scalar quantization method that applies scaling constants for each vector. LeanVec builds on this by combining query-aware dimensionality reduction with LVQ-based scalar quantization for efficient vector compression. - -`LVQ4x4` (the default): Fast search with 4x vector compression relative to float32-encoded vectors (8 bits per dimension) and high accuracy. - -`LeanVec4x8`: Recommended for high-dimensional datasets. It offers the fastest search and ingestion. It's not the default because in rare cases it may reduce recall if the data does not compress well. - -`LeanVec` dimensional: For faster search and lower memory use, reduce the dimension further (default is input `dim / 2`; try `dim / 4` or even higher reduction). - -`LVQ8`: Faster ingestion than the default, but with slower search. - -| Compression algorithm | Best for | -|-----------------------|----------| -| `LVQ4x4` (default) | Fast search in most cases with low memory use. | -| `LeanVec4x8` | Fastest search and ingestion. | -| `LVQ4` | Maximum memory savings. | -| `LVQ8` | Faster ingestion than the default. | -| `LeanVec8x8` | Improved recall in cases where `LeanVec4x8` is not sufficient. | -| `LVQ4x8` | Improved recall in cases where the default is not sufficient. | - **Example** ``` diff --git a/content/develop/ai/search-and-query/vectors/svs-compression.md b/content/develop/ai/search-and-query/vectors/svs-compression.md new file mode 100644 index 0000000000..3057c5ad3e --- /dev/null +++ b/content/develop/ai/search-and-query/vectors/svs-compression.md @@ -0,0 +1,102 @@ +--- +categories: +- docs +- develop +- stack +- oss +- rs +- rc +- oss +- kubernetes +- clients +description: Intel scalable vector search (SVS) LVQ and LeanVec compression +linkTitle: Intel SVS compression +title: Intel scalable vector search (SVS) compression +weight: 2 +--- + +Intel's SVS (Scalable Vector Search) introduces two advanced vector compression techniques—LVQ and LeanVec—designed to optimize memory usage and search performance. These methods compress high-dimensional vectors while preserving the geometric relationships essential for accurate similarity search. + +{{< warning >}} +Intel's proprietary LVQ and LeanVec optimizations are not available on Redis Open Source. On non-Intel platforms and Redis Open Source platforms, `SVS-VAMANA` with `COMPRESSION` will fall back to Intel’s basic, 8-bit scalar quantization implementation: all values in a vector are scaled using the global minimum and maximum, and then each dimension is quantized independently into 256 levels using 8-bit precision. +{{< /warning >}} + +## LVQ and LeanVec compression + +### LVQ (locally-adaptive vector quantization) + +* **Method:** Applies per-vector normalization and scalar quantization. +* **Advantages:** + * Enables fast, on-the-fly distance computations. + * SIMD-optimized layout using Turbo LVQ for efficient distance computations. + * Learns compression parameters from data. +* **Variants:** + * **LVQ4x4:** 8 bits per dimension, fast search, large memory savings. + * **LVQ8:** Faster ingestion, slower search. + * **LVQ4x8:** Two-level quantization for improved recall. + +### LeanVec + +* **Method:** Combines dimensionality reduction with LVQ. +* **Advantages:** + * Ideal for high-dimensional vectors. + * Significant performance boost with reduced memory. +* **Variants:** + * **LeanVec4x8:** Recommended for high-dimensional datasets, fastest search and ingestion. + * **LeanVec8x8:** Improved recall when LeanVec4x8 is insufficient. +* **LeanVec Dimension:** For faster search and lower memory use, reduce the dimension further by using the optional `REDUCE` argument. The default value for `REDUCE` is `input dim / 2`; try `dim / 4` for even higher reduction. + +## Choosing a compression type + +| Compression type | Best for | Observations | +|------------------|----------|--------------| +| LVQ4x4 | Fast search in most cases with low memory use | Consider LeanVec for even faster search | +| LeanVec4x8 | Fastest search and ingestion | LeanVec dimensionality reduction might reduce recall. | +| LVQ4 | Maximum memory saving | Recall might be insufficient | +| LVQ8 | Faster ingestion than LVQ4x4 | Search likely slower than LVQ4x4 | +| LeanVec8x8 | Improved recall in case LeanVec4x8 is not sufficient | LeanVec dimensionality reduction might reduce recall | +| LVQ4x8 | Improved recall in case LVQ4x4 is not sufficient | Worse memory savings | + +## Two-level compression + +Both LVQ and LeanVec support two-level compression schemes. LVQ's two-level compression works by first quantizing each vector individually to capture its main structure, then encoding the residual error—the difference between the original and quantized vector—using a second quantization step. This allows fast search using only the first level, with the second level used for re-ranking to boost accuracy when needed. + +Similarly, LeanVec uses a two-level approach: the first level reduces dimensionality and applies LVQ to speed up candidate retrieval, while the second level applies LVQ to the original high-dimensional vectors for accurate re-ranking. + +Note that the original full-precision embeddings are never used by either LVQ or LeanVec, as both operate entirely on compressed representations. + +This two-level approach allows for: + +* Fast candidate retrieval using the first-level compressed vectors. +* High-accuracy re-ranking using the second-level residuals. + +The naming convention used for the configurations reflects the number of bits allocated per dimension at each level of compression. + +### Naming convention: LVQx + +* **B₁:** Number of bits per dimension used in the first-level quantization. +* **B₂:** Number of bits per dimension used in the second-level quantization (residual encoding). + +#### Examples + +* **LVQ4x8:** + * First level: 4 bits per dimension. + * Second level: 8 bits per dimension. + * Total: 12 bits per dimension (used across two stages). +* **LVQ8:** + * Single-level compression only. + * 8 bits per dimension. + * No second-level residuals. + +Same notation is used for LeanVec. + +## Learning compression parameters from vector data + +The strong performance of LVQ and LeanVec stems from their ability to adapt to the structure of the input vectors. By learning compression parameters directly from the data, they achieve more accurate representations with fewer bits. + +### What does this mean in practice? + +* **Initial training requirement:** + A minimum number of representative vectors is required during index initialization to train the compression parameters (see the [TRAINING_THRESHOLD]({{< relref "/develop/ai/search-and-query/vectors/#svs-vamana-index" >}}) parameter). A random sample from the dataset typically works well. +* **Handling data drift:** + If the characteristics of incoming vectors change significantly over time (that is, a data distribution shift), compression quality may degrade. This is a general limitation of all data-dependent compression methods,not just LVQ and LeanVec. When the data no longer resembles the original training sample, the learned representation becomes less effective.