diff --git a/java-recipes/README.md b/java-recipes/README.md index 4909d9f..c8ba21f 100644 --- a/java-recipes/README.md +++ b/java-recipes/README.md @@ -32,9 +32,10 @@ Notebooks require a Jupyter Notebook environment to run. Check out the [Setup In ## Applications -| Application | Description | -|-----------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------| -| [applications/vector-similarity-search/spring_boot](./applications/vector-similarity-search/spring_boot_redis_om_spring.md) | Demonstrates building a vector similarity search application using Spring Boot and Redis OM Spring | +| Application | Description | +|-------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------| +| [applications/vector-similarity-search/redis-om-spring](./applications/vector-similarity-search/redis-om-spring/spring_boot_redis_om_spring.md) | Demonstrates building a vector similarity search application using Spring Boot and Redis OM Spring | +| [applications/vector-similarity-search/spring-ai](./applications/vector-similarity-search/spring-ai/spring_boot_spring_ai.md) | Demonstrates building a vector similarity search application using Spring Boot and Spring AI | ## Example Notebooks & Applications diff --git a/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/autocomplete.png b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/autocomplete.png new file mode 100644 index 0000000..37b5858 Binary files /dev/null and b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/autocomplete.png differ diff --git a/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/index-redis-insight.png b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/index-redis-insight.png new file mode 100644 index 0000000..42089ac Binary files /dev/null and b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/index-redis-insight.png differ diff --git a/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/pre-filtered-vector-search.png b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/pre-filtered-vector-search.png new file mode 100644 index 0000000..4db1b0a Binary files /dev/null and b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/pre-filtered-vector-search.png differ diff --git a/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/redis-insight.png b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/redis-insight.png new file mode 100644 index 0000000..313e4e4 Binary files /dev/null and b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/redis-insight.png differ diff --git a/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/vector-search.png b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/vector-search.png new file mode 100644 index 0000000..bb33e9c Binary files /dev/null and b/java-recipes/applications/vector-similarity-search/redis-om-spring/readme-assets/vector-search.png differ diff --git a/java-recipes/applications/vector-similarity-search/redis-om-spring/spring_boot_redis_om_spring.md b/java-recipes/applications/vector-similarity-search/redis-om-spring/spring_boot_redis_om_spring.md new file mode 100644 index 0000000..fa5e873 --- /dev/null +++ b/java-recipes/applications/vector-similarity-search/redis-om-spring/spring_boot_redis_om_spring.md @@ -0,0 +1,227 @@ +# Vector Search with Redis OM Spring (SpringBoot) + +Vector similarity search (also known as semantic search) is a powerful technique that allows you to find items based on their semantic meaning rather than exact keyword matches. Redis Query Engine supports vector similarity search through its vector indexing capabilities, enabling you to implement semantic search applications with high performance and low latency. + +This demo showcases how to implement vector similarity search using Redis OM Spring, a library that simplifies working with Redis data models and the Redis Query Engine. + +## Learning resources: + +- Article: [Semantic Search with Spring Boot & Redis](https://raphaeldelio.com/2025/04/29/semantic-search-with-spring-boot-redis/) +- Video: [Autocomplete in Spring with Redis](https://www.youtube.com/watch?v=rjaR1PR5gVk) +- Video: [What is an embedding model?](https://youtu.be/0U1S0WSsPuE) +- Video: [Exact vs Approximate Nearest Neighbors - What's the difference?](https://youtu.be/9NvO-VdjY80) +- Video: [What is semantic search?](https://youtu.be/o3XN4dImESE) +- Video: [What is a vector database?](https://youtu.be/Yhv19le0sBw) + + +## Repository + +The repository for this demo can be found [here](https://github.com/redis-developer/redis-springboot-resources/tree/main/search/vector-search) + +## Requirements + +To run this demo, you’ll need the following installed on your system: +- Docker – [Install Docker](https://docs.docker.com/get-docker/) +- Docker Compose – Included with Docker Desktop or available via CLI installation guide + +## Running the demo + +The easiest way to run the demo is with Docker Compose, which sets up all required services in one command. + +### Step 1: Clone the repository + +If you haven’t already: + +```bash +git clone https://github.com/redis-developer/redis-springboot-recipes.git +cd redis-springboot-recipes/search/full-text-search-and-autocomplete +``` + +### Step 2: Start the services + +```bash +docker compose up --build +``` + +This will start: + +- redis: for storing documents +- redis-insight: a UI to explore the Redis data +- vector-search-app: the Spring Boot app that implements vector search + +## Using the demo + +When all of your services are up and running. Go to `localhost:8080` to access the demo. + +If you search using the extract box, the system will perform semantic search and find items on the database that are semantically similar to your query: + +![Screenshot of a movie search app using vector similarity search. The user searches for “movie about a clownfish who searches for his son.” The top result is Finding Nemo, with a similarity score of 0.505, followed by Big Fish and Swordfish. Each result includes a poster, title, year, cast, genres, and description snippet.](readme-assets/vector-search.png) + +You can also apply filters for pre-filtering the results before applying semantic search: + +![Screenshot of a movie search app using vector similarity search with filters applied: cast = Albert Brooks, genre = animated. The query is “movie about a clownfish who searches for his son.” Results include Finding Nemo, Finding Nemo 3D, and Finding Dory, each with similarity scores, posters, cast, genres, and descriptions.](readme-assets/pre-filtered-vector-search.png) + +This demo also supports autocompletion of the title: + +![Close-up screenshot of a movie search app’s autocomplete feature. The user types “Finding” in the “Movie Title” field, triggering a dropdown with suggestions like Finding You, Finding Nemo, Finding Dory, Finding Bliss, and Finding Amanda. Autocomplete response time is shown as 8 ms.](readme-assets/autocomplete.png) + +### Redis Insight + +RedisInsight is a graphical tool developed by Redis to help developers and administrators interact with and manage Redis databases more efficiently. It provides a visual interface for exploring keys, running commands, analyzing memory usage, and monitoring performance metrics in real-time. RedisInsight supports features like full-text search, time series, streams, and vector data structures, making it especially useful for working with more advanced Redis use cases. With its intuitive UI, it simplifies debugging, optimizing queries, and understanding data patterns without requiring deep familiarity with the Redis CLI. + +The Docker Compose file will also spin up an instance of Redis Insight. We can access it by going to `localhost:5540`: + +If we go to Redis Insight, we will be able to see the data stored in Redis: + +![Screenshot of RedisInsight showing 10,000 JSON movie documents in the com.redis.vectorsearch.domain.Movie namespace. The selected document is for Star Trek III: The Search for Spock, displaying fields like title, year, genres, extract, and a thumbnail URL. The embeddedExtract vector field is also included.](readme-assets/redis-insight.png) + +And if run the command `FT.INFO 'com.redis.fulltextsearchandautocomplete.domain.MovieIdx'`, we'll be able to see the schema that was created for indexing our documents efficiently: + +![Screenshot of RedisInsight displaying the schema of the MovieIdx vector search index. The index is built on JSON documents and includes fields like title, year, cast, genres, embeddedExtract (VECTOR), and id. The vector field uses the HNSW algorithm with FLOAT32 data type, 384 dimensions, COSINE distance metric, M=16, and EF_CONSTRUCTION=200.](readme-assets/index-redis-insight.png) + +## How It Is Implemented + +The application uses Redis OM Spring to vectorize documents and perform vector similarity search. Here's how it works: + +### Defining Vector Fields with Redis OM Spring Annotations + +Documents are defined as Java classes with Redis OM Spring annotations that specify how they should be vectorized and indexed: + +```java +@Document +public class Movie { + // Other fields... + + @Vectorize( + destination = "embeddedExtract", + embeddingType = EmbeddingType.SENTENCE + ) + private String extract; + + @Indexed( + schemaFieldType = SchemaFieldType.VECTOR, + algorithm = VectorField.VectorAlgorithm.HNSW, + type = VectorType.FLOAT32, + dimension = 384, + distanceMetric = DistanceMetric.COSINE, + initialCapacity = 10 + ) + private float[] embeddedExtract; + + // Getters and setters... +} +``` + +Let's break down the annotations: + +- `@Vectorize`: Automatically generates vector embeddings for the text field + - `destination`: Specifies the field where the embedding will be stored + - `embeddingType`: Defines the granularity of the embedding (SENTENCE in this case) + +- `@Indexed` with vector parameters: + - `schemaFieldType = SchemaFieldType.VECTOR`: Marks this as a vector field + - `algorithm = VectorField.VectorAlgorithm.HNSW`: Uses the Hierarchical Navigable Small World algorithm for efficient approximate nearest neighbor search + - `type = VectorType.FLOAT32`: Specifies the vector data type + - `dimension = 384`: Sets the vector dimension (must match the number of dimensions output by the embedding model) + - `distanceMetric = DistanceMetric.COSINE`: Uses cosine similarity for distance calculation + +### Storing and Vectorizing Documents + +When documents are saved to Redis using the repository, Redis OM Spring automatically generates vector embeddings: + +```java +public void loadAndSaveMovies(String filePath) throws Exception { + // Load movies from JSON file + List movies = objectMapper.readValue(is, new TypeReference<>() {}); + + // Save movies in batches + int batchSize = 500; + for (int i = 0; i < unprocessedMovies.size(); i += batchSize) { + int end = Math.min(i + batchSize, unprocessedMovies.size()); + List batch = unprocessedMovies.subList(i, end); + movieRepository.saveAll(batch); + } +} +``` + +When `movieRepository.saveAll(batch)` is called: +1. Redis OM Spring generates vector embeddings for the `extract` field +2. The embeddings are stored in the `embeddedExtract` field +3. The documents are saved to Redis with their vector embeddings +4. Redis creates a vector index for efficient similarity search + +### Performing Vector Similarity Search + +Vector similarity search is implemented using Redis OM Spring's EntityStream API: + +```java +public Map search( + String title, + String extract, + List actors, + Integer year, + List genres, + Integer numberOfNearestNeighbors +) { + SearchStream stream = entityStream.of(Movie.class); + + if (extract != null) { + // Convert search query to vector embedding + float[] embeddedQuery = embedder.getTextEmbeddingsAsFloats(List.of(extract), Movie$.EXTRACT).getFirst(); + + // Perform KNN search with the embedded query + stream = stream.filter(Movie$.EMBEDDED_EXTRACT.knn(numberOfNearestNeighbors, embeddedQuery)) + .sorted(Movie$._EMBEDDED_EXTRACT_SCORE); + } + + // Apply additional filters + List> matchedMovies = stream + .filter(Movie$.TITLE.containing(title)) + .filter(Movie$.CAST.eq(actors)) + .filter(Movie$.YEAR.eq(year)) + .filter(Movie$.GENRES.eq(genres)) + .map(Fields.of(Movie$._THIS, Movie$._EMBEDDED_EXTRACT_SCORE)) + .collect(Collectors.toList()); + + return result; +} +``` + +This method: +1. Converts the search query text into a vector embedding using the same embedding model +2. Performs a K-Nearest Neighbors (KNN) search to find the most similar vectors +3. Applies additional filters to narrow down the results (pre-filtering) +4. Returns the matched movies along with their similarity scores + +### Combining Vector Search with Autocomplete + +The application also supports autocomplete functionality alongside vector search: + +```java +public interface MovieRepository extends RedisDocumentRepository { + List autoCompleteTitle(String title, AutoCompleteOptions options); +} +``` + +The `autoCompleteTitle` method is automatically implemented by Redis OM Spring based on the `@AutoComplete` annotation on the `title` field in the Movie class. + +### How Redis Indexes the Vectors + +When the application starts, Redis OM Spring creates a vector index in Redis based on the annotations: + +``` +FT.CREATE idx:com.redis.vectorsearch.domain.Movie ON JSON PREFIX 1 com.redis.vectorsearch.domain.Movie: SCHEMA + $.title AS title TEXT SORTABLE + $.year AS year NUMERIC SORTABLE + $.cast AS cast TAG + $.genres AS genres TAG + $.embeddedExtract AS embeddedExtract VECTOR HNSW 6 TYPE FLOAT32 DIM 384 DISTANCE_METRIC COSINE INITIAL_CAP 10 +``` + +This index enables efficient vector similarity search with the following features: +- HNSW algorithm for approximate nearest neighbor search +- 384-dimensional FLOAT32 vectors +- Cosine similarity as the distance metric +- Additional text and tag fields for filtering + +This approach allows for high-performance semantic search operations, even with large datasets, by leveraging Redis's in-memory data structures and the Redis Query Engine's vector search capabilities. diff --git a/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/index-redis-insight.png b/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/index-redis-insight.png new file mode 100644 index 0000000..42089ac Binary files /dev/null and b/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/index-redis-insight.png differ diff --git a/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/pre-filtered-vector-search.png b/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/pre-filtered-vector-search.png new file mode 100644 index 0000000..4db1b0a Binary files /dev/null and b/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/pre-filtered-vector-search.png differ diff --git a/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/redis-insight.png b/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/redis-insight.png new file mode 100644 index 0000000..313e4e4 Binary files /dev/null and b/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/redis-insight.png differ diff --git a/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/vector-search.png b/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/vector-search.png new file mode 100644 index 0000000..bb33e9c Binary files /dev/null and b/java-recipes/applications/vector-similarity-search/spring-ai/readme-assets/vector-search.png differ diff --git a/java-recipes/applications/vector-similarity-search/spring-ai/spring_boot_spring_ai.md b/java-recipes/applications/vector-similarity-search/spring-ai/spring_boot_spring_ai.md new file mode 100644 index 0000000..75ad153 --- /dev/null +++ b/java-recipes/applications/vector-similarity-search/spring-ai/spring_boot_spring_ai.md @@ -0,0 +1,231 @@ +# Vector Search with Spring AI (SpringBoot) + +Vector similarity search (semantic search) allows you to find items based on their semantic meaning rather than exact keyword matches. Spring AI provides a standardized way to work with AI models and vector embeddings across different providers. This demo showcases how to integrate Redis Vector Search with Spring AI to implement semantic search applications. + +## Learning resources: + +- Article: [Semantic Search with Spring Boot & Redis](https://raphaeldelio.com/2025/04/29/semantic-search-with-spring-boot-redis/) +- Video: [What is an embedding model?](https://youtu.be/0U1S0WSsPuE) +- Video: [What is semantic search?](https://youtu.be/o3XN4dImESE) +- Video: [What is a vector database?](https://youtu.be/Yhv19le0sBw) + +## Repository + +The repository for this demo can be found [here](https://github.com/redis-developer/redis-springboot-resources/tree/main/search/vector-search-spring-ai) + +## Requirements + +To run this demo, you’ll need the following installed on your system: +- Docker – [Install Docker](https://docs.docker.com/get-docker/) +- Docker Compose – Included with Docker Desktop or available via CLI installation guide + +## Running the demo + +The easiest way to run the demo is with Docker Compose, which sets up all required services in one command. + +### Step 1: Clone the repository + +If you haven’t already: + +```bash +git clone https://github.com/redis-developer/redis-springboot-recipes.git +cd redis-springboot-recipes/search/full-text-search-and-autocomplete +``` + +### Step 2: Start the services + +```bash +docker compose up --build +``` + +This will start: + +- redis: for storing documents +- redis-insight: a UI to explore the Redis data +- vector-search-spring-ai-app: the Spring Boot app that implements vector search + +## Using the demo + +When all of your services are up and running. Go to `localhost:8080` to access the demo. + +If you search using the extract box, the system will perform semantic search and find items on the database that are semantically similar to your query: + +![Screenshot of a movie search app using vector similarity search. The user searches for “movie about a clownfish who searches for his son.” The top result is Finding Nemo, with a similarity score of 0.505, followed by Big Fish and Swordfish. Each result includes a poster, title, year, cast, genres, and description snippet.](readme-assets/vector-search.png) + +You can also apply filters for pre-filtering the results before applying semantic search: + +![Screenshot of a movie search app using vector similarity search with filters applied: cast = Albert Brooks, genre = animated. The query is “movie about a clownfish who searches for his son.” Results include Finding Nemo, Finding Nemo 3D, and Finding Dory, each with similarity scores, posters, cast, genres, and descriptions.](readme-assets/pre-filtered-vector-search.png) + +### Redis Insight + +RedisInsight is a graphical tool developed by Redis to help developers and administrators interact with and manage Redis databases more efficiently. It provides a visual interface for exploring keys, running commands, analyzing memory usage, and monitoring performance metrics in real-time. RedisInsight supports features like full-text search, time series, streams, and vector data structures, making it especially useful for working with more advanced Redis use cases. With its intuitive UI, it simplifies debugging, optimizing queries, and understanding data patterns without requiring deep familiarity with the Redis CLI. + +The Docker Compose file will also spin up an instance of Redis Insight. We can access it by going to `localhost:5540`: + +If we go to Redis Insight, we will be able to see the data stored in Redis: + +![Screenshot of RedisInsight showing 10,000 JSON movie documents in the com.redis.vectorsearch.domain.Movie namespace. The selected document is for Star Trek III: The Search for Spock, displaying fields like title, year, genres, extract, and a thumbnail URL. The embeddedExtract vector field is also included.](readme-assets/redis-insight.png) + +And if run the command `FT.INFO 'com.redis.fulltextsearchandautocomplete.domain.MovieIdx'`, we'll be able to see the schema that was created for indexing our documents efficiently: + +![Screenshot of RedisInsight displaying the schema of the MovieIdx vector search index. The index is built on JSON documents and includes fields like title, year, cast, genres, embeddedExtract (VECTOR), and id. The vector field uses the HNSW algorithm with FLOAT32 data type, 384 dimensions, COSINE distance metric, M=16, and EF_CONSTRUCTION=200.](readme-assets/index-redis-insight.png) + +## How It Is Implemented + +The application uses Spring AI's `RedisVectorStore` to store and search vector embeddings of movie descriptions. + +### Configuring the Vector Store + +```kotlin +@Bean +fun movieVectorStore( + embeddingModel: EmbeddingModel, + jedisPooled: JedisPooled +): RedisVectorStore { + return RedisVectorStore.builder(jedisPooled, embeddingModel) + .indexName("movieIdx") + .contentFieldName("extract") + .embeddingFieldName("extractEmbedding") + .metadataFields( + RedisVectorStore.MetadataField("title", Schema.FieldType.TEXT), + RedisVectorStore.MetadataField("year", Schema.FieldType.NUMERIC), + RedisVectorStore.MetadataField("cast", Schema.FieldType.TAG), + RedisVectorStore.MetadataField("genres", Schema.FieldType.TAG), + RedisVectorStore.MetadataField("thumbnail", Schema.FieldType.TEXT), + ) + .prefix("movies:") + .initializeSchema(true) + .vectorAlgorithm(RedisVectorStore.Algorithm.HSNW) + .build() +} +``` + +Let's break this down: + +- **Index Name**: `movieIdx` - Redis will create an index with this name for searching movies +- **Content Field**: `extract` - The movie description that will be embedded +- **Embedding Field**: `extractEmbedding` - The field that will store the resulting vector embedding +- **Metadata Fields**: Additional fields for filtering and retrieval (title, year, cast, genres, thumbnail) +- **Prefix**: `movies:` - All keys in Redis will be prefixed with this to organize the data +- **Vector Algorithm**: `HSNW` - Hierarchical Navigable Small World algorithm for efficient approximate nearest neighbor search + +### Configuring the Embedding Model + +Spring AI provides a standardized way to work with different embedding models. In this application, we use the Transformers embedding model: + +```kotlin +@Bean +fun embeddingModel(): EmbeddingModel { + return TransformersEmbeddingModel() +} +``` + +The `TransformersEmbeddingModel` is a local embedding model based on the Hugging Face Transformers library, which allows us to generate vector embeddings without relying on external API calls. + +### Storing and Vectorizing Documents + +When the application starts, it loads movie data from a JSON file and stores it in Redis with vector embeddings: + +```kotlin +fun storeMovies(movies: List) { + val documents = movies.map { movie -> + val text = movie.extract ?: "" + val metadata = mapOf( + "title" to (movie.title ?: ""), + "year" to movie.year, + "cast" to movie.cast, + "genres" to movie.genres, + "thumbnail" to (movie.thumbnail ?: "") + ) + Document(text, metadata) + } + movieVectorStore.add(documents) +} +``` + +This process: +1. Converts each Movie object to a Spring AI Document +2. Sets the movie extract as the document content +3. Adds metadata fields for filtering and retrieval +4. Adds the documents to the RedisVectorStore, which automatically: + - Generates vector embeddings for the content + - Stores the documents in Redis with their embeddings + - Updates the vector index for efficient search + +### Performing Vector Similarity Search + +When a user enters a search query, the application performs vector similarity search to find semantically similar movies: + +```kotlin +fun searchMovies( + title: String, + extract: String, + actors: List, + year: Int? = null, + genres: List, + numberOfNearestNeighbors: Int +): Map { + val b = FilterExpressionBuilder() + val filterList = mutableListOf() + + // Add filters for title, actors, year, and genres + if (title.isNotBlank()) { + filterList.add(b.`in`("title", title)) + } + + // ... other filters ... + + val filterExpression = when (filterList.size) { + 0 -> null + 1 -> filterList[0] + else -> filterList.reduce { acc, expr -> b.and(acc, expr) } + }?.build() + + val searchResults = movieVectorStore.similaritySearch( + SearchRequest.builder() + .query(extract) + .topK(numberOfNearestNeighbors) + .filterExpression(filterExpression) + .build() + ) ?: emptyList() + + // Transform results to Movie objects + // ... +} +``` + +This search process: +1. Builds filter expressions for pre-filtering based on metadata (title, actors, year, genres) +2. Creates a search request with: + - The extract text as the query (which will be embedded into a vector) + - A topK parameter to limit the number of results + - Optional filter expressions for pre-filtering +3. Performs vector similarity search using the RedisVectorStore +4. Transforms the search results back into Movie objects with similarity scores + +### Pre-filtering with Vector Search + +One powerful feature of Redis vector search is the ability to pre-filter results before performing vector similarity search. This allows for more efficient and targeted searches: + +```kotlin +val filterExpression = when (filterList.size) { + 0 -> null + 1 -> filterList[0] + else -> filterList.reduce { acc, expr -> b.and(acc, expr) } +}?.build() + +val searchResults = movieVectorStore.similaritySearch( + SearchRequest.builder() + .query(extract) + .topK(numberOfNearestNeighbors) + .filterExpression(filterExpression) + .build() +) +``` + +Pre-filtering works by: +1. First applying traditional filters on metadata fields (e.g., year, cast, genres) +2. Then performing vector similarity search only on the filtered subset +3. Returning the top K most similar results from the filtered set + +This approach combines the precision of traditional filtering with the semantic understanding of vector search, allowing users to find movies that are both semantically similar to their query and match specific criteria. diff --git a/java-recipes/applications/vector-similarity-search/spring_boot_redis_om_spring.md b/java-recipes/applications/vector-similarity-search/spring_boot_redis_om_spring.md deleted file mode 100644 index db8d6d4..0000000 --- a/java-recipes/applications/vector-similarity-search/spring_boot_redis_om_spring.md +++ /dev/null @@ -1,358 +0,0 @@ -# Vector Similarity Search with Redis OM Spring (Spring Boot) - -![Redis](https://redis.io/wp-content/uploads/2024/04/Logotype.svg?auto=webp&quality=85,75&width=120) - -This guide demonstrates how to build a Vector Similarity Search (VSS) system using Spring Boot and Redis OM Spring. The example allows movies to be searched by their synopses based on semantic similarity rather than keyword matching. - -## Prerequisites - -- Java 21 -- Maven for dependency management -- Docker and Docker Compose (for running Redis) -- OpenAI API key for text embeddings - -## Repository - -The repository for this demo can be found [here](https://github.com/redis/redis-om-spring/tree/main/demos/roms-vss-movies) - -## Project Structure Overview - -The project implements a system that demonstrates vector similarity search using Redis 8's built-in capabilities together with Redis OM Spring: - -The repository can be found at: - -``` -roms-vss-movies/ -├── src/main/java/dev/raphaeldelio/redis8demovectorsimilaritysearch/ -│ ├── controller/ -│ │ └── SearchController.java # REST endpoints for search -│ ├── domain/ -│ │ └── Movie.java # Entity with vector annotations -│ ├── repository/ -│ │ └── MovieRepository.java # Redis repository interface -│ ├── service/ -│ │ ├── MovieService.java # Service for data loading -│ │ └── SearchService.java # Service for vector search -│ └── RomsVectorSimilaritySearchMovies.java # Main application -└── src/main/resources/ - ├── application.properties # Application configuration - └── movies.json # Sample dataset -``` - -## Getting Started - -1. **Start the Redis instance**: - ```bash - docker-compose up -d redis-vector-search - ``` - -2. **Build and run the application**: - ```bash - mvn spring-boot:run - ``` - -## Dependencies - -This application uses the following key dependencies: - -```xml - - - com.redis.om.spring - redis-om-spring - 0.9.11 - - - - - org.springframework.ai - spring-ai-openai - 1.0.0-M6 - - - org.springframework.ai - spring-ai-transformers - 1.0.0-M6 - -``` - -## Implementation Details - -### 1. Define the `Movie` entity - -Redis OM Spring provides two annotations that makes it easy to vectorize data and perform vector similarity search from within Spring Boot. -- `@Vectorize`: Automatically generates vector embeddings from the text field -- `@Indexed`: Enables vector indexing on the field for efficient search - -The core of the implementation is the `Movie` class with Redis vector indexing annotations: - -```java -@RedisHash // This annotation is used by Redis OM Spring to store the entity as a hash in Redis -public class Movie { - - @Id // IDs are automatically generated by Redis OM Spring as ULID - private String title; - - @Indexed(sortable = true) // This annotation enables indexing on the field for filtering and sorting - private int year; - - @Indexed - private List cast; - - @Indexed - private List genres; - - private String href; - - // This annotation automatically generates vector embeddings from the text - @Vectorize( - destination = "embeddedExtract", // The field where the embedding will be stored - embeddingType = EmbeddingType.SENTENCE, // Type of embedding to generate (Sentence, Image, face, or word) - provider = EmbeddingProvider.OPENAI, // The provider for generating embeddings (OpenAI, Transformers, VertexAI, etc.) - openAiEmbeddingModel = OpenAiApi.EmbeddingModel.TEXT_EMBEDDING_3_LARGE // The specific OpenAI model to use for embeddings - ) - private String extract; - - // This defines the vector field that will store the embeddings - // The indexed annotation enables vector search on this field - @Indexed( - schemaFieldType = SchemaFieldType.VECTOR, // Defines the field type as a vector - algorithm = VectorField.VectorAlgorithm.FLAT, // The algorithm used for vector search (FLAT or HNSW) - type = VectorType.FLOAT32, - dimension = 3072, // The dimension of the vector (must match the embedding model) - distanceMetric = DistanceMetric.COSINE, // The distance metric used for similarity search (Cosine or Euclidean) - initialCapacity = 10 - ) - private byte[] embeddedExtract; - - private String thumbnail; - private int thumbnailWidth; - private int thumbnailHeight; - - // Getters and setters... -} -``` - -### 2. Repository Interface - -A simple repository interface extends `RedisEnhancedRepository`. This will be used to load the data into Redis using the saveAll() method: - -```java -public interface MovieRepository extends RedisEnhancedRepository { -} -``` - -This provides basic CRUD operations for `Movie` entities, with the first generic parameter being the entity type and the second being the ID type. - -### 3. Search Service - -The search service uses two beans provided by Redis OM Spring: -- `EntityStream`: For creating a stream of entities to perform searches. The Entity Stream must not be confused with the Java Streams API. The Entity Stream will generate a Redis Command that will be sent to Redis so that Redis can perform the searching, filtering and sorting efficiently on its side. -- `Embedder`: Used for generating the embedding for the query sent by the user. It will be generated following the configuration of the `@Vectorize` annotation defined in the `Movie` class/ - -The search functionality is implemented in the `SearchService`: - -```java -@Service -public class SearchService { - - private static final Logger logger = LoggerFactory.getLogger(SearchService.class); - private final EntityStream entityStream; - private final Embedder embedder; - - public SearchService(EntityStream entityStream, Embedder embedder) { - this.entityStream = entityStream; - this.embedder = embedder; - } - - public List> search( - String query, - Integer yearMin, - Integer yearMax, - List cast, - List genres, - Integer numberOfNearestNeighbors) { - logger.info("Received text: {}", query); - logger.info("Received yearMin: {} yearMax: {}", yearMin, yearMax); - logger.info("Received cast: {}", cast); - logger.info("Received genres: {}", genres); - - if (numberOfNearestNeighbors == null) numberOfNearestNeighbors = 3; - if (yearMin == null) yearMin = 1900; - if (yearMax == null) yearMax = 2100; - - // Convert query text to vector embedding - byte[] embeddedQuery = embedder.getTextEmbeddingsAsBytes(List.of(query), Movie$.EXTRACT).getFirst(); - - // Perform vector search with additional filters - SearchStream stream = entityStream.of(Movie.class); - return stream - // KNN search for nearest vectors - .filter(Movie$.EMBEDDED_EXTRACT.knn(numberOfNearestNeighbors, embeddedQuery)) - // Additional metadata filters (hybrid search) - .filter(Movie$.YEAR.between(yearMin, yearMax)) - .filter(Movie$.CAST.eq(cast)) - .filter(Movie$.GENRES.eq(genres)) - // Sort by similarity score - .sorted(Movie$._EMBEDDED_EXTRACT_SCORE) - // Return both the movie and its similarity score - .map(Fields.of(Movie$._THIS, Movie$._EMBEDDED_EXTRACT_SCORE)) - .collect(Collectors.toList()); - } -} -``` - -Key features of the search service: -- Uses `EntityStream` to create a search stream for `Movie` entities -- Converts the text query into a vector embedding -- Uses K-nearest neighbors (KNN) search to find similar vectors -- Applies additional filters for hybrid search (combining vector and traditional search) -- Returns pairs of movies and their similarity scores - -### 4. Movie Service for Data Loading - -The `MovieService` handles loading movie data into Redis. It reads a JSON file containing movie date and save the movies into Redis. -It may take one or two minutes to load the data for the 36 thousand movies in the file because the embedding generation is done in the background. -The `@Vectorize` annotation will generate the embeddings for the `extract` field when the movie is saved into Redis.: - -```java -@Service -public class MovieService { - - private static final Logger log = LoggerFactory.getLogger(MovieService.class); - private final ObjectMapper objectMapper; - private final ResourceLoader resourceLoader; - private final MovieRepository movieRepository; - - public MovieService(ObjectMapper objectMapper, ResourceLoader resourceLoader, MovieRepository movieRepository) { - this.objectMapper = objectMapper; - this.resourceLoader = resourceLoader; - this.movieRepository = movieRepository; - } - - public void loadAndSaveMovies(String filePath) throws Exception { - Resource resource = resourceLoader.getResource("classpath:" + filePath); - try (InputStream is = resource.getInputStream()) { - List movies = objectMapper.readValue(is, new TypeReference<>() {}); - List unprocessedMovies = movies.stream() - .filter(movie -> !movieRepository.existsById(movie.getTitle()) && - movie.getYear() > 1980 - ).toList(); - long systemMillis = System.currentTimeMillis(); - movieRepository.saveAll(unprocessedMovies); - long elapsedMillis = System.currentTimeMillis() - systemMillis; - log.info("Saved " + movies.size() + " movies in " + elapsedMillis + " ms"); - } - } - - public boolean isDataLoaded() { - return movieRepository.count() > 0; - } -} -``` - -### 5. Search Controller - -The REST controller exposes the search endpoint: - -```java -@RestController -public class SearchController { - - private final SearchService searchService; - - public SearchController(SearchService searchService) { - this.searchService = searchService; - } - - @GetMapping("/search") - public Map search( - @RequestParam(required = false) String text, - @RequestParam(required = false) Integer yearMin, - @RequestParam(required = false) Integer yearMax, - @RequestParam(required = false) List cast, - @RequestParam(required = false) List genres, - @RequestParam(required = false) Integer numberOfNearestNeighbors - ) { - List> matchedMovies = searchService.search( - text, - yearMin, - yearMax, - cast, - genres, - numberOfNearestNeighbors - ); - return Map.of( - "matchedMovies", matchedMovies, - "count", matchedMovies.size() - ); - } -} -``` - -### 6. Application Bootstrap - -The main application class initializes Redis OM Spring and loads data: - -```java -@SpringBootApplication -@EnableRedisEnhancedRepositories(basePackages = {"dev.raphaeldelio.redis8demo*"}) -public class Redis8DemoVectorSimilaritySearchApplication { - - public static void main(String[] args) { - SpringApplication.run(Redis8DemoVectorSimilaritySearchApplication.class, args); - } - - @Bean - CommandLineRunner loadData(MovieService movieService) { - return args -> { - if (movieService.isDataLoaded()) { - System.out.println("Data already loaded. Skipping data load."); - return; - } - movieService.loadAndSaveMovies("movies.json"); - }; - } -} -``` - -The `@EnableRedisEnhancedRepositories` annotation activates Redis OM Spring's repository support. - -## Example API Requests - -You can make requests to the search endpoint: - -``` -GET http://localhost:8082/search?text=A movie about a young boy who goes to a wizardry school - -GET http://localhost:8082/search?numberOfNearestNeighbors=1&yearMin=1970&yearMax=1990&text=A movie about a kid and a scientist who go back in time - -GET http://localhost:8082/search?cast=Dee Wallace,Henry Thomas&text=A boy who becomes friend with an alien -``` - -## Sample Response - -```json -{ - "count": 1, - "matchedMovies": [ - { - "first": { - "title": "Back to the Future", - "year": 1985, - "cast": [ - "Michael J. Fox", - "Christopher Lloyd" - ], - "genres": [ - "Science Fiction" - ], - "extract": "Back to the Future is a 1985 American science fiction film directed by Robert Zemeckis and written by Zemeckis, and Bob Gale. It stars Michael J. Fox, Christopher Lloyd, Lea Thompson, Crispin Glover, and Thomas F. Wilson. Set in 1985, it follows Marty McFly (Fox), a teenager accidentally sent back to 1955 in a time-traveling DeLorean automobile built by his eccentric scientist friend Emmett \"Doc\" Brown (Lloyd), where he inadvertently prevents his future parents from falling in love – threatening his own existence – and is forced to reconcile them and somehow get back to the future.", - "thumbnail": "https://upload.wikimedia.org/wikipedia/en/d/d2/Back_to_the_Future.jpg" - }, - "second": 0.463297247887 - } - ] -} -``` -