-
Notifications
You must be signed in to change notification settings - Fork 53
Adds self-contained docker environment for java-recipes #68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
+952
−1
Merged
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -5,3 +5,4 @@ | |
| 05_nvidia_ai_rag_redis | ||
| 01_routing_optimization | ||
| 02_semantic_cache_optimization | ||
| spring_ai_redis_rag.ipynb | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -220,3 +220,4 @@ pip-selfcheck.json | |
| libs/redis/docs/.Trash* | ||
| .python-version | ||
| .idea/* | ||
| java-recipes/.* | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,148 @@ | ||
| <div align="center"> | ||
| <div><img src="../assets/redis-logo.svg" style="width: 130px"> </div> | ||
| <h1>Redis AI Java Resources</h1> | ||
| <div align="center"> | ||
|
|
||
| [](https://opensource.org/licenses/MIT) | ||
|  | ||
|  | ||
|
|
||
| </div> | ||
| <div> | ||
| ✨ Java-based code examples, notebooks, and resources for using Redis in AI and ML applications. ✨ | ||
| </div> | ||
|
|
||
| <div></div> | ||
| <br> | ||
|
|
||
| [**Setup**](#setup) | [**Running the Project**](#running-the-project) | [**Notebooks**](#notebooks) | [**Project Structure**](#project-structure) | [**Implementation Details**](#implementation-details) | ||
|
|
||
| </div> | ||
| <br> | ||
|
|
||
| ## Setup | ||
|
|
||
| This project uses Docker Compose to set up a complete environment for running Java-based AI applications with Redis. The environment includes: | ||
|
|
||
| - A Jupyter Notebook server with Java kernel support | ||
| - Redis Stack (includes Redis and RedisInsight) | ||
| - Pre-installed dependencies for AI/ML workloads | ||
|
|
||
| ### Prerequisites | ||
|
|
||
| - [Docker](https://docs.docker.com/get-docker/) and [Docker Compose](https://docs.docker.com/compose/install/) | ||
| - OpenAI API key (for notebooks that use OpenAI services) | ||
|
|
||
| ### Environment Configuration | ||
|
|
||
| 1. Create a `.env` file in the project root with your OpenAI API key: | ||
|
|
||
| ```bash | ||
| OPENAI_API_KEY=your_openai_api_key_here | ||
| ``` | ||
|
|
||
| ## Running the Project | ||
|
|
||
| 1. Clone the repository (if you haven't already): | ||
|
|
||
| ```bash | ||
| git clone https://github.com/redis-developer/redis-ai-resources.git | ||
| cd redis-ai-resources/java-resources | ||
| ``` | ||
|
|
||
| 2. Start the Docker containers: | ||
|
|
||
| ```bash | ||
| docker-compose up -d | ||
| ``` | ||
|
|
||
| 3. Access the Jupyter environment: | ||
| - Open your browser and navigate to [http://localhost:8888](http://localhost:8888) | ||
| - The token is usually shown in the docker-compose logs. You can view them with: | ||
|
|
||
| ```bash | ||
| docker-compose logs jupyter | ||
| ``` | ||
|
|
||
| 4. Access RedisInsight: | ||
| - Open your browser and navigate to [http://localhost:8001](http://localhost:8001) | ||
| - Connect to Redis using the following details: | ||
| - Host: redis-java | ||
| - Port: 6379 | ||
| - No password (unless configured) | ||
|
|
||
| 5. When finished, stop the containers: | ||
|
|
||
| ```bash | ||
| docker-compose down | ||
| ``` | ||
|
|
||
| ## Notebooks | ||
|
|
||
| | Notebook | Description | | ||
| | --- | --- | | ||
| | [RAG/spring_ai_redis_rag.ipynb](./RAG/spring_ai_redis_rag.ipynb) | Demonstrates building a RAG-based beer recommendation chatbot using Spring AI and Redis as the vector store | | ||
|
|
||
| ## Project Structure | ||
|
|
||
| ```bash | ||
| java-recipes/ | ||
| ├── .env # Environment variables (create this) | ||
| ├── docker-compose.yml # Docker Compose configuration | ||
| ├── jupyter/ # Jupyter configuration files | ||
| │ ├── Dockerfile # Dockerfile for Jupyter with Java kernel | ||
| │ ├── environment.yml # Conda environment specification | ||
| │ ├── install.py # JJava kernel installation script | ||
| │ ├── kernel.json # Kernel specification | ||
| │ └── java/ # Java dependencies and configuration | ||
| │ └── pom.xml # Maven project file with dependencies | ||
| └── resources/ # Data files for notebooks | ||
| └── beers.json.gz # Compressed beer dataset | ||
| ``` | ||
|
|
||
| ## Implementation Details | ||
|
|
||
| ### Java Jupyter Kernel | ||
|
|
||
| The project uses [JJava](https://github.com/dflib/jjava), a Jupyter kernel for Java based on JShell. This allows for interactive Java development in Jupyter notebooks. | ||
|
|
||
| Key components: | ||
|
|
||
| - Java 21 for modern Java features | ||
| - Maven for dependency management | ||
| - JJava kernel for Jupyter integration | ||
|
|
||
| ### Spring AI Integration | ||
|
|
||
| The Spring AI notebooks showcase how to use Spring's AI capabilities with Redis: | ||
| - **Spring AI**: Framework for building AI-powered applications | ||
| - **Redis Vector Store**: Used for storing and querying vector embeddings | ||
| - **Transformer Models**: For generating embeddings locally | ||
| - **RAG Pattern**: Demonstrates the Retrieval Augmented Generation pattern | ||
| ### Docker Configuration | ||
| The Docker setup includes: | ||
| 1. **Jupyter Container**: | ||
| - Based on minimal Jupyter notebook image | ||
| - Adds Java 21, Maven, and the JJava kernel | ||
| - Includes Python environment with PyTorch and other ML libraries | ||
| 2. **Redis Container**: | ||
| - Uses Redis Stack image with Vector Search capabilities | ||
| - Persists data using Docker volumes | ||
| - Exposes Redis on port 6379 and RedisInsight on port 8001 | ||
| ## Example Applications | ||
| ### Beer Recommendation Chatbot | ||
| The `spring-ai-rag.ipynb` notebook demonstrates: | ||
| - Loading and embedding beer data into Redis Vector Store | ||
| - Using local transformer models for generating embeddings | ||
| - Connecting to OpenAI for LLM capabilities | ||
| - Building a RAG pipeline to answer beer-related queries | ||
| - Semantic search over beer properties and descriptions |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| name: redis-ai-java | ||
| services: | ||
| jupyter: | ||
| build: | ||
| context: . | ||
| dockerfile: ./jupyter/Dockerfile | ||
| ports: | ||
| - "8888:8888" | ||
| environment: | ||
| - JUPYTER_ENABLE_LAB=yes | ||
| env_file: | ||
| - .env | ||
| volumes: | ||
| - ./:/home/jovyan/ | ||
| - ./resources:/home/jovyan/resources | ||
| redis-java: | ||
| image: redis/redis-stack:latest | ||
| ports: | ||
| - "6379:6379" # Redis database port | ||
| - "8001:8001" # RedisInsight port | ||
| volumes: | ||
| - redis-data:/data # Persist Redis data | ||
|
|
||
| volumes: | ||
| redis-data: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,59 @@ | ||
| FROM quay.io/jupyter/minimal-notebook:latest | ||
|
|
||
| RUN mkdir /home/jovyan/resources | ||
|
|
||
| USER root | ||
| WORKDIR /home/jovyan | ||
|
|
||
| # Install dependencies: Java 21 and Maven | ||
| RUN apt-get update && apt-get install -y openjdk-21-jdk maven | ||
|
|
||
| # Copy the pre-created Maven project and jjava-glue project | ||
| COPY ./jupyter/java /home/jovyan/java | ||
| COPY ./jupyter/install.py /home/jovyan/install.py | ||
|
|
||
| # Use Maven to download dependencies for JJava | ||
| WORKDIR /home/jovyan/java | ||
|
|
||
| # Download the JJava jar directly | ||
| RUN mvn dependency:get -Dartifact=org.dflib.jjava:jjava:1.0-M3 -Ddest=./ -Dtransitive=false | ||
| RUN mv jjava-1.0-M3.jar jjava.jar | ||
|
|
||
| # Pre-download Spring AI Dependencies | ||
| RUN mvn dependency:get -Dartifact=org.springframework.ai:spring-ai-openai:1.0.0-M6 | ||
| RUN mvn dependency:get -Dartifact=org.springframework.ai:spring-ai-transformers:1.0.0-M6 | ||
| RUN mvn dependency:get -Dartifact=org.springframework.ai:spring-ai-redis-store:1.0.0-M6 | ||
| # Pre-download Jedis | ||
| RUN mvn dependency:get -Dartifact=redis.clients:jedis:5.2.0 | ||
| # Download all dependencies | ||
| RUN mvn dependency:copy-dependencies -DoutputDirectory=./lib | ||
|
|
||
| # Create a list of dependencies for the classpath | ||
| RUN find ./lib -name "*.jar" | tr '\n' ':' > classpath.txt | ||
| # Add the jjava.jar to the classpath | ||
| RUN echo -n "/home/jovyan/java/jjava.jar:" >> classpath.txt | ||
|
|
||
| # Install the kernel with classpath configuration | ||
| WORKDIR /home/jovyan | ||
| RUN python install.py --prefix /opt/conda/ --classpath $(cat /home/jovyan/java/classpath.txt) | ||
|
|
||
| # Pre-download Transformer Models | ||
| RUN pip install transformers torch | ||
| RUN mkdir -p /home/jovyan/.cache/huggingface/hub | ||
| # Pre-download the specific model used in Spring AI Transformers | ||
| RUN python -c "from transformers import AutoModel; AutoModel.from_pretrained('sentence-transformers/all-MiniLM-L6-v2')" | ||
|
|
||
| # Clean up Maven artifacts but keep the jjava.jar and lib directory | ||
| RUN rm -rf /home/jovyan/java/target /home/jovyan/java/.m2 /home/jovyan/java/pom.xml \ | ||
| /home/jovyan/java/classpath.txt \ | ||
| && rm -f /home/jovyan/install.py | ||
|
|
||
| # Install conda packages from environment.yml | ||
| COPY ./jupyter/environment.yml /tmp/ | ||
| RUN conda env update -f /tmp/environment.yml && \ | ||
| conda clean --all -f -y && \ | ||
| fix-permissions "${CONDA_DIR}" && \ | ||
| fix-permissions "/home/${NB_USER}" | ||
|
|
||
| WORKDIR /home/jovyan | ||
| USER $NB_UID |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| name: base | ||
| channels: | ||
| - pytorch | ||
| - conda-forge | ||
| - defaults | ||
| dependencies: | ||
| - pytorch | ||
| - torchtext | ||
| - gensim |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.