Medical-RAG-using-Bio-Mistral-7B

This is a RAG implementation using Open Source stack. BioMistral 7B has been used to build this app along with PubMedBert as an embedding model, Qdrant as a self hosted Vector DB, and Langchain & Llama CPP as an orchestration frameworks.

Reference Implementation on Intel AI PC

Disclaimer

This demo is intended only for the purpose of exploring new LLM use cases at the edge and not recommended for production-grade medical chatbot

Device Under Test

Processor: Intel® Core™ Ultra 7 165H
OS: Windows 11 Pro 23H2
RAM: 64GB
Python 3.11.9

Prerequisite

Install Microsoft Visual C++ compiler toolset required for installing llama-cpp-python

Steps

1. Clone this repository
2. Create a Python virtual environment and install the dependencies

   python -m venv biomistral_rag   
   biomistral_rag\Scripts\activate   
   python -m pip install pip --upgrade  
   cd <folder_name>
   pip install -r requirements.txt   
   pip install qdrant-client --upgrade   
   

3. Download the INT4 version of BioMistral-7B model in GGUF format
4. Download the embedding model

   git lfs install
   git clone https://huggingface.co/NeuML/pubmedbert-base-embeddings
   

5. Install Docker Desktop for Windows with optional proxy settings
6. Create the Qdrant container

   docker pull qdrant/qdrant
   docker run -p 6333:6333 -v .\qdrant_db\:/qdrant/storage qdrant/qdrant
   

   
   Access the Dashboard using http://localhost:6333/dashboard
7. Create embeddings for the new documents in data

   python ingest.py
   

   Check the new Collection on the Qdrant Dashboard
   
8. Run the application

   uvicorn app:app   
   

   

Sample Outputs