Recipe Recommender using CrewAI, Qdrant, and Neon

This notebook demonstrates how to build a simple recipe recommendation system using a combination of technologies:

• CrewAI: For orchestrating the agent and task.
• Qdrant: As a vector database for storing and searching recipe embeddings.
• Neon: As a serverless PostgreSQL database to store recipe metadata.
• Langchain-Google-GenAI: To generate embeddings using Google's models.
• Psycopg2-binary: Python adapter for PostgreSQL.
• Qdrant-client: Python client for Qdrant.
• Crewai-tools: Provides tools for CrewAI agents.
• Scikit-learn: Used for TF-IDF vectorization.

The system allows you to search for recipes based on natural language queries, leveraging both semantic search (using embeddings) and keyword search (using TF-IDF) for improved relevance.

Prerequisites

To run this notebook, you will need:

• A Google Cloud Project with the Gemini API enabled.
• A Neon account and a PostgreSQL database instance.
• A Qdrant Cloud account and a free cluster.
• Python 3.7 or higher.

Setup

1. Clone the Repository:

   git clone <repository_url>
   cd <repository_directory>

2. Install Dependencies: Run the first few cells of the notebook to install the required Python packages:

   !pip install psycopg2-binary qdrant-client crewai crewai-tools langchain-google-genai -U qdrant-client

3. Set up Environment Variables: You will need to provide your credentials for the databases and the Gemini API. You can either set these as environment variables before running the notebook or directly within the notebook itself (though using environment variables is recommended for security).

   • GEMINI_API_KEY: Your API key for the Google Gemini API.
   • DB_HOST: Your Neon PostgreSQL host.
   • DB_NAME: Your Neon PostgreSQL database name.
   • DB_USER: Your Neon PostgreSQL database user.
   • DB_PASSWORD: Your Neon PostgreSQL database password.
   • QDRANT_URL: Your Qdrant Cloud cluster URL.
   • QDRANT_API_KEY: Your Qdrant Cloud API key.

4. Run the Notebook: Execute the cells in the notebook sequentially. The setup_database_and_qdrant() function will create the necessary tables in your Neon database, insert sample data, and index this data in your Qdrant collection.

How it Works

1. Data Loading and Indexing: The setup_database_and_qdrant() function connects to your Neon database, creates a recipes table, inserts sample data, and then indexes this data in Qdrant. Each recipe is indexed using both a dense vector (generated by the Gemini embedding model) for semantic similarity and a sparse vector (generated by TF-IDF) for keyword matching.
2. Recipe Search Tool: The RecipeSearchTool is a custom tool for the CrewAI agent. When called with a user query, it performs a hybrid search in Qdrant, combining the results from both the dense and sparse vector searches using RRF (Reciprocal Rank Fusion) to get the best results.
3. CrewAI Agent and Task: A Recipe Expert agent is defined with the RecipeSearchTool. A task is assigned to this agent to find a recipe based on the user's query.
4. Crew Execution: The CrewAI Crew is created with the agent and task. The kickoff method starts the process, the agent uses the RecipeSearchTool to search Qdrant, and then generates a friendly response recommending the found recipe(s).

Usage

After running the setup cells, the notebook will prompt you to enter a recipe query: