refactor: modularize codebase and add comprehensive RAG functionality

[Wirasm]

Summary

This PR represents a complete refactoring of the crawl4ai MCP server, transforming it from a single monolithic file into a well-structured, modular application with comprehensive RAG (Retrieval-Augmented Generation) capabilities and proper testing infrastructure.

Motivation and Context

The original implementation was a single 1,054-line file that was becoming difficult to maintain and extend. This refactoring addresses several key issues:

• Maintainability: Code is now organized into logical modules following vertical slice architecture
• Testability: Comprehensive test suite with tests co-located with their respective modules
• Extensibility: Clear separation of concerns makes it easy to add new features
• Developer Experience: Added proper tooling, linting, and development workflows

Changes Made

Architecture & Structure

• ✨ Migrated from monolithic crawl4ai_mcp.py to modular package structure
• 📁 Implemented vertical slice architecture with co-located tests
• 🏗️ Created clear service, tool, and utility layers
• 📦 Properly packaged as crawl4ai_mcp with namespace imports

Core Features

• 🤖 RAG Server Implementation: Full MCP server with crawling and search capabilities
• 🔍 Smart Crawling: Depth control, URL filtering, and metadata tracking
• 🧠 Semantic Search: Embedding-based search with reranking algorithms
• 💻 Code Search: Specialized search for code examples and documentation

Developer Experience

• 📚 Added comprehensive development documentation (CLAUDE.md)
• 🛠️ Integrated UV package manager for dependency management
• ✅ Added pytest-based test suite with async support
• 🎨 Configured ruff for linting and code formatting
• 🔍 Set up mypy for type checking
• 📋 Created PRP (Project Refinement Protocol) framework

Services Added

• services/crawling.py: Async web crawling with Crawl4AI
• services/database.py: SQLite persistence layer
• services/embeddings.py: Text embedding generation
• services/search.py: Search and RAG query implementation

Tools Added

• crawl_single_page: Single page crawling
• smart_crawl_url: Intelligent multi-page crawling
• perform_rag_query: RAG query execution
• search_code_examples: Code-specific search
• get_available_sources: List crawled sources

Utilities Added

• text_processing.py: Text chunking and processing
• reranking.py: Search result ranking algorithms
• metadata.py: Metadata extraction utilities

Type of Change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation update
• Performance improvement
• Code refactoring

Testing

How has this been tested?

• ✅ Comprehensive unit test suite with 15 test files
• ✅ All tests passing (119 tests total)
• ✅ Async tests for crawling functionality
• ✅ Mock-based tests for external dependencies

Test configuration details

• Python 3.11+
• pytest with async support
• Test fixtures in conftest.py
• Tests co-located with implementation files

Instructions for reviewers to test

# Install dependencies
uv sync

# Run tests
uv run pytest

# Run with coverage
uv run pytest --cov=src

# Run linting
uv run ruff check .

# Run type checking
uv run mypy .

Screenshots/Recordings

N/A - Backend service changes only

Breaking Changes

While the core functionality remains the same, import paths have changed:

• Old: from src.crawl4ai_mcp import server
• New: from crawl4ai_mcp.mcp_server import server

The CLI entry point remains the same: crawl4ai-mcp

Performance Impact

• ✅ Async operations throughout for better concurrency
• ✅ Efficient text chunking with configurable parameters
• ✅ Database indexing for faster searches
• ✅ Caching of embeddings to avoid recomputation

Security Considerations

• ✅ Input validation on all user-provided URLs
• ✅ Safe SQL query construction (no raw string interpolation)
• ✅ Proper error handling to avoid information leakage
• ✅ No hardcoded secrets or credentials

Checklist

• My code follows the project's style guidelines
• I have performed a self-review
• I have commented hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective
• New and existing unit tests pass locally
• Any dependent changes have been merged

Dependencies

• This PR depends on #XXX
• No dependencies

Deploy Notes

No special deployment requirements. The package can be installed directly with:

uv pip install -e .

Additional Notes

Future Improvements

While this PR significantly improves the codebase, there are areas for future enhancement:

• Complete type annotations (currently at ~70% coverage)
• Increase test coverage from 57% to 80%+
• Add integration tests for end-to-end workflows
• Consider adding more AI provider options

Development Workflow

This PR also introduces a comprehensive development workflow documented in CLAUDE.md, including:

• Coding standards and conventions
• Testing requirements
• Git workflow (develop → main)
• Claude.ai integration for AI-assisted development

The PRP (Project Refinement Protocol) framework provides templates for:

• Feature development
• Code refactoring
• Performance optimization
• Bug fixes

This refactoring lays a solid foundation for future development while maintaining backward compatibility for existing users.

[btyeung]

Wish I saw this earlier, your refactor looks a ton better than mine (was quick and dirty).

[coleam00]

What a PR haha, thanks @Wirasm! It's going to take a while to review this, I am thinking maybe there would be some opinionated reachitectures that I would want to do differently, so we will see. Even if I rearchitect things a bit different, I would love to use a lot of this work as a base though!

[fpytloun]

I looked into this project thinking to add support for anything else then Supabase (in my case PostgreSQL + Qdrant) but not doable until refactored with some amount of abstraction, nice change 👍