Merge pull request #5 from MeshJS/feat/py-rag

Add Python RAG functionality

Author: smutyala1at

README.md

@@ -1,19 +1,19 @@
-# Mimir
+## Mimir
 
-A comprehensive **contextual RAG** (Retrieval Augmented Generation) system with MCP (Model Context Protocol) integration for both **documentation and TypeScript codebases**. Mimir ingests documentation and TypeScript code from GitHub repositories into a Supabase vector store and provides powerful querying capabilities through both REST API and MCP protocol. Unlike basic RAG, contextual RAG provides rich context around each code entity, including full file content, imports, and surrounding code.
+A comprehensive **contextual RAG** (Retrieval Augmented Generation) system with MCP (Model Context Protocol) integration for both **documentation and codebases**. Mimir ingests documentation and source code (currently **TypeScript** and **Python**, with more languages planned) from GitHub repositories into a Supabase vector store and provides powerful querying capabilities through both REST API and MCP protocol. Unlike basic RAG, contextual RAG provides rich context around each code entity, including full file content, imports, and surrounding code.
 
 ## Projects
 
 This repository contains two main components:
 
 ### [mimir-rag](./mimir-rag)
 
-The core RAG server that handles ingestion and querying of both **documentation (MDX)** and **TypeScript codebases**.
+The core RAG server that handles ingestion and querying of both **documentation (MDX)** and **codebases** (TypeScript, Python, and easily extensible to more languages).
 
 **Features:**
-- Ingests documentation and TypeScript code from GitHub repositories into Supabase vector store
+- Ingests documentation and source code from GitHub repositories into Supabase vector store
 - Supports separate repositories for code and documentation
-- Automatically extracts TypeScript entities (functions, classes, interfaces, exported const functions)
+- Automatically extracts code entities (e.g., TypeScript: functions, classes, interfaces, exported const functions; Python: functions, classes, methods, module-level context)
 - Supports multiple LLM providers (OpenAI, Anthropic, Google, Mistral)
 - OpenAI-compatible chat completions endpoint (`/v1/chat/completions`)
 - MCP endpoint for semantic document search (`/mcp/ask`)
@@ -98,8 +98,8 @@ See the [mimir-mcp README](./mimir-mcp/README.md) for detailed setup instruction
 ## Workflow
 
 1. **Ingestion Phase:**
-   - mimir-rag fetches documentation (MDX) and TypeScript code from configured GitHub repository(ies)
-   - TypeScript files are parsed to extract entities (functions, classes, interfaces, exported const functions)
+   - mimir-rag fetches documentation (MDX) and code from configured GitHub repository(ies)
+   - Code files are parsed to extract language-specific entities (TypeScript entities, Python functions/classes/methods, etc.)
    - **Contextual RAG**: Each entity is enriched with surrounding context - full file content, imports, parent classes, and related code
    - Documents are chunked into smaller segments with rich contextual information
    - Chunks are embedded using your chosen LLM provider
@@ -121,9 +121,9 @@ See the [mimir-mcp README](./mimir-mcp/README.md) for detailed setup instruction
 
 ## Use Cases
 
-- **AI-Powered Code Assistant**: Let your AI coding assistant query your TypeScript codebase in real-time - find functions, classes, and understand code structure
+- **AI-Powered Code Assistant**: Let your AI coding assistant query your codebase in real-time - find functions, classes, and understand code structure (supports TypeScript, Python, and more)
 - **AI-Powered Documentation Assistant**: Let your AI coding assistant query your docs in real-time
-- **Codebase Understanding**: Index your entire TypeScript project - functions, classes, interfaces, and exported const functions
+- **Codebase Understanding**: Index your entire codebase - functions, classes, interfaces, and other language-specific entities
 - **Internal Knowledge Base**: Index internal wikis, API docs, or technical documentation
 - **Customer Support**: Provide accurate, context-aware answers from your documentation
 - **Developer Onboarding**: Help new developers quickly find information in your codebase and documentation
@@ -135,7 +135,7 @@ See the [mimir-mcp README](./mimir-mcp/README.md) for detailed setup instruction
 - **Node.js**: 20 or later
 - **Supabase**: Vector store for embeddings and document storage
 - **LLM Provider**: API key for OpenAI, Anthropic, Google, or Mistral
-- **GitHub**: Repository with documentation (MDX) and/or TypeScript code to ingest (optional)
+- **GitHub**: Repository with documentation (MDX) and/or code (TypeScript, Python, etc.) to ingest (optional)
 
 ## Getting Started
 

mimir-rag/README.md

@@ -1,6 +1,6 @@
-# mimir-rag
+## mimir-rag
 
-Utility CLI + API that ingests **documentation (MDX) and TypeScript codebases** into Supabase using **contextual RAG** and exposes OpenAI-compatible chat completions, MCP endpoints, and ingestion endpoints. Perfect for making your entire codebase and documentation queryable by AI assistants with rich contextual understanding.
+Utility CLI + API that ingests **documentation (MDX) and codebases** into Supabase using **contextual RAG** and exposes OpenAI-compatible chat completions, MCP endpoints, and ingestion endpoints. It currently supports **TypeScript** and **Python** code, and is designed to be easily extensible to additional languages. Perfect for making your entire codebase and documentation queryable by AI assistants with rich contextual understanding.
 
 ## Quick Start
 
@@ -139,9 +139,9 @@ Key configuration variables include:
 
 - **Server**: `MIMIR_SERVER_API_KEY` (required), `MIMIR_SERVER_GITHUB_WEBHOOK_SECRET`, `MIMIR_SERVER_FALLBACK_INGEST_INTERVAL_MINUTES`
 - **Supabase**: `MIMIR_SUPABASE_URL` (required), `MIMIR_SUPABASE_SERVICE_ROLE_KEY` (required), `MIMIR_SUPABASE_TABLE` (optional, default: "docs")
-- **GitHub**: 
+- **GitHub** (language-agnostic code + docs ingestion): 
   - `MIMIR_GITHUB_URL` - Main repository URL (fallback if separate repos not set)
-  - `MIMIR_GITHUB_CODE_URL` - Separate repository for TypeScript code (optional)
+  - `MIMIR_GITHUB_CODE_URL` - Separate repository for code (TypeScript, Python, etc.) (optional)
   - `MIMIR_GITHUB_DOCS_URL` - Separate repository for MDX documentation (optional)
   - `MIMIR_GITHUB_TOKEN`, `MIMIR_GITHUB_DIRECTORY`, `MIMIR_GITHUB_BRANCH`
   - `MIMIR_GITHUB_CODE_DIRECTORY`, `MIMIR_GITHUB_CODE_INCLUDE_DIRECTORIES` - Code repo specific settings
@@ -160,24 +160,24 @@ Key configuration variables include:
 
 ### Separate Code and Documentation Repositories
 
-You can configure separate repositories for TypeScript code and MDX documentation:
+You can configure separate repositories for code and MDX documentation. Code repositories can contain TypeScript, Python, or any other supported language – the ingestion pipeline is language-agnostic at the repository level.
 
 ```bash
 # Main repository (fallback)
 MIMIR_GITHUB_URL=https://github.com/user/main-repo
 
-# Separate code repository
+# Separate code repository (TypeScript, Python, etc.)
 MIMIR_GITHUB_CODE_URL=https://github.com/user/code-repo
 MIMIR_GITHUB_CODE_DIRECTORY=src
 MIMIR_GITHUB_CODE_INCLUDE_DIRECTORIES=src,lib
 
-# Separate documentation repository
+# Separate documentation repository (MD/MDX)
 MIMIR_GITHUB_DOCS_URL=https://github.com/user/docs-repo
 MIMIR_GITHUB_DOCS_DIRECTORY=docs
 MIMIR_GITHUB_DOCS_INCLUDE_DIRECTORIES=docs,guides
 ```
 
-When configured, TypeScript files will be ingested from the code repository and MDX files from the docs repository. Source URLs for TypeScript files will automatically use the code repository URL.
+When configured, code files will be ingested from the code repository and MDX files from the docs repository. Source URLs for code files will automatically use the code repository URL.
 
 ### Parser Configuration
 
@@ -191,27 +191,34 @@ Control what gets extracted from your codebase:
 
   Example: `MIMIR_EXCLUDE_PATTERNS=*.test.ts,*.spec.ts,test/,__tests__/,tests/`
 
-### TypeScript Entity Extraction
+### Code Entity Extraction (TypeScript, Python, and more)
 
-mimir-rag automatically extracts and indexes TypeScript entities from your codebase:
+mimir-rag automatically extracts and indexes language-specific code entities from your codebase:
 
-- **Functions**: `export function myFunction() {}`
-- **Exported Const Functions**: `export const myFunction = () => {}` (always extracted)
-- **Classes**: `export class MyClass {}`
-- **Interfaces**: `export interface MyInterface {}`
-- **Types**: `export type MyType = ...`
-- **Enums**: `export enum MyEnum {}`
-- **Methods**: Class methods (if `MIMIR_EXTRACT_METHODS=true`)
+- **TypeScript**:
+  - Functions: `export function myFunction() {}`
+  - Exported const functions: `export const myFunction = () => {}` (always extracted)
+  - Classes: `export class MyClass {}`
+  - Interfaces: `export interface MyInterface {}`
+  - Types: `export type MyType = ...`
+  - Enums: `export enum MyEnum {}`
+  - Methods: class methods (if `MIMIR_EXTRACT_METHODS=true`)
+
+- **Python**:
+  - Top-level functions
+  - Classes
+  - Methods (functions inside classes)
+  - Module-level context entity for each file
 
 Each entity is stored as a separate chunk with **rich contextual information**:
 - Full code snippet
-- **Contextual RAG**: Surrounding file content, imports, and parent class context
-- JSDoc comments (if present)
-- Parameters and return types
+- **Contextual RAG**: Surrounding file content, imports, and parent/module/class context
+- Language-native doc comments (e.g., TypeScript JSDoc, Python docstrings)
+- Parameters and return types when available
 - Line numbers for source linking
 - GitHub URL for direct code access
 
-This contextual RAG approach allows the AI to understand not just the entity itself, but also how it fits into the larger codebase - what it imports, what it's part of, and how it's used. This enables more accurate and contextually-aware answers with direct links to source code.
+This contextual RAG approach allows the AI to understand not just the entity itself, but also how it fits into the larger codebase—what it imports, what it's part of, and how it's used. This enables more accurate and contextually-aware answers with direct links to source code.
 
 ## API Endpoints
 

mimir-rag/package-lock.json

@@ -22,7 +22,6 @@
     "tsx": "^4.20.6"
   },
   "dependencies": {
-    "typescript": "^5.9.3",
     "@ai-sdk/anthropic": "^2.0.50",
     "@ai-sdk/google": "^2.0.44",
     "@ai-sdk/mistral": "^2.0.25",
@@ -36,6 +35,9 @@
     "p-limit": "^7.2.0",
     "p-retry": "^7.1.0",
     "pino": "^10.1.0",
-    "tiktoken": "^1.0.22"
+    "tiktoken": "^1.0.22",
+    "tree-sitter-python": "^0.25.0",
+    "typescript": "^5.9.3",
+    "web-tree-sitter": "^0.26.3"
   }
 }

mimir-rag/package.json

@@ -122,7 +122,17 @@ export async function loadAppConfig(configPath?: string): Promise<AppConfig> {
         parser: {
             extractVariables: getEnvBoolean("MIMIR_EXTRACT_VARIABLES", false),
             extractMethods: getEnvBoolean("MIMIR_EXTRACT_METHODS", true),
-            excludePatterns: getEnv("MIMIR_EXCLUDE_PATTERNS", false)?.split(",").map(p => p.trim()).filter(Boolean),
+            excludePatterns: [
+                // Default Python test patterns
+                "test_*.py",
+                "*_test.py",
+                "*_tests.py",
+                "tests/",
+                "test/",
+                "__tests__/",
+                // User-defined patterns from env
+                ...(getEnv("MIMIR_EXCLUDE_PATTERNS", false)?.split(",").map(p => p.trim()).filter(Boolean) ?? []),
+            ],
             includeDirectories: getEnv("MIMIR_GITHUB_INCLUDE_DIRECTORIES", false)?.split(",").map(p => p.trim()).filter(Boolean),
         },
         docs: {

mimir-rag/src/config/loadConfig.ts

@@ -24,7 +24,7 @@ export interface GithubConfig {
     githubUrl: string;
     directory?: string; // Directory for main repo (fallback if separate not set)
     includeDirectories?: string[]; // Include directories for main repo (fallback if separate not set)
-    codeUrl?: string; // Optional: separate repo for TypeScript code
+    codeUrl?: string; // Optional: separate repo for code (TypeScript, Python, etc.)
     codeDirectory?: string; // Optional: directory for code repo
     codeIncludeDirectories?: string[]; // Optional: include directories for code repo
     docsUrl?: string; // Optional: separate repo for MDX docs

mimir-rag/src/config/types.ts

@@ -1,7 +1,7 @@
 import { Buffer } from "node:buffer";
-import { extractTitle } from "../utils/extractTitle";
-import { calculateChecksum } from "../utils/calculateChecksum";
-import { countTokens, getEncoder } from "../utils/tokenEncoder";
+import { extractTitle } from "../../utils/extractTitle";
+import { calculateChecksum } from "../../utils/calculateChecksum";
+import { countTokens, getEncoder } from "../../utils/tokenEncoder";
 
 export interface MdxChunk {
     chunkTitle: string;