added annotations and docstrings

Author: codinglabsong

.gitignore

@@ -9,6 +9,7 @@ __pycache__/
 build/
 .gradio
 *.tar.gz
+.benchmarks
 
 # secrets
 .env

ETHICS.md

@@ -2,9 +2,9 @@
 
 Anyfile-Agent helps users explore their local documents with the assistance of an external language model. I designed the software with the following principles:
 
-- **User Control and Privacy** – Files remain on the local machine. Processing uses open‑source libraries and the configured language model API. No uploaded content is sent elsewhere by the application.
+- **User Control and Privacy** – Files remain on the local machine. Processing uses open-source libraries and the configured language model API. No uploaded content is sent elsewhere by the application.
 - **Transparency** – Indexing creates temporary representations of the documents (e.g., embeddings, OCR text) so the agent can search them. These artifacts are stored locally and users may delete them at any time.
-- **Responsible Use** – The agent can generate or execute SQL queries over the user’s data. Only read‑only commands are permitted, but users should review outputs before acting on them. Do not rely on the agent for legal, medical, or safety‑critical decisions.
+- **Responsible Use** – The agent can generate or execute SQL queries over the user’s data. Only read-only commands are permitted, but users should review outputs before acting on them. Do not rely on the agent for legal, medical, or safety-critical decisions.
 - **Bias and Limitations** – Responses may reflect biases of the underlying language model or the provided data. Users should validate critical information from original sources.
 - **Open Development** – The project is MIT licensed so that others may inspect, modify, and improve the code. Contributions must follow these ethical guidelines.
 

MODEL_CARD.md

@@ -11,7 +11,7 @@ Anyfile-Agent is a retrieval-based assistant that helps users search and analyze
 ## Intended Use
 - **Primary uses**: Searching personal documents, extracting structured summaries, and answering questions via natural language.
 - **Users**: Individuals or teams who want a local assistant for their files. Requires a valid Google Gemini API key.
-- **Out-of-scope uses**: Do not use the agent for generating legal, medical, or safety‑critical advice. It should not be used to process data that violates privacy regulations or third‑party terms of service.
+- **Out-of-scope uses**: Do not use the agent for generating legal, medical, or safety-critical advice. It should not be used to process data that violates privacy regulations or third-party terms of service.
 
 ## Data and Training
 Anyfile-Agent does not train a new model. It indexes user-provided documents locally and sends text chunks to a Google Gemini model for embedding and chat responses. The quality of answers depends on that service and the content of the uploaded data.

README.md

@@ -1,5 +1,5 @@
 # Anyfile-Agent
-Anyfile-Agent lets you query your own documents using natural language. It indexes a folder of files, converts CSV and Excel sheets into a DuckDB database, and performs semantic search via vector retrieval. Built with LangChain/LangGraph, this interactive LLM agent combines RAG‑based retrieval and SQL querying so you can “chat” with your data.
+Anyfile-Agent lets you query your own documents using natural language. It indexes a folder of files, converts CSV and Excel sheets into a DuckDB database, and performs semantic search via vector retrieval. Built with LangChain/LangGraph, this interactive LLM agent combines RAG-based retrieval and SQL querying so you can “chat” with your data.
 
 ## Features
 - **Multi-format ingestion** – Images are processed through OCR so their text is indexed. PDFs, Word docs, PowerPoint, Markdown, HTML, and plain text are split into searchable chunks. 
@@ -50,7 +50,7 @@ python app.py
    * For best results with XLSX, use a simple tabular layout—one header row, uniform columns, and no merged cells or custom formatting. You can have multiple sheets.
 
 ## Example Results
-### Multi‑Step Reasoning with Tool Use
+### Multi-Step Reasoning with Tool Use
 <div style="max-height:400px; overflow-y:auto; border:1px solid #ccc; padding:8px;">
   <pre><code class="language-bash">
 ================================ Human Message =================================

app.py

@@ -1,3 +1,5 @@
+"""Gradio-based web app for Anyfile-Agent: upload documents, index them, and chat interactively."""
+
 import atexit
 import asyncio
 import gc
@@ -25,7 +27,10 @@
 
 
 class Session:
+    """Session state for file uploads, indexing, and chat history."""
+
     def __init__(self):
+        """Initialize session IDs, paths, and database connections."""
         self.sid = uuid.uuid4().hex
         self.db_path = TMP_DIR / "csv_excel_to_db.duckdb"
         self.index_path = TMP_DIR / "faiss_index"
@@ -35,6 +40,7 @@ def __init__(self):
         self.sql_engines: List = []
 
     def cleanup(self):
+        """Dispose SQL engines, close history DB, reset agent, and remove tmp directory."""
         # dispose any SQLAlchemy/SQL-toolkit engines
         for eng in self.sql_engines:
             try:
@@ -60,10 +66,12 @@ def cleanup(self):
 # shutdown hook that is called when session ends
 @atexit.register
 def _purge_all():
+    """Cleanup session data on program exit."""
     sess.cleanup()
 
 
 def _safe_copy(src: Path, dst_dir: Path):
+    """Copy a file to dst_dir, avoiding name collisions by appending a random suffix."""
     dst = dst_dir / src.name
     if dst.exists():
         dst = dst.with_name(f"{dst.stem}_{uuid.uuid4().hex[:4]}{dst.suffix}")
@@ -72,6 +80,14 @@ def _safe_copy(src: Path, dst_dir: Path):
 
 # upload & sync
 def cb_upload_and_sync(files: List[gr.File]) -> Generator[Tuple[str, list], None, None]:
+    """Handle uploaded files: copy to temp, index docs, build agent, and yield status updates.
+
+    Args:
+        files: List of uploaded files from the Gradio interface.
+
+    Yields:
+        Tuples of (status_message, chat_history).
+    """
     # GUARDRAIL FOR EMPTY FILES
     if not files:
         yield "⚠️ No files selected.", []
@@ -132,6 +148,15 @@ def cb_upload_and_sync(files: List[gr.File]) -> Generator[Tuple[str, list], None
 
 # chat
 def cb_chat(hist: List[dict], msg: str) -> Tuple[List[dict], str]:
+    """Handle user messages: stream agent response and update conversation history.
+
+    Args:
+        hist: Conversation history as a list of role/content dicts.
+        msg: New user message.
+
+    Returns:
+        A tuple of (updated_history, clear_input_str).
+    """
     if sess.agent is None:
         hist.append(
             {

src/any_chatbot/agent.py

@@ -1,3 +1,5 @@
+"""CLI entry-point for running Anyfile-Agent in streaming mode."""
+
 import argparse
 import sqlite3
 import logging
@@ -18,6 +20,7 @@
 
 
 def parse_args() -> argparse.Namespace:
+    """Parse command-line options for the agent."""
     p = argparse.ArgumentParser()
 
     p.add_argument(
@@ -61,6 +64,7 @@ def parse_args() -> argparse.Namespace:
 
 
 def main() -> None:
+    """Entry-point invoked by `python agent.py`."""
     logging.basicConfig(level=logging.INFO)
     cfg = parse_args()
     load_environ_vars()

src/any_chatbot/indexing.py

@@ -1,3 +1,5 @@
+"""Data-ingestion pipeline: load docs, build DuckDB tables, create FAISS index."""
+
 import os
 import re
 import logging
@@ -6,6 +8,7 @@
 import shutil
 from dotenv import load_dotenv
 from pathlib import Path
+from typing import List, Tuple
 
 from langchain_community.vectorstores import FAISS
 from langchain_google_genai import GoogleGenerativeAIEmbeddings
@@ -20,7 +23,8 @@
 DATA = BASE / "data"
 
 
-def load_and_split_text_docs(data_dir):
+def load_and_split_text_docs(data_dir: Path) -> List[Document]:
+    """Load PDFs, DOCX, PPTX, etc. and split into chunks suitable for embeddings."""
     text_chunks = []
     globs = [
         "**/*.pdf",
@@ -60,7 +64,8 @@ def load_and_split_text_docs(data_dir):
     return text_chunks
 
 
-def load_image_docs_as_text(data_dir):
+def load_image_docs_as_text(data_dir: Path) -> List[Document]:
+    """Run OCR on images and return one Document per image."""
     image_text_docs = []
     globs = [
         "**/*.png",
@@ -90,7 +95,7 @@ def load_image_docs_as_text(data_dir):
 
 
 def _tbl(name: str) -> str:
-    """make a safe SQL table name"""
+    """Sanitize an arbitrary string so it can be used as a SQL table name."""
     name = re.sub(r"[^0-9a-zA-Z_]+", "_", name).strip("_")
     if not name or name[0].isdigit():
         name = f"t_{name}"
@@ -101,6 +106,7 @@ def build_duckdb_and_summary_cards(
     data_dir: Path,
     db_path: Path,
 ) -> list[Document]:
+    """Create DuckDB tables for CSV/XLSX files and return vector-searchable summary cards."""
     summary_cards = []
     # skip if there are no .csv/.xlsx/.xls files
     patterns = ("*.csv", "*.xlsx", "*.xls")
@@ -197,7 +203,8 @@ def embed_and_index_all_docs(
     db_path: Path = DATA / "generated_db" / "csv_excel_to_db.duckdb",
     index_path: Path = DATA / "generated_db" / "faiss_index",
     load_data: bool = False,
-):
+) -> Tuple[GoogleGenerativeAIEmbeddings, FAISS]:
+    """Return (embeddings, vector_store). Build or load FAISS & DuckDB as needed."""
     # load embeedings and vector store
     embeddings = GoogleGenerativeAIEmbeddings(model="models/embedding-001")
 

src/any_chatbot/prompts.py

@@ -1,3 +1,5 @@
+"""Prompt templates used by Anyfile-Agent."""
+
 system_message = """
 You are a agent designed to conduct semantic search on the uploaded user documents 
 and/or also interact with a SQL database.

src/any_chatbot/tools.py

@@ -1,3 +1,5 @@
+"""Utility helpers that turn a FAISS vector store or DuckDB database into LangChain tools usable by the agent."""
+
 from typing import Tuple, List, Literal
 from pathlib import Path
 
@@ -12,6 +14,15 @@
 
 
 def initialize_retrieve_tool(vector_store: VectorStore):
+    """Return a LangChain `@tool` that performs semantic search.
+
+    Args:
+        vector_store: A pre-built FAISS (or compatible) vector store.
+
+    Returns:
+        The decorated `retrieve` function ready to be passed into an agent.
+    """
+
     @tool(
         description=(
             """
@@ -41,7 +52,14 @@ def retrieve(
 
 
 def is_safe_sql(query: str) -> bool:
-    """Filters out destructive sql queries"""
+    """Reject queries that contain DML/DDL keywords.
+
+    Args:
+        query: Arbitrary SQL supplied by the LLM.
+
+    Returns:
+        True if the query looks safe (SELECT/PRAGMA), else False.
+    """
     forbidden = ["insert", "update", "delete", "drop", "alter", "create", "replace"]
     # Make sure to only block whole words (e.i., don't block 'updated_at')
     return not any(f" {word} " in f" {query.lower()} " for word in forbidden)
@@ -51,6 +69,15 @@ def initialize_sql_toolkit(
     llm,
     db_path: Path = DATA / "generated_db" / "csv_excel_to_db.duckdb",
 ):
+    """Wrap DuckDB in a LangChain `SQLDatabaseToolkit` with a safety filter.
+
+    Args:
+        llm: The chat model that will power SQL-aware tools.
+        db_path: Location of the DuckDB file created during indexing.
+
+    Returns:
+        A list of LangChain tools for schema look-up and SELECT queries.
+    """
     db = SQLDatabase.from_uri(f"duckdb:///{db_path}")
 
     # Monkey-path the run method to include safety filter

src/any_chatbot/utils.py

@@ -1,3 +1,5 @@
+"""Small utility helpers not tied to LangChain."""
+
 import getpass
 import os