feat: Add export domain with LlamaIndex support and batching

- Add new export/ domain for conversation export formats
- Implement export_for_llamaindex() with optional batch_size parameter
- Fix discover_claude_files() to properly return file paths
- Export all filtering functions in main module
- Update README with v2.1.0 features and export roadmap
- Add batching using more_itertools.chunked() for memory efficiency

BREAKING CHANGES: None - fully backward compatible

🤖 Generated with Claude Code

Co-Authored-By: Claude <noreply@anthropic.com>

Author: alicoding

CHANGELOG.md

@@ -1,3 +1,29 @@
+## [2.1.0] - 2025-09-16
+
+### Added
+
+- **Export Domain**: New `export/` module for converting conversations to different formats (#5)
+  - `export_for_llamaindex()` - Export conversations to LlamaIndex document format for semantic search
+  - Returns list of documents with `text` and `metadata` fields
+  - Filters out tool operations, keeping only pure conversation
+  - **Batching Support**: Optional `batch_size` parameter for memory-efficient processing
+  - Uses `more_itertools.chunked()` for 100% framework delegation
+- **Complete API Export**: All filtering functions now properly exported from main module (#6)
+  - `filter_messages_by_type()`, `filter_messages_by_tool()`, `search_messages_by_content()`, `exclude_tool_operations()`
+
+### Fixed
+
+- **discover_claude_files()**: Now properly returns file paths instead of empty list (#7)
+  - Extracts `transcript_path` from session metadata
+  - Returns empty list for non-existent search paths
+  - Enables downstream services to find and index conversations
+
+### Documentation
+
+- Updated README with export functionality and roadmap for future formats
+- Added 10+ planned export formats: Mem0, ChromaDB, Pinecone, Markdown, JSON-LD, OpenAI, Anthropic, LangChain, Haystack
+- Corrected architecture diagram to show all 19 domains (was showing 15)
+
 ## [2.0.1] - 2025-09-15
 
 # Changelog

README.md

@@ -1,4 +1,4 @@
-# Claude Parser v2.0.0 🚀
+# Claude Parser v2.1.0 🚀
 
 [![PyPI version](https://badge.fury.io/py/claude-parser.svg)](https://badge.fury.io/py/claude-parser)
 [![Documentation](https://img.shields.io/badge/docs-mkdocs-blue)](https://alicoding.github.io/claude-parser/)
@@ -8,9 +8,15 @@
 
 Claude Parser treats every Claude API call as a git commit, enabling powerful recovery and analysis capabilities when things go wrong.
 
-## 🎉 What's New in v2.0.0
+## 🎉 What's New in v2.1.0
 
-### Major Changes
+### New Features
+- **🔍 Export Domain** - Export conversations to different formats for indexing
+- **📚 LlamaIndex Export** - `export_for_llamaindex()` for semantic search
+- **🛠️ Fixed Discovery** - `discover_claude_files()` now properly returns file paths
+- **📦 Complete API** - All filtering functions now properly exported
+
+## 📋 v2.0.0 Major Changes
 - **🎯 Complete API Redesign** - Clean, intuitive Python API with 30+ functions
 - **📚 15 Domain Architecture** - Organized into focused, composable modules
 - **🔧 CG Commands** - Full Git-like CLI for disaster recovery
@@ -106,7 +112,10 @@ from claude_parser import (
 
     # Filtering (NEW in v2!)
     filter_messages_by_type, filter_messages_by_tool,
-    search_messages_by_content, exclude_tool_operations
+    search_messages_by_content, exclude_tool_operations,
+
+    # Export (NEW in v2.1!)
+    export_for_llamaindex  # Export conversations for semantic search
 )
 ```
 
@@ -155,26 +164,44 @@ def on_assistant(message):
 watch("~/.claude/projects/current/session.jsonl", on_assistant=on_assistant)
 ```
 
+### 5. Export for Semantic Search (NEW in v2.1!)
+```python
+from claude_parser import export_for_llamaindex
+
+# Export conversations to LlamaIndex format
+docs = export_for_llamaindex("session.jsonl")
+# Returns: [{"text": "message", "metadata": {...}}, ...]
+
+# Use with semantic search services
+for doc in docs:
+    print(f"Text: {doc['text'][:50]}...")
+    print(f"Speaker: {doc['metadata']['speaker']}")
+```
+
 ## 🏗️ Architecture
 
-### Clean Domain Organization (15 modules)
+### Clean Domain Organization (19 modules)
 ```
 claude_parser/
-├── analytics/      # Session and tool analysis
+├── analytics/     # Session and tool analysis
+├── api/           # API utilities
 ├── cli/           # CG and CH commands
+├── core/          # Core utilities
 ├── discovery/     # File and project discovery
-├── filtering/     # Message filtering (NEW!)
+├── export/        # Export formats (NEW in v2.1!)
+├── extensions/    # Extension system
+├── filtering/     # Message filtering
 ├── hooks/         # Hook system and API
 ├── loaders/       # Session loading
-├── messages/      # Message utilities (NEW!)
-├── models/        # Data models (NEW!)
+├── messages/      # Message utilities
+├── models/        # Data models
 ├── navigation/    # Timeline and UUID navigation
 ├── operations/    # File operations
 ├── queries/       # DuckDB SQL queries
 ├── session/       # Session management
 ├── storage/       # DuckDB engine
 ├── tokens/        # Token counting and billing
-└── watch/         # Real-time monitoring (NEW!)
+└── watch/         # Real-time monitoring
 ```
 
 ### LNCA Principles
@@ -229,6 +256,33 @@ twine upload dist/*
 ### Documentation
 Documentation auto-deploys to GitHub Pages on every push to main.
 
+## 🗺️ Export Format Roadmap
+
+### Currently Available (v2.1)
+- ✅ **LlamaIndex** - `export_for_llamaindex()` - For semantic search indexing
+
+### Planned Export Formats
+- 🔜 **Mem0** - Long-term memory for AI agents
+- 🔜 **ChromaDB** - Vector database format
+- 🔜 **Pinecone** - Cloud vector database
+- 🔜 **Markdown** - Human-readable conversation logs
+- 🔜 **JSON-LD** - Structured data with context
+- 🔜 **OpenAI Messages** - Direct OpenAI API format
+- 🔜 **Anthropic Messages** - Direct Anthropic API format
+- 🔜 **LangChain Documents** - LangChain document format
+- 🔜 **Haystack Documents** - Haystack NLP framework
+
+### Export Domain Architecture
+```python
+claude_parser/export/
+├── __init__.py        # Export registry
+├── llamaindex.py      # LlamaIndex format (DONE)
+├── mem0.py           # Mem0 format (TODO)
+├── chroma.py         # ChromaDB format (TODO)
+├── markdown.py       # Markdown format (TODO)
+└── ...               # More formats
+```
+
 ## 🤝 Contributing
 
 We welcome contributions! Please ensure:
@@ -249,18 +303,18 @@ MIT License - See [LICENSE](LICENSE) file for details.
 
 ## 📊 Stats
 
-- **15** specialized domains
-- **30+** public functions
+- **19** specialized domains
+- **35+** public functions
 - **<80** lines per file
 - **100%** framework delegation
 - **0** custom error handling
 
 ---
 
-**Ready to never lose code again?** Install v2.0.0 and experience the power of Git-like recovery for Claude Code!
+**Ready to never lose code again?** Install v2.1.0 and experience the power of Git-like recovery for Claude Code!
 
 ```bash
-pip install claude-parser==2.0.0
+pip install claude-parser==2.1.0
 ```
 
 [Documentation](https://alicoding.github.io/claude-parser/) | [GitHub](https://github.com/alicoding/claude-parser) | [PyPI](https://pypi.org/project/claude-parser/)

claude_parser/__init__.py

@@ -6,17 +6,19 @@
 
 # LNCA Core API - 100% Framework Delegation
 from .main import load_session, load_latest_session, discover_all_sessions
-from .analytics import analyze_session, analyze_project_contexts, analyze_tool_usage  
+from .analytics import analyze_session, analyze_project_contexts, analyze_tool_usage
 from .discovery import discover_claude_files, group_by_projects, analyze_project_structure, discover_current_project_files
 from .operations import restore_file_content, generate_file_diff, compare_files, backup_file
 from .navigation import find_message_by_uuid, get_message_sequence, get_timeline_summary
 from .tokens import count_tokens, analyze_token_usage, estimate_cost, token_status
 from .tokens.context import calculate_context_window
 from .tokens.billing import calculate_session_cost
 from .session import SessionManager
+from .export import export_for_llamaindex
+from .filtering import filter_messages_by_type, filter_messages_by_tool, search_messages_by_content, exclude_tool_operations
 
 # Version info
-__version__ = "2.0.1"
+__version__ = "2.1.0"
 
 # Message types for filtering
 class MessageType:
@@ -41,13 +43,14 @@ def find_current_transcript():
 
 # Clean exports - API only
 __all__ = [
-    'load_session', 'load_latest_session', 'discover_all_sessions', 
+    'load_session', 'load_latest_session', 'discover_all_sessions',
     'analyze_session', 'analyze_project_contexts', 'analyze_tool_usage',
     'discover_claude_files', 'group_by_projects', 'analyze_project_structure', 'discover_current_project_files',
     'restore_file_content', 'generate_file_diff', 'compare_files', 'backup_file',
     'find_message_by_uuid', 'get_message_sequence', 'get_timeline_summary',
     'count_tokens', 'analyze_token_usage', 'estimate_cost', 'token_status',
     'calculate_context_window', 'calculate_session_cost',
-    'load_many', 'find_current_transcript', 
+    'filter_messages_by_type', 'filter_messages_by_tool', 'search_messages_by_content', 'exclude_tool_operations',
+    'load_many', 'find_current_transcript', 'export_for_llamaindex',
     'MessageType', '__version__'
 ]

claude_parser/discovery/core.py

@@ -42,12 +42,20 @@ def discover_claude_files(search_path: str = None) -> List[Path]:
             for pattern in ["*.jsonl", "*.claude", "*.transcript"]:
                 files.extend(search_dir.rglob(pattern))
             return sorted(files, key=lambda p: p.stat().st_mtime, reverse=True)
+        else:
+            # Non-existent path returns empty list
+            return []
 
-    # Default: discover all sessions
+    # Default: discover all sessions and extract file paths
     sessions = discover_all_sessions()
-    # Extract file paths from session data
-    # Sessions are dicts with metadata, need to extract paths
-    return []  # Simplified for now
+    # Extract transcript_path from each session's metadata
+    paths = []
+    for session in sessions:
+        if session and 'metadata' in session:
+            transcript_path = session['metadata'].get('transcript_path')
+            if transcript_path:
+                paths.append(Path(transcript_path))
+    return paths
 
 
 def group_by_projects(files: List[Path]) -> Dict[Path, List[Path]]:

claude_parser/export/__init__.py

@@ -0,0 +1,9 @@
+"""
+Export domain - 100% framework delegation
+@FRAMEWORK_FIRST: Only use existing functions
+@ZERO_CUSTOM_CODE: No loops, no manual parsing
+"""
+
+from .llamaindex import export_for_llamaindex
+
+__all__ = ['export_for_llamaindex']

claude_parser/export/llamaindex.py

@@ -0,0 +1,74 @@
+"""
+LlamaIndex export format - 100% framework delegation
+@ZERO_CUSTOM_CODE: No loops, only map/filter
+@FRAMEWORK_FIRST: Reuse existing functions
+@DRY_FIRST: Don't duplicate text extraction logic
+@LOC_ENFORCEMENT: <80 lines
+"""
+
+from typing import List, Dict, Any, Union, Iterator
+from operator import methodcaller
+from functools import partial
+from more_itertools import chunked
+
+from ..main import load_session
+from ..filtering import filter_pure_conversation
+from ..messages.utils import get_text
+
+
+def _extract_document(msg: Dict[str, Any]) -> Dict[str, Any]:
+    """Transform message to LlamaIndex document format
+
+    @SEMANTIC_INTERFACE: Returns simple text + metadata
+    @NO_IMPLEMENTATION_EXPOSURE: Hides message complexity
+    """
+    return {
+        'text': get_text(msg),  # Reuse existing text extractor
+        'metadata': {
+            'speaker': msg.get('type', 'unknown'),
+            'uuid': msg.get('uuid', ''),
+            'timestamp': msg.get('timestamp', ''),
+            'session_id': msg.get('sessionId', '')
+        }
+    }
+
+
+def export_for_llamaindex(jsonl_path: str, batch_size: int = None) -> Union[List[Dict[str, Any]], Iterator[List[Dict[str, Any]]]]:
+    """Export conversation for LlamaIndex indexing with optional batching
+
+    @API_FIRST: Public interface for semantic-search
+    @FRAMEWORK_FIRST: 100% delegation to existing functions
+    @ZERO_CUSTOM_CODE: No manual loops, only map and chunked
+
+    Args:
+        jsonl_path: Path to JSONL conversation file
+        batch_size: Optional batch size for memory-efficient processing
+
+    Returns:
+        If batch_size=None: List of all documents
+        If batch_size>0: Iterator yielding batches of documents
+        Each document has:
+        - text: Plain text content
+        - metadata: speaker, uuid, timestamp, session_id
+    """
+    # Load using existing SDK
+    session = load_session(jsonl_path)
+    if not session:
+        return [] if not batch_size else iter([])
+
+    # Get messages
+    messages = session.get('messages', [])
+
+    # Filter using existing function (returns generator)
+    clean_messages = filter_pure_conversation(messages)
+
+    # Transform using map (no custom loops!)
+    documents = map(_extract_document, clean_messages)
+
+    # Return batched or full list based on batch_size
+    if batch_size:
+        # more-itertools handles all batching logic!
+        return chunked(documents, batch_size)
+    else:
+        # Current behavior - return full list
+        return list(documents)

docs/api.md

@@ -1,23 +1,9 @@
-To generate API documentation for the Claude Parser project, you can follow these steps:
+To generate API documentation, you can follow these steps:
 
-1. **Documentation Tool**: Choose a documentation tool like Sphinx, MkDocs, or similar that suits your project requirements.
+1. Document each module, function, and class in your codebase using docstrings that describe their purpose, parameters, and return values.
+2. Utilize tools like Sphinx or MkDocs to automatically generate documentation from the docstrings.
+3. Organize the documentation into sections based on the different components of your project, such as hooks, tokens, CLI commands, etc.
+4. Include examples and usage scenarios to help users understand how to interact with the API.
+5. Provide information on any configuration settings, environment variables, or dependencies required to use the API.
 
-2. **Setup Documentation**: Initialize the documentation tool in the project directory and configure it to generate documentation from the source code.
-
-3. **Documenting Modules**: Use the tool to automatically generate documentation from the Python source files in the project. This will include information about modules, classes, functions, and their docstrings.
-
-4. **Include Descriptions**: Ensure that the generated documentation includes descriptions of modules, classes, functions, and their parameters. This will help users understand the purpose and usage of each component.
-
-5. **API Endpoints**: If the project includes API endpoints, document them with details on request/response formats, parameters, and expected outputs.
-
-6. **Hooks and Handlers**: Document the hooks and handlers used in the project, including their purpose, input parameters, and expected behavior.
-
-7. **Settings and Configurations**: Include documentation for settings and configurations used in the project, such as API keys, database URLs, and other important settings.
-
-8. **Token Analysis**: Document the token analysis functions, including details on estimating costs, analyzing token usage, and status tracking.
-
-9. **Analytics and Tools**: Provide documentation for the analytics functions, tool usage analysis, and any other data analysis operations in the project.
-
-10. **CLI Commands**: Document the CLI commands available in the project, including their usage, options, and expected outcomes.
-
-By following these steps and ensuring comprehensive documentation across all aspects of the project, you can generate detailed API documentation that will be helpful for users and developers interacting with the Claude Parser project.
+Additionally, you can install Sphinx using pip, navigate to the root directory of your project, initialize Sphinx, modify the `conf.py` file to include paths to your Python modules, write docstrings in reStructuredText format, use Sphinx directives like `autodoc` to generate documentation, run the Sphinx build command, and find the generated HTML documentation in the specified output directory. By customizing the generated documentation with additional details as needed, you can create comprehensive API documentation for your Python project.

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "claude-parser"
-version = "2.0.1"
+version = "2.1.0"
 description = "Parse and analyze Claude Code JSONL exports"
 authors = ["Your Name <you@example.com>"]
 readme = "README.md"