feat: Add TOON format, relevance filtering, and token budget management

Major Features:
- TOON format as default output (30-60% token reduction)
  - Token-Oriented Object Notation inspired by johannschopplich/toon
  - Compact tabular arrays and inline dependencies
  - Auto-detection from .toon file extension

- Relevance filtering with multi-factor scoring
  - Prioritize files by keywords (--relevant flag)
  - Filename (10x), directory (5x), imports (3x), content (1x) weights
  - Smart file prioritization based on entry points and depth

- Token budget management (--max-tokens flag)
  - Limit output to stay within AI model context windows
  - Automatic exclusion of lower-priority files
  - Detailed exclusion summary with token breakdown

- Format auto-detection
  - Detect format from output file extension (.toon, .md, .xml)
  - Conflict warnings when flag and extension mismatch

- Enhanced output display
  - Show "included/total" file counts when budget applied
  - Display full project tokens vs included tokens
  - List first 5 excluded files with token counts
  - Filter directory tree to show only included files

Documentation:
- Updated README with new features and examples
- Created relevance-filtering.md documentation
- Updated output-formats.md with TOON format details
- Updated getting-started.md with new CLI examples
- Updated index.md with feature highlights

Technical:
- New internal/relevance package for scoring
- New internal/format/toon.go encoder
- Enhanced processor with file prioritization
- Comprehensive test coverage for all new features

Author: 1broseidon

README.md

@@ -11,14 +11,18 @@ promptext is a code context extraction tool designed for AI assistant interactio
 
 ## Key Features
 
-- Smart file filtering with .gitignore support and intelligent defaults
-- Accurate token counting using tiktoken (GPT-3.5/4 compatible)
-- Comprehensive project analysis (entry points, configs, core files, tests, docs)
-- Multiple output formats (Markdown, XML)
-- Configurable via CLI flags or configuration files
-- Project metadata extraction (language, version, dependencies)
-- Git repository information extraction
-- Performance monitoring and debug logging
+- **TOON Format Output** - Default token-optimized format (30-60% smaller than JSON/Markdown), inspired by [johannschopplich/toon](https://github.com/johannschopplich/toon)
+- **Smart Relevance Filtering** - Multi-factor scoring prioritizes files by keywords (filename, directory, imports, content)
+- **Token Budget Management** - Limit output to specific token count, automatically excluding lower-priority files
+- **Format Auto-Detection** - Automatically detects output format from file extension (.toon, .md, .xml)
+- **Smart File Filtering** - .gitignore support and intelligent defaults
+- **Accurate Token Counting** - Using tiktoken (GPT-3.5/4 compatible)
+- **Comprehensive Project Analysis** - Entry points, configs, core files, tests, docs
+- **Multiple Output Formats** - TOON (default), Markdown, XML
+- **Flexible Configuration** - CLI flags or configuration files
+- **Project Metadata Extraction** - Language, version, dependencies
+- **Git Repository Information** - Branch, commit, message
+- **Performance Monitoring** - Debug logging and timing analysis
 
 ## Install
 
@@ -45,7 +49,7 @@ See our [documentation](https://1broseidon.github.io/promptext/) for more instal
 ## Basic Usage
 
 ```bash
-# Process current directory (output copied to clipboard)
+# Process current directory (TOON format copied to clipboard)
 prx
 
 # Process specific directory with positional argument
@@ -57,8 +61,13 @@ prx -e .go,.js,.ts
 # Show project summary only
 prx -i
 
-# Export as XML to file
-prx -f xml -o project.xml
+# Auto-detect format from file extension
+prx -o context.toon     # TOON format
+prx -o context.md       # Markdown format
+prx -o project.xml      # XML format
+
+# Explicit format specification
+prx -f markdown -o context.md
 
 # Process with custom exclusions and view output in terminal
 prx -x "test/,vendor/" --verbose
@@ -70,6 +79,58 @@ prx --dry-run -e .go
 prx -q -o output.md
 ```
 
+## Advanced Features
+
+### Relevance Filtering
+
+Prioritize files matching specific keywords using multi-factor scoring:
+
+```bash
+# Prioritize authentication-related files
+prx --relevant "auth login OAuth"
+prx -r "database SQL postgres"
+
+# Multi-factor scoring weights:
+# - Filename matches: 10x
+# - Directory matches: 5x
+# - Import matches: 3x
+# - Content matches: 1x
+```
+
+### Token Budget Management
+
+Limit output to stay within token limits for AI models:
+
+```bash
+# Limit to 8000 tokens (fits Claude Haiku context)
+prx --max-tokens 8000
+
+# Combine with relevance to prioritize important files
+prx -r "api routes handlers" --max-tokens 5000
+
+# Cost-optimized queries
+prx --max-tokens 3000 -o quick-context.toon
+```
+
+When the budget is exceeded, promptext:
+- Shows which files were included vs excluded
+- Displays token breakdown for excluded files
+- Filters directory tree to show only included files
+
+```
+╭───────────────────────────────────────────────╮
+│ 📦 promptext (Go)                             │
+│    Included: 7/18 files • ~4,847 tokens       │
+│    Full project: 18 files • ~19,512 tokens    │
+╰───────────────────────────────────────────────╯
+
+⚠️  Excluded 11 files due to token budget:
+    • internal/cli/commands.go (~784 tokens)
+    • internal/app/app.go (~60 tokens)
+    ... and 9 more files (~8,453 tokens)
+    Total excluded: ~9,297 tokens
+```
+
 ## Configuration
 
 Configuration is loaded with the following precedence (highest to lowest):
@@ -90,7 +151,7 @@ excludes:
   - vendor/
   - node_modules/
   - "*.test.go"
-format: markdown
+format: toon        # Options: toon, markdown, xml
 verbose: false
 ```
 
@@ -106,7 +167,7 @@ extensions:
 excludes:
   - vendor/
   - __pycache__/
-format: markdown
+format: toon
 ```
 
 ## Documentation
@@ -115,10 +176,12 @@ Visit our [documentation site](https://1broseidon.github.io/promptext/) for comp
 
 - Getting Started Guide
 - Configuration Options
-- File Filtering Rules  
-- Token Analysis
+- File Filtering Rules
+- **Relevance Filtering** - Smart file prioritization
+- **Token Budget Management** - Optimize for AI model context windows
+- Token Analysis & Counting
 - Project Analysis Features
-- Output Format Specifications
+- Output Format Specifications (TOON, Markdown, XML)
 - Performance Tips
 
 ## License

cmd/promptext/main.go

@@ -4,6 +4,8 @@ import (
 	"fmt"
 	"log"
 	"os"
+	"path/filepath"
+	"strings"
 
 	"github.com/1broseidon/promptext/internal/processor"
 	"github.com/spf13/pflag"
@@ -43,7 +45,7 @@ FILTERING OPTIONS:
                               Examples: vendor/,node_modules/  or  *.test.go,dist/
 
 OUTPUT OPTIONS:
-    -f, --format FORMAT       Output format: markdown, md, xml (default: markdown)
+    -f, --format FORMAT       Output format: toon, markdown, md, xml (default: toon for AI-optimized structure)
     -o, --output FILE         Write output to file instead of clipboard
     -n, --no-copy            Don't copy output to clipboard
     -i, --info               Show only project summary (no file contents)
@@ -53,6 +55,12 @@ PROCESSING OPTIONS:
         --dry-run            Preview files that would be processed without reading content
     -q, --quiet              Suppress non-essential output for scripting
 
+RELEVANCE & TOKEN BUDGET:
+    -r, --relevant KEYWORDS  Keywords to prioritize files (comma or space separated)
+                             Uses multi-factor scoring: filename (10x), directory (5x), imports (3x), content (1x)
+        --max-tokens NUMBER  Maximum token budget for output (excludes lower-priority files when exceeded)
+                             Works best with --relevant to prioritize important files first
+
 DEBUG OPTIONS:
     -D, --debug              Enable debug logging and timing information
     -h, --help               Show this help message
@@ -71,6 +79,9 @@ EXAMPLES:
     # Export specific file types to XML with debug info
     prx -e .js,.ts,.json -f xml -o project.xml -D
 
+    # Use TOON format for AI-optimized structure (better scannability)
+    prx -f toon -o project.toon
+
     # Process with custom exclusions and see output in terminal
     prx -x "vendor/,*.test.go,dist/" -v
 
@@ -86,6 +97,19 @@ EXAMPLES:
     # Quiet mode for use in scripts (minimal output)
     prx -q -f xml -o output.xml
 
+    # Auto-detect format from output file extension
+    prx -o context.toon                    # Automatically uses TOON format
+    prx -o context.md                      # Automatically uses markdown format
+
+    # Prioritize authentication-related files
+    prx --relevant "auth login OAuth"
+
+    # Limit output to 8000 tokens, prioritizing database files
+    prx --relevant "database" --max-tokens 8000
+
+    # Combined: relevant files with token budget
+    prx -r "api routes handlers" --max-tokens 5000 -o api-context.toon
+
 CONFIGURATION:
     Create a .promptext.yml file in your project root for persistent settings:
 
@@ -96,7 +120,7 @@ CONFIGURATION:
     excludes:
       - vendor/
       - node_modules/
-    format: markdown
+    format: toon
     verbose: false
 
     CLI flags override configuration file settings.
@@ -131,7 +155,7 @@ func main() {
 	exclude := pflag.StringP("exclude", "x", "", "Patterns to exclude (comma-separated, e.g., vendor/,*.test.go)")
 
 	// Output options
-	format := pflag.StringP("format", "f", "markdown", "Output format: markdown, md, or xml")
+	format := pflag.StringP("format", "f", "toon", "Output format: toon, markdown, md, or xml (default: toon)")
 	outFile := pflag.StringP("output", "o", "", "Write output to file instead of clipboard")
 	noCopy := pflag.BoolP("no-copy", "n", false, "Don't copy output to clipboard")
 	infoOnly := pflag.BoolP("info", "i", false, "Show only project summary without file contents")
@@ -141,6 +165,11 @@ func main() {
 	dryRun := pflag.Bool("dry-run", false, "Preview files that would be processed without reading content")
 	quiet := pflag.BoolP("quiet", "q", false, "Suppress non-essential output for scripting")
 
+	// Relevance and token budget options
+	relevant := pflag.StringP("relevant", "r", "", "Keywords to prioritize files (comma or space separated, multi-factor scoring)")
+	maxTokens := pflag.Int("max-tokens", 0, "Maximum token budget for output (excludes lower-priority files when exceeded)")
+	explainSelection := pflag.Bool("explain-selection", false, "Show detailed priority scoring breakdown for file selection")
+
 	// Debug options
 	debug := pflag.BoolP("debug", "D", false, "Enable debug logging and timing information")
 
@@ -162,7 +191,34 @@ func main() {
 		*dirPath = args[0]
 	}
 
-	if err := processor.Run(*dirPath, *extension, *exclude, *noCopy, *infoOnly, *verbose, *format, *outFile, *debug, *gitignore, *useDefaultRules, *dryRun, *quiet); err != nil {
+	// Format auto-detection from output file extension
+	if *outFile != "" {
+		ext := strings.ToLower(filepath.Ext(*outFile))
+		detectedFormat := ""
+		switch ext {
+		case ".toon":
+			detectedFormat = "toon"
+		case ".md", ".markdown":
+			detectedFormat = "markdown"
+		case ".xml":
+			detectedFormat = "xml"
+		}
+
+		// Check for format conflict and warn
+		if detectedFormat != "" && *format != detectedFormat {
+			// User explicitly set format flag
+			formatFlag := pflag.Lookup("format")
+			if formatFlag.Changed {
+				// Warn about conflict
+				fmt.Fprintf(os.Stderr, "⚠️  Warning: format flag '%s' conflicts with output extension '%s' - using '%s' (flag takes precedence)\n", *format, ext, *format)
+			} else {
+				// Auto-detect format from extension since flag wasn't explicitly set
+				*format = detectedFormat
+			}
+		}
+	}
+
+	if err := processor.Run(*dirPath, *extension, *exclude, *noCopy, *infoOnly, *verbose, *format, *outFile, *debug, *gitignore, *useDefaultRules, *dryRun, *quiet, *relevant, *maxTokens, *explainSelection); err != nil {
 		log.Fatal(err)
 	}
 }

docs-astro/src/content/docs/getting-started.md

@@ -34,17 +34,22 @@ Download binaries from [GitHub Releases](https://github.com/1broseidon/promptext
 ### Simple Commands
 
 ```bash
-# Process current directory
+# Process current directory (TOON format to clipboard)
 promptext
 
+# Use alias for convenience
+prx
+
 # Process specific directory
 promptext -d /path/to/project
 
 # Show project overview only
-promptext -info
+promptext -i
 
-# Export to file
-promptext -o output.md
+# Export to file (format auto-detected from extension)
+promptext -o context.toon
+promptext -o context.md
+promptext -o project.xml
 ```
 
 ### Common Options
@@ -54,10 +59,13 @@ promptext -o output.md
 | `-d` | Directory to process |
 | `-e` | File extensions (`.go,.js`) |
 | `-x` | Exclude patterns |
-| `-f` | Format (`markdown`, `xml`) |
-| `-o` | Output file |
+| `-f` | Format (`toon`, `markdown`, `xml`) |
+| `-o` | Output file (auto-detects format) |
 | `-i` | Info mode only |
+| `-r` | Relevant keywords for prioritization |
+| `--max-tokens` | Token budget limit |
 | `-v` | Verbose output |
+| `-q` | Quiet mode for scripting |
 
 ### Examples
 
@@ -71,9 +79,74 @@ promptext -e .go,.js,.ts
 promptext -x "node_modules/,vendor/,test/"
 ```
 
-**Generate XML report:**
+**Generate reports:**
 ```bash
+# TOON format (default, token-optimized)
+promptext -o context.toon
+
+# Markdown format
+promptext -f markdown -o context.md
+
+# XML format for automation
 promptext -f xml -o report.xml
 ```
 
-Continue with [Configuration](configuration) to customize behavior.
+**Prioritize relevant files:**
+```bash
+# Focus on authentication code
+promptext -r "auth login OAuth"
+
+# Database-related files
+promptext -r "database SQL migration"
+```
+
+**Stay within token budgets:**
+```bash
+# Limit to 8000 tokens (Claude Haiku)
+promptext --max-tokens 8000
+
+# Combine with relevance for smart selection
+promptext -r "api routes" --max-tokens 5000
+```
+
+## Quick Workflows
+
+### For AI Queries
+
+```bash
+# Quick context (3k tokens)
+prx -r "auth" --max-tokens 3000
+
+# Standard context (8k tokens)
+prx -r "api database" --max-tokens 8000
+
+# Full codebase (within limits)
+prx --max-tokens 50000
+```
+
+### For Documentation
+
+```bash
+# Export project overview
+prx -i -o overview.md
+
+# Export full context in markdown
+prx -f markdown -o full-context.md
+```
+
+### For CI/CD
+
+```bash
+# Machine-readable XML
+prx -f xml -o build/context.xml
+
+# Quiet mode for scripting
+prx -q -o context.toon
+```
+
+## Next Steps
+
+- [Output Formats](output-formats) - TOON, Markdown, and XML formats
+- [Relevance Filtering](relevance-filtering) - Smart file prioritization
+- [Configuration](configuration) - Customize behavior
+- [File Filtering](file-filtering) - Advanced filtering rules