🚀 RAFT Toolkit v0.0.1

🚀 Added

Analysis Tools Suite

• 🛠️ Six comprehensive evaluation tools integrated into web interface
  • Dataset Evaluation: Model performance analysis with configurable metrics
  • Answer Generation: High-quality answer generation using various LLMs
  • PromptFlow Analysis: Multi-dimensional evaluation (relevance, groundedness, fluency, coherence)
  • Dataset Analysis: Statistical analysis and quality metrics
  • Model Comparison: Side-by-side performance comparison
  • Batch Processing: Automated workflows for multiple datasets

Enhanced Web Interface

• Analysis Tools Tab: Complete tool integration with visual interface
• Job Management: Real-time monitoring with progress indicators
• Results Visualization: Comprehensive display of metrics and statistics
• File Upload: Enhanced drag-and-drop with validation
• Download Capabilities: Direct download of analysis results

Comprehensive Documentation

• 📚 Complete documentation overhaul:
  • RAFT methodology explanation with pros/cons vs traditional RAG
  • Fine-tuning process documentation with best practices
  • Chunking strategies guide with document-type recommendations, overlap guidance, and configuration examples
  • Web Interface Guide (docs/WEB_INTERFACE.md)
  • Deployment Guide (docs/DEPLOYMENT.md) with cloud platform instructions
  • Configuration Reference (docs/CONFIGURATION.md)
  • Enhanced tools documentation (tools/README.md)

Advanced Docker & CI/CD

• 🐳 Multi-stage Docker builds:
  • Production-optimized images with security hardening
  • Development images with debugging support
  • Testing images with coverage reporting
  • CLI-only lightweight images
• 🔧 Comprehensive GitHub workflows:
  • Build workflow with linting, security scanning, and multi-platform builds
  • Test workflow with dependency on successful builds
  • Release workflow with dependency on successful tests
  • Security workflow with automated dependency updates

Testing Infrastructure

• 🧪 Complete test suite with multiple categories:
  • Unit tests across Python versions (3.9, 3.10, 3.11)
  • Integration tests with service dependencies
  • API tests with web interface validation
  • CLI tests for command-line functionality
  • Docker-based testing environment
• 📊 Coverage reporting with Codecov integration
• 🔍 Security scanning with Trivy, Bandit, and Safety

🔧 Enhanced

Developer Experience

• Visual Process Flow: Mermaid diagram showing RAFT training process
• Tool Integration: Clear documentation of where each tool fits
• Installation Guides: Step-by-step for all components
• Performance Optimization: Guidelines and benchmarks

Documentation Quality

• Factual Content: Validated information about RAFT methodology
• Best Practices: Comprehensive fine-tuning guidelines
• Decision Framework: When to use RAFT vs traditional RAG
• Examples and Workflows: Complete end-to-end examples

🛡️ Security & Operations

Critical Security Fixes

• 🔒 Cryptographically secure random generation: Replaced random with secrets.SystemRandom for security-sensitive operations
• 🛡️ File upload security: Added comprehensive validation, sanitization, and size limits
• 🚫 Path traversal protection: Implemented secure file path validation and sanitization
• 📁 File permissions: Restrictive permissions on uploaded files and directories (0o600/0o700)
• 🌐 CORS hardening: Restricted origins, methods, and headers; disabled credentials
• 🔐 Security headers: Added comprehensive HTTP security headers (XSS, CSRF, content-type protection)
• ⚡ Input validation: Enhanced subprocess execution with command validation
• 📦 Dependency updates: Updated vulnerable packages (transformers, PyPDF2→pypdf, langchain)

Container Security

• Non-root user execution in all Docker images
• Health checks for all services
• Vulnerability scanning with automated reporting
• SBOM generation for supply chain security

CI/CD Pipeline

• Dependency-based workflows: Tests only run after successful builds
• Automated releases: Only after successful testing
• Security monitoring: Daily scans and automated dependency updates
• Quality gates: Comprehensive checks before deployment

Configuration Management

• Environment-based configuration with validation
• Secret management best practices
• Multi-environment support: Development, testing, production

📦 Dependencies

Critical Dependency Migration

• PromptFlow Evaluation: Migrated from deprecated promptflow.eval to azure-ai-evaluation
  • Updated all evaluation tools to use new Azure AI Evaluation SDK
  • Fixed parameter names: question → query, answer → response
  • Resolved installation failures with missing promptflow.eval package
• Security Updates: Upgraded vulnerable dependencies
  • transformers==4.37.2 → transformers>=4.44.0,<5.0.0 (fixed 7 CVEs)
  • PyPDF2==3.0.1 → pypdf>=4.0.0,<5.0.0 (fixed 1 CVE)
  • langchain-experimental → ==0.3.4 (resolved CVE-2024-46946 completely)
• Final Security Remediation: Addressed remaining vulnerability
  • CVE-2024-46946: Pinned langchain-experimental==0.3.4 (safe version, vulnerability only affects LLMSymbolicMathChain not SemanticChunker)
  • Safety CLI: Updated from deprecated safety check to safety scan command
• Dependency Resolution: Fixed multiple package version conflicts
  • FastAPI Conflict: Updated fastapi==0.104.1 → fastapi>=0.109.0,<1.0.0 (required by promptflow-core 1.18.0)
  • OpenAI: openai==1.30.1 → openai>=1.68.2,<2.0.0 (compatible with langchain-openai)
  • Web Dependencies: Added version bounds to prevent future conflicts (uvicorn, redis, celery)
  • Core Dependencies: Added version constraints to promptflow-core, azure-ai-evaluation, jsonlines
  • Added upper bounds to prevent breaking changes across all dependencies

New Development Dependencies

• Testing: pytest-cov, pytest-asyncio, httpx for API testing
• Security: bandit, safety, semgrep for security scanning
• Code Quality: flake8, black, isort, mypy for linting and formatting
• Documentation: mkdocs, mkdocs-material for documentation generation
• CI/CD: Actions for automated workflows

Updated Core Dependencies

• Docker: Updated base images to latest security patches
• Python: Support for Python 3.9, 3.10, and 3.11
• FastAPI: Latest version with enhanced security features

🔍 Quality Assurance

Code Quality

• Automated linting with flake8, black, and isort
• Security scanning with multiple tools
• License compliance monitoring
• Dependency vulnerability tracking

Testing Coverage

• Multi-platform testing (linux/amd64, linux/arm64)
• Cross-version compatibility testing
• Service integration testing with Redis
• End-to-end workflow validation

📈 Performance

Docker Optimization

• Layer caching for faster builds
• Multi-stage builds for smaller production images
• Platform-specific optimizations

Workflow Efficiency

• Parallel job execution where possible
• Intelligent caching strategies
• Minimal resource usage optimization

🐛 Fixed

Workflow Dependencies

• Strict dependency chain: Build → Test → Release workflow gating
• Test gating: Tests only run after successful builds via workflow_run trigger
• Release gating: Releases only after successful tests with proper status checks
• Pull request support: Tests run directly for PRs while maintaining dependency chain
• Error handling: Comprehensive failure management with clear status messages
• Artifact management: Proper cleanup and retention
• Deprecated actions: Updated to latest versions (upload-artifact@v4, action-gh-release@v1, upload-pages-artifact@v3)

Docker Build Issues

• Missing file references: Removed non-existent run_cli.py from Dockerfile COPY commands
• Correct entry points: CLI uses raft.py, web uses run_web.py, tests use run_tests.py
• Trivy scanner: Fixed multiple image tags issue by using single image reference for vulnerability scanning
• Build resilience: Added error handling and dependency verification in Docker builds
• Workflow robustness: Added fallback scanning and continue-on-error for security transitions
• Linting tools: Added missing flake8, black, isort, mypy to requirements-test.txt
• Code quality config: Added .flake8 and pyproject.toml for consistent formatting and linting
• Docker Compose compatibility: Updated workflows to use docker compose instead of deprecated docker-compose
• Test runner enhancement: Added --output-dir support and improved Python executable detection
• Docker test volumes: Simplified using bind mounts for easier CI/CD result extraction
• Configurable test directories: Added support for custom temp, output, and coverage directories via CLI args and environment variables
  • Added --temp-dir, --coverage-dir parameters to test runner
  • Added TEST_OUTPUT_DIR, TEST_TEMP_DIR, TEST_COVERAGE_DIR environment variable support
  • Added HOST_TEST_RESULTS_DIR, HOST_COVERAGE_DIR, HOST_TEMP_DIR for Docker environments
  • Created .env.test.example and docs/TEST_DIRECTORIES.md for configuration guidance
• Enhanced documentation: Comprehensive updates to README.md and project documentation
  • Added detailed testing and CI/CD integration sections
  • Included deployment guides for Docker, Kubernetes, and cloud platforms
  • Added security best practices and monitoring guidance
  • Updated installation instructions with multi-target Docker builds
  • Enhanced project structure documentation with clear file organization
• Dependency management tools: Created comprehensive dependency verification system
  • Added scripts/check_dependencies.py for automated dependency conflict detection
  • Enhanced Dockerfile with improved dependency resolution error handling
  • Added pip check validation in Docker builds to catch conflicts early
• Dockerfile improvements: Enhanced Docker build process and compliance
  • Fixed FROM/AS casing inconsistencies for Docker linting compliance
  • Added proper metadata labels (maintainer, description, version)
  • Improved apt cache cleanup with apt-get clean and cache removal
  • Created comprehensive .dockerignore file to optimize build context
  • Added scripts/dockerfile_lint.py for automated Dockerfile quality checks

Documentation Issues

• Mermaid diagram: Improved readability with black text on light backgrounds
• Installation clarity: Step-by-step instructions for all components
• Configuration examples: Environment-specific templates
• Broken links: Fixed non-existent deployment docs links, consolidated to existing DEPLOYMENT.md

---

📦 Installation

Docker

docker pull ghcr.io/MakerCorn/raft-toolkit:0.0.1-production

Python Package

pip install raft-toolkit==0.0.1

🔗 Links

• Documentation
• Docker Images
• Full Changelog