madengine v2 with unified framework for local and distribution

.gitignore

@@ -6,6 +6,22 @@ __pycache__/
 # C extensions
 *.so
 
+# OS generated files
+.DS_Store
+.DS_Store?
+._*
+.Spotlight-V100
+.Trashes
+ehthumbs.db
+Thumbs.db
+
+# IDE files
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
 # Distribution / packaging
 .Python
 build/
@@ -36,7 +52,7 @@ MANIFEST
 pip-log.txt
 pip-delete-this-directory.txt
 
-# Unit test / coverage reports
+# Testing and coverage
 htmlcov/
 .tox/
 .nox/
@@ -49,6 +65,23 @@ coverage.xml
 *.py,cover
 .hypothesis/
 .pytest_cache/
+
+# MADEngine specific
+credential.json
+data.json
+*.log
+*.csv
+*.html
+library_trace.csv
+library_perf.csv
+perf.csv
+perf.html
+
+# Temporary and build files
+temp/
+tmp/
+*.tmp
+.pytest_cache/
 cover/
 
 # Translations
@@ -101,4 +134,10 @@ scripts/
 .*_env/
 .vscode/
 
-tmp/
+build_manifest.json
+tmp/
+k8s_manifests/
+k8s_results/
+rocprof_output/
+slurm_output/
+MagicMock/

.pre-commit-config.yaml

@@ -0,0 +1,36 @@
+# Pre-commit hooks configuration
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.4.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-json
+      - id: check-toml
+      - id: check-added-large-files
+      - id: check-merge-conflict
+      - id: debug-statements
+
+  - repo: https://github.com/psf/black
+    rev: 23.3.0
+    hooks:
+      - id: black
+        language_version: python3
+
+  - repo: https://github.com/pycqa/isort
+    rev: 5.12.0
+    hooks:
+      - id: isort
+
+  - repo: https://github.com/pycqa/flake8
+    rev: 6.0.0
+    hooks:
+      - id: flake8
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.3.0
+    hooks:
+      - id: mypy
+        additional_dependencies: [types-requests, types-PyYAML]
+        exclude: ^(tests/|scripts/)

CHANGELOG.md

@@ -0,0 +1,116 @@
+# Changelog
+
+All notable changes to madengine will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+### Fixed
+- **Error Pattern Detection**: Fixed false failure detection in HuggingFace GPT2/BERT models
+  - ROCProf logging messages (E20251230/W20251230 prefixes) no longer trigger false failures
+  - Added benign pattern list to exclude profiling tool output from error detection
+  - Made error patterns more specific (e.g., `RuntimeError:` instead of `Error:`)
+  - Improved performance metric extraction robustness to prevent bash segfaults during profiling
+  - Tests: Added `TestErrorPatternMatching` class in `tests/unit/test_error_handling.py`
+- Removed stale compiled Python file (`__init__.pyc`) from source tree
+- Cleaned up unused `typing_extensions` import in `core/console.py`
+- Improved type hint accuracy in `Console.sh()` method docstring
+
+### Breaking Changes
+- **CLI Unification**: Simplified command-line interface
+  - ✅ `madengine` is now the unified CLI command (previously `madengine-cli`)
+  - ❌ Removed legacy `madengine` v1.x CLI (previously `mad.py`)
+  - ❌ Removed `madengine-cli` alias (use `madengine` instead)
+  - **Migration**: Simply replace `madengine-cli` with `madengine` in your scripts
+  - All functionality remains identical, just cleaner command naming
+
+### Removed
+- **Legacy CLI Components**:
+  - `src/madengine/mad.py` - Legacy CLI entry point (v1.x)
+  - `src/madengine/tools/run_models.py` - Legacy model runner
+  - `docs/legacy-cli.md` - Legacy CLI documentation
+- Justification: Modern `madengine` CLI (formerly `madengine-cli`) provides all functionality plus K8s, SLURM, and distributed support
+
+### Security
+- **CRITICAL:** Fixed SQL injection vulnerability in legacy database module (`src/madengine/db/database_functions.py`)
+  - Replaced string formatting with parameterized queries using SQLAlchemy `text()`
+  - Prevents potential SQL injection attacks in `get_matching_db_entries()` function
+- Fixed 4 instances of bare `except:` blocks that could mask critical exceptions
+  - `kubernetes.py`: Replaced with specific exception types (`ConfigException`, `FileNotFoundError`, `ApiException`)
+  - `console.py`: Replaced with specific exception types (`OSError`, `ValueError`) for resource cleanup
+
+### Added
+- **Comprehensive Launcher Support**: Full K8s and SLURM support for 6 distributed frameworks
+  - TorchTitan: LLM pre-training with FSDP2+TP+PP+CP parallelism
+  - vLLM: High-throughput LLM inference with continuous batching
+  - SGLang: Fast LLM inference with structured generation
+  - DeepSpeed: ZeRO optimization training (K8s support added)
+  - Megatron-LM: Large-scale transformer training (K8s + SLURM)
+  - torchrun: Standard PyTorch DDP/FSDP
+- **Centralized Launcher Documentation**: `docs/distributed-launchers.md` with comprehensive guide
+- **Example Configurations**: 6 new minimal configs for distributed launchers (K8s)
+- Comprehensive development tooling and configuration
+- Pre-commit hooks for code quality
+- Makefile for common development tasks
+- Developer guide with coding standards
+- Type checking with mypy
+- Code formatting with black and isort
+- Enhanced .gitignore for better file exclusions
+- CI/CD configuration templates
+- **Major Documentation Refactor**: Complete integration of distributed execution and CLI guides into README.md
+- Professional open-source project structure with badges and table of contents
+- Comprehensive MAD package integration documentation
+- Enhanced model discovery and tag system documentation
+- Modern deployment scenarios and configuration examples
+
+### Changed
+- **README.md**: Added launcher ecosystem highlights to v2.0 features
+- **K8s README**: Updated with new launcher configs and comprehensive launcher section
+- **Documentation Structure**: Consolidated all launcher docs into single comprehensive guide
+- Improved package initialization and imports
+- Replaced print statements with proper logging in main CLI
+- Enhanced error handling and logging throughout codebase
+- Cleaned up setup.py for better maintainability
+- Updated development dependencies in pyproject.toml
+- **Complete README.md overhaul**: Merged all documentation into a single, comprehensive source
+- Restructured documentation to emphasize MAD package integration
+- Enhanced CLI usage examples and distributed execution workflows
+- Improved developer contribution guidelines and legacy compatibility notes
+
+### Changed (Previous)
+- Removed Python cache files from repository
+- Fixed import organization and structure
+- Improved docstring formatting and consistency
+- Cleaned up documentation fragmentation
+
+### Removed
+- Unnecessary debug print statements
+- Python cache files and build artifacts
+- **Legacy documentation files**: `docs/distributed-execution-solution.md` and `docs/madengine-cli-guide.md`
+- **Duplicate documentation**: `docs/TORCHTITAN_LAUNCHER.md` (consolidated into distributed-launchers.md)
+- Redundant documentation scattered across multiple files
+
+## [Previous Versions]
+
+For changes in previous versions, please refer to the git history.
+
+---
+
+## Guidelines for Changelog Updates
+
+### Categories
+- **Added** for new features
+- **Changed** for changes in existing functionality
+- **Deprecated** for soon-to-be removed features
+- **Removed** for now removed features
+- **Fixed** for any bug fixes
+- **Security** for vulnerability fixes
+
+### Format
+- Keep entries brief but descriptive
+- Include ticket/issue numbers when applicable
+- Group related changes together
+- Use present tense ("Add feature" not "Added feature")
+- Target audience: users and developers of the project