Merge remote-tracking branch 'origin/main' into existing-test-pr-comment

Author: aseembits93

.github/workflows/deploy-docs-to-azure.yaml

@@ -11,7 +11,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.9", "3.10", "3.11", "3.12"]
+        python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"]
     continue-on-error: true
     runs-on: ubuntu-latest
     steps:
@@ -30,4 +30,4 @@ jobs:
         run: uv sync
 
       - name: Unit tests
-        run: uv run pytest tests/ --benchmark-skip -m "not ci_skip"
+        run: uv run pytest tests/

.github/workflows/unit-tests.yaml

@@ -1,7 +1,8 @@
 repos:
-  - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: "v0.11.0"
-    hooks:
-      - id: ruff
-        args: [--fix, --exit-non-zero-on-fix, --config=pyproject.toml]
-      - id: ruff-format
+- repo: https://github.com/astral-sh/ruff-pre-commit
+  rev: v0.12.7
+  hooks:
+    # Run the linter.
+    - id: ruff-check
+    # Run the formatter.
+    - id: ruff-format

.pre-commit-config.yaml

@@ -65,6 +65,13 @@ For detailed installation and usage instructions, visit our documentation at [do
 
 https://github.com/user-attachments/assets/38f44f4e-be1c-4f84-8db9-63d5ee3e61e5
 
+- Optiming a workflow end to end automatically with `codeflash optimize`
+
+
+https://github.com/user-attachments/assets/355ba295-eb5a-453a-8968-7fb35c70d16c
+
+
+
 ## Support
 
 Join our community for support and discussions. If you have any questions, feel free to reach out to us using one of the following methods:

README.md

@@ -0,0 +1,19 @@
+# Security Policy
+
+This document outlines Codeflash's vulnerability disclosure policy. For more information about Codeflash's approach to security, please visit [codeflash.ai/security](https://www.codeflash.ai/security).
+
+## Supported Versions
+
+Since Codeflash is moving quickly, we can only commit to fixing security issues for the latest version of codeflash client.
+If a vulnerability is discovered in our backend, we will release the fix for all the users.
+
+## Reporting a Vulnerability
+
+
+Please do not report security vulnerabilities through public GitHub issues.
+
+Instead, please report them to our [GitHub Security page](https://github.com/codeflash-ai/codeflash/security). If you prefer to submit one without using GitHub, you can also email us at [email protected].
+
+We commit to acknowledging vulnerability reports immediately, and will work to fix active vulnerabilities as soon as we can. We will publish resolved vulnerabilities in the form of security advisories on our GitHub security page. Critical incidents will be communicated both on the GitHub security page and via email to all affected users.
+
+We appreciate your help in making Codeflash more secure for everyone. Thank you for your support and responsible disclosure.

SECURITY.md

@@ -0,0 +1,191 @@
+# WARP.md
+
+This file provides guidance to WARP (warp.dev) when working with code in this repository.
+
+## Project Overview
+
+Codeflash is a general-purpose optimizer for Python that helps improve code performance while maintaining correctness. It uses advanced LLMs to generate optimization ideas, tests them for correctness, and benchmarks them for performance, then creates merge-ready pull requests.
+
+## Development Environment Setup
+
+### Prerequisites
+- Python 3.9+ (project uses uv for dependency management)
+- Git (for version control and PR creation)
+- Codeflash API key (for AI services)
+
+### Initial Setup
+```bash
+# Install dependencies using uv (preferred over pip)
+uv sync
+
+# Initialize codeflash configuration
+uv run codeflash init
+```
+
+## Core Development Commands
+
+### Code Quality & Linting
+```bash
+# Format code with ruff (includes check and format)
+uv run ruff check --fix codeflash/
+uv run ruff format codeflash/
+
+# Type checking with mypy
+uv run mypy codeflash/
+
+# Pre-commit hooks (ruff check + format)
+uv run pre-commit run --all-files
+```
+
+### Testing
+```bash
+# Run all tests
+uv run pytest
+
+# Run specific test file
+uv run pytest tests/test_specific_file.py
+
+# Run tests matching pattern
+uv run pytest -k "pattern"
+
+```
+
+### Running Codeflash
+```bash
+# Optimize entire codebase
+uv run codeflash --all
+
+# Optimize specific file
+uv run codeflash --file path/to/file.py
+
+# Optimize specific function
+uv run codeflash --function "module.function"
+
+# Optimize a script end-to-end
+uv run codeflash optimize script.py
+
+# Run with benchmarking
+uv run codeflash --benchmark
+
+# Verify setup
+uv run codeflash --verify-setup
+```
+
+## Architecture Overview
+
+### Main Components
+
+**Core Modules:**
+- `codeflash/main.py` - CLI entry point and command coordination
+- `codeflash/cli_cmds/` - Command-line interface implementations
+- `codeflash/optimization/` - Core optimization engine and algorithms
+- `codeflash/verification/` - Code correctness verification
+- `codeflash/benchmarking/` - Performance measurement and comparison
+- `codeflash/discovery/` - Code analysis and function discovery
+- `codeflash/tracing/` - Runtime tracing and profiling
+- `codeflash/context/` - Code context extraction and analysis
+- `codeflash/result/` - Result processing, PR creation, and explanations
+
+**Supporting Systems:**
+- `codeflash/api/` - Backend API communication
+- `codeflash/github/` - GitHub integration for PR creation
+- `codeflash/models/` - Data models and schemas
+- `codeflash/telemetry/` - Analytics and error reporting
+- `codeflash/code_utils/` - Code parsing, formatting, and manipulation utilities
+
+### Key Workflows
+
+1. **Code Discovery**: Analyzes codebase to identify optimization candidates
+2. **Context Extraction**: Extracts relevant code context and dependencies
+3. **Optimization Generation**: Uses LLMs to generate optimization candidates
+4. **Verification**: Tests optimizations for correctness using existing tests
+5. **Benchmarking**: Measures performance improvements
+6. **Result Processing**: Creates explanations and pull requests
+
+### Configuration
+
+Configuration is stored in `pyproject.toml` under `[tool.codeflash]`:
+- `module-root` - Source code location (default: "codeflash")
+- `tests-root` - Test location (default: "tests") 
+- `benchmarks-root` - Benchmark location (default: "tests/benchmarks")
+- `test-framework` - Testing framework ("pytest" or "unittest")
+- `formatter-cmds` - Commands for code formatting
+
+## Project Structure
+
+```
+codeflash/
+├── api/                 # Backend API communication
+├── benchmarking/        # Performance measurement
+├── cli_cmds/           # CLI command implementations
+├── code_utils/         # Code analysis and manipulation
+├── context/            # Code context extraction
+├── discovery/          # Function and test discovery  
+├── github/             # GitHub API integration
+├── lsp/                # Language server protocol support
+├── models/             # Data models and schemas
+├── optimization/       # Core optimization engine
+├── result/             # Result processing and PR creation
+├── telemetry/          # Analytics and monitoring
+├── tracing/            # Runtime tracing and profiling
+├── verification/       # Correctness verification
+└── main.py            # CLI entry point
+
+tests/                  # Test suite
+├── benchmarks/         # Performance benchmarks
+└── scripts/           # Test utilities
+
+docs/                   # Documentation
+code_to_optimize/       # Example code for optimization
+codeflash-benchmark/    # Benchmark workspace member
+```
+
+## Development Notes
+
+### Code Style
+- Uses ruff for linting and formatting (configured in pyproject.toml)
+- Strict mypy type checking enabled
+- Pre-commit hooks enforce code quality
+
+### Testing
+- pytest-based test suite with extensive coverage
+- Parameterized tests for multiple scenarios
+- Benchmarking tests for performance validation
+- Test discovery supports both pytest and unittest frameworks
+
+### Workspace Structure
+- Uses uv workspace with `codeflash-benchmark` as a member
+- Dependencies managed through uv.lock
+- Dynamic versioning from git tags using uv-dynamic-versioning
+
+### Build & Distribution
+- Uses hatchling as build backend
+- BSL-1.1 license
+- Excludes development files from distribution packages
+
+### CI/CD Integration
+- GitHub Actions workflow for automatic optimization of PR code
+- Pre-commit hooks for code quality enforcement
+- Automated testing and benchmarking
+
+## Important Patterns
+
+### Error Handling
+- Uses `either.py` for functional error handling patterns
+- Comprehensive error tracking through Sentry integration
+- Graceful degradation when AI services are unavailable
+
+### Instrumentation
+- Extensive tracing capabilities for performance analysis
+- Line profiler integration for detailed performance metrics
+- Custom tracer implementation for code execution analysis
+
+### AI Integration
+- Structured prompts and response handling for LLM interactions
+- Critic module for evaluating optimization quality
+- Context-aware code generation and explanation
+
+### Git Integration 
+- GitPython for repository operations
+- Automated PR creation with detailed explanations
+- Branch management for optimization experiments

WARP.md

@@ -1,8 +1,2 @@
 DEFAULT_API_URL = "https://api.galileo.ai/"
 DEFAULT_APP_URL = "https://app.galileo.ai/"
-
-
-# function_names: GalileoApiClient.get_console_url
-# module_abs_path : /home/mohammed/Work/galileo-python/src/galileo/api_client.py
-# preexisting_objects: {('GalileoApiClient', ()), ('_set_destination', ()), ('get_console_url', (FunctionParent(name='GalileoApiClient', type='ClassDef'),))}
-# project_root_path: /home/mohammed/Work/galileo-python/src

code_to_optimize/code_directories/circular_deps/constants.py

@@ -0,0 +1,3 @@
+"""CodeFlash Benchmark - Pytest benchmarking plugin for codeflash.ai."""
+
+__version__ = "0.1.0"

codeflash-benchmark/codeflash_benchmark/__init__.py

@@ -0,0 +1,94 @@
+from __future__ import annotations
+
+import importlib.util
+
+import pytest
+
+from codeflash.benchmarking.plugin.plugin import codeflash_benchmark_plugin
+
+PYTEST_BENCHMARK_INSTALLED = importlib.util.find_spec("pytest_benchmark") is not None
+
+benchmark_options = [
+    ("--benchmark-columns", "store", None, "Benchmark columns"),
+    ("--benchmark-group-by", "store", None, "Benchmark group by"),
+    ("--benchmark-name", "store", None, "Benchmark name pattern"),
+    ("--benchmark-sort", "store", None, "Benchmark sort column"),
+    ("--benchmark-json", "store", None, "Benchmark JSON output file"),
+    ("--benchmark-save", "store", None, "Benchmark save name"),
+    ("--benchmark-warmup", "store", None, "Benchmark warmup"),
+    ("--benchmark-warmup-iterations", "store", None, "Benchmark warmup iterations"),
+    ("--benchmark-min-time", "store", None, "Benchmark minimum time"),
+    ("--benchmark-max-time", "store", None, "Benchmark maximum time"),
+    ("--benchmark-min-rounds", "store", None, "Benchmark minimum rounds"),
+    ("--benchmark-timer", "store", None, "Benchmark timer"),
+    ("--benchmark-calibration-precision", "store", None, "Benchmark calibration precision"),
+    ("--benchmark-disable", "store_true", False, "Disable benchmarks"),
+    ("--benchmark-skip", "store_true", False, "Skip benchmarks"),
+    ("--benchmark-only", "store_true", False, "Only run benchmarks"),
+    ("--benchmark-verbose", "store_true", False, "Verbose benchmark output"),
+    ("--benchmark-histogram", "store", None, "Benchmark histogram"),
+    ("--benchmark-compare", "store", None, "Benchmark compare"),
+    ("--benchmark-compare-fail", "store", None, "Benchmark compare fail threshold"),
+]
+
+
+def pytest_configure(config: pytest.Config) -> None:
+    """Register the benchmark marker and disable conflicting plugins."""
+    config.addinivalue_line("markers", "benchmark: mark test as a benchmark that should be run with codeflash tracing")
+
+    if config.getoption("--codeflash-trace"):
+        # When --codeflash-trace is used, ignore all benchmark options by resetting them to defaults
+        for option, _, default, _ in benchmark_options:
+            option_name = option.replace("--", "").replace("-", "_")
+            if hasattr(config.option, option_name):
+                setattr(config.option, option_name, default)
+
+        if PYTEST_BENCHMARK_INSTALLED:
+            config.pluginmanager.set_blocked("pytest_benchmark")
+            config.pluginmanager.set_blocked("pytest-benchmark")
+
+
+def pytest_addoption(parser: pytest.Parser) -> None:
+    parser.addoption(
+        "--codeflash-trace", action="store_true", default=False, help="Enable CodeFlash tracing for benchmarks"
+    )
+    # These options are ignored when --codeflash-trace is used
+    for option, action, default, help_text in benchmark_options:
+        help_suffix = " (ignored when --codeflash-trace is used)"
+        parser.addoption(option, action=action, default=default, help=help_text + help_suffix)
+
+
+@pytest.fixture
+def benchmark(request: pytest.FixtureRequest) -> object:
+    """Benchmark fixture that works with or without pytest-benchmark installed."""
+    config = request.config
+
+    # If --codeflash-trace is enabled, use our implementation
+    if config.getoption("--codeflash-trace"):
+        return codeflash_benchmark_plugin.Benchmark(request)
+
+    # If pytest-benchmark is installed and --codeflash-trace is not enabled,
+    # return the normal pytest-benchmark fixture
+    if PYTEST_BENCHMARK_INSTALLED:
+        from pytest_benchmark.fixture import BenchmarkFixture as BSF  # pyright: ignore[reportMissingImports]  # noqa: I001, N814
+
+        bs = getattr(config, "_benchmarksession", None)
+        if bs and bs.skip:
+            pytest.skip("Benchmarks are skipped (--benchmark-skip was used).")
+
+        node = request.node
+        marker = node.get_closest_marker("benchmark")
+        options = dict(marker.kwargs) if marker else {}
+
+        if bs:
+            return BSF(
+                node,
+                add_stats=bs.benchmarks.append,
+                logger=bs.logger,
+                warner=request.node.warn,
+                disabled=bs.disabled,
+                **dict(bs.options, **options),
+            )
+        return lambda func, *args, **kwargs: func(*args, **kwargs)
+
+    return lambda func, *args, **kwargs: func(*args, **kwargs)