Skip to content

Conversation

@KRRT7
Copy link
Contributor

@KRRT7 KRRT7 commented Jul 23, 2025

PR Type

Enhancement


Description

  • Add debugging prints in trace wrapper

  • Introduce frozen BenchmarkKey dataclass

  • Extend module path traversal logic

  • Refine pytest marker handling and skip logic


Diagram Walkthrough

flowchart LR
  A["pytest marker detection"]
  B["BenchmarkKey dataclass"]
  C["set CODEFLASH env vars"]
  D["codeflash_trace debug prints"]
  E["module path traversal fallback"]
  F["optimizer run & result printing"]
  A --> B --> C --> D --> E --> F
Loading

File Walkthrough

Relevant files
Enhancement
codeflash_trace.py
Add debug prints in trace wrapper                                               

codeflash/benchmarking/codeflash_trace.py

  • Added debug print markers around trace invocation.
  • Logged environment variables for benchmark context.
+4/-0     
plugin.py
Introduce BenchmarkKey and marker handling                             

codeflash/benchmarking/plugin/plugin.py

  • Defined frozen BenchmarkKey dataclass with __str__.
  • Updated pytest marker logic to skip unmarked tests.
  • Extended fixture docstring for clarity.
  • Passed traverse_up=True to module name resolution.
+16/-15 
code_utils.py
Enhance module path traversal logic                                           

codeflash/code_utils/code_utils.py

  • Added traverse_up parameter to fallback relative path logic.
  • Implemented upward traversal when file outside project root.
  • Raise clear error if traversal fails.
+15/-3   
pytest_new_process_discovery.py
Register benchmark marker in discovery plugin                       

codeflash/discovery/pytest_new_process_discovery.py

  • Added pytest_configure to register benchmark marker.
+3/-0     
optimizer.py
Refactor run_benchmarks flow                                                         

codeflash/optimization/optimizer.py

  • Added console.rule() before benchmarks.
  • Refactored run logic to raise on no benchmarks.
  • Unified benchmark results printing.
+8/-7     

@github-actions
Copy link

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Debug prints

Debug print statements are added in the trace wrapper, which may clutter output in production. Consider replacing with a configurable logger at an appropriate log level or removing them before release.

print("XXX CALLED THE CODEFLASH_TRACE MARKER")
# Initialize thread-local active functions set if it doesn't exist
if not hasattr(self._thread_local, "active_functions"):
    self._thread_local.active_functions = set()
# If it's in a recursive function, just return the result
if func_id in self._thread_local.active_functions:
    return func(*args, **kwargs)
# Track active functions so we can detect recursive functions
self._thread_local.active_functions.add(func_id)
# Measure execution time
start_time = time.thread_time_ns()
result = func(*args, **kwargs)
end_time = time.thread_time_ns()
# Calculate execution time
execution_time = end_time - start_time
self.function_call_count += 1

# Check if currently in pytest benchmark fixture
if os.environ.get("CODEFLASH_BENCHMARKING", "False") == "False":
    self._thread_local.active_functions.remove(func_id)
    return result
# Get benchmark info from environment
benchmark_function_name = os.environ.get("CODEFLASH_BENCHMARK_FUNCTION_NAME", "")
benchmark_module_path = os.environ.get("CODEFLASH_BENCHMARK_MODULE_PATH", "")
benchmark_line_number = os.environ.get("CODEFLASH_BENCHMARK_LINE_NUMBER", "")
print("XXX benchmark_function_name", benchmark_function_name)
print("XXX benchmark_module_path", benchmark_module_path)
print("XXX benchmark_line_number", benchmark_line_number)
Abrupt exit

Raising SystemExit for control flow when no benchmarks are found may terminate the application unexpectedly. Consider returning gracefully or handling this case without exiting the process.

    raise SystemExit  # noqa: TRY301
function_benchmark_timings = CodeFlashBenchmarkPlugin.get_function_benchmark_timings(trace_file)
total_benchmark_timings = CodeFlashBenchmarkPlugin.get_benchmark_timings(trace_file)
Module path traversal

The traverse_up logic in module_name_from_file_path may produce incorrect module names or miss boundary conditions when files lie outside the project root. Verify edge cases and ensure consistent module path resolution.

def module_name_from_file_path(file_path: Path, project_root_path: Path, *, traverse_up: bool = True) -> str:
    try:
        relative_path = file_path.relative_to(project_root_path)
        return relative_path.with_suffix("").as_posix().replace("/", ".")
    except ValueError:
        if traverse_up:
            parent = file_path.parent
            while parent not in (project_root_path, parent.parent):
                try:
                    relative_path = file_path.relative_to(parent)
                    return relative_path.with_suffix("").as_posix().replace("/", ".")
                except ValueError:
                    parent = parent.parent
        msg = f"File {file_path} is not within the project root {project_root_path}."
        raise ValueError(msg)  # noqa: B904

@github-actions
Copy link

PR Code Suggestions ✨

Explore these optional code suggestions:

CategorySuggestion                                                                                                                                    Impact
General
Replace prints with debug logs

Remove these unconditional print statements and use a logger at debug level instead
to avoid noisy standard output. Replace all print(...) calls with logger.debug(...)
so they can be enabled or disabled via logging configuration. This will keep
benchmark output clean in production.

codeflash/benchmarking/codeflash_trace.py [111]

-print("XXX CALLED THE CODEFLASH_TRACE MARKER")
+logger.debug("CODEFLASH_TRACE marker called")
Suggestion importance[1-10]: 6

__

Why: The unconditional print is too noisy and using logger.debug allows configurable verbosity and cleaner production output.

Low
Possible issue
Simplify ancestor traversal

Simplify and harden the upward traversal by iterating over file_path.parents instead
of a custom while loop. This ensures you search all ancestor directories up to the
filesystem root and avoids subtle loop‐termination bugs.

codeflash/code_utils/code_utils.py [116-124]

 except ValueError:
     if traverse_up:
-        parent = file_path.parent
-        while parent not in (project_root_path, parent.parent):
+        for parent in file_path.parents:
             try:
                 relative_path = file_path.relative_to(parent)
                 return relative_path.with_suffix("").as_posix().replace("/", ".")
             except ValueError:
-                parent = parent.parent
+                continue
Suggestion importance[1-10]: 5

__

Why: Iterating file_path.parents both simplifies the code and ensures thorough ancestor traversal without custom loop logic.

Low

codeflash-ai bot added a commit that referenced this pull request Jul 23, 2025
…items` by 670% in PR #574 (`easier-benchmark`)

Here’s how you can optimize your code for **speed** and **memory** while maintaining return values, behavior, and comments. I focused on making `pytest_collection_modifyitems` faster by.

- **Avoiding function calls/attribute checks inside tight loops:**  
  Since ideally `get_closest_marker` existence is consistent across items, fetch it once per item and cache the lookup.
- **Reducing repeated work:**  
  Move more work (e.g. creation of the skip marker) outside the inner loop.




### Changes and Optimizations.
- **Cache Attribute**:  
  `getattr(item, "get_closest_marker", None)` is used to avoid repeated `hasattr` checks or repeated attribute lookups.
- **Reuse Marker Instance**:  
  `skip_marker` is created once, not in every loop iteration.
- **Skip on missing attribute** instead of raising.

**This will speed up the loop by:**
- Reducing per-item attribute lookup
- Reducing decorator construction
- Reducing function calls on items which don’t have `get_closest_marker` (if any)

---

Let me know if you want further or different optimizations!
@codeflash-ai
Copy link
Contributor

codeflash-ai bot commented Jul 23, 2025

⚡️ Codeflash found optimizations for this PR

📄 670% (6.70x) speedup for CodeFlashBenchmarkPlugin.pytest_collection_modifyitems in codeflash/benchmarking/plugin/plugin.py

⏱️ Runtime : 14.8 milliseconds 1.93 milliseconds (best of 465 runs)

I created a new dependent PR with the suggested changes. Please review:

If you approve, it will be merged into this PR (branch easier-benchmark).

@github-actions github-actions bot added the workflow-modified This PR modifies GitHub Actions workflows label Jul 24, 2025
@KRRT7 KRRT7 force-pushed the easier-benchmark branch from e81834f to 3159233 Compare July 24, 2025 02:08
@KRRT7 KRRT7 requested a review from misrasaurabh1 July 24, 2025 02:25
@KRRT7 KRRT7 enabled auto-merge July 24, 2025 20:52
@KRRT7 KRRT7 merged commit 34c3aa9 into main Jul 24, 2025
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Review effort 3/5 workflow-modified This PR modifies GitHub Actions workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants