pydantic ai benchmarking support

[KRRT7]

PR Type

Enhancement

---

Description

• Add debugging prints in trace wrapper

• Introduce frozen BenchmarkKey dataclass

• Extend module path traversal logic

• Refine pytest marker handling and skip logic

---

Diagram Walkthrough

flowchart LR
  A["pytest marker detection"]
  B["BenchmarkKey dataclass"]
  C["set CODEFLASH env vars"]
  D["codeflash_trace debug prints"]
  E["module path traversal fallback"]
  F["optimizer run & result printing"]
  A --> B --> C --> D --> E --> F

Loading

File Walkthrough

Relevant files

Enhancement

codeflash_trace.py Add debug prints in trace wrapper                                                codeflash/benchmarking/codeflash_trace.py Added debug print markers around trace invocation. Logged environment variables for benchmark context. +4/-0      plugin.py Introduce BenchmarkKey and marker handling                              codeflash/benchmarking/plugin/plugin.py Defined frozen BenchmarkKey dataclass with __str__. Updated pytest marker logic to skip unmarked tests. Extended fixture docstring for clarity. Passed traverse_up=True to module name resolution. +16/-15  code_utils.py Enhance module path traversal logic                                            codeflash/code_utils/code_utils.py Added traverse_up parameter to fallback relative path logic. Implemented upward traversal when file outside project root. Raise clear error if traversal fails. +15/-3    pytest_new_process_discovery.py Register benchmark marker in discovery plugin                        codeflash/discovery/pytest_new_process_discovery.py Added pytest_configure to register benchmark marker. +3/-0      optimizer.py Refactor run_benchmarks flow                                                          codeflash/optimization/optimizer.py Added console.rule() before benchmarks. Refactored run logic to raise on no benchmarks. Unified benchmark results printing. +8/-7

---

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Debug prints Debug print statements are added in the trace wrapper, which may clutter output in production. Consider replacing with a configurable logger at an appropriate log level or removing them before release. print("XXX CALLED THE CODEFLASH_TRACE MARKER") # Initialize thread-local active functions set if it doesn't exist if not hasattr(self._thread_local, "active_functions"): self._thread_local.active_functions = set() # If it's in a recursive function, just return the result if func_id in self._thread_local.active_functions: return func(*args, **kwargs) # Track active functions so we can detect recursive functions self._thread_local.active_functions.add(func_id) # Measure execution time start_time = time.thread_time_ns() result = func(*args, **kwargs) end_time = time.thread_time_ns() # Calculate execution time execution_time = end_time - start_time self.function_call_count += 1 # Check if currently in pytest benchmark fixture if os.environ.get("CODEFLASH_BENCHMARKING", "False") == "False": self._thread_local.active_functions.remove(func_id) return result # Get benchmark info from environment benchmark_function_name = os.environ.get("CODEFLASH_BENCHMARK_FUNCTION_NAME", "") benchmark_module_path = os.environ.get("CODEFLASH_BENCHMARK_MODULE_PATH", "") benchmark_line_number = os.environ.get("CODEFLASH_BENCHMARK_LINE_NUMBER", "") print("XXX benchmark_function_name", benchmark_function_name) print("XXX benchmark_module_path", benchmark_module_path) print("XXX benchmark_line_number", benchmark_line_number) Abrupt exit Raising SystemExit for control flow when no benchmarks are found may terminate the application unexpectedly. Consider returning gracefully or handling this case without exiting the process. raise SystemExit # noqa: TRY301 function_benchmark_timings = CodeFlashBenchmarkPlugin.get_function_benchmark_timings(trace_file) total_benchmark_timings = CodeFlashBenchmarkPlugin.get_benchmark_timings(trace_file) Module path traversal The traverse_up logic in module_name_from_file_path may produce incorrect module names or miss boundary conditions when files lie outside the project root. Verify edge cases and ensure consistent module path resolution. def module_name_from_file_path(file_path: Path, project_root_path: Path, *, traverse_up: bool = True) -> str: try: relative_path = file_path.relative_to(project_root_path) return relative_path.with_suffix("").as_posix().replace("/", ".") except ValueError: if traverse_up: parent = file_path.parent while parent not in (project_root_path, parent.parent): try: relative_path = file_path.relative_to(parent) return relative_path.with_suffix("").as_posix().replace("/", ".") except ValueError: parent = parent.parent msg = f"File {file_path} is not within the project root {project_root_path}." raise ValueError(msg) # noqa: B904

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Replace prints with debug logs Remove these unconditional print statements and use a logger at debug level instead to avoid noisy standard output. Replace all print(...) calls with logger.debug(...) so they can be enabled or disabled via logging configuration. This will keep benchmark output clean in production. codeflash/benchmarking/codeflash_trace.py [111] -print("XXX CALLED THE CODEFLASH_TRACE MARKER") +logger.debug("CODEFLASH_TRACE marker called") Suggestion importance[1-10]: 6 __ Why: The unconditional print is too noisy and using logger.debug allows configurable verbosity and cleaner production output.	Low
Possible issue	Simplify ancestor traversal Simplify and harden the upward traversal by iterating over file_path.parents instead of a custom while loop. This ensures you search all ancestor directories up to the filesystem root and avoids subtle loop‐termination bugs. codeflash/code_utils/code_utils.py [116-124] except ValueError: if traverse_up: - parent = file_path.parent - while parent not in (project_root_path, parent.parent): + for parent in file_path.parents: try: relative_path = file_path.relative_to(parent) return relative_path.with_suffix("").as_posix().replace("/", ".") except ValueError: - parent = parent.parent + continue Suggestion importance[1-10]: 5 __ Why: Iterating file_path.parents both simplifies the code and ensures thorough ancestor traversal without custom loop logic.	Low

[codeflash-ai]

⚡️ Codeflash found optimizations for this PR

📄 670% (6.70x) speedup for CodeFlashBenchmarkPlugin.pytest_collection_modifyitems in codeflash/benchmarking/plugin/plugin.py

⏱️ Runtime : 14.8 milliseconds → 1.93 milliseconds (best of 465 runs)

I created a new dependent PR with the suggested changes. Please review:

• ⚡️ Speed up method CodeFlashBenchmarkPlugin.pytest_collection_modifyitems by 670% in PR #574 (easier-benchmark) #575

If you approve, it will be merged into this PR (branch easier-benchmark).