⚡️ Speed up method CodeFlashBenchmarkPlugin.pytest_collection_modifyitems by 161% in PR #59 (codeflash-trace-decorator)

[codeflash-ai]

⚡️ This pull request contains optimizations for PR #59

If you approve this dependent PR, these changes will be merged into the original PR branch codeflash-trace-decorator.

This PR will be automatically closed if the original PR is merged.

---

📄 161% (1.61x) speedup for CodeFlashBenchmarkPlugin.pytest_collection_modifyitems in codeflash/benchmarking/plugin/plugin.py

⏱️ Runtime : 167 milliseconds → 64.0 milliseconds (best of 19 runs)

📝 Explanation and details

Explanation of Changes.

1. Optimize the Loop in pytest_collection_modifyitems Method.
   • Instead of checking the fixture names and conditionally adding the marker within the same loop, we separate the items into two lists: one for items with the benchmark fixture and one for items without it.
   • This optimization helps to reduce the possible overhead of repeatedly calling add_marker by first classifying the items and then applying the marker only to those necessary.
   • Finally, we concatenate the lists to retain the original order, except with tests without benchmark fixtures getting the skip marker.

Note: The item list ordering at the end of the method may not be necessary depending on the context, but this ensures that we process markers efficiently without changing the original relative order of test items.

✅ Correctness verification report:

Test	Status
⚙️ Existing Unit Tests	🔘 None Found
🌀 Generated Regression Tests	✅ 8 Passed
⏪ Replay Tests	🔘 None Found
🔎 Concolic Coverage Tests	🔘 None Found
📊 Tests Coverage

🌀 Generated Regression Tests Details

from __future__ import annotations

from unittest.mock import Mock

# imports
import pytest  # used for our unit tests
from codeflash.benchmarking.plugin.plugin import CodeFlashBenchmarkPlugin

# unit tests

# Basic Functionality


def test_codeflash_trace_option():
    # Mock config and items
    config_mock = Mock()
    items_mock = [Mock(fixturenames=[]) for _ in range(5)]
    
    # Set the config option to True
    config_mock.getoption.return_value = True
    
    # Call the function
    CodeFlashBenchmarkPlugin.pytest_collection_modifyitems(config_mock, items_mock)

# Test Items with and without Benchmark Fixture

def test_with_benchmark_fixture():
    # Mock config and items
    config_mock = Mock()
    items_mock = [Mock(fixturenames=['benchmark']) for _ in range(5)]
    
    # Set the config option to True
    config_mock.getoption.return_value = True
    
    # Call the function
    CodeFlashBenchmarkPlugin.pytest_collection_modifyitems(config_mock, items_mock)

def test_without_benchmark_fixture():
    # Mock config and items
    config_mock = Mock()
    items_mock = [Mock(fixturenames=[]) for _ in range(5)]
    
    # Set the config option to True
    config_mock.getoption.return_value = True
    
    # Call the function
    CodeFlashBenchmarkPlugin.pytest_collection_modifyitems(config_mock, items_mock)

# Edge Cases


def test_items_no_fixturenames():
    # Mock config and items
    config_mock = Mock()
    items_mock = [Mock() for _ in range(5)]
    
    # Remove fixturenames attribute from items
    for item in items_mock:
        del item.fixturenames
    
    # Set the config option to True
    config_mock.getoption.return_value = True
    
    # Call the function
    CodeFlashBenchmarkPlugin.pytest_collection_modifyitems(config_mock, items_mock)

# Mixed Test Items

def test_mixed_items():
    # Mock config and items
    config_mock = Mock()
    items_mock = [Mock(fixturenames=['benchmark']), Mock(fixturenames=[])]
    
    # Set the config option to True
    config_mock.getoption.return_value = True
    
    # Call the function
    CodeFlashBenchmarkPlugin.pytest_collection_modifyitems(config_mock, items_mock)

# Large Scale Test Cases



from __future__ import annotations

from unittest.mock import MagicMock

# imports
import pytest  # used for our unit tests
from codeflash.benchmarking.plugin.plugin import CodeFlashBenchmarkPlugin

# unit tests

To edit these changes git checkout codeflash/optimize-pr59-2025-04-03T00.13.14 and push.