[testing, CI] fix coverage statistics issue caused by test_common.py tracer patching

[icfaust]

Description

Python code coverage has been introduced in #2222 and #2225. Some code, while properly covered with tests, was showing incorrect coverage. It was discovered that the tracing used in test_common.py interferes with the tracing collected by pytest-cov. Any tests alphabetically after test_common.py have been neglected in codecov and by coverage.py. Correcting this can be done by tracing the estimators in a separate process and passing the data back to the pytest-cov parent process.

This has been implemented using python multiprocessing due to the time overhead of loading sklearnex for each estimator and method. The process persists for the period of all test_common.py tests as a daemon process. The test passing/xfail/etc rate has been verified to match to main.

No performance metrics are generated due to purely for testing and CI.

The effectiveness of this change can be observed by the ~1.2% improvement in code coverage (as indirect changes).

---

PR should start as a draft, then move to ready for review state after CI is passed and all applicable checkboxes are closed.
This approach ensures that reviewers don't spend extra time asking for regular requirements.

You can remove a checkbox as not applicable only if it doesn't relate to this PR in any way.
For example, PR with docs update doesn't require checkboxes for performance while PR with any change in actual code should have checkboxes and justify how this code change is expected to affect performance (or justification should be self-evident).

Checklist to comply with before moving PR from draft:

PR completeness and readability

• I have reviewed my changes thoroughly before submitting this pull request.
• I have commented my code, particularly in hard-to-understand areas.
• I have updated the documentation to reflect the changes or created a separate PR with update and provided its number in the description, if necessary.
• Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
• I have added a respective label(s) to PR if I have a permission for that.
• I have resolved any merge conflicts that might occur with the base branch.

Testing

• I have run it locally and tested the changes extensively.
• All CI jobs are green or I have provided justification why they aren't.
• I have extended testing suite if new functionality was introduced in this PR.

Performance

• I have measured performance for affected algorithms using scikit-learn_bench and provided at least summary table with measured data, if performance change is expected.
• I have provided justification why performance has changed or why changes are not expected.
• I have provided justification why quality metrics have changed or why changes are not expected.
• I have extended benchmarking suite and provided corresponding scikit-learn_bench PR if new measurable functionality was introduced in this PR.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Flag	Coverage Δ
azure	78.01% <ø> (+1.22%)	⬆️
github	71.04% <ø> (+0.85%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

see 14 files with indirect coverage changes

[icfaust]

/intelci: run

[icfaust]

/intelci: run

[ethanglaser]

Is 1.2% increase what you would expect? In general, where does the other missing coverage come from?

[icfaust]

Is 1.2% increase what you would expect? In general, where does the other missing coverage come from?

@ethanglaser good question! Its mainly 2 aspects: 1) The utils/validation.py tests coverage was not being recorded, 2) centralized testing (especially test_patching.py) was not being recorded, which covers all other methods of estimators (especially score) which otherwise might not be specifically tested in the individual estimator tests. With the fix these are now recorded, and that means these tests provide an additional 1.2% of the codebase coverage not seen in sklearn conformance or estimator/onedal tests.

[ahuber21]

Suggested change

	try:
	est = PATCHED_MODELS[estimator]()
	except KeyError:
	est = SPECIAL_INSTANCES[estimator]
	estimator = (SPECIAL_INSTANCES | PATCHED_MODELS)[estimator_name]

[icfaust]

I had to revert this, because SPECIAL_INSTANCES is a special dictionary of estimators which uses sklearn's clone (in order to guarantee that there is no hysteresis between uses of the instance). And the patched models are classes. To be honest, the difference is tech debt that I introduced at the beginning of 2024, as I was trying to unify the centralized testing. Hindsight I would structure things like SPECIAL_INSTANCES.

[ahuber21]

Then I'd go with PATCHED_MODELS.get(estimator_name, None) or SPECIAL_INSTANCES[estimator_name]. I don't want to waste 4 lines on something that doesn't contribute to the function logic.

[icfaust]

Took me some time to figure this out, turns out the ensemble algorithms of sklearn break the suggestion construction.

from sklearn.ensemble import RandomForestRegressor
RandomForestRegressor() or 3

will yield:
AttributeError: 'RandomForestRegressor' object has no attribute 'estimators_'. Did you mean: 'estimator'?

which means I cannot use this in this case, its definitely doing something with the or operator and checking if its an iterable. Its not something on our side, but comes from sklearn conformance. It comes from sklearn for whatever reason defining a __len__ for ensemble estimators thats only valid after fitting.

[ahuber21]

Sorry for sending you down a rabbit hole. I would still prefer a different implementation because I think try/except is a bit of an overkill

estimator = PATCHED_MODELS[estimator_name] if estimator_name in PATCHED_MODELS else SPECIAL_INSTANCES[estimator_name]

[icfaust]

no apologies necessary, it showed i hadnt handled a failure case properly and would lead to pytest hanging, which would have been a nightmare to debug. now it should error 'gracefully' in CI (testing it now)

[icfaust]

test if new failure system works on last commit

[icfaust]

/intelci: run