feat(flaky-detection): Add pytest-xdist support to flaky detection system by remyduthu · Pull Request #384 · Mergifyio/pytest-mergify

remyduthu · 2026-03-19T13:07:57Z

Add full flaky detection support under pytest-xdist using controller-
orchestrated pre-computed deadlines and xdist's built-in IPC.

Architecture:

Controller fetches flaky detection context from API once, distributes
it to workers via workerinput
Each worker constructs a FlakyDetector from the serialized context,
computes the same global budget, runs tests with reruns
Workers send metrics back via workeroutput
Controller aggregates metrics and generates the terminal report

Changes:

Add FlakyDetector.from_context() classmethod for worker-side construction
Add to_serializable_metrics() for xdist IPC (plain builtins only)
Add make_report_from_aggregated() for controller-side reporting
Add static deadline mode (_is_xdist flag) preserving dynamic for non-xdist
Add MergifyCIInsights.load_flaky_detector_from_context()
Add controller hooks (pytest_configure_node, pytest_testnodedown)
Add worker hooks (workerinput loading, workeroutput export)
Disable flaky detection under 'each' scheduling mode
Add pytest-xdist dev dependency and integration tests

Fixes: MRGFY-6296

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

mergify · 2026-03-19T13:08:08Z

Copilot

Pull request overview

Adds pytest-xdist compatibility to the flaky detection feature by moving context fetching/reporting responsibilities to the controller process and using xdist IPC (workerinput/workeroutput) for worker-side execution and metric aggregation.

Changes:

Add worker-safe FlakyDetector.from_context() construction and to_serializable_metrics() for IPC payloads.
Add controller-side aggregation/reporting via make_report_from_aggregated() and xdist hooks (pytest_configure_node, pytest_testnodedown).
Introduce tests/docs and add pytest-xdist as a dev dependency.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
`pytest_mergify/flaky_detection.py`	Adds xdist-aware construction/serialization, static deadline mode, and aggregated report generation.
`pytest_mergify/__init__.py`	Implements controller/worker xdist hooks, context distribution, worker loading, and metrics export.
`pytest_mergify/ci_insights.py`	Adds worker-path loader to build the flaky detector from controller-provided context.
`tests/test_flaky_detection.py`	Adds unit tests for new serialization/from-context/reporting/static-deadline behavior.
`tests/test_xdist.py`	Adds xdist-focused tests and lifecycle simulations around controller/worker behavior.
`pyproject.toml`	Adds `pytest-xdist` to dev dependencies.
`uv.lock`	Locks `pytest-xdist` and transitive deps (`execnet`) with Python version markers.
`docs/superpowers/specs/2026-03-19-xdist-flaky-detection-design.md`	Documents the design and architecture for xdist support.
`docs/superpowers/plans/2026-03-19-xdist-flaky-detection.md`	Implementation plan for the xdist flaky detection work.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

pytest_mergify/flaky_detection.py

+    available_budget_seconds = available_budget_duration_ms / 1000
+    used_budget_ms = sum(m["total_duration_ms"] for m in test_metrics.values())
+    used_budget_seconds = used_budget_ms / 1000
+    result += (
+        f"{os.linesep}- Used {used_budget_seconds / available_budget_seconds * 100:.2f} % of the budget "
+        f"({used_budget_seconds:.2f} s/{available_budget_seconds:.2f} s)"
+    )


pytest_mergify/flaky_detection.py

+    if tests_prevented_from_timeout:
+        result += (
+            f"{os.linesep}⚠️ Reduced reruns for the following "
+            f"test{'s' if len(tests_prevented_from_timeout) else ''} to respect 'pytest-timeout':"


pytest_mergify/flaky_detection.py

+    result += (
+        f"{os.linesep}- Active for {len(test_metrics)} {mode} "
+        f"test{'s' if len(test_metrics) > 1 else ''}:"
+    )
+    for test, m in test_metrics.items():
+        if m["rerun_count"] < context.min_test_execution_count:


pytest_mergify/__init__.py


-        if self.mergify_ci.flaky_detector:
+        if _is_xdist_controller(terminalreporter.config):
+            if self._xdist_flaky_context:


tests/test_xdist.py

+        """
+    )
+
+    result = pytester.runpytest_inprocess(


remyduthu · 2026-03-19T13:18:35Z

pytest_mergify/__init__.py

+        # xdist controller state.
+        self._xdist_flaky_context: typing.Optional[typing.Dict[str, typing.Any]] = None
+        self._xdist_flaky_mode: typing.Optional[str] = None
+        self._xdist_aggregated_metrics: typing.Dict[str, typing.Any] = {
+            "test_metrics": {},
+            "over_length_tests": [],
+            "debug_logs": [],
+        }
+        self._xdist_available_budget_duration_ms: float = 0.0
+
+        # On xdist controller, reuse the already-loaded detector's context
+        # for distribution to workers. No extra API call needed since
+        # MergifyCIInsights.__post_init__ already calls _load_flaky_detector().
+        if _is_xdist_controller(config) and self.mergify_ci.flaky_detector:
+            self._xdist_flaky_context = dataclasses.asdict(
+                self.mergify_ci.flaky_detector._context
+            )
+            self._xdist_flaky_mode = self.mergify_ci.flaky_detector.mode
+
+        # xdist worker: load flaky detector from controller-provided context.
+        if _is_xdist_worker(config):
+            workerinput = getattr(config, "workerinput", {})
+            context = workerinput.get("flaky_detection_context")
+            mode = workerinput.get("flaky_detection_mode")
+            if context is not None and mode is not None:
+                self.mergify_ci.load_flaky_detector_from_context(context, mode)


Extract in another method to keep the main function small.

remyduthu · 2026-03-19T13:19:09Z

pytest_mergify/__init__.py

+        dist_mode = getattr(node.config.option, "dist", None)
+        if dist_mode == "each":


Suggested change

dist_mode = getattr(node.config.option, "dist", None)

if dist_mode == "each":

if getattr(node.config.option, "dist", None) == "each":

remyduthu · 2026-03-19T13:20:16Z

pytest_mergify/__init__.py

        self.mergify_ci = MergifyCIInsights(**kwargs)

+        # xdist controller state.
+        self._xdist_flaky_context: typing.Optional[typing.Dict[str, typing.Any]] = None


We should probably move all this state in the flaky detector class to separate the concerns properly.

remyduthu · 2026-03-19T13:40:26Z

Code review

Found 4 issues:

ZeroDivisionError in make_report_from_aggregated() when available_budget_duration_ms is 0.0. On xdist workers, _available_budget_duration starts as timedelta() (zero) and is only updated in prepare_for_session(). If the worker's budget stays zero but test_metrics is non-empty, the division crashes. Since make_report() now delegates to make_report_from_aggregated(), both paths are affected. (confidence: 85/100)

pytest-mergify/pytest_mergify/flaky_detection.py

Lines 493 to 501 in a84cc23

    
               return result 
        
           available_budget_seconds = available_budget_duration_ms / 1000 
        
           used_budget_ms = sum(m["total_duration_ms"] for m in test_metrics.values()) 
        
           used_budget_seconds = used_budget_ms / 1000 
        
           result += ( 
        
               f"{os.linesep}- Used {used_budget_seconds / available_budget_seconds * 100:.2f} % of the budget " 
        
               f"({used_budget_seconds:.2f} s/{available_budget_seconds:.2f} s)" 
        
           )

_is_xdist_controller check may fire too early. xdist registers dsession in its pytest_configure hook marked with @pytest.hookimpl(trylast=True), meaning it runs AFTER most plugins. When PytestMergify.pytest_configure runs, dsession may not be registered yet, so _is_xdist_controller() returns False. This leaves _xdist_flaky_context = None, and pytest_configure_node distributes nothing to workers — silently disabling xdist flaky detection. (confidence: 75/100)

pytest-mergify/pytest_mergify/__init__.py

Lines 48 to 56 in a84cc23

    
           # On xdist controller, reuse the already-loaded detector's context 
        
           # for distribution to workers. No extra API call needed since 
        
           # MergifyCIInsights.__post_init__ already calls _load_flaky_detector(). 
        
           if _is_xdist_controller(config) and self.mergify_ci.flaky_detector: 
        
               self._xdist_flaky_context = dataclasses.asdict( 
        
                   self.mergify_ci.flaky_detector._context 
        
               ) 
        
               self._xdist_flaky_mode = self.mergify_ci.flaky_detector.mode

Comment leaks implementation details. # No extra API call needed since MergifyCIInsights.__post_init__ already calls _load_flaky_detector(). references private methods, violating CLAUDE.md rules "Do not leak implementation details" and "Explain WHAT and WHY, not HOW." (confidence: 75/100)

pytest-mergify/pytest_mergify/__init__.py

Lines 48 to 50 in a84cc23

    
           # On xdist controller, reuse the already-loaded detector's context 
        
           # for distribution to workers. No extra API call needed since 
        
           # MergifyCIInsights.__post_init__ already calls _load_flaky_detector().

pytest_runtest_protocol tracer guard blocks reruns if tracer is unavailable. The if not self.tracer: return None guard means that if the tracer fails to initialize on a worker (e.g. git unavailable, repo name unresolvable), the entire rerun loop is skipped — flaky detection becomes a no-op. In practice this is unlikely since xdist workers inherit env vars, but the guard is overly aggressive. (confidence: 50/100)

pytest-mergify/pytest_mergify/__init__.py

Lines 289 to 292 in a84cc23

    
           # See: 
        
           # https://docs.pytest.org/en/7.1.x/how-to/writing_hook_functions.html#firstresult 
        
           if not self.tracer: 
        
               return None

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}

…stem Add full flaky detection support under pytest-xdist using controller- orchestrated pre-computed deadlines and xdist's built-in IPC. Architecture: - Controller fetches flaky detection context from API once, distributes it to workers via workerinput - Each worker constructs a FlakyDetector from the serialized context, computes the same global budget, runs tests with reruns - Workers send metrics back via workeroutput - Controller aggregates metrics and generates the terminal report Changes: - Add FlakyDetector.from_context() classmethod for worker-side construction - Add to_serializable_metrics() for xdist IPC (plain builtins only) - Add make_report_from_aggregated() for controller-side reporting - Add static deadline mode (_is_xdist flag) preserving dynamic for non-xdist - Add MergifyCIInsights.load_flaky_detector_from_context() - Add controller hooks (pytest_configure_node, pytest_testnodedown) - Add worker hooks (workerinput loading, workeroutput export) - Disable flaky detection under 'each' scheduling mode - Add pytest-xdist dev dependency and integration tests Fixes: MRGFY-6296 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Change-Id: I2811d2999ee3b51fb4649e2f60c33c200635435b

Copilot AI review requested due to automatic review settings March 19, 2026 13:07

mergify bot had a problem deploying to Mergify Merge Protections March 19, 2026 13:08 Failure

mergify bot requested a review from a team March 19, 2026 13:09

Copilot AI reviewed Mar 19, 2026

View reviewed changes

mergify bot added the review threads unresolved label Mar 19, 2026

remyduthu force-pushed the devs/remyduthu/remyduthu/mrgfy-6296-add-support-for-pytest-xdist-to-our-flaky-detection-system/I2811d2999ee3b51fb4649e2f60c33c200635435b branch from 2a21e1d to a84cc23 Compare March 19, 2026 13:16

mergify bot had a problem deploying to Mergify Merge Protections March 19, 2026 13:17 Failure

remyduthu commented Mar 19, 2026

View reviewed changes

remyduthu self-assigned this Mar 19, 2026

remyduthu force-pushed the devs/remyduthu/remyduthu/mrgfy-6296-add-support-for-pytest-xdist-to-our-flaky-detection-system/I2811d2999ee3b51fb4649e2f60c33c200635435b branch from a84cc23 to 145f31c Compare March 19, 2026 13:47

mergify bot had a problem deploying to Mergify Merge Protections March 19, 2026 13:47 Failure

remyduthu force-pushed the devs/remyduthu/remyduthu/mrgfy-6296-add-support-for-pytest-xdist-to-our-flaky-detection-system/I2811d2999ee3b51fb4649e2f60c33c200635435b branch from 145f31c to 08a7051 Compare March 19, 2026 14:12

mergify bot had a problem deploying to Mergify Merge Protections March 19, 2026 14:12 Failure

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(flaky-detection): Add pytest-xdist support to flaky detection system#384

feat(flaky-detection): Add pytest-xdist support to flaky detection system#384
remyduthu wants to merge 1 commit intomainfrom
devs/remyduthu/remyduthu/mrgfy-6296-add-support-for-pytest-xdist-to-our-flaky-detection-system/I2811d2999ee3b51fb4649e2f60c33c200635435b

remyduthu commented Mar 19, 2026

Uh oh!

mergify bot commented Mar 19, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

remyduthu Mar 19, 2026

Uh oh!

remyduthu Mar 19, 2026

Uh oh!

remyduthu Mar 19, 2026

Uh oh!

remyduthu commented Mar 19, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

		dist_mode = getattr(node.config.option, "dist", None)
		if dist_mode == "each":

	dist_mode = getattr(node.config.option, "dist", None)
	if dist_mode == "each":
	if getattr(node.config.option, "dist", None) == "each":

Uh oh!

Conversation

remyduthu commented Mar 19, 2026

Uh oh!

mergify bot commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🔴 Approval

🔴 🔎 Reviews

🟢 Continuous Integration

🟢 Enforce conventional commit

🟢 📕 PR description

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

remyduthu Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

remyduthu Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

remyduthu Mar 19, 2026

Choose a reason for hiding this comment

Uh oh!

remyduthu commented Mar 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code review

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

mergify bot commented Mar 19, 2026 •

edited

Loading

remyduthu commented Mar 19, 2026 •

edited

Loading