Fix EADDRINUSE errors in distributed tests with port manager and retry logic

[littlebullGit]

Pull Request Description

What does this PR do?

Fixes #21308

This PR eliminates EADDRINUSE (address already in use) errors in distributed tests by implementing a two-layer port management strategy.

Problem

Distributed tests were experiencing frequent EADDRINUSE errors in CI because:

• OS keeps released ports in TIME_WAIT state for 30-120 seconds
• Tests immediately tried to reuse the same ports → collision
• Spawned processes set MASTER_PORT without tracking
• No mechanism to prevent rapid port reallocation

Solution

Layer 1: Proactive Prevention (Deque-based tracking)

New PortManager class (src/lightning/fabric/utilities/port_manager.py):

• 1024-slot queue to track recently released ports
• Prevents reallocation until ports cycle out
• Thread-safe singleton for process-wide coordination
• Tracks externally assigned ports via reserve_existing_port()

Environment integration (src/lightning/fabric/plugins/environments/lightning.py):

• Check and reserve pre-existing MASTER_PORT on startup
• Properly release and clean up ports in teardown()

Test fixtures (tests/*/conftest.py):

• Capture and track MASTER_PORT set by spawned child processes
• Clean up environment variables between tests

Layer 2: Reactive Recovery (Retry logic)

Automatic retry on EADDRINUSE (tests/*/conftest.py):

• pytest_runtest_makereport() hook detects EADDRINUSE errors
• Automatically retries up to 3 times with 0.5s delay
• Clears port manager state between retries
• Logs retry attempts for debugging

Why Both Layers?

Deque alone: Would fail if >1024 ports allocated during TIME_WAIT window
Retry alone: Would be slow and mask the underlying problem

Together: Deque prevents 99% of conflicts, retry handles the 1% edge case

Changes

Core Implementation

• ✅ New: src/lightning/fabric/utilities/port_manager.py (212 lines, 100% test coverage)
• ✅ Modified: src/lightning/fabric/plugins/environments/lightning.py (port reservation/cleanup)
• ✅ Modified: src/lightning/fabric/utilities/__init__.py (export port manager)

Test Infrastructure

• ✅ New: tests/tests_fabric/utilities/test_port_manager.py (45 tests)
• ✅ Modified: tests/tests_fabric/conftest.py (retry logic)
• ✅ Modified: tests/tests_pytorch/conftest.py (retry logic)

Test Results

tests/tests_fabric/utilities/test_port_manager.py::45 tests PASSED
tests/tests_fabric/plugins/environments/test_lightning.py::10 tests PASSED

Does your PR introduce any breaking changes?

No - All changes are internal to the test infrastructure. No user-facing API changes.

Before submitting

• Was this discussed/approved via a GitHub issue? (yes, EADDRINUSE errors in distributed tests #21308)
• Did you write any new necessary tests?
• Did you make sure all tests pass locally?
• Did you update the documentation (if necessary)?
  • Not necessary - internal test infrastructure only

---

📚 Documentation preview 📚: https://pytorch-lightning--21309.org.readthedocs.build/en/21309/

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87%. Comparing base (f58a176) to head (fa8c278).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #21309    +/-   ##
========================================
- Coverage      87%      87%    -0%     
========================================
  Files         269      270     +1     
  Lines       23707    23801    +94     
========================================
- Hits        20679    20643    -36     
- Misses       3028     3158   +130     

[Borda]

@deependujha let's merge this one as it is to the unblock master and prepare another/follow-up PR with your suggestions 🦩

[deependujha]

interesting. seems more work is required. Also copying logs from litbot UI doesn't work properly.

Only copies what's visible on screen, not the selection.

[Borda]

Only copies what's visible on screen, not the selection.

E           torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 50379, useIpv6: false, code: -98, name: EADDRINUSE, message: address already in use

../.venv/lib/python3.12/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError

[deependujha]

@Borda try selecting longer piece of text. In my case, it copies only a small portion of the selected text.

[deependujha]

thanks for the great work @littlebullGit 💜⚡️

[deependujha]

Hi @littlebullGit, we're still sometimes getting the error: https://lightning.ai/lightning-ai/ci/jobs/ci-run-lightning-ai-pytorch-lightning-213433f-pytorch-yml-lit-job-lightning-3-12-c7ac01cf?app_id=jobs&job_detail_tab=logs

We run gpu tests in a batch of 5. The current implementation of PortManager is thread-safe. I believe, we need to make it process safe.

Maybe it should track used ports via a file (or similar) to coordinate across processes?

[littlebullGit]

Hi @littlebullGit, we're still sometimes getting the error: https://lightning.ai/lightning-ai/ci/jobs/ci-run-lightning-ai-pytorch-lightning-213433f-pytorch-yml-lit-job-lightning-3-12-c7ac01cf?app_id=jobs&job_detail_tab=logs

We run gpu tests in a batch of 5. The current implementation of PortManager is thread-safe. I believe, we need to make it process safe.

Maybe it should track used ports via a file (or similar) to coordinate across processes?

Saw your PR. Will take a look and see if I can help.

[littlebullGit]

Hi @littlebullGit, we're still sometimes getting the error: https://lightning.ai/lightning-ai/ci/jobs/ci-run-lightning-ai-pytorch-lightning-213433f-pytorch-yml-lit-job-lightning-3-12-c7ac01cf?app_id=jobs&job_detail_tab=logs
We run gpu tests in a batch of 5. The current implementation of PortManager is thread-safe. I believe, we need to make it process safe.
Maybe it should track used ports via a file (or similar) to coordinate across processes?

Saw your PR. Will take a look and see if I can help.

#21313
@deependujha please see above PR