[BUG] macOS: `make test-fast` hangs indefinitely

[RecreationalMath]

Description

Running make test-fast on macOS hangs indefinitely during tests/utils/test_dist__gather__losses.py. The pytest process enters a sleeping state and never makes further progress. I left it running for over 70 minutes and observed only about 1 minute of cumulative CPU time.

There is no exception and no traceback, so nothing is printed when the hang occurs. Nothing in the terminal indicates which file is responsible.

Reproduction

On macOS:

git clone https://github.com/lightly-ai/lightly.git
cd lightly
uv venv
source .venv/bin/activate
make install-dev
make test-fast

Or isolated to the offending file:

pytest tests/utils/test_dist__gather__losses.py -v

Expected: the suite completes in a few minutes.

Actual: hangs after 1 to 2 minutes of apparent progress, at or during tests/utils/test_dist__gather__losses.py. The hang is presistant on macOS across repeated runs.

Orphaned child processes after kill

If the hung pytest is terminated (Ctrl-C or kill), the spawned child process is frequently orphaned. The orphan remains in a sleeping state with roughly 500 MB to 1 GB resident memory and does not exit on its own, it must be cleaned up manually with kill -9 <pid>. Observed repeatedly during investigation.

Not observed in CI

The four tests in TestGatherLayer_Losses are guarded by a class-level pytest.mark.skipif decorator gated on the GITHUB_ACTIONS environment variable, so they are always skipped on GitHub Actions runners. The existing skip reason attributes the skip to a codecov compatibility concern, not a platform compatibility concern. Local macOS contributors have no equivalent skip applied.

Environment

• OS: macOS 26.4.1 (arm64)
• Python: 3.12.13
• torch: 2.11.0
• pytorch-lightning: 2.6.1
• Installed via: make install-dev at commit c792812 on master
• Invocation: make test-fast

Impact

• Affected: contributors running the full test suite on macOS locally.
• Not affected: GitHub Actions CI and Linux contributors.
• Severity: high friction for new and returning macOS contributors. There is no error, no progress indication, and the hang is easy to mistake for a slow test. Cleanup after a kill also requires manual intervention to remove orphaned child processes.

[RecreationalMath]

FYI: I ran into this bug while I was testing #1916 locally on my MacBook.

[RecreationalMath]

Spent some time looking further into this and ran pytest tests/utils/test_dist__gather__losses.py -v ~40 times on the same environment as the issue body:

• 35 hung indefinitely (87.5%)
• 3 died with silent SIGKILL exit 137 inside ~15s
• 2 completed in under 5s with an assertion failure on test_loss_dcl
• 0 passed cleanly

Every hang looked the same, parent pytest plus exactly one spawned DDP child, both in SN (sleeping, nice) state, zero CPU, orphans-on-kill. The test that hangs varies across runs (30 hangs on test_loss_tico, 3 on test_loss_vicreg, 2 on test_loss_dcl). test_loss_ntxent passed every time.

Where the hang lives in the code: All four tests delegate to the private helper _test_ddp at line 138. The hang is the trainer.fit call on line 176, where the Trainer is built with DDPStrategy(find_unused_parameters=False) on accelerator="cpu", devices=2. The bug is in the DDP trainer lifecycle, not in any specific loss.

Potential root cause (spawn vs fork): DDPStrategy launches a worker process. Linux defaults to fork, so the child inherits the parent's memory and sees pytest's already-loaded state. macOS defaults to spawn since Python 3.8, so the child starts a fresh interpreter and re-imports the test module, re-entering pytest's fixture machinery in an unexpected context. The child blocks on IPC waiting for the parent, the parent waits for the child to join the DDP process group, and neither makes progress.

Workaround for macOS contributors until a fix lands: GITHUB_ACTIONS=true make test-fast reuses the existing skipif, or pytest tests --ignore=tests/utils/test_dist__gather__losses.py to be more precise.

Recommended fix for now: broaden the existing skipif

import sys

@pytest.mark.skipif(
    os.getenv("GITHUB_ACTIONS") is not None or sys.platform == "darwin",
    reason=(
        "Skipped on GitHub Actions (breaks codecov under multi-process runs) "
        "and on macOS (DDPStrategy + spawn start method deadlocks inside pytest)."
    ),
)
class TestGatherLayer_Losses:
    ...

One-line change. Loses macOS coverage of the gather path, but that coverage is already absent in CI, so no net regression.

Alternatives:

• Force fork inside this test file. fork is unsafe on macOS: CoreFoundation and Objective-C runtimes abort on first use after fork without a subsequent exec, which crashes any library that touches macOS-native code. High risk of introducing new flakes.
• Rewrite the test to avoid DDPStrategy by mocking world_size like test_lejepa_sigreg_loss.py::test_forward_gather_distributed_world_size_gt_one does. Preserves coverage on all platforms, but preserving the single-device reference comparison is non-trivial.
• Document the workaround in CONTRIBUTING.md. Contributors still hit the hang on their first run.

One thing to note: even if the skip lands, in the rare runs where DDP completes on macOS, the trained params diverge from the hardcoded reference far beyond the rtol=1e-3 tolerance, so the test fails anyway. The class docstring states the reference was generated from a single-device run, but the platform used is not documented. If someone revisits macOS coverage, the reference may need to be regenerated.

Happy to open a PR for the skipif change if it makes sense for now.

[liopeer]

Thanks, I agree on the recommendation with just skipping those tests on macOS. I don't think anyone cares about DDP support on macOS anyway (since you simply don't get the hardware for it). Created an internal ticket to fix this, but if you'd like to open a PR we're also happy to receive that one.

[RecreationalMath]

Sure, I'll send a PR shortly.

[gabrielfruet]

Regarding this, my macOS doesn't hang with these tests. In fact, I have another problem where DCL loss targets are not proper.

Could you please give more steps on how to reproduce this issue @RecreationalMath ?

[RecreationalMath]

Regarding this, my macOS doesn't hang with these tests.
Could you please give more steps on how to reproduce this issue @RecreationalMath ?

Re-verified the hang just now on the current upstream master HEAD (7675ddae) before drafting this. Same fingerprint as the original analysis above: parent pytest plus exactly one spawned DDP child, both in SN state, log freezes mid-test_loss_vicreg after test_loss_ntxent and test_loss_tico pass.

Steps that triggered the hang on my machine (even on a fresh clone):

git clone https://github.com/lightly-ai/lightly.git
cd lightly
# current master HEAD -> 7675ddae at time of writing
uv venv
source .venv/bin/activate
make install-dev
pytest tests/utils/test_dist__gather__losses.py -v

Expected: 4 tests run.
Actual on my machine: hangs after 1 to 2 tests pass. No output, no traceback. After about a minute, ps shows both parent and child in SN state with near-zero CPU.

In fact, I have another problem where DCL loss targets are not proper.

In the 40-run characterisation from my earlier comment, 2 of my 40 runs completed in under 5 seconds with exactly this assertion failure on test_loss_dcl. The trained params on macOS DDP completion diverged from the hardcoded reference far beyond the rtol=1e-3 tolerance. The class docstring states the reference was generated on a single-device run, but the platform used is not documented, so macOS multi-device DDP produces different numerics when it does happen to complete. Out of the 40 runs the breakdown was 35 hang, 3 silent SIGKILL exit 137, 2 DCL assertion failure, 0 clean pass. Your environment may simply land in the "DDP completes" bucket where my environment lands in the "hang" bucket.

Could you share your environment (OS version + Python + torch + pytorch-lightning versions)? That may help pin down which axis flips the failure mode from hang to assertion failure, from one machine to another. Either way, the skipif fix in #1922 sidesteps both on macOS without affecting CI.