[MAINT] run tests on parallel threads

[Remi-Gau]

please ignore for now: just using this to get a listing of the failing tesrts in CI

• relates to nibabel thread safe? #1454

[Remi-Gau]

dirty script used to automatically mark as thread_unsafe tests parallel failiing in CI

mostly putting that here in case I nuke my local repo

from pathlib import Path
from rich import print

dry_run = False

with (Path(__file__).parent / "tmp2.txt").open() as f:
    content = f.readlines()

test_files = {}

is_summary = False
for l in content:
    if "short test summary info" in l:
        is_summary = True
        continue

    if is_summary:
        if "PARALLEL FAILED" not in l:
            continue

        file_fn_msg = l.split("PARALLEL FAILED\x1b[0m ", maxsplit=1)[1]
        file_fn = file_fn_msg.split("\x1b[0m - ", maxsplit=1)[0]
        file_fn = file_fn.replace("\n", "")

        tmp = file_fn.split("::")

        file = tmp[0]
        if file not in test_files:
            test_files[file] = {"fn":[], "cls": {}}

        if len(tmp) == 2:
            fn = tmp[1].replace("\x1b[1m", "")
            if fn not in test_files[file]["fn"]:
                test_files[file]["fn"].append(fn)
            del fn

        elif len(tmp) == 3:
            cls = tmp[1].replace("\x1b[1m", "")
            meth = tmp[2].replace("\x1b[1m", "")
            if cls not in test_files[file]["cls"]:
                test_files[file]["cls"][cls] = []
            if meth not in test_files[file]["cls"][cls]:
                test_files[file]["cls"][cls].append(meth)
            del cls
            del meth

root_dir = Path(__file__).parent / ".."

for x in test_files:
    with (root_dir / x).open("r") as f:
        content = f.readlines()
    
    with (root_dir / x).open("w") as f:

        in_class = False
        current_class = None

        skip = False
        for l in content:

            if "pytest.mark.thread_unsafe" in l:
                skip = True
                f.write(l)
                continue

            for cls in test_files[x]["cls"]:
                if l.startswith(f"class {cls}"):
                    in_class = True
                    current_class = cls
                    break

            fn_to_remove = None
            for fn in test_files[x]["fn"]:
                if  l.startswith(f"def {fn}"):
                    in_class = False
                    current_class = None
                    fn_to_remove = fn
                    print(f"marking {x}::{fn}")
                    if not skip and not dry_run:
                        f.write("@pytest.mark.thread_unsafe\n")
                        skip = False
                    break
            if fn_to_remove is not None:
                test_files[x]["fn"].remove(fn_to_remove)

            if in_class:
                meth_to_remove = None
                for meth in test_files[x]["cls"][current_class]:
                    if  f"def {meth}" in l:
                        meth_to_remove = meth
                        print(f"marking {x}::{cls}::{meth}")
                        if not skip and not dry_run:
                            f.write("    @pytest.mark.thread_unsafe\n")
                            skip = False
                        break
                if meth_to_remove is not None:
                    test_files[x]["cls"][current_class].remove(meth_to_remove)

            f.write(l)

print(test_files)

[effigies]

Do you want CI to run? LMK make you a member of the org...

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 95.43%. Comparing base (f105777) to head (efccdb6).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1455      +/-   ##
==========================================
+ Coverage   95.42%   95.43%   +0.01%     
==========================================
  Files         209      209              
  Lines       29822    29968     +146     
  Branches     4483     4483              
==========================================
+ Hits        28457    28601     +144     
- Misses        930      932       +2     
  Partials      435      435              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Remi-Gau]

Do you want CI to run?

getting it to run on my fork is mostly fine for now

[Remi-Gau]

@effigies
things seem green but from my nilearn experience things may be a bit flaky with parallel thread testing, so I would not be surprised if some things came out red once in a while

Next step: do you want to try to integrate this into tox.ini and use the pre-existing CI workflow instead of adding a new one?