Skip to content

refactor(tests): split up test suites, update docs #278

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

vados-cosmonic
Copy link
Contributor

No description provided.

@vados-cosmonic vados-cosmonic force-pushed the refactor(tests)=separate-suites branch 8 times, most recently from c0a4cb6 to 0c43f12 Compare July 31, 2025 10:09
@vados-cosmonic vados-cosmonic changed the title refactor(tests): separate test suites refactor(tests): split up test suties, update docs Jul 31, 2025
@vados-cosmonic vados-cosmonic force-pushed the refactor(tests)=separate-suites branch from 0c43f12 to 6d76607 Compare July 31, 2025 10:19
@vados-cosmonic vados-cosmonic changed the title refactor(tests): split up test suties, update docs refactor(tests): split up test suites, update docs Jul 31, 2025
@vados-cosmonic
Copy link
Contributor Author

OK, so this PR is ready for review, but I'm a little dissatisfied with the consistency of CI -- running with more concurrency seems to have made tests more unstable.

Going to re-run the latest job a few more times to make sure it passes most of the time so that the average PR should pass without issues.

@vados-cosmonic
Copy link
Contributor Author

OK unfortunately there were still a couple failures -- (mac/windows + weval). I wonder if we can get away with disabling test concurrency for weval runs... If that's more consistent then I'll go with that, otherwise I'll just disable test-level concurrency all together (suite level concurrency still happens)

@vados-cosmonic
Copy link
Contributor Author

vados-cosmonic commented Aug 1, 2025

OK, going to rip out concurrency here -- tests consistently passing is more important I think.

[EDIT] - Actually the test that is still failing is a Weval one which is timing out... Updating test config to run only one concurrent suite at a time if we're doing that.

@tschneidereit
Copy link
Member

@vados-cosmonic same request I often have: can you update the description here to add some motivation for this? It's so very much easier to review PRs when I know which lens to look through. It's also very useful for looking at changes later on.

@vados-cosmonic
Copy link
Contributor Author

Hey @tschneidereit yeah it's been a while on this PR -- I believe I was trying to finish the switch to vitest in this PR, in particular a follow up to this PR:

#277

In particular this comment here:

#277 (comment)

When not in CI (on really slow machines), this PR makes the tests finish faster by separating one large suite into multiple (that can run in parallel under vitest).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants