test: Pre-compile common wheels in base test env#13715
Merged
Conversation
The test suite disables Python bytecode caching so if we don't pre-compile the bytecode for coverage, setuptools, and pytest-subket which are all preinstalled in the test environments, we pay a heavy import time penalty (e.g., setuptools during a PEP 517 build). pip is already pre-compiled, FWIW. This is doubly bad for coverage and pytest-subket since they are imported on every Python process startup. Locally, this shaves a full minute from a full test suite run down to 2:45 from 3:45.
ichard26
added
C: tests
Member
Author
|
Since the individual job times are highly variable, I've taken to use the total billing minutes as a way to determine whether we're saving time in aggregate or not. This PR is at 3 hours and 15 minutes. This is an improvement from:
For reference, this scheduled CI run on main from Nov 23 is at 4 hours and 8 minutes and this scheduled run from Jun 29 is at 3 hours and 21 minutes. So, it looks like we still have some progress to make. I do wonder what made our test suite so much slower all of a sudden. Probably the removal of legacy builds, but I would have do some testing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The test suite disables Python bytecode caching so if we don't pre-compile the bytecode for coverage, setuptools, and pytest-subket which are all preinstalled in the test environments, we pay a heavy import time penalty (e.g., setuptools during a PEP 517 build). pip is already pre-compiled FWIW.
This is doubly bad for coverage and pytest-subket since they are imported on every Python process startup.
Locally, this shaves a full minute from a full test suite run down to 2:45 from 3:45.
Towards #13707.
Another idea I have is to stop copying the test data for every test that requests it. This will need some work since AFAIU some tests depend on being to write freely to the temporary test data folder, but there's no reason why they can't use the normal temporary directory pytest provides. This will hopefully partially alleviate the issues we have with abysmal file I/O performance on Windows.