Test suite: When skipping tests, skip them ONLY in GitHub Actions CI; do NOT skip them in other CI (such as PkgEval) #984
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The purpose of skipping tests in GitHub Actions CI is to reduce the noise here. When there are lots of CI failures in this repo, people have a hard time knowing which CI failures are "real" (due to their PR), and which ones are not related to their PR. This sea of red ❌'s increases the risk that someone merges a PR that actually does introduce a new bug or new test failure.
Therefore, in the GitHub Actions CI on this repo, we skip various tests, based on the Julia version. When we do so, we:
@infoor@warnsaying that we're skipping a test, and ideally that log message also mentions the GitHub Issue number (or includes the URL of the GitHub Issue).@test_skip falseor@test_broken false, just for completion.This allows us to have green CI on this repo, even though there are known bugs.
In other CI providers, such as Julia PkgEval.jl (which sets the
CIenvironment variable), we intentionally do NOT want to skip these tests. Because if we skip these tests in PkgEval, we greatly reduce the effectiveness of PkgEval on this package. Therefore, if PkgEval is running the PackageCompiler tests, we should be running all (or almost all) of our test suite.This is why, in this PR, we specifically skip tests based on the value of
is_gha_ci, which we set based on the value of theGITHUB_ACTIONSenvironment variable, which is specific to GitHub Actions.