Skip to content

Conversation

@DilumAluthge
Copy link
Member

The purpose of skipping tests in GitHub Actions CI is to reduce the noise here. When there are lots of CI failures in this repo, people have a hard time knowing which CI failures are "real" (due to their PR), and which ones are not related to their PR. This sea of red ❌'s increases the risk that someone merges a PR that actually does introduce a new bug or new test failure.

Therefore, in the GitHub Actions CI on this repo, we skip various tests, based on the Julia version. When we do so, we:

  1. Add a comment in the code explaining why we're skipping it, and providing the URL of the relevant GitHub Issue for the underlying bug
  2. Ideally, also print an @info or @warn saying that we're skipping a test, and ideally that log message also mentions the GitHub Issue number (or includes the URL of the GitHub Issue).
  3. Do a @test_skip false or @test_broken false, just for completion.

This allows us to have green CI on this repo, even though there are known bugs.

In other CI providers, such as Julia PkgEval.jl (which sets the CI environment variable), we intentionally do NOT want to skip these tests. Because if we skip these tests in PkgEval, we greatly reduce the effectiveness of PkgEval on this package. Therefore, if PkgEval is running the PackageCompiler tests, we should be running all (or almost all) of our test suite.

This is why, in this PR, we specifically skip tests based on the value of is_gha_ci, which we set based on the value of the GITHUB_ACTIONS environment variable, which is specific to GitHub Actions.

… do NOT skip them in other CI (such as PkgEval)
@codecov
Copy link

codecov bot commented Oct 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 84.56%. Comparing base (cdde346) to head (90f1b6f).
Report is 1 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master     #984   +/-   ##
=======================================
  Coverage   84.56%   84.56%           
=======================================
  Files           3        3           
  Lines         823      823           
=======================================
  Hits          696      696           
  Misses        127      127           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@DilumAluthge DilumAluthge merged commit 87e1e77 into master Oct 21, 2024
44 checks passed
@DilumAluthge DilumAluthge deleted the dpa/is-gha branch October 21, 2024 21:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant