Buildkite CI builds that include certain projects are testing clang twice

[DavidSpickett]

I noticed this looking into #110239, and its build logs https://buildkite.com/llvm-project/github-pull-requests/builds/104650#0192330a-3b1e-45ca-845c-1c442bc2f321.

If you look at the ninja targets used on Linux:

ninja -C <...> -k 0 check-all check-clang check-clang-tools

It's testing everything and then testing clang and clang-tools for a second time. Or rather, it's reversed due to however ninja orders targets, but the point is that it doesn't know that "check-all" is a superset of "check-anything-else".

We can see it finish one check and start lit again for the next one:

GTotal Discovered Tests: 38089^M
G  Skipped          :     4 (0.01%)^M
G  Unsupported      :    88 (0.23%)^M
G  Passed           : 37966 (99.68%)^M
G  Expectedly Failed:    31 (0.08%)^M
G^M[8000/8003] Generating GwpAsan-x86_64-Test^[[K^[_bk;t=1727436048237^G^M[8001/8003] Running all regression tests^[[K^[_bk;t=1727436048237^G^M
<...>
Gllvm-lit: /var/lib/buildkite-agent/builds/linux-56-59b8f5d88-f4kc6-1/llvm-project/github-pull-requests/llvm/utils/lit/lit/main.py:72: note: The test suite configuration requested an individual test timeout of 0 seconds but a timeout of 1200 seconds was requested on the command line. Forcing timeout to be 1200 seconds.^M
G-- Testing: 87609 tests, 56 workers --^M
GPASS: Clang :: Driver/clang-offload-bundler.c (1 of 87609)^M
GPASS: Clang :: Driver/darwin-ld.c (2 of 87609)^M

And confirm that the same test is run twice:

$ cat github-pull-requests_build_104650_linux-linux-x64\ \(1\).log | grep "clang-offload-bundler.c"
clang-offload-bundler.c (10908 of 21210)
clang-offload-bundler.c (1 of 87609)

This is about a 24% more tests than it needs to run, at least for projects alone. About 6 extra minutes on my development box, 45% more time spent than just check-all.

This does not happen for Windows but only by chance:

-- Build files have been written to: C:/ws/src/build
+ echo '--- ninja'
--- ninja
+ ninja -C C:/ws/src/build -k 0 check-clang check-clang-tools

The check-all target is added by compiler-rt in .ci/generate-buildkite-pipeline-premerge:

function check-targets() {
  projects=${@}
  for project in ${projects}; do
    case ${project} in
    clang-tools-extra)
      echo "check-clang-tools"
    ;;
    compiler-rt)
      echo "check-all"
    ;;

However compiler-rt testing is disabled for Windows:

function exclude-windows() {
  projects=${@}
  for project in ${projects}; do
    case ${project} in
    cross-project-tests) ;; # tests failing

This will happen with builds that include pstl and libclc also:

$ cat .ci/generate-buildkite-pipeline-premerge | grep "check-all" -C 1
    compiler-rt)
      echo "check-all"
    ;;
--
    pstl)
      echo "check-all"
    ;;
    libclc)
      echo "check-all"
    ;;

I think this pipeline builder script needs to know that check-all is a superset of all the enabled check targets.

[DavidSpickett]

I originally looked at the logs to see if I could combine the XML results files into one final report so that early failures aren't buried in the log. If more runs are just check-all that will solve that problem for at least those runs.

[DavidSpickett]

Address compiler-rt: #111515
Address libclc: #111547

pstl remains, looking for someone who knows how to build the test suite.