[clangd] flaky tests on clang-aarch64-quick buildbot

[vbvictor]

This clang-aarch64-quick buildbot somewhat consistently fails due to failed Clangd tests. This could lead to unwanted messaged like this: #165175 (comment).

Here are some test runs with failures:

• https://lab.llvm.org/buildbot/#/builders/65/builds/24598 - Clangd Unit Tests::ClangdTests/242
• https://lab.llvm.org/buildbot/#/builders/65/builds/24594 - Clangd Unit Tests::ClangdTests/ClangdServerTest/Reparse
• https://lab.llvm.org/buildbot/#/builders/65/builds/24578 - Clangd Unit Tests::ClangdTests/243
• https://lab.llvm.org/buildbot/#/builders/65/builds/24576 - Clangd Unit Tests::ClangdTests/112
• https://lab.llvm.org/buildbot/#/builders/65/builds/24543 - Clangd Unit Tests::ClangdTests/ClangdServer/TidyOverrideTest

[llvmbot]

@llvm/issue-subscribers-clangd

Author: Baranov Victor (vbvictor)

This [clang-aarch64-quick buildbot](https://lab.llvm.org/buildbot/#/builders/65) somewhat consistently fails due to failed Clangd tests. This could lead to unwanted messaged like this: https://github.com//pull/165175#issuecomment-3453642794.

Here are some test runs with failures:

• https://lab.llvm.org/buildbot/#/builders/65/builds/24598 - Clangd Unit Tests::ClangdTests/242
• https://lab.llvm.org/buildbot/#/builders/65/builds/24594 - Clangd Unit Tests::ClangdTests/ClangdServerTest/Reparse
• https://lab.llvm.org/buildbot/#/builders/65/builds/24578 - Clangd Unit Tests::ClangdTests/243
• https://lab.llvm.org/buildbot/#/builders/65/builds/24576 - Clangd Unit Tests::ClangdTests/112
• https://lab.llvm.org/buildbot/#/builders/65/builds/24543 - Clangd Unit Tests::ClangdTests/ClangdServer/TidyOverrideTest

[DavidSpickett]

Thanks for the report. This happens to these "quick" bots sometimes because they "roam free" so to speak, on a shared server with many other bots, using whatever CPU is free at the time. So if we get unlucky with scheduling, processes can be stalled for long periods of time. We could assign all cores to specific bots, but for various reasons, mainly to make best usage of the whole machine's power, we prefer to have a few that are flexible.

All the failures here are not flakes in clangd itself, it's the default timeout of blockUntilIdleForTest running out. A sample of a few failed builds:

Built preamble of size 817532 for file /clangd-test/foo.h version null in 12.09 seconds
Built preamble of size 737152 for file /clangd-test/Foo1.cpp version null in 16.24 seconds
Built preamble of size 737152 for file /clangd-test/Foo2.cpp version null in 18.43 seconds
Built preamble of size 817632 for file /clangd-test/bar.cpp version null in 21.11 seconds
Built preamble of size 737140 for file /clangd-test/foo.cpp version 1 in 22.06 seconds
Built preamble of size 817616 for file /clangd-test/baz.cpp version null in 19.21 seconds
Built preamble of size 817632 for file /clangd-test/bar.cpp version null in 21.71 seconds
Built preamble of size 817632 for file /clangd-test/foo.cpp version null in 12.68 seconds
Built preamble of size 817632 for file /clangd-test/foo.cpp version null in 37.32 seconds

With 37.32 being the high score. I could bump the timeout to 40 seconds, but why not 50 and so on, it never ends. In theory the OS could stall a process for hours in the right (/wrong) circumstance.

As this timeout has been reasonable for everyone else, I'm going to disable clangd testing on our "quick" bots. Tests will still be run on our other bots, the fastest of which being around 30 minutes which I think is ok. Those other bots have not shown this flake to my knowledge.

[DavidSpickett]

We've had flaky lit tests for the same reason. I'm disabling those with llvm/llvm-zorg#637.

[vbvictor]

Thank you for taking a look at this quickly!

[DavidSpickett]

I've merged the zorg changes but it will be a few days before they show up on the build bots themselves.

[DavidSpickett]

Changes have now been applied:

cmake -G Ninja ../llvm/llvm -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=True '-DLLVM_LIT_ARGS='"'"'-v'"'"'' -DCMAKE_INSTALL_PREFIX=../stage1.install '-DLLVM_TARGETS_TO_BUILD='"'"'AArch64'"'"'' '-DLLVM_LIT_ARGS=-v --filter-out '"'"'(googletest-timeout|timeout-hang|max-time)\.py'"'"'' '-DLLVM_ENABLE_PROJECTS=llvm;clang'

Please raise another issue if you see other flaky tests.