Skip to content

Conversation

bd1976bris
Copy link
Collaborator

The Clang driver tests tried to match the Clang executable name via a regular expression. This failed on some buildbots where the name was not anticipated.

Make the test more robust by extracting the filename with Python and appending a line to the output. This is then captured into a FileCheck variable and used directly in the check for the executable name.

Should fix buildbot failures caused by the merge of PR #159129.

The Clang driver tests tried to match the Clang executable name via a
regular expression. This failed on some buildbots where the name was not
anticipated.

Make the test more robust by extracting the filename with Python and
appending a line to the output. This is then captured into a FileCheck
variable and used directly in the check for the executable name.

Should fix buildbot failures caused by the merge of PR llvm#159129.
@llvmbot llvmbot added clang Clang issues not falling into any other category clang:driver 'clang' and 'clang++' user-facing binaries. Not 'clang-cl' labels Sep 16, 2025
@llvmbot
Copy link
Member

llvmbot commented Sep 16, 2025

@llvm/pr-subscribers-clang-driver

@llvm/pr-subscribers-clang

Author: bd1976bris (bd1976bris)

Changes

The Clang driver tests tried to match the Clang executable name via a regular expression. This failed on some buildbots where the name was not anticipated.

Make the test more robust by extracting the filename with Python and appending a line to the output. This is then captured into a FileCheck variable and used directly in the check for the executable name.

Should fix buildbot failures caused by the merge of PR #159129.


Full diff: https://github.com/llvm/llvm-project/pull/159151.diff

3 Files Affected:

  • (modified) clang/test/Driver/DTLTO/dtlto.c (+15-11)
  • (added) clang/test/Driver/DTLTO/filename.py (+3)
  • (modified) clang/test/Driver/DTLTO/ps5-dtlto.c (+17-12)
diff --git a/clang/test/Driver/DTLTO/dtlto.c b/clang/test/Driver/DTLTO/dtlto.c
index 053955a9e3d4a..8f086584e6ba7 100644
--- a/clang/test/Driver/DTLTO/dtlto.c
+++ b/clang/test/Driver/DTLTO/dtlto.c
@@ -6,14 +6,16 @@
 /// Check DTLTO options are forwarded to the linker.
 
 /// Check that options are forwarded as expected with --thinlto-distributor=.
+// RUN: %python %S/filename.py %clang > %t_forward.log
 // RUN: %clang -flto=thin %s -### -fuse-ld=lld --target=x86_64-linux-gnu \
 // RUN:   -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 \
-// RUN:   -fthinlto-distributor=d.exe -Werror 2>&1 | \
-// RUN:   FileCheck %s --check-prefix=FORWARD
+// RUN:   -fthinlto-distributor=d.exe -Werror >>%t_forward.log 2>&1
+// RUN: FileCheck %s --input-file=%t_forward.log --check-prefix=FORWARD
 
+// FORWARD: filename.py:[[CLANG:.*]]
 // FORWARD: ld.lld
 // FORWARD-SAME: "--thinlto-distributor=d.exe"
-// FORWARD-SAME: "--thinlto-remote-compiler={{.*}}clang{{[^\"]*}}"
+// FORWARD-SAME: "--thinlto-remote-compiler={{.*}}[[CLANG]]"
 // FORWARD-SAME: "--thinlto-distributor-arg=a1"
 // FORWARD-SAME: "--thinlto-distributor-arg=a2"
 // FORWARD-SAME: "--thinlto-distributor-arg=a3"
@@ -22,8 +24,8 @@
 /// that a warning is issued for unused -Xthinlto-distributor options.
 // RUN: %clang -flto=thin %s -### -fuse-ld=lld --target=x86_64-linux-gnu \
 // RUN:   -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 2>&1 | \
-// RUN:   FileCheck %s --check-prefix=NODIST --implicit-check-not=distributor \
-// RUN:     --implicit-check-not=remote-compiler
+// RUN: FileCheck %s --check-prefix=NODIST --implicit-check-not=distributor \
+// RUN:   --implicit-check-not=remote-compiler
 
 // NODIST: warning: argument unused during compilation: '-Xthinlto-distributor=a1'
 // NODIST: warning: argument unused during compilation: '-Xthinlto-distributor=a2,a3'
@@ -31,10 +33,11 @@
 
 /// Check the expected arguments are forwarded by default with only
 /// --thinlto-distributor=.
+// RUN: %python %S/filename.py %clang > %t_default.log
 // RUN: %clang -flto=thin %s -### -fuse-ld=lld --target=x86_64-linux-gnu \
-// RUN:   -fthinlto-distributor=d.exe -Werror 2>&1 | \
-// RUN:   FileCheck %s --check-prefix=DEFAULT --implicit-check-not=distributor \
-// RUN:     --implicit-check-not=remote-compiler
+// RUN:   -fthinlto-distributor=d.exe -Werror >>%t_default.log 2>&1
+// RUN: FileCheck %s --input-file=%t_default.log --check-prefix=DEFAULT \
+// RUN:   --implicit-check-not=distributor --implicit-check-not=remote-compiler
 
 // DEFAULT: ld.lld
 // DEFAULT-SAME: "--thinlto-distributor=d.exe"
@@ -42,10 +45,11 @@
 
 /// Check that nothing is forwarded when the compiler is not in LTO mode, and that
 /// appropriate unused option warnings are issued.
+// RUN: %python %S/filename.py %clang > %t_noflto.log
 // RUN: %clang %s -### -fuse-ld=lld --target=x86_64-linux-gnu \
-// RUN:   -fthinlto-distributor=d.exe 2>&1 | \
-// RUN:   FileCheck %s --check-prefix=NOFLTO --implicit-check-not=distributor \
-// RUN:     --implicit-check-not=remote-compiler
+// RUN:   -fthinlto-distributor=d.exe  >>%t_noflto.log 2>&1
+// RUN: FileCheck %s --input-file=%t_noflto.log --check-prefix=NOFLTO \
+// RUN:   --implicit-check-not=distributor --implicit-check-not=remote-compiler
 
 // NOFLTO: warning: argument unused during compilation: '-fthinlto-distributor=d.exe'
 // NOFLTO: ld.lld
diff --git a/clang/test/Driver/DTLTO/filename.py b/clang/test/Driver/DTLTO/filename.py
new file mode 100644
index 0000000000000..6b6d483a2fd2a
--- /dev/null
+++ b/clang/test/Driver/DTLTO/filename.py
@@ -0,0 +1,3 @@
+from pathlib import Path
+import sys
+print(f"filename.py:{Path(sys.argv[1]).name}")
diff --git a/clang/test/Driver/DTLTO/ps5-dtlto.c b/clang/test/Driver/DTLTO/ps5-dtlto.c
index eb4691da9d2fc..b52765db5b1c7 100644
--- a/clang/test/Driver/DTLTO/ps5-dtlto.c
+++ b/clang/test/Driver/DTLTO/ps5-dtlto.c
@@ -6,14 +6,16 @@
 /// Check DTLTO options are forwarded to the linker.
 
 /// Check that options are forwarded as expected with --thinlto-distributor=.
+// RUN: %python %S/filename.py %clang > %t_forward.log
 // RUN: %clang -flto=thin %s -### --target=x86_64-sie-ps5 \
 // RUN:   -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 \
-// RUN:   -fthinlto-distributor=d.exe -Werror 2>&1 | \
-// RUN:   FileCheck %s --check-prefix=FORWARD
+// RUN:   -fthinlto-distributor=d.exe -Werror >>%t_forward.log 2>&1
+// RUN: FileCheck %s --input-file=%t_forward.log --check-prefix=FORWARD
 
+// FORWARD: filename.py:[[CLANG:.*]]
 // FORWARD: prospero-lld
 // FORWARD-SAME: "--thinlto-distributor=d.exe"
-// FORWARD-SAME: "--thinlto-remote-compiler={{.*}}clang{{[^\"]*}}"
+// FORWARD-SAME: "--thinlto-remote-compiler={{.*}}[[CLANG]]"
 // FORWARD-SAME: "--thinlto-distributor-arg=a1"
 // FORWARD-SAME: "--thinlto-distributor-arg=a2"
 // FORWARD-SAME: "--thinlto-distributor-arg=a3"
@@ -22,8 +24,8 @@
 /// that a warning is issued for unused -Xthinlto-distributor options.
 // RUN: %clang -flto=thin %s -### --target=x86_64-sie-ps5 \
 // RUN:   -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 2>&1 | \
-// RUN:   FileCheck %s --check-prefix=NODIST --implicit-check-not=distributor \
-// RUN:     --implicit-check-not=remote-compiler
+// RUN: FileCheck %s --check-prefix=NODIST --implicit-check-not=distributor \
+// RUN:   --implicit-check-not=remote-compiler
 
 // NODIST: warning: argument unused during compilation: '-Xthinlto-distributor=a1'
 // NODIST: warning: argument unused during compilation: '-Xthinlto-distributor=a2,a3'
@@ -31,18 +33,21 @@
 
 /// Check the expected arguments are forwarded by default with only
 /// --thinlto-distributor=.
+// RUN: %python %S/filename.py %clang > %t_default.log
 // RUN: %clang -flto=thin %s -### --target=x86_64-sie-ps5 \
-// RUN:   -fthinlto-distributor=d.exe -Werror 2>&1 | \
-// RUN:   FileCheck %s --check-prefix=DEFAULT --implicit-check-not=distributor \
-// RUN:     --implicit-check-not=remote-compiler
+// RUN:   -fthinlto-distributor=d.exe -Werror >>%t_default.log 2>&1
+// RUN: FileCheck %s --input-file=%t_default.log --check-prefix=DEFAULT \
+// RUN:   --implicit-check-not=distributor --implicit-check-not=remote-compiler
 
+// DEFAULT: filename.py:[[CLANG:.*]]
 // DEFAULT: prospero-lld
 // DEFAULT-SAME: "--thinlto-distributor=d.exe"
-// DEFAULT-SAME: "--thinlto-remote-compiler={{.*}}clang{{[^\"]*}}"
+// DEFAULT-SAME: "--thinlto-remote-compiler={{.*}}[[CLANG]]"
 
 /// Check that the arguments are forwarded unconditionally even when the
 /// compiler is not in LTO mode.
+// RUN: %python %S/filename.py %clang > %t_noflto.log
 // RUN: %clang %s -### --target=x86_64-sie-ps5 \
-// RUN:   -fthinlto-distributor=d.exe -Werror 2>&1 | \
-// RUN:   FileCheck %s --check-prefix=DEFAULT --implicit-check-not=distributor \
-// RUN:     --implicit-check-not=remote-compiler
+// RUN:   -fthinlto-distributor=d.exe -Werror >>%t_noflto.log 2>&1
+// RUN: FileCheck %s --input-file=%t_noflto.log --check-prefix=DEFAULT \
+// RUN:   --implicit-check-not=distributor --implicit-check-not=remote-compiler

Copy link

github-actions bot commented Sep 16, 2025

✅ With the latest revision this PR passed the Python code formatter.

@jplehr
Copy link
Contributor

jplehr commented Sep 16, 2025

This appears to still break the testing. Can we please revert the original PR to get our bot back to green?

@bd1976bris
Copy link
Collaborator Author

Should fix buildbot failures caused by the merge of PR #159129.

It would involve reverting a number of commits. Can you approve #159158? @petrhosek approved the same commit (although with an incorrect description) earlier: #159050.

@bd1976bris bd1976bris requested a review from nga888 September 16, 2025 20:06
bd1976bris added a commit that referenced this pull request Sep 16, 2025
)

Make the test regexes more permissive to fix buildbot failures caused by
the merge of PR #159129. This mirrors the earlier fix in PR #148908. We
retain cross-project-test coverage to verify that the path content is
correct.

A follow-up will update the tests to robustly check the Clang executable
filename, likely along the lines of PR #159151.

Short-term unbreak to keep the buildbots green without reverting a chain
of commits.
@jplehr
Copy link
Contributor

jplehr commented Sep 16, 2025

it seems that this needs rebasing.

@bd1976bris bd1976bris merged commit 2ef58b6 into llvm:main Sep 17, 2025
7 of 9 checks passed

// DEFAULT: ld.lld
// DEFAULT-SAME: "--thinlto-distributor=d.exe"
// DEFAULT-SAME: "--thinlto-remote-compiler={{[^"]+}}"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rebasing via the GitHub GUI seems to have dropped this part of the change. We should be capturing and checking: "--thinlto-remote-compiler={{.*}}[[CLANG]]" as in the other cases. I don't expect this to cause a build bot breakage as the code we have ended up with here is a revert to the original code that has lived upstream for several weeks previous to the latest round of changes.

Copy link
Collaborator Author

@bd1976bris bd1976bris Sep 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in: #159418.

@petrhosek
Copy link
Member

petrhosek commented Sep 17, 2025

This change broke our builders again:

XPASS: Clang :: Driver/DTLTO/ps5-dtlto.c (11550 of 22550)
 ******************** TEST 'Clang :: Driver/DTLTO/ps5-dtlto.c' FAILED ********************
 Exit Code: 0
 Command Output (stderr):
 --
 "/b/s/w/ir/cipd_bin_packages/cpython3/bin/python3.11" /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/filename.py /b/s/w/ir/x/w/llvm_build/bin/clang > /b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_forward.log # RUN: at line 9
 + /b/s/w/ir/cipd_bin_packages/cpython3/bin/python3.11 /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/filename.py /b/s/w/ir/x/w/llvm_build/bin/clang
 /b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c -### --target=x86_64-sie-ps5    -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3    -fthinlto-distributor=d.exe -Werror >>/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_forward.log 2>&1 # RUN: at line 10
 + /b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c -### --target=x86_64-sie-ps5 -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 -fthinlto-distributor=d.exe -Werror
 /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c --input-file=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_forward.log --check-prefix=FORWARD # RUN: at line 13
 + /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c --input-file=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_forward.log --check-prefix=FORWARD
 /b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c -### --target=x86_64-sie-ps5    -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 2>&1 |  /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c --check-prefix=NODIST --implicit-check-not=distributor    --implicit-check-not=remote-compiler # RUN: at line 25
 + /b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c -### --target=x86_64-sie-ps5 -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3
 + /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c --check-prefix=NODIST --implicit-check-not=distributor --implicit-check-not=remote-compiler
 "/b/s/w/ir/cipd_bin_packages/cpython3/bin/python3.11" /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/filename.py /b/s/w/ir/x/w/llvm_build/bin/clang > /b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_default.log # RUN: at line 36
 + /b/s/w/ir/cipd_bin_packages/cpython3/bin/python3.11 /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/filename.py /b/s/w/ir/x/w/llvm_build/bin/clang
 /b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c -### --target=x86_64-sie-ps5    -fthinlto-distributor=d.exe -Werror >>/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_default.log 2>&1 # RUN: at line 37
 + /b/s/w/ir/x/w/llvm_build/bin/clang -flto=thin /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c -### --target=x86_64-sie-ps5 -fthinlto-distributor=d.exe -Werror
 /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c --input-file=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_default.log --check-prefix=DEFAULT    --implicit-check-not=distributor --implicit-check-not=remote-compiler # RUN: at line 39
 + /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c --input-file=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_default.log --check-prefix=DEFAULT --implicit-check-not=distributor --implicit-check-not=remote-compiler
 "/b/s/w/ir/cipd_bin_packages/cpython3/bin/python3.11" /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/filename.py /b/s/w/ir/x/w/llvm_build/bin/clang > /b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_noflto.log # RUN: at line 49
 + /b/s/w/ir/cipd_bin_packages/cpython3/bin/python3.11 /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/filename.py /b/s/w/ir/x/w/llvm_build/bin/clang
 /b/s/w/ir/x/w/llvm_build/bin/clang /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c -### --target=x86_64-sie-ps5    -fthinlto-distributor=d.exe -Werror >>/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_noflto.log 2>&1 # RUN: at line 50
 + /b/s/w/ir/x/w/llvm_build/bin/clang /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c -### --target=x86_64-sie-ps5 -fthinlto-distributor=d.exe -Werror
 /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c --input-file=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_noflto.log --check-prefix=DEFAULT    --implicit-check-not=distributor --implicit-check-not=remote-compiler # RUN: at line 52
 + /b/s/w/ir/x/w/llvm_build/bin/FileCheck /b/s/w/ir/x/w/llvm-llvm-project/clang/test/Driver/DTLTO/ps5-dtlto.c --input-file=/b/s/w/ir/x/w/llvm_build/tools/clang/test/Driver/DTLTO/Output/ps5-dtlto.c.tmp_noflto.log --check-prefix=DEFAULT --implicit-check-not=distributor --implicit-check-not=remote-compiler
 --
 ********************

Looks like we need to remove the XFAIL that was added in #159129?

petrhosek added a commit to petrhosek/llvm-project that referenced this pull request Sep 17, 2025
DTLTO is still incompatible with llvm-driver, but the test started
passing after llvm#159151.
bd1976bris added a commit to bd1976bris/llvm-project that referenced this pull request Sep 17, 2025
Remove XFAILs for Multicall. DTLTO is still incompatible with
llvm-driver, but these tests now pass after llvm#159151.

Modify a missed regex to use filename.py (missed in llvm#159151).

Tighten overly greedy regexes to prevent spurious failures.
bd1976bris added a commit that referenced this pull request Sep 17, 2025
Remove XFAILs for llvm-driver. DTLTO is still incompatible with
llvm-driver, but these tests now pass after #159151.

Modify a missed regex to use filename.py (missed in #159151).

Tighten overly greedy regexes to prevent spurious failures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

clang:driver 'clang' and 'clang++' user-facing binaries. Not 'clang-cl' clang Clang issues not falling into any other category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants