[ATfL] Make sure internal shell is not used by lit

[pawosm-arm]

This is to mitigate the effects of this commit:
llvm/llvm-project@ee0f86d

[pawosm-arm]

The error fixed by this commit:

23:40:55  -- Testing: 96738 tests, 4 workers --
23:47:36  Testing:  0.. 10
23:47:36  FAIL: Clang :: Driver/env.c (13142 of 96738)
23:47:36  ******************** TEST 'Clang :: Driver/env.c' FAILED ********************
23:47:36  Exit Code: 1
23:47:36  
23:47:36  Command Output (stdout):
23:47:36  --
23:47:36  # RUN: at line 5
23:47:36  bash -c env | grep LD_LIBRARY_PATH | sed -ne 's/^.*=//p' | tr -d '\n' > /workspace/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.ld_library_path
23:47:36  # executed command: bash -c env
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # executed command: grep LD_LIBRARY_PATH
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # executed command: sed -ne 's/^.*=//p'
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # executed command: tr -d '\n'
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # RUN: at line 7
23:47:36  env -i LC_ALL=C LD_LIBRARY_PATH="/workspace/build/stage/product_build/lib:/workspace/build/atfl/lib:/workspace/build/atfl/lib/aarch64-unknown-linux-gnu:." CLANG_NO_DEFAULT_CONFIG=1    /workspace/build/stage/product_build/bin/clang /workspace/src/clang/test/Driver/env.c -### -o /workspace/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.o --target=i386-unknown-linux      --sysroot=/workspace/src/clang/test/Driver/Inputs/basic_linux_tree      --rtlib=platform --unwindlib=platform -no-pie      2>&1 | /workspace/build/stage/product_build/bin/FileCheck --check-prefix=CHECK-LD-32 /workspace/src/clang/test/Driver/env.c
23:47:36  # executed command: env -i LC_ALL=C 'LD_LIBRARY_PATH=%{readfile:/workspace/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.ld_library_path}' CLANG_NO_DEFAULT_CONFIG=1 /workspace/build/stage/product_build/bin/clang /workspace/src/clang/test/Driver/env.c '-###' -o /workspace/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.o --target=i386-unknown-linux --sysroot=/workspace/src/clang/test/Driver/Inputs/basic_linux_tree --rtlib=platform --unwindlib=platform -no-pie
23:47:36  # note: command had no output on stdout or stderr
23:47:36  # error: command failed with exit status: 127
23:47:36  # executed command: /workspace/build/stage/product_build/bin/FileCheck --check-prefix=CHECK-LD-32 /workspace/src/clang/test/Driver/env.c
23:47:36  # .---command stderr------------
23:47:36  # | /workspace/src/clang/test/Driver/env.c:20:17: error: CHECK-LD-32: expected string not found in input
23:47:36  # | // CHECK-LD-32: "{{.*}}ld{{(.exe)?}}" "--sysroot=[[SYSROOT:[^"]+]]"
23:47:36  # |                 ^
23:47:36  # | <stdin>:1:1: note: scanning from here
23:47:36  # | /workspace/build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
23:47:36  # | ^
23:47:36  # | <stdin>:1:140: note: possible intended match here
23:47:36  # | /workspace/build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
23:47:36  # |                                                                                                                                            ^
23:47:36  # | 
23:47:36  # | Input file: <stdin>
23:47:36  # | Check file: /workspace/src/clang/test/Driver/env.c
23:47:36  # | 
23:47:36  # | -dump-input=help explains the following input dump.
23:47:36  # | 
23:47:36  # | Input was:
23:47:36  # | <<<<<<
23:47:36  # |             1: /workspace/build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory 
23:47:36  # | check:20'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
23:47:36  # | check:20'1                                                                                                                                                ?                  possible intended match
23:47:36  # | >>>>>>
23:47:36  # `-----------------------------
23:47:36  # error: command failed with exit status: 1
23:47:36  
23:47:36  --
23:47:36  
23:47:36  ********************
23:57:24  Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90.. 
23:57:24  ********************
23:57:24  Failed Tests (1):
23:57:24    Clang :: Driver/env.c
23:57:24  
23:57:24  
23:57:24  Testing Time: 984.80s
23:57:24  
23:57:24  Total Discovered Tests: 130955
23:57:24    Skipped          :   219 (0.17%)
23:57:24    Unsupported      : 45532 (34.77%)
23:57:24    Passed           : 85110 (64.99%)
23:57:24    Expectedly Failed:    93 (0.07%)
23:57:24    Failed           :     1 (0.00%)
23:57:24  FAILED: CMakeFiles/check-all /workspace/build/stage/product_build/CMakeFiles/check-all

[boomanaiden154]

It would be better if someone could fix the test. I don't think anyone else so far has relied on the LD_LIBRARY_PATH pass through in this test. It should work but given it hasn't been validated, there's probably some small detail for why it doesn't.

Support for the external shell is going to be removed eventually and there is a very real risk that new tests do not work with the external shell and there will be very little upstream coverage of that configuration.

[pawosm-arm]

It would be better if someone could fix the test. I don't think anyone else so far has relied on the LD_LIBRARY_PATH pass through in this test. It should work but given it hasn't been validated, there's probably some small detail for why it doesn't.

Support for the external shell is going to be removed eventually and there is a very real risk that new tests do not work with the external shell and there will be very little upstream coverage of that configuration.

What worries me more is how low is the coverage of the 'using just-built libc++ for building LLVM' scenario.

[DavidTruby]

LGTM.

I mainly use the internal shell for lit, but I still regularly see errors I just know to ignore. I don't think it's ready for a product. I do strongly think we should document this somewhere and reconsider in future. I'm also sure issue reports for breaking tests upstream would be much appreciated!

[DavidTruby]

What worries me more is how low is the coverage of the 'using just-built libc++ for building LLVM' scenario.

I've also been concerned about this for a long time. There's even supposedly "built-in" bootstrapping builds you can turn on that I've never been able to get to work.

Is there any hope we could ask eg. Linaro to look at a build bot for these?

[pawosm-arm]

I'm trying to figure out what issue are we facing here. I've triggered a manual build, and, as expected, it fails on the env.c test case:

env -i LC_ALL=C LD_LIBRARY_PATH="/build/stage/product_build/lib:/build/atfl/lib:/build/atfl/lib/aarch64-unknown-linux-gnu:." CLANG_NO_DEFAULT_CONFIG=1    /build/stage/product_build/bin/clang /llvm/clang/test/Driver
/env.c -### -o /build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.o --target=i386-unknown-linux      --sysroot=/llvm/clang/test
/Driver/Inputs/basic_linux_tree      --rtlib=platform --unwindlib=platform -no-pie      2>&1 | /build/stage/product_build/bin/FileCheck --check-prefix=CHECK-LD-32 /llvm/clang/test/Driver/env.c
# executed command: env -i LC_ALL=C 'LD_LIBRARY_PATH=%{readfile:/build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.ld_library_path}' CLANG_NO_DEFAULT_CONFIG=1 /dsg_space/projectscratch_dsg_space/pawosm
01/atfl-development/arm-toolchain.git/arm-software/linux/build/stage/product_build/bin/clang /llvm/clang/test/Driver/env.c '-###' -o /build/stage/product_build/tools/clang/test/Driver/Output/env.c.tmp.o --target=i386-unknown-linux --sysroot=/llvm/clang/test/Driver/Inputs/basic_linux_tree --rtlib=platform --unwindlib=platform -no-pie
# note: command had no output on stdout or stderr
# error: command failed with exit status: 127
# executed command: /build/stage/product_build/bin/FileCheck --check-prefix=CHECK-LD-32 /llvm/clang/test/Driver/env.c
# .---command stderr------------
# | /llvm/clang/test/Driver/env.c:20:17: error: CHECK-LD-32: expected string not found in input
# | // CHECK-LD-32: "{{.*}}ld{{(.exe)?}}" "--sysroot=[[SYSROOT:[^"]+]]"
# |                 ^
# | <stdin>:1:1: note: scanning from here
# | /build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
# | ^
# | <stdin>:1:228: note: possible intended match here
# | /build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
# |                                                                                                                                                                                                                                    ^
# |
# | Input file: <stdin>
# | Check file: /llvm/clang/test/Driver/env.c
# |
# | -dump-input=help explains the following input dump.
# |
# | Input was:
# | <<<<<<
# |             1: /build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
# | check:20'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
# | check:20'1                                                                                                                                                                                                                                        ?                  possible intended match
# | >>>>>>
# `-----------------------------
# error: command failed with exit status: 1

And, having all those files still there, I could confirm that it doesn't happen in the command line (did the positive test case first, that clang actually needs LD_LIBRARY_PATH to be set):

$ /build/stage/product_build/bin/clang
/build/stage/product_build/bin/clang: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory

$ LD_LIBRARY_PATH="/build/stage/product_build/lib:/build/atfl/lib:/build/atfl/lib/aarch64-unknown-linux-gnu:." /build/stage/product_build/bin/clang
clang: error: no input files

$ env -i LC_ALL=C LD_LIBRARY_PATH="/build/stage/product_build/lib:/build/atfl/lib:/build/atfl/lib/aarch64-unknown-linux-gnu:." CLANG_NO_DEFAULT_CONFIG=1    /build/stage/product_build/bin/clang
clang: error: no input files

Now the questions are:

• what is the lit internal shell anyway? do we build something bash-alike for running the check-all tests in LLVM?
• is env in lit something other than the system-provided env? Does it filter LD_LIBRARY_PATH to get rid of all of the non-lit-blessed content?

[boomanaiden154]

I think llvm/llvm-project#165140 might fix this.

what is the lit internal shell anyway? do we build something bash-alike for running the check-all tests in LLVM?

Not really bash like. It runs the binaries directly inside Python using subprocess.run instead of writing out a shell script and then executing that.

is env in lit something other than the system-provided env? Does it filter LD_LIBRARY_PATH to get rid of all of the non-lit-blessed content?

Yes. It's a builtin function. It by doesn't do anything special to LD_LIBRARY_PATH though.

[pawosm-arm]

I think llvm/llvm-project#165140 might fix this.

I've tried this locally, and it seems it does really help. Thing is, the way this PR has been opened does not make it easy to just cherry-pick it, I had to rework this patch manually in order to try it out.

[pawosm-arm]

Closing it as the upstream fix has proven to work.