Clang failed 100 tests on llvmorg-19.1.7

[leecommamichael]

The windows binaries do not contain a target for wasm, so I decided to build it myself. This is a fresh install of Windows 11 using Visual Studio 2022. I compiled with CMake and Ninja as advised on https://clang.llvm.org/get_started.html

I ran cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_PROJECTS=clang ..\llvm
then ninja check as advised on the "get started" page.

Testing Time: 893.79s

Total Discovered Tests: 98695
  Skipped          :    32 (0.03%)
  Unsupported      :  2898 (2.94%)
  Passed           : 95462 (96.72%)
  Expectedly Failed:   203 (0.21%)
  Failed           :   100 (0.10%)

This is on tag llvmorg-19.1.7 commit cd708029e0b2869e80abe31ddb175f7c35361f90

Is this to be expected?

[nikic]

Can you please share the output for the test failures (from your build log)?

[leecommamichael]

********************
Testing:  0.. 10.. 20.. 30.. 40.. 50.. 60.. 70.. 80.. 90..
********************
Failed Tests (100):
  Clang :: ARCMT/GC-no-finalize-removal.m
  Clang :: ARCMT/allowlisted/objcmt-with-allowlist-impl.m
  Clang :: ARCMT/allowlisted/objcmt-with-allowlist.m
  Clang :: ARCMT/atautorelease.m
  Clang :: ARCMT/designated-init-in-header/designated-init-in-header.m
  Clang :: ARCMT/objcmt-arc-cf-annotations.m
  Clang :: ARCMT/objcmt-atomic-property.m
  Clang :: ARCMT/objcmt-migrate-all.m
  Clang :: ARCMT/objcmt-ns-macros.m
  Clang :: ARCMT/objcmt-ns-nonatomic-iosonly.m
  Clang :: ARCMT/objcmt-property-availability.m
  Clang :: ARCMT/objcmt-property.m
  Clang :: ARCMT/objcmt-undefined-ns-macros.m
  Clang :: ARCMT/protected-scope.m
  Clang :: AST/ast-dump-attr-json.cpp
  Clang :: AST/ast-dump-default-arg-json.cpp
  Clang :: AST/ast-dump-default-init-json.cpp
  Clang :: AST/ast-dump-pragma-json.c
  Clang :: AST/ast-dump-stmt-json.cpp
  Clang :: AST/ast-dump-template-json-win32-mangler-crash.cpp
  Clang :: Analysis/diagnostics/sarif-diagnostics-taint-test.c
  Clang :: Analysis/diagnostics/sarif-multi-diagnostic-test.c
  Clang :: CXX/drs/cwg0xx.cpp
  Clang :: CXX/drs/cwg23xx.cpp
  Clang :: CXX/drs/cwg3xx.cpp
  Clang :: CXX/drs/cwg6xx.cpp
  Clang :: CodeGen/debug-info-embed-source.c
  Clang :: CodeGen/debug-info-file-checksum.c
  Clang :: Driver/at_file.c
  Clang :: ExtractAPI/emit-symbol-graph/multi_file.c
  Clang :: ExtractAPI/emit-symbol-graph/single_file.c
  Clang :: ExtractAPI/macros.c
  Clang :: Format/xmloutput.cpp
  Clang :: Index/cxx11-lambdas.cpp
  Clang :: Index/cxx14-lambdas.cpp
  Clang :: Index/get-cursor.cpp
  Clang :: Index/index-templates.cpp
  Clang :: Index/pch-opaque-value.cpp
  Clang :: Index/record-completion-invocation.c
  Clang :: Index/record-parsing-invocation.c
  Clang :: Index/retain-comments-from-system-headers.c
  Clang :: Index/usrs.cpp
  Clang :: Index/usrs.m
  Clang :: Lexer/SourceLocationsOverflow.c
  Clang :: Lexer/conflict-marker.c
  Clang :: Lexer/eof-conflict-marker.c
  Clang :: Lexer/raw-string-dlim-invalid.cpp
  Clang :: Misc/diag-template-diffing-cxx26.cpp
  Clang :: PCH/preamble.c
  Clang :: Preprocessor/embed_art.c
  Clang :: Preprocessor/embed_codegen.cpp
  Clang :: Preprocessor/embed_constexpr.cpp
  Clang :: Preprocessor/embed_parameter_offset.c
  Clang :: Preprocessor/traditional-cpp.c
  Clang :: Refactor/Extract/ExtractExprIntoFunction.cpp
  Clang :: Refactor/tool-test-support.c
  Clang :: SemaCXX/type-traits.cpp
  Clang :: SemaCXX/warn-unsafe-buffer-usage-fixits-parm-span.cpp
  Clang :: Tooling/clang-diff-json.cpp
  Clang :: clang-rename/ClassAsTemplateArgument.cpp
  Clang :: clang-rename/ClassSimpleRenaming.cpp
  Clang :: clang-rename/ClassTestMulti.cpp
  Clang :: clang-rename/ComplexFunctionOverride.cpp
  Clang :: clang-rename/ComplicatedClassType.cpp
  Clang :: clang-rename/Ctor.cpp
  Clang :: clang-rename/CtorInitializer.cpp
  Clang :: clang-rename/DeclRefExpr.cpp
  Clang :: clang-rename/FunctionMacro.cpp
  Clang :: clang-rename/FunctionOverride.cpp
  Clang :: clang-rename/FunctionTemplate.cpp
  Clang :: clang-rename/InvalidOffset.cpp
  Clang :: clang-rename/MemberExprMacro.cpp
  Clang :: clang-rename/TemplateClassInstantiation.cpp
  Clang :: clang-rename/TemplateTypename.cpp
  Clang :: clang-rename/TemplatedClassFunction.cpp
  Clang :: clang-rename/Typedef.cpp
  Clang :: clang-rename/UserDefinedConversion.cpp
  Clang :: clang-rename/Variable.cpp
  Clang :: clang-rename/VariableMacro.cpp
  Clang :: clang-rename/VariableTemplate.cpp
  Clang :: clang-rename/YAMLInput.cpp
  Clang :: utils/update_cc_test_checks/annotations.test
  Clang :: utils/update_cc_test_checks/basic-cplusplus.test
  Clang :: utils/update_cc_test_checks/check_attrs.test
  Clang :: utils/update_cc_test_checks/def-and-decl.test
  Clang :: utils/update_cc_test_checks/exec-all-runlines.test
  Clang :: utils/update_cc_test_checks/explicit-template-instantiation.test
  Clang :: utils/update_cc_test_checks/generated-funcs-regex.test
  Clang :: utils/update_cc_test_checks/generated-funcs.test
  Clang :: utils/update_cc_test_checks/global-hex-value-regex.test
  Clang :: utils/update_cc_test_checks/global-value-regex.test
  Clang :: utils/update_cc_test_checks/ifdef.test
  Clang :: utils/update_cc_test_checks/mangled_names.test
  Clang :: utils/update_cc_test_checks/on_the_fly_arg_change.test
  Clang :: utils/update_cc_test_checks/prefix-never-matches.test
  Clang :: utils/update_cc_test_checks/replace-value-regex-across-runs.test
  Clang :: utils/update_cc_test_checks/resolve-tmp-conflict.test
  LLVM :: MC/ELF/warn-newline-in-escaped-string.s
  LLVM :: TableGen/x86-fold-tables.td
  lit :: shtest-shell.py

Oh sorry, of course you'd need that. Reading up on grabbing the build log, but here are the failures.

[leecommamichael]

I reran the tests and piped STDOUT and STDERR to logs below.

test.out.log
test.err.log

Well this is concerning... different results. But still over 100 failures.

[nikic]

Did you set core.autocrlf=false for the checkout? I think the test failures are due to Windows newline issues.

From https://clang.llvm.org/get_started.html:

Note: Some Clang tests are sensitive to the line endings. Ensure that checking out the files does not convert LF line endings to CR+LF. If you're using git on Windows, make sure your core.autocrlf setting is false.

[leecommamichael]

Sigh. autocrlf must've resurrected when I installed a git gui. Thanks for the help, sorry for the waste.