Amazon Linux 2023 - LLVM out of memory after merging stable/21.x

[shahmishal]

Build command:

[2025-10-07T09:53:57.165Z] + docker run --name swift-bootstrap-amazonlinux-2023-x86_64 --security-opt=no-new-privileges --cap-add=SYS_PTRACE --security-opt seccomp=unconfined -e TOOLCHAIN_VERSION -e OS_VERSION -e ARCHIVE -e SWIFT_SOURCE_ROOT -e GIT_ALLOW_PROTOCOL -w /home/build-user/ -v /tmp/jenkins-bd22abac/workspace/swift-bootstrap-amazonlinux-2023-x86_64:/source -v swift-bootstrap-amazonlinux-2023-x86_64:/home/build-user 608094965728.dkr.ecr.us-west-2.amazonaws.com/main-ci-ecr-8ce5c7b:swift-ci-amazonlinux2023-bootstrap /bin/bash -lc 'env &&                           cp -r /source/* /home/build-user;                           ./swift/utils/build-script --preset bootstrap_stage0 build_subdir=bootstrap_stage0 install_destdir=/home/build-user/stages/stage0 &&                           PATH="/home/build-user/stages/stage0/usr/bin:$PATH" ./swift/utils/build-script --preset bootstrap_stage1 build_subdir=bootstrap_stage1 install_destdir=/home/build-user/stages/stage1 &&                           PATH="/home/build-user/stages/stage1/usr/bin:$PATH" ./swift/utils/build-script --preset bootstrap_stage2 build_subdir=bootstrap_stage2 install_destdir=/home/build-user/stages/stage2 &&                           PATH="/home/build-user/stages/stage2/usr/bin:$PATH" ./swift/utils/build-script --preset buildbot_linux install_destdir=/home/build-user/swift-nightly-install installable_package=/home/build-user/swift-DEVELOPMENT-SNAPSHOT-2025-10-07-a-amazonlinux2023.tar.gz'

Out of memory:

[2025-10-07T11:59:52.627Z] LLVM ERROR: out of memory
[2025-10-07T11:59:52.627Z] Allocation failed
[2025-10-07T11:59:52.627Z] Please submit a bug report (https://swift.org/contributing/#reporting-bugs) and include the crash backtrace.
[2025-10-07T11:59:52.627Z] Stack dump:
[2025-10-07T11:59:52.627Z] 0.	Program arguments: /home/build-user/stages/stage1/usr/bin/swift-frontend -frontend -c /home/build-user/swift/lib/ASTGen/Sources/ASTGen/ASTGen.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/ASTGen+CompilerBuildConfiguration.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Availability.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Bridge.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/BuiltinPound.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/CompilerBuildConfiguration.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/DeclAttrs.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Decls.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Diagnostics.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/DiagnosticsBridge.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/EmbeddedSupport.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Exprs.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Fingerprint.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Generics.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/LexicalLookup.swift -primary-file /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Literals.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/ParameterClause.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Patterns.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Regex.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/SourceFile.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/StableHasher.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Stmts.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/TypeAttrs.swift /home/build-user/swift/lib/ASTGen/Sources/ASTGen/Types.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-importedHeaderDependencies-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerSwiftCompilerPluginMessageHandling-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerRegexParser-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerSwiftSyntax-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerSwiftIfConfig-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerSwiftLexicalLookup-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerSwiftOperators-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerSwiftSyntaxBuilder-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerSwiftParser-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerSwiftParserDiagnostics-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-_CompilerSwiftDiagnostics-dep.swift /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen/forced-swiftBasicSwift-dep.swift -emit-module-path lib/ASTGen/Sources/ASTGen/CMakeFiles/swiftASTGen.dir/Literals.swift.swiftmodule -emit-module-doc-path lib/ASTGen/Sources/ASTGen/CMakeFiles/swiftASTGen.dir/Literals.swift.swiftdoc -emit-module-source-info-path lib/ASTGen/Sources/ASTGen/CMakeFiles/swiftASTGen.dir/Literals.swift.swiftsourceinfo -emit-dependencies-path lib/ASTGen/Sources/ASTGen/CMakeFiles/swiftASTGen.dir/Literals.swift.o.d -target x86_64-unknown-linux-gnu -disable-objc-interop -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/ASTGen -I /home/build-user/swift/lib/ASTGen/Sources/ASTGen -I /home/build-user/swift/stdlib/public/SwiftShims -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/include -I /home/build-user/swift/include -I /home/build-user/llvm-project/llvm/include -I /home/build-user/build/bootstrap_stage2/llvm-linux-x86_64/include -I /home/build-user/llvm-project/clang/include -I /home/build-user/build/bootstrap_stage2/llvm-linux-x86_64/tools/clang/include -I /home/build-user/cmark/src/include -I /home/build-user/build/bootstrap_stage2/cmark-linux-x86_64/src -I /home/build-user/swift-corelibs-libdispatch/src/BlocksRuntime -I /home/build-user/swift-corelibs-libdispatch -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftCompilerPluginMessageHandling -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftSyntax -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftBasicFormat -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftDiagnostics -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftIfConfig -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftParser -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftSyntaxMacros -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftSyntaxMacroExpansion -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftOperators -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/modules -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftSyntaxBuilder -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftLexicalLookup -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/_deps/compilerswiftsyntax-build/Sources/SwiftParserDiagnostics -I /home/build-user/build/bootstrap_stage2/swift-linux-x86_64/lib/ASTGen/Sources/BasicSwift/modules -I /home/build-user/swift-syntax/Sources/_SwiftSyntaxCShims/include -I /usr/include -color-diagnostics -module-link-name swiftASTGen -static -O -cxx-interoperability-mode=default -D CMARK_STATIC_DEFINE -D SWIFT_BUILD_SWIFT_SYNTAX -D _DEBUG -D _GLIBCXX_ASSERTIONS -D __STDC_CONSTANT_MACROS -D __STDC_FORMAT_MACROS -D __STDC_LIMIT_MACROS -disable-target-os-checking -in-process-plugin-server-path /home/build-user/stages/stage1/usr/lib/swift/host/libSwiftInProcPluginServer.so -plugin-path /home/build-user/stages/stage1/usr/lib/swift/host/plugins -plugin-path /home/build-user/stages/stage1/usr/local/lib/swift/host/plugins -Xcc -UNDEBUG -Xcc -std=c++17 -Xcc -DCOMPILED_WITH_SWIFT -Xcc -UIBOutlet -Xcc -UIBAction -Xcc -UIBInspectable -parse-as-library -module-name swiftASTGen -num-threads 36 -o lib/ASTGen/Sources/ASTGen/CMakeFiles/swiftASTGen.dir/Literals.swift.o -runtime-compatibility-version none
[2025-10-07T11:59:52.627Z] 1.	Swift version 6.3-dev (LLVM 26c9331e7a08196, Swift e4fb8819d25700e)
[2025-10-07T11:59:52.627Z] 2.	Compiling with effective version 5.10
[2025-10-07T11:59:52.627Z] 3.	While evaluating request BindExtensionsRequest(swiftASTGen)
[2025-10-07T11:59:52.627Z] 4.	While evaluating request UnderlyingTypeDeclsReferencedRequest(swiftASTGen.(file).Identifier@/home/build-user/swift/lib/ASTGen/Sources/ASTGen/Bridge.swift:18:18)
[2025-10-07T11:59:52.627Z] 5.	While evaluating request QualifiedLookupRequest(0x55731a709be8 TypeAliasDecl name=Identifier, {__ObjC.(file).swift@/home/build-user/swift/include/swift/AST/Stmt.h:34:11}, 'Identifier', { NL_RemoveNonVisible, NL_OnlyTypes })
[2025-10-07T11:59:52.627Z] 6.	While evaluating request DirectLookupRequest(directly looking up 'Identifier' on __ObjC.(file).swift@/home/build-user/swift/include/swift/AST/Stmt.h:34:11 with options {  })
[2025-10-07T11:59:52.627Z] 7.	While evaluating request CXXNamespaceMemberLookup(Looking up 'Identifier' in __ObjC.(file).swift@/home/build-user/swift/include/swift/AST/Stmt.h:34:11)
[2025-10-07T11:59:52.627Z] 8.	/home/build-user/swift/include/swift/AST/Identifier.h:72:7: importing 'swift::Identifier'
[2025-10-07T11:59:52.627Z] 9.	/home/build-user/swift/include/swift/AST/Identifier.h:154:15: importing 'swift::Identifier::isEditorPlaceholder'
[2025-10-07T11:59:52.628Z] 10.	While importing a clang type UsingType 0x55731b3ec480 'llvm::StringRef' sugar imported

Full log:

stack_dump.log
Jenkins logs
https://ci.swift.org/job/swift-bootstrap-amazonlinux-2023-x86_64/249/console

[beccadax]

It looks like this process runs out of memory because ClangImporter repeatedly imports unrelated == operators when trying to import the template class __gnu_cxx::__normal_iterator:

25.	/usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11: importing '__gnu_cxx::__normal_iterator'
26.	While conforming to UnsafeCxxInputIterator '__normal_iterator<UnsafeMutablePointer<CChar>, std.__cxx11.basic_string<CChar, std.char_traits<CChar>, std.allocator<CChar>>>' (at /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11)
27.	/home/build-user/swift/include/swift/AST/Witness.h:241:15: importing 'swift::operator=='
<snip>
64.	/usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11: importing '__gnu_cxx::__normal_iterator'
65.	While conforming to UnsafeCxxInputIterator '__normal_iterator<UnsafeMutablePointer<swift.AnyAttrKind>, std.vector<swift.AnyAttrKind, std.allocator<swift.AnyAttrKind>>>' (at /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11)
66.	/home/build-user/swift/include/swift/AST/SearchPathOptions.h:320:17: importing 'swift::operator=='
<snip>
80.	/usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11: importing '__gnu_cxx::__normal_iterator'
81.	While conforming to UnsafeCxxInputIterator '__normal_iterator<UnsafeMutablePointer<swift.SearchPathOptions.SearchPath>, std.vector<swift.SearchPathOptions.SearchPath, std.allocator<swift.SearchPathOptions.SearchPath>>>' (at /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11)
82.	/home/build-user/swift/include/swift/Basic/Fingerprint.h:87:15: importing 'swift::operator=='
<snip>
99.	/usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11: importing '__gnu_cxx::__normal_iterator'
100.	While conforming to UnsafeCxxInputIterator '__normal_iterator<UnsafeMutablePointer<CWideChar>, std.__cxx11.basic_string<CWideChar, std.char_traits<CWideChar>, std.allocator<CWideChar>>>' (at /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11)
101.	/home/build-user/swift/include/swift/AST/DeclContext.h:774:15: importing 'swift::operator=='
<snip>
155.	/usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11: importing '__gnu_cxx::__normal_iterator'
156.	While conforming to UnsafeCxxInputIterator '__normal_iterator<UnsafeMutablePointer<CUnsignedLong>, std.vector<CUnsignedLong, std.allocator<CUnsignedLong>>>' (at /usr/lib/gcc/x86_64-amazon-linux/11/../../../../include/c++/11/bits/stl_iterator.h:1004:11)
157.	/home/build-user/llvm-project/llvm/include/llvm/Support/LineIterator.h:86:15: importing 'llvm::operator=='

(There's also one at 174 but I'm not sure that one is incorrect in itself.)

Notice how, for instance, 26 says we're importing the iterator for std::basic_string but then 27 is importing a == in swift/AST/Witness.h. This is probably the operator for TypeWitnessAndDecl, a completely unrelated type.

Seems like this either needs to be made lazier or more selective somehow.