[flang] Building flang on Macs requires manually passing a flag with path to runtime libraries

[ktras]

I have recently installed flang on an Apple M1 Mac that is running Sonoma 14.5. I installed with the latest from the main branch at the time (the commit hash fd86b2e26c0933c2af61). When invoking the installed flang executable to build a basic Fortran program, I get the following error:

ld: library 'flang_rt.runtime' not found
flang-22: error: linker command failed with exit code 1 (use -v to see invocation)

I am able to resolve this by manually passing the path to the runtime library with the following command:

flang -L`flang -print-resource-dir`/lib/darwin hello-world.f90

I believe this to be an issue that other mac users also face and hope that it can be resolved so the runtime libraries do not have to be manually passed.

[ktras]

For added context, when I build on my mac, I do use the -DDEFAULT_SYSROOT="$(xcrun --show-sdk-path flag. And the problem described in this issue only started occurring after the introduction of flang-rt.

[PHHargrove]

@ktras
While I don't have any suggestions for fixing the actual problem, I can suggest a post-installation work-around to eliminate the need to pass the -L option every time flang is invoked for linking. The "trick" is to use configuration files to make that option a default.

With an appropriate replacement for [prefix] you can try the following:

echo -L`flang -print-resource-dir`/lib/darwin >> [prefix]/bin/flang.cfg

If root permissions are required to write to the install, then you likely need the following alternative:

echo -L`flang -print-resource-dir`/lib/darwin | sudo tee -a [prefix]/bin/flang.cfg

EDIT: I've corrected both instances of libexec originally appearing above to bin. This after confirming by experiment that bin is the correct location by default. My original libexec came from a non-default directory layout used in Homebrew's build of flang.

[Meinersbur]

As someone not owning a Mac, I cannot reproduce this. If you want to want to create a PR, here are some hints:

This is likely a inconsistency of how flang (the driver) searches for libraries and where CMake puts it.

1. Driver code here:

   llvm-project/clang/lib/Driver/ToolChain.cpp

   Lines 1004 to 1015 in 20f6ec4

   	std::optional<std::string> ToolChain::getRuntimePath() const {
   	SmallString<128> P(D.ResourceDir);
   	llvm::sys::path::append(P, "lib");
   	if (auto Ret = getTargetSubDirPath(P))
   	return Ret;
   	// Darwin does not use per-target runtime directory.
   	if (Triple.isOSDarwin())
   	return {};
   	llvm::sys::path::append(P, Triple.str());
   	return std::string(P);
   	}

2. CMake code here:

   llvm-project/flang-rt/CMakeLists.txt

   Lines 115 to 116 in 20f6ec4

   	set(FLANG_RT_INSTALL_RESOURCE_PATH "${FLANG_RT_INSTALL_RESOURCE_PATH_DEFAULT}"
   	CACHE PATH "Path to install runtime libraries to (default: clang resource dir)")

This is intended to mimic the location for compiler-rt (libclang_rt.builtins.a), but with LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=ON on all platforms. You probably need to modify https://github.com/llvm/llvm-project/blob/main/flang-rt/cmake/modules/GetToolchainDirs.cmake to make CMake put the library into the location where the driver expects it.

As you see, workaround it to set cmake -DFLANG_RT_INSTALL_RESOURCE_PATH=... to the location where the driver is expecting it.

[PHHargrove]

@Meinersbur
I've crafted the patch below, but it did not resolve the problem @ktras reported (and I am able to reproduce).
Fwiw, I did try (not show) debug output which confirmed that my modified ToolChain::getRuntimePath() is now returning the directory where libflang_rt.runtime.a is installed.

I've noticed that the ld command in the output of flang -v ... does not include the path returned from ToolChain::getRuntimePath(). So, there might be more to the problem then you've inferred, such as something missing in the flang driver.

diff --git a/clang/lib/Driver/ToolChain.cpp b/clang/lib/Driver/ToolChain.cpp
index 25c6b5a486fd..d95bf62d706f 100644
--- a/clang/lib/Driver/ToolChain.cpp
+++ b/clang/lib/Driver/ToolChain.cpp
@@ -1006,11 +1006,11 @@ std::optional<std::string> ToolChain::getRuntimePath() const {
   llvm::sys::path::append(P, "lib");
   if (auto Ret = getTargetSubDirPath(P))
     return Ret;
-  // Darwin does not use per-target runtime directory.
-  if (Triple.isOSDarwin())
-    return {};

-  llvm::sys::path::append(P, Triple.str());
+  // Darwin uses "darwin" rather than a per-target runtime directory.
+  auto target = Triple.isOSDarwin() ? std::string("darwin") : Triple.str();
+
+  llvm::sys::path::append(P, target);
   return std::string(P);
 }
 

[PHHargrove]

Regarding this portion of my previous comment:

So, there might be more to the problem then you've inferred, such as something missing in the flang driver.

I think I've answered my own question/concern. The link command is generated, in part, by ToolChain::addFortranRuntimeLibraryPath(), which is not using ToolChain::getRuntimePath().

[PHHargrove]

@ktras
Please try the following (it works for me)

diff --git a/clang/lib/Driver/ToolChain.cpp b/clang/lib/Driver/ToolChain.cpp
index 25c6b5a486fd..993701cc47f0 100644
--- a/clang/lib/Driver/ToolChain.cpp
+++ b/clang/lib/Driver/ToolChain.cpp
@@ -864,6 +864,8 @@ void ToolChain::addFortranRuntimeLibraryPath(const llvm::opt::ArgList &Args,
   llvm::sys::path::append(DefaultLibPath, "lib");
   if (getTriple().isKnownWindowsMSVCEnvironment())
     CmdArgs.push_back(Args.MakeArgString("-libpath:" + DefaultLibPath));
+  else if (getTriple().isOSDarwin())
+    CmdArgs.push_back(Args.MakeArgString("-L" + getCompilerRTPath()));
   else
     CmdArgs.push_back(Args.MakeArgString("-L" + DefaultLibPath));
 }

If this is the correct fix, then the comment immediately proceeding the diff context will need to be updated before this is suitable to PR.

[Meinersbur]

Fwiw, I did try (not show) debug output which confirmed that my modified ToolChain::getRuntimePath() is now returning the directory where libflang_rt.runtime.a is installed.

Sorry for the confusion, what's actually is the effective path is pretty complex, including that it has been refractored in #131041. See this comment:

llvm-project/flang-rt/cmake/modules/GetToolchainDirs.cmake

Lines 23 to 27 in 507b879

	# directories have to match what Clang is looking for, which is done in
	# ToolChain::getArchSpecificLibPaths(..), ToolChain::getRuntimePath(),
	# ToolChain::getCompilerRTPath(), and ToolChain::getCompilerRT(..), not entirely
	# consistent between these functions, Compiler-RT's CMake code, and overrides
	# in different toolchains.

Please try the following (it works for me)

ToolChain::addFortranRuntimeLibraryPath gets a dir to ../lib which is a path for user-installed libraries. Your code may cause other libraries to be not found. flang_rt.runtime is located in the resource directory, under ../lib/clang/<version>/lib/<triple-or-os>. ToolChain::addFlangRTLibPath is responsible for that.

I would suggest to modify GetToolchainDirs.cmake instead to whatever clang::driver::toolchains::MachO's result of getArchSpecificLibPaths() might be.

GetToolchainDirs.cmake was extracted from compiler-rt. If it is wrong there, it might also be wrong for compiler-rt. Or you are using different settings of LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.

[PHHargrove]

Your code may cause other libraries to be not found.
[...]
I would suggest to modify GetToolchainDirs.cmake instead to whatever clang::driver::toolchains::MachO's result of getArchSpecificLibPaths() might be.

GetToolchainDirs.cmake was extracted from compiler-rt. If it is wrong there, it might also be wrong for compiler-rt. Or you are using different settings of LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.

Sorry, but this now sounds over my head. In particular, I have no experience modifying CMake files nor am I confident I'd know how to test for "other libraries" not being found. Perhaps most importantly, I cannot justify spending more of my employer's time on this than I already have.
So, I am no longer attempting to provide a solution.

Fwiw, I am not passing any -DLLVM_ENABLE_PER_TARGET_RUNTIME_DIR to cmake.

For whoever is able to pick this up, my reproducer is use of flang configured with the following cmake command on a macOS Sonoma system with Xcode CLT (version 16.?). Linking any trivial Fortran program is sufficient.

cmake [SOMETHING]/llvm \
    -DDEFAULT_SYSROOT="$(xcrun --show-sdk-path)" \
    -DCMAKE_INSTALL_PREFIX=/opt/llvm \
    -DCMAKE_BUILD_TYPE=Release \
    -DLLVM_PARALLEL_LINK_JOBS=1 \
    -DLLVM_ENABLE_PROJECTS="flang;mlir;openmp" \
    -DLLVM_INCLUDE_BENCHMARKS=OFF \
    -DLLVM_INCLUDE_EXAMPLES=OFF \
    -DLLVM_INCLUDE_TESTS=OFF \
    -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON \
    -DLLVM_TARGETS_TO_BUILD=host

I've verified that a build without -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON exhibits the same issue.