Implement -static-stdlib for Linux (#1891)

[Driver] implement -static-stdlib for Linux

Implement the -static-stdlib driver flag for Linux, allowing the static
linking of the standard library.

This implementation largely follows the Darwin implementation in #1817,
although some pecularities warrant extended discussion.

The original "link with stdlib" implementation had some redundancies
with getRuntimeLibraryPath; these redundancies are resolved, and the
implementation alternates between getRuntimeLibraryPath and
getStaticRuntimeLibraryPath cleanly as appropriate.

A variety of libraries are required to link statically on Linux.  The
implementation currently dynamically links with them.  We should
probably support static linking of those as well, but I think that is
beyond the scope of a -static-stdlib flag.

The test coverage uses ldd here, as otool is not available on Linux.  As
a result, we currently have separate tests for Linux vs the other
platforms; that isn't ideal, but it seems necessary.

Perhaps the oddest part, and the one worth the most discussion, is the
use of --dynamic-list.  Inside
stdlib/public/runtime/ProtocolConformances.cpp appears the following
code:

    #elif defined(__ELF__)
    static int _addImageProtocolConformances(struct dl_phdr_info *info,
                                              size_t size, void *data) {
      // inspectArgs contains addImage*Block function and the section name
      InspectArgs *inspectArgs = reinterpret_cast<InspectArgs *>(data);

      void *handle;
      if (!info->dlpi_name || info->dlpi_name[0] == '\0') {
        handle = dlopen(nullptr, RTLD_LAZY);
      } else
        handle = dlopen(info->dlpi_name, RTLD_LAZY | RTLD_NOLOAD);
      auto conformances = reinterpret_cast<const uint8_t*>(
          dlsym(handle, inspectArgs->sectionName));

The effect of this is to search for protocol_conformances_start inside
the images.  However, dlsym only finds symbols that exist in the dynamic
table.  Failure to find the protocol conformances can be diagnosed by a
"hello world" program printing

    String(_core: Swift._StringCore(_baseAddress: Swift.OpaquePointer(_rawValue: (Opaque Value)), _countAndFlags: Swift.UInt(_value: (Opaque Value)), _owner: Swift.Optional<Swift.AnyObject>.none))

instead of "hello world".  (And also by the test coverage in this commit.)

Surprisingly, this behavior can still occur on ELF platforms even if
`objdump -t` reports a valid `.protocol_conformances_start`.  This is
because `objdump -t` searches the global table, not the dynamic table,
while dlsym only searches the dynamic table.  To configure objdump to
search only the dynamic table, use `-T`.

Inquiring minds may wonder whether dynamically-linked programs (e.g. all
Linux binaries up until now) also have a broken protocol conformance
table on ELF.  The answer is, surprisingly, no; I checked, and ordinary
ELF programs are fine.  The distinction is probably this, from the ld
manpage:

> the dynamic symbol table will normally contain only those
symbols which are referenced by some dynamic object mentioned in the
link.

I think the linker sees `.protocol_conformances_start` inside
libswiftCore.so and erroneously concludes the one in *the executable* is
"referenced by some dynamic object" (e.g. the standard library).  This
behavior seems to save the dyanmically-linked executable from a broken
protocol conformance table.  I wonder if it would be wise to apply a
similar fix to dynamically-linked programs to avoid relying on the
linker "helping" us here, but that's out of scope of this commit.

The linker manpage reflects that many people have been bitten by dlsym
"surprise", and encourages the use of `--export-dynamic`:

> If you use "dlopen" to load a dynamic object which needs to refer back
> to the symbols defined by the program, rather than some other dynamic
> object, then you will probably need to use [--export-dynamic] when
> linking the program itself.

However in this situation, the use of `--export-dynamic` causes the
entire stdlib to be exported, which is not ideal.  However, by combining
with the `--exclude-libs ALL` argument, we avoid exporting the entire stdlib.

Author: drewcrawford

lib/Driver/ToolChains.cpp

@@ -1317,8 +1317,40 @@ toolchains::GenericUnix::constructInvocation(const LinkJobAction &job,
     Arguments.push_back(context.Args.MakeArgString(context.OI.SDKPath));
   }
 
+  // Link the standard library.
   Arguments.push_back("-L");
-  Arguments.push_back(context.Args.MakeArgString(RuntimeLibPath));
+  if (context.Args.hasFlag(options::OPT_static_stdlib,
+                            options::OPT_no_static_stdlib,
+                            false)) {
+    SmallString<128> StaticRuntimeLibPath;
+    getRuntimeStaticLibraryPath(StaticRuntimeLibPath, context.Args, *this);
+    Arguments.push_back(context.Args.MakeArgString(StaticRuntimeLibPath));
+    // The following libraries are required to build a satisfactory
+    // static program
+    Arguments.push_back("-ldl");
+    Arguments.push_back("-lpthread");
+    Arguments.push_back("-lbsd");
+    Arguments.push_back("-licui18n");
+    Arguments.push_back("-licuuc");
+    // The runtime uses dlopen to look for the protocol conformances.
+    // Therefore, we need to ensure they appear in the dynamic table.
+    // This happens automatically for dynamically-linked programs, but
+    // in this case we have to take additional measures.
+    Arguments.push_back("-Xlinker");
+    Arguments.push_back("-export-dynamic");
+    Arguments.push_back("-Xlinker");
+    Arguments.push_back("--exclude-libs");
+    Arguments.push_back("-Xlinker");
+    Arguments.push_back("ALL");
+
+  }
+  else {
+    Arguments.push_back(context.Args.MakeArgString(RuntimeLibPath));
+  }
+
+
+  // Explicitly pass the target to the linker
+  Arguments.push_back(context.Args.MakeArgString("--target=" + getTriple().str()));
 
   if (context.Args.hasArg(options::OPT_profile_generate)) {
     SmallString<128> LibProfile(RuntimeLibPath);

test/Driver/static-stdlib-linux.swift

@@ -0,0 +1,10 @@
+// Statically link a "hello world" program
+// REQUIRES: OS=linux-gnu
+// REQUIRES: static_stdlib
+print("hello world!")
+// RUN: rm -rf %t && mkdir %t
+// RUN: %target-swiftc_driver -static-stdlib -o %t/static-stdlib %s
+// RUN: %t/static-stdlib | FileCheck %s
+// RUN: ldd %t/static-stdlib | FileCheck %s --check-prefix=LDD
+// CHECK: hello world!
+// LDD-NOT: libswiftCore.so