Introducing NativeLoader (#9625)

* Introducing NativeLoader

Introducing NativeLoader API intended to standardize how native libraries are loaded in dd-trace-java

NativeLoader allows for using different file layout and file resolution strategies

* spotless

* Eliminate var-args warning

Switched Objects.hash -> Arrays.hashCode

* Remove unused import

* Added preloaded test

* Javadoc and misc clean-up

* Removed errant comment

* Tweaking Javadoc

* More tests & javadoc

Enabling the NativeLoader tests - using dummy lib files, but enough to test resolution process and linkage failures

More Javadoc to help explain the various pluggable bits of the API

* Solidifying exception handling

Changed NativeLoader to raise LibraryLoadException rather than returning null when it encounters an unsupported OS or architecture

Allowing LibraryResolver and PathLocator to both raise Exceptions

Creating PathLocatorHelper to help with exception handling in LibraryResolver-s when multiple locator requests are being performed

* CapturingPathResolver -> CapturingPathLocator

Clean-up to account for prior name change of PathResolver to PathLocator

* nativeloader -> native-loader

* Adding FunctionalInterface

* Removing public from function

* Forbidden API fix

* Overloading withPreloaded to take a Set

- addressing review comments
- overloading withPreloaded to have a version that takes a Set<String>

* Hooking into arch detection code relocated to :components:environment

Also added javadoc to PlatformSpec

* Code coverage - PathUtils

* spotless

* More PathUtils coverage

* PathLocatorHelper coverage

* Coverage PlatformSpec

* Adding ability to record multiple locate requests to CapturingPathLocator

* Coverage of LibraryResolvers

- reworked CapturingPathLocator to capture all requested paths
- updated tests to cover all fallback paths
- added external OS (without arch) fallback

* Coverage PathLocators

Coverage of anonymous inner class that adds preloaded support

* Coverage LibDirBasedPathLocator

Covering multi-dir case

* PathLocators coverage

More coverage of LibDirBasedPathLocator - including component support

* Fixed bug with subResource loading in ClassLoaderResourcePathLocator

More coverage

* More coverage

Covering DefaultPlatformSpec - now renamed IntrospectPlatformSpec

* NativeLoader.Builder coverage

* spotless

* ClassLoaderBasedPathLocator coverage

* LibFile coverage

Aded more check to the assert helpers to improve LibFile coverage

* Coverage NativeLoader.Builder - added tempDir tests

* Testing tempDir functionality

Fixed problem with tempDir support when tempDir doesn't already exist

* Coverage - Testing PlatformSpec override variations of resolveDynamic

* Temp file coverage

Creating temp dirs with varying permissions to trigger problem cases

Also added simulated case for being unable to immediately clean-up the temp file

* spotless

* Update components/native-loader/src/main/java/datadog/nativeloader/FlatDirLibraryResolver.java

Co-authored-by: Brice Dutheil <[email protected]>

* Addressing review comments - lowering visibility on a few classes

* Addressing review comments - explicit equals call in test

* Addressing review comments - be more explicit about equivalence tests

* fixing merge conflict

* spotless

* Addressing review comments - LibraryResolver test clean up

Added testFailOnExceptions helper to CapturingPathLocator
Updated nested & flat resolvers tests to use new helper method
Renamed test classes to match their corresponding classes

* Addressing review comments - removing redundant final on methods

* Addressing review comments - removing final on methods

* spotless

* Addressing review comments - check a not preloaded case for Builder

* spotless

* Adding explanatory comment for review question

* Tweaking assertions for readability

* Improving path concatenation handling

Added a 2-arg version of PathUtils.concatPath
Using new concatPath version in LibDirBasedPathLocator

Added some header doc to both versions of concatPath

* Removed test jar file

Switched to creating test jar from test-data directory
Added various utility functions to make tests easier to read

---------

Co-authored-by: Brice Dutheil <[email protected]>

Author: dougqh

components/native-loader/build.gradle.kts

@@ -0,0 +1,9 @@
+plugins {
+  `java-library`
+}
+
+apply(from = "$rootDir/gradle/java.gradle")
+
+dependencies {
+  implementation(project(":components:environment"))
+}

components/native-loader/src/main/java/datadog/nativeloader/ClassLoaderResourcePathLocator.java

@@ -0,0 +1,34 @@
+package datadog.nativeloader;
+
+import java.net.URL;
+import java.util.Objects;
+
+/** ClassLoaderResourcePathLocator locates library paths inside a {@link ClassLoader} */
+final class ClassLoaderResourcePathLocator implements PathLocator {
+  private final ClassLoader classLoader;
+  private final String baseResource;
+
+  public ClassLoaderResourcePathLocator(final ClassLoader classLoader, final String baseResource) {
+    this.classLoader = classLoader;
+    this.baseResource = baseResource;
+  }
+
+  @Override
+  public URL locate(String component, String path) {
+    return this.classLoader.getResource(PathUtils.concatPath(component, this.baseResource, path));
+  }
+
+  @Override
+  public int hashCode() {
+    return Objects.hash(this.classLoader, this.baseResource);
+  }
+
+  @Override
+  public boolean equals(Object obj) {
+    if (!(obj instanceof ClassLoaderResourcePathLocator)) return false;
+
+    ClassLoaderResourcePathLocator that = (ClassLoaderResourcePathLocator) obj;
+    return this.classLoader.equals(that.classLoader)
+        && Objects.equals(this.baseResource, that.baseResource);
+  }
+}

components/native-loader/src/main/java/datadog/nativeloader/FlatDirLibraryResolver.java

@@ -0,0 +1,55 @@
+package datadog.nativeloader;
+
+import java.net.URL;
+
+/**
+ * FlatDirLibraryResolver - uses flat directories to provide more specific libraries to load <code>
+ * {os}-{arch}-{libc/musl}</code>
+ */
+public final class FlatDirLibraryResolver implements LibraryResolver {
+  public static final FlatDirLibraryResolver INSTANCE = new FlatDirLibraryResolver();
+
+  private FlatDirLibraryResolver() {}
+
+  @Override
+  public final URL resolve(
+      PathLocator pathLocator, String component, PlatformSpec platformSpec, String libName)
+      throws Exception {
+    PathLocatorHelper pathLocatorHelper = new PathLocatorHelper(libName, pathLocator);
+
+    String libFileName = PathUtils.libFileName(platformSpec, libName);
+
+    String osPath = PathUtils.osPartOf(platformSpec);
+    String archPath = PathUtils.archPartOf(platformSpec);
+    String libcPath = PathUtils.libcPartOf(platformSpec);
+
+    URL url;
+    String regularPath = osPath + "-" + archPath;
+
+    if (libcPath != null) {
+      String specializedPath = regularPath + "-" + libcPath;
+      url = pathLocatorHelper.locate(component, specializedPath + "/" + libFileName);
+      if (url != null) return url;
+    }
+
+    url = pathLocatorHelper.locate(component, regularPath + "/" + libFileName);
+    if (url != null) return url;
+
+    url = pathLocatorHelper.locate(component, osPath + "/" + libFileName);
+    if (url != null) return url;
+
+    // fallback to searching at top-level, mostly concession to good out-of-box behavior
+    // with java.library.path
+    url = pathLocatorHelper.locate(component, libFileName);
+    if (url != null) return url;
+
+    if (component != null) {
+      url = pathLocatorHelper.locate(null, libFileName);
+      if (url != null) return url;
+    }
+
+    pathLocatorHelper.tryThrow();
+
+    return null;
+  }
+}

components/native-loader/src/main/java/datadog/nativeloader/IntrospectPlatformSpec.java

@@ -0,0 +1,65 @@
+package datadog.nativeloader;
+
+import datadog.environment.OperatingSystem;
+import datadog.environment.OperatingSystem.Architecture;
+
+/*
+ * Default PlatformSpec used in dd-trace-java -- wraps detection code in component:environment
+ */
+final class IntrospectPlatformSpec extends PlatformSpec {
+  static final PlatformSpec INSTANCE = new IntrospectPlatformSpec();
+
+  @Override
+  public boolean isLinux() {
+    return OperatingSystem.isLinux();
+  }
+
+  @Override
+  public boolean isMac() {
+    return OperatingSystem.isMacOs();
+  }
+
+  @Override
+  public boolean isWindows() {
+    return OperatingSystem.isWindows();
+  }
+
+  @Override
+  public boolean isMusl() {
+    return OperatingSystem.isMusl();
+  }
+
+  @Override
+  public boolean isAarch64() {
+    return isArch(Architecture.ARM64);
+  }
+
+  @Override
+  public boolean isArm32() {
+    return isArch(Architecture.ARM);
+  }
+
+  @Override
+  public boolean isX86_32() {
+    return isArch(Architecture.X86);
+  }
+
+  @Override
+  public boolean isX86_64() {
+    return isArch(Architecture.X64);
+  }
+
+  static final boolean isArch(OperatingSystem.Architecture arch) {
+    return (OperatingSystem.architecture() == arch);
+  }
+
+  @Override
+  public int hashCode() {
+    return IntrospectPlatformSpec.class.hashCode();
+  }
+
+  @Override
+  public boolean equals(Object obj) {
+    return (obj instanceof IntrospectPlatformSpec);
+  }
+}

components/native-loader/src/main/java/datadog/nativeloader/LibDirBasedPathLocator.java

@@ -0,0 +1,49 @@
+package datadog.nativeloader;
+
+import java.io.File;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.util.Arrays;
+
+/** LibDirBasedPathLocator locates libraries inside a list of library directories */
+final class LibDirBasedPathLocator implements PathLocator {
+  private final File[] libDirs;
+
+  public LibDirBasedPathLocator(File... libDirs) {
+    this.libDirs = libDirs;
+  }
+
+  @Override
+  public URL locate(String component, String path) {
+    String fullPath = PathUtils.concatPath(component, path);
+
+    for (File libDir : this.libDirs) {
+      File libFile = new File(libDir, fullPath);
+      if (libFile.exists()) return toUrl(libFile);
+    }
+
+    return null;
+  }
+
+  @SuppressWarnings("deprecation")
+  private static final URL toUrl(File file) {
+    try {
+      return file.toURL();
+    } catch (MalformedURLException e) {
+      return null;
+    }
+  }
+
+  @Override
+  public int hashCode() {
+    return Arrays.hashCode(this.libDirs);
+  }
+
+  @Override
+  public boolean equals(Object obj) {
+    if (!(obj instanceof LibDirBasedPathLocator)) return false;
+
+    LibDirBasedPathLocator that = (LibDirBasedPathLocator) obj;
+    return Arrays.equals(this.libDirs, that.libDirs);
+  }
+}

components/native-loader/src/main/java/datadog/nativeloader/LibFile.java

@@ -0,0 +1,81 @@
+package datadog.nativeloader;
+
+import java.io.File;
+import java.nio.file.Path;
+
+/**
+ * Represents a resolved library
+ *
+ * <ul>
+ *   <li>library may be preloaded - with no backing file
+ *   <li>regular file - that doesn't require clean-up
+ *   <li>temporary file - copying from another source - that does require clean-up
+ * </ul>
+ */
+public final class LibFile implements AutoCloseable {
+  static final boolean NO_CLEAN_UP = false;
+  static final boolean CLEAN_UP = true;
+
+  static final LibFile preloaded(String libName) {
+    return new LibFile(libName, null, NO_CLEAN_UP);
+  }
+
+  static final LibFile fromFile(String libName, File file) {
+    return new LibFile(libName, file, NO_CLEAN_UP);
+  }
+
+  static final LibFile fromTempFile(String libName, File file) {
+    return new LibFile(libName, file, CLEAN_UP);
+  }
+
+  final String libName;
+
+  final File file;
+  final boolean needsCleanup;
+
+  LibFile(String libName, File file, boolean needsCleanup) {
+    this.libName = libName;
+
+    this.file = file;
+    this.needsCleanup = needsCleanup;
+  }
+
+  /** Indicates if this library was "preloaded" */
+  public boolean isPreloaded() {
+    return (this.file == null);
+  }
+
+  /** Loads the underlying library into the JVM */
+  public void load() throws LibraryLoadException {
+    if (this.isPreloaded()) return;
+
+    try {
+      Runtime.getRuntime().load(this.getAbsolutePath());
+    } catch (Throwable t) {
+      throw new LibraryLoadException(this.libName, t);
+    }
+  }
+
+  /** Provides a File to the library -- returns null for pre-loaded libraries */
+  public final File toFile() {
+    return this.file;
+  }
+
+  /** Provides a Path to the library -- return null for pre-loaded libraries */
+  public final Path toPath() {
+    return this.file == null ? null : this.file.toPath();
+  }
+
+  /** Provides the an absolute path to the library -- returns null for pre-loaded libraries */
+  public final String getAbsolutePath() {
+    return this.file == null ? null : this.file.getAbsolutePath();
+  }
+
+  /** Schedules clean-up of underlying file -- if the file is a temp file */
+  @Override
+  public void close() {
+    if (this.needsCleanup) {
+      NativeLoader.delete(this.file);
+    }
+  }
+}

components/native-loader/src/main/java/datadog/nativeloader/LibraryLoadException.java

@@ -0,0 +1,29 @@
+package datadog.nativeloader;
+
+/** Exception raised when NativeLoader fails to resolve or load a library */
+public class LibraryLoadException extends Exception {
+  static final String UNSUPPORTED_OS = "Unsupported OS";
+  static final String UNSUPPORTED_ARCH = "Unsupported arch";
+
+  private static final long serialVersionUID = 1L;
+
+  public LibraryLoadException(String libName) {
+    super(message(libName));
+  }
+
+  public LibraryLoadException(String libName, Throwable cause) {
+    this(message(libName), cause.getMessage(), cause);
+  }
+
+  public LibraryLoadException(String libName, String message) {
+    super(message(libName) + " - " + message);
+  }
+
+  public LibraryLoadException(String libName, String message, Throwable cause) {
+    super(message(libName) + " - " + message, cause);
+  }
+
+  static final String message(String libName) {
+    return "Unable to resolve library " + libName;
+  }
+}

components/native-loader/src/main/java/datadog/nativeloader/LibraryResolver.java

@@ -0,0 +1,20 @@
+package datadog.nativeloader;
+
+import java.net.URL;
+
+/**
+ * LibraryResolver encapsulates a library resolution strategy
+ *
+ * <p>The LibraryResolver should use the provided {@link PathLocator} to locate the desired
+ * resources. The LibraryResolver may try multiple locations to find the best possible library to
+ * use.
+ */
+@FunctionalInterface
+public interface LibraryResolver {
+  default boolean isPreloaded(PlatformSpec platform, String libName) {
+    return false;
+  }
+
+  URL resolve(PathLocator pathLocator, String component, PlatformSpec platformSpec, String libName)
+      throws Exception;
+}

components/native-loader/src/main/java/datadog/nativeloader/LibraryResolvers.java

@@ -0,0 +1,44 @@
+package datadog.nativeloader;
+
+import java.net.URL;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.Set;
+
+public final class LibraryResolvers {
+  private LibraryResolvers() {}
+
+  public static final LibraryResolver defaultLibraryResolver() {
+    return flatDirs();
+  }
+
+  public static final LibraryResolver withPreloaded(
+      LibraryResolver baseResolver, String... preloadedLibNames) {
+    return withPreloaded(baseResolver, new HashSet<>(Arrays.asList(preloadedLibNames)));
+  }
+
+  public static final LibraryResolver withPreloaded(
+      LibraryResolver baseResolver, Set<String> preloadedLibNames) {
+    return new LibraryResolver() {
+      @Override
+      public boolean isPreloaded(PlatformSpec platform, String libName) {
+        return preloadedLibNames.contains(libName);
+      }
+
+      @Override
+      public URL resolve(
+          PathLocator pathLocator, String component, PlatformSpec platformSpec, String libName)
+          throws Exception {
+        return baseResolver.resolve(pathLocator, component, platformSpec, libName);
+      }
+    };
+  }
+
+  public static final LibraryResolver flatDirs() {
+    return FlatDirLibraryResolver.INSTANCE;
+  }
+
+  public static final LibraryResolver nestedDirs() {
+    return NestedDirLibraryResolver.INSTANCE;
+  }
+}