[GR-57838] Include language resources in native image.

PullRequest: graal/19430

Author: jchalou

docs/reference-manual/embedding/embed-languages.md

@@ -538,13 +538,17 @@ To build a native executable with the above configuration, run:
 mvn -Pnative package
 ```
 
-To build a native executable from a polyglot application, for example, a Java-host application embedding Python, a `./resources` directory containing all the required files is created by default.
-By default, the language runtime will look for the resources directory relative to the native executable or library image that was built.
+Building a native executable from a polyglot application, for example, a Java-host application embedding Python, automatically captures all the internal resources required by the included languages and tools.
+By default, the resources are included in the native executable itself.
+The inclusion of resources in the native executable can be disabled by `-H:-IncludeLanguageResources`.
+Another option is a separate _resources_ directory containing all the required files.
+To switch to this option, use `-H:+CopyLanguageResources`. This is the default behavior when `-H:+IncludeLanguageResources` is not supported, i.e., with Graal Languages earlier than 24.2.x (see the [versions roadmap](https://www.graalvm.org/release-calendar/)).
+When `-H:+CopyLanguageResources` is used, the language runtime will look for the resources directory relative to the native executable or the shared library.
 At run time, the lookup location may be customized using the `-Dpolyglot.engine.resourcePath=path/to/resources` option.
-To disable the resource creation, the `-H:-CopyLanguageResources` build-time option may be used.
-Note that some languages may not support running without a resources directory.
+To disable the capturing of resources altogether, add both `-H:-IncludeLanguageResources` and `-H:-CopyLanguageResources` to build-time options.
+Note that some languages may not support running without their resources.
 
-With Polyglot version 23.1 the language home options like `-Dorg.graalvm.home` should no longer be used and were replaced with the resource directory option.
+With Graal Languages version 23.1 and newer the language home options like `-Dorg.graalvm.home` should no longer be used and were replaced with the resource directory option.
 The language home options remain functional for compatibility reasons but may be removed in future releases.
 
 ### Configuring Native Host Reflection

sdk/CHANGELOG.md

@@ -10,6 +10,7 @@ For usages of the module-path pass the `--enable-native-access=org.graalvm.truff
 * GR-54300 `Context` and `Engine` is now automatically closed when no longer strongly referenced. A reachable `Value` or `PolyglotException` will keep the associated `Context` reachable. Additionally, the `Context` remains reachable when explicitly entered or if there is an active polyglot thread within it. The `Engine` remains reachable when there is a strongly reachable `Language`, `Instrument`, or `Context` instance. However, it is still recommended not to rely on garbage collection for closing. Instead, use the try-with-resources pattern for explicit context and engine management. For more information, refer to the [Automatic Close on GC documentation](https://github.com/oracle/graal/blob/master/truffle/docs/CloseOnGc.md).
 * GR-60022 Introduced the [FileSystem.newCompositeFileSystem](https://www.graalvm.org/truffle/javadoc/org/graalvm/polyglot/io/FileSystem.html#newCompositeFileSystem(org.graalvm.polyglot.io.FileSystem,org.graalvm.polyglot.io.FileSystem.Selector...)) method, which creates a composite `FileSystem` delegating operations to the specified delegate FileSystem instances based on the provided selectors.
 * GR-60022 Introduced the [FileSystem.newDenyIOFileSystem](https://www.graalvm.org/truffle/javadoc/org/graalvm/polyglot/io/FileSystem.html#newDenyIOFileSystem()) method, which creates a `FileSystem` that denies all file operations except for path parsing.
+* GR-57838 Added automatic inclusion of language and instrument resources for embedding Truffle languages in native image. We no longer produce a _resources_ folder next to the image by default. Documentation available [here](https://www.graalvm.org/reference-manual/embed-languages/#build-native-executables-from-polyglot-applications).
 
 ## Version 24.1.0
 * GR-51177 Enable random offsets of runtime compiled function entry points for the UNTRUSTED polyglot sandbox policy.

sdk/mx.sdk/mx_sdk_vm_impl.py

@@ -2516,6 +2516,7 @@ def get_build_args(self):
             '-H:APIFunctionPrefix=truffle_isolate_',
         ] + svm_experimental_options([
             '-H:+IgnoreMaxHeapSizeWhileInVMOperation',
+            '-H:+CopyLanguageResources',
             '-H:+GenerateBuildArtifactsFile',  # generate 'build-artifacts.json'
         ]) + mx.get_runtime_jvm_args(self.subject.native_image_jar_distributions) + \
         project.native_image_config.build_args + project.native_image_config.build_args_enterprise

substratevm/src/com.oracle.svm.core/src/com/oracle/svm/core/SubstrateOptions.java

@@ -1331,10 +1331,11 @@ public static boolean useClosedTypeWorld() {
 
     public static class TruffleStableOptions {
 
-        @Option(help = "Automatically copy the necessary language resources to the resources/languages directory next to the produced image." +
-                        "Language resources for each language are specified in the native-image-resources.filelist file located in the language home directory." +
-                        "If there is no native-image-resources.filelist file in the language home directory or the file is empty, then no resources are copied.", type = User, stability = OptionStability.STABLE)//
-        public static final HostedOptionKey<Boolean> CopyLanguageResources = new HostedOptionKey<>(true);
+        @Option(help = "Automatically copy the necessary language resources to the resources directory next to the produced image.", type = User, stability = OptionStability.STABLE)//
+        public static final HostedOptionKey<Boolean> CopyLanguageResources = new HostedOptionKey<>(false);
+
+        @Option(help = "Automatically include the necessary language internal resources in the produced image.", type = User, stability = OptionStability.STABLE)//
+        public static final HostedOptionKey<Boolean> IncludeLanguageResources = new HostedOptionKey<>(true);
     }
 
     @Option(help = "Reduce the amount of metadata in the image for implicit exceptions by removing inlining information from the stack trace. " +

substratevm/src/com.oracle.svm.truffle/src/com/oracle/svm/truffle/TruffleBaseFeature.java

@@ -246,6 +246,9 @@ public boolean getAsBoolean() {
 
     private boolean needsAllEncodings;
 
+    private boolean copyLanguageResources;
+    private boolean includeLanguageResources;
+
     private static void initializeTruffleReflectively(ClassLoader imageClassLoader) {
         invokeStaticMethod("com.oracle.truffle.api.impl.Accessor", "getTVMCI", Collections.emptyList());
         invokeStaticMethod("com.oracle.truffle.polyglot.InternalResourceCache", "initializeNativeImageState",
@@ -282,6 +285,16 @@ static <T> T invokeStaticMethod(String className, String methodName, Collection<
         }
     }
 
+    static boolean isStaticMethodPresent(String className, String methodName, Collection<Class<?>> parameterTypes) {
+        try {
+            Class<?> clazz = Class.forName(className);
+            Method method = ReflectionUtil.lookupMethod(clazz, methodName, parameterTypes.toArray(new Class<?>[0]));
+            return method != null;
+        } catch (ReflectiveOperationException e) {
+            throw VMError.shouldNotReachHere(e);
+        }
+    }
+
     /**
      * Register all fields accessed by a InlinedField for an instance field or a static field as
      * unsafe accessed, which is necessary for correctness of the static analysis.
@@ -469,8 +482,31 @@ public void duringSetup(DuringSetupAccess a) {
 
         StaticObjectSupport.duringSetup(a);
 
+        if (SubstrateOptions.TruffleStableOptions.CopyLanguageResources.hasBeenSet() && SubstrateOptions.TruffleStableOptions.IncludeLanguageResources.hasBeenSet() &&
+                        SubstrateOptions.TruffleStableOptions.CopyLanguageResources.getValue() && SubstrateOptions.TruffleStableOptions.IncludeLanguageResources.getValue()) {
+            throw VMError.shouldNotReachHere("The options CopyLanguageResources and IncludeLanguageResources can't both be set to true!");
+        }
+
+        copyLanguageResources = SubstrateOptions.TruffleStableOptions.CopyLanguageResources.getValue();
+        includeLanguageResources = SubstrateOptions.TruffleStableOptions.IncludeLanguageResources.getValue();
+
+        if (includeLanguageResources && (!isStaticMethodPresent("com.oracle.truffle.polyglot.InternalResourceCache", "includeResourcesForNativeImage", List.of(Path.class, BiConsumer.class)) ||
+                        (HomeFinder.getInstance() != null && HomeFinder.getInstance().getLanguageHomes() != null && !HomeFinder.getInstance().getLanguageHomes().isEmpty()))) {
+            // IncludeLanguageResources == true, but it is not supported.
+            if (SubstrateOptions.TruffleStableOptions.IncludeLanguageResources.hasBeenSet()) {
+                throw VMError.shouldNotReachHere("The option IncludeLanguageResources has been set to true, but it is not supported!");
+            }
+            if (!SubstrateOptions.TruffleStableOptions.CopyLanguageResources.hasBeenSet()) {
+                copyLanguageResources = true;
+            }
+        }
+
+        if (copyLanguageResources) {
+            includeLanguageResources = false;
+        }
+
         HomeFinder hf = HomeFinder.getInstance();
-        if (SubstrateOptions.TruffleStableOptions.CopyLanguageResources.getValue()) {
+        if (copyLanguageResources) {
             if (!(hf instanceof DefaultHomeFinder)) {
                 VMError.shouldNotReachHere(String.format("HomeFinder %s cannot be used if CopyLanguageResources option of TruffleBaseFeature is enabled", hf.getClass().getName()));
             }
@@ -521,6 +557,38 @@ public void duringSetup(DuringSetupAccess a) {
         if (needsAllEncodings) {
             RuntimeResourceSupport.singleton().addResources(ConfigurationCondition.alwaysTrue(), "org/graalvm/shadowed/org/jcodings/tables/.*bin$", "Truffle needsAllEncodings flag is set");
         }
+
+        if (includeLanguageResources) {
+            Path resourcesForNativeImageTempDir;
+            try {
+                resourcesForNativeImageTempDir = Files.createTempDirectory("resources_for_native_image");
+            } catch (IOException e) {
+                throw VMError.shouldNotReachHere("Unable to create temporary directory for truffle language resources", e);
+            }
+            try {
+                invokeStaticMethod("com.oracle.truffle.polyglot.InternalResourceCache", "includeResourcesForNativeImage", List.of(Path.class, BiConsumer.class), resourcesForNativeImageTempDir,
+                                new BiConsumer<Module, Pair<String, byte[]>>() {
+                                    @Override
+                                    public void accept(Module module, Pair<String, byte[]> resource) {
+                                        RuntimeResourceSupport.singleton().injectResource(module, resource.getLeft(), resource.getRight(), "Truffle Language Internal Resources");
+                                    }
+                                });
+            } catch (Exception e) {
+                throw VMError.shouldNotReachHere("Unable to include truffle language resources in the image.", e);
+            } finally {
+                try (Stream<Path> filesToDelete = Files.walk(resourcesForNativeImageTempDir)) {
+                    filesToDelete.sorted(Comparator.reverseOrder()).forEach(f -> {
+                        try {
+                            Files.deleteIfExists(f);
+                        } catch (IOException ioe) {
+                            throw VMError.shouldNotReachHere("Unable to delete temporary directory for truffle language resources", ioe);
+                        }
+                    });
+                } catch (IOException ioe) {
+                    throw VMError.shouldNotReachHere("Unable to delete temporary directory for truffle language resources", ioe);
+                }
+            }
+        }
     }
 
     static void checkTruffleFile(ClassInitializationSupport classInitializationSupport, TruffleFile file) {
@@ -570,7 +638,7 @@ public void beforeAnalysis(BeforeAnalysisAccess access) {
         registerInternalResourceFieldValueTransformers(config);
     }
 
-    private static void registerInternalResourceFieldValueTransformers(BeforeAnalysisAccessImpl config) {
+    private void registerInternalResourceFieldValueTransformers(BeforeAnalysisAccessImpl config) {
         Class<?> internalResourceCacheClass = config.findClassByName("com.oracle.truffle.polyglot.InternalResourceCache");
         Class<?> internalResourceRootsClass = config.findClassByName("com.oracle.truffle.polyglot.InternalResourceRoots");
         Class<?> resetableCacheRootClass = config.findClassByName("com.oracle.truffle.polyglot.InternalResourceCache$ResettableCachedRoot");
@@ -587,6 +655,9 @@ private static void registerInternalResourceFieldValueTransformers(BeforeAnalysi
             config.registerFieldValueTransformer(ReflectionUtil.lookupField(false, internalResourceCacheClass, "path"), ResetFieldValueTransformer.INSTANCE);
             config.registerFieldValueTransformer(ReflectionUtil.lookupField(false, internalResourceRootsClass, "roots"), ResetFieldValueTransformer.INSTANCE);
         }
+        if (!copyLanguageResources && !includeLanguageResources) {
+            config.registerFieldValueTransformer(ReflectionUtil.lookupField(false, internalResourceCacheClass, "useInternalResources"), (receiver, originalValue) -> false);
+        }
     }
 
     private static final class ResetFieldValueTransformer implements FieldValueTransformer {
@@ -1133,7 +1204,7 @@ static void onBuildInvocation(Class<?> storageSuperClass, Class<?> factoryInterf
 
     @Override
     public void afterImageWrite(AfterImageWriteAccess access) {
-        if (SubstrateOptions.TruffleStableOptions.CopyLanguageResources.getValue()) {
+        if (copyLanguageResources) {
             Path buildDir = access.getImagePath();
             if (buildDir != null) {
                 Path parent = buildDir.getParent();
@@ -1150,14 +1221,13 @@ private void copyResources(Path buildDir) {
             Path languagesDir = resourcesDir.resolve("languages");
             if (Files.exists(languagesDir)) {
                 try (Stream<Path> filesToDelete = Files.walk(languagesDir)) {
-                    filesToDelete.sorted(Comparator.reverseOrder())
-                                    .forEach(f -> {
-                                        try {
-                                            Files.deleteIfExists(f);
-                                        } catch (IOException ioe) {
-                                            throw VMError.shouldNotReachHere("Deletion of previous language resources directory failed.", ioe);
-                                        }
-                                    });
+                    filesToDelete.sorted(Comparator.reverseOrder()).forEach(f -> {
+                        try {
+                            Files.deleteIfExists(f);
+                        } catch (IOException ioe) {
+                            throw VMError.shouldNotReachHere("Deletion of previous language resources directory failed.", ioe);
+                        }
+                    });
                 } catch (IOException ioe) {
                     throw VMError.shouldNotReachHere("Deletion of previous language resources directory failed.", ioe);
                 }
@@ -1464,20 +1534,6 @@ final class Target_com_oracle_truffle_polyglot_LanguageCache {
     private String languageHome;
 }
 
-@TargetClass(className = "com.oracle.truffle.polyglot.InternalResourceCache", onlyWith = TruffleBaseFeature.IsEnabled.class)
-final class Target_com_oracle_truffle_polyglot_InternalResourceCache {
-
-    @Alias @RecomputeFieldValue(kind = Kind.Custom, declClass = UseInternalResourcesComputer.class, isFinal = true) //
-    private static boolean useInternalResources;
-
-    private static final class UseInternalResourcesComputer implements FieldValueTransformer {
-        @Override
-        public Object transform(Object receiver, Object originalValue) {
-            return SubstrateOptions.TruffleStableOptions.CopyLanguageResources.getValue();
-        }
-    }
-}
-
 @TargetClass(className = "com.oracle.truffle.polyglot.PolyglotEngineImpl", onlyWith = TruffleBaseFeature.IsEnabled.class)
 final class Target_com_oracle_truffle_polyglot_PolyglotEngineImpl {
     @Substitute

truffle/CHANGELOG.md

@@ -37,6 +37,8 @@ This changelog summarizes major changes between Truffle versions relevant to lan
 * GR-32682 Added `InstrumentableNode.findProbe()` to specify how an instrumentable node finds its associated probe. This is useful for bytecode interpreters to store the probe nodes in a separate data structure without associated wrapper nodes.
 * GR-32682 Added `InstrumentableNode.createProbe(SourceSection)` method, which allows to create eager probe nodes for an instrumentable node. Eager probes do not wait for a probe to be inserted and for example all probes for statements can be inserted in a batch. Eager probes are typically used in combination with overriding the `InstrumentableNode.findProbe()` method.
 * GR-32682 Added detection of boxing overloads to support state sharing and better boxing elimination. See `Specialization#rewriteOn` for details. 
+    * `TruffleSafepoint#poll(Node)` does not require a non-null location anymore. However, it is still recommended to always pass a location node, if available. 
+* GR-57838 Added `InternalResource#unpackResourceFiles(Path, Path, Path, Predicate)` to allow filtering of resources to unpack. 
 
 ## Version 24.1.0
 * GR-43839 Added optional parameter to TruffleString.ByteIndexOfCodePointSetNode to choose whether the node may calculate the input string's precise code range.

truffle/mx.truffle/mx_truffle.py

@@ -1483,7 +1483,10 @@ def _qualname(distribution_name):
                 deps=[],
                 layout={
                     f'{resource_base_folder}/': f'dependency:{build_library.name}',
-                    f'{resource_base_folder}/resources': f'dependency:{build_library.name}/resources',
+                    f'{resource_base_folder}/resources': {"source_type": "dependency",
+                                                          "dependency": f'{build_library.name}',
+                                                          "path": 'resources',
+                                                          "optional": True},
                 },
                 path=None,
                 platformDependent=True,

truffle/src/com.oracle.truffle.api/snapshot.sigtest

@@ -190,6 +190,7 @@ meth public com.oracle.truffle.api.InternalResource$CPUArchitecture getCPUArchit
 meth public com.oracle.truffle.api.InternalResource$OS getOS()
 meth public java.util.List<java.lang.String> readResourceLines(java.nio.file.Path) throws java.io.IOException
 meth public void unpackResourceFiles(java.nio.file.Path,java.nio.file.Path,java.nio.file.Path) throws java.io.IOException
+meth public void unpackResourceFiles(java.nio.file.Path,java.nio.file.Path,java.nio.file.Path,java.util.function.Predicate<java.nio.file.Path>) throws java.io.IOException
 supr java.lang.Object
 hfds contextPreinitializationCheck,owner,resourceClass