Skip to content

Commit d41774d

Browse files
tomasstupkatomasstupka
committed
[GR-57944] GraalPy embedding: better experience when not embedding Python home.
PullRequest: graalpython/3617
2 parents 84d39c4 + 0ced85e commit d41774d

File tree

19 files changed

+454
-657
lines changed

19 files changed

+454
-657
lines changed

‎ci.jsonnet

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
{ "overlay": "80c82ac2e9a0455373dc290cffb8f3fac2ebe4f7" }
1+
{ "overlay": "b78a4cdbe55c57fc0148bf3fff9ba2ade3f1d2d2" }

‎docs/user/Embedding-Build-Tools.md

Lines changed: 2 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,6 @@ The GraalPy **Maven** and **Gradle** plugins provide functionality to manage Pyt
1111
required for embedding Python code in Java-based applications:
1212
- *Python application files* provided by the user, for example, Python sources which are part of the project.
1313
- *Third-party Python packages* installed by the plugin during the build according to the plugin configuration.
14-
- *The Python standard library*, which is necessary to make Native Image generated executables self-contained.
1514

1615
Apart from physically managing and deploying those files, it is also necessary to make them available in Python at runtime by configuring the **GraalPy Context** in your Java code accordingly.
1716
The [GraalPyResources](https://github.com/oracle/graalpython/blob/master/graalpython/org.graalvm.python.embedding/src/org/graalvm/python/embedding/utils/GraalPyResources.java) API provides factory methods to create a Context preconfigured for accessing Python, embedding relevant resources with a **Virtual Filesystem** or from a dedicated **external directory**.
@@ -53,13 +52,10 @@ The factory methods in [GraalPyResources](https://github.com/oracle/graalpython/
5352
- `${root}/venv`: used for the Python virtual environment holding installed third-party Python packages.
5453
The Context will be configured as if it is executed from this virtual environment. Notably packages installed in this
5554
virtual environment will be automatically available for importing.
56-
- `${root}/home`: used for the Python standard library (equivalent to `PYTHONHOME` environment variable).
5755

58-
The Maven or Gradle plugin will fully manage the contents of the `venv` and `home` subdirectories.
59-
Any manual changes in these directories will be overridden by the plugin during the build.
56+
The Maven or Gradle plugin will fully manage the contents of the `venv` subdirectory.
57+
Any manual change will be overridden by the plugin during the build.
6058
- `${root}/venv`: the plugin creates a virtual environment and installs required packages according to the plugin configuration in _pom.xml_ or _build.gradle_.
61-
- `${root}/home`: the plugin copies the required (also configurable) parts of the Python standard library into this directory.
62-
By default, the full standard library is used.
6359

6460
The _src_ subdirectory is left to be manually populated by the user with custom Python scripts or modules.
6561

@@ -88,24 +84,6 @@ The **packages** element declares a list of third-party Python packages to be do
8884
...
8985
</configuration>
9086
```
91-
- The **pythonHome** subsection declares what parts of the standard library should be deployed.
92-
93-
Each `include` and `exclude` element is interpreted as a Java-like regular expression specifying which file paths should be included or excluded.
94-
```xml
95-
<configuration>
96-
<pythonHome>
97-
<includes>
98-
<include>.*</include>
99-
...
100-
</includes>
101-
<excludes>
102-
<exclude></exclude>
103-
...
104-
</excludes>
105-
</pythonHome>
106-
...
107-
</configuration>
108-
```
10987
- If the **pythonResourcesDirectory** element is specified, then the given directory is used as an [external directory](#external-directory) and no Java resources are embedded.
11088
Remember to use the appropriate `GraalPyResources` API to create the Context.
11189
```xml
@@ -140,17 +118,6 @@ The plugin can be configured in the `graalPy` block:
140118
...
141119
}
142120
```
143-
- The **pythonHome** subsection declares what parts of the standard library should be deployed.
144-
Each element in the `includes` and `excludes` list is interpreted as a Java-like regular expression specifying which file paths should be included or excluded.
145-
```
146-
graalPy {
147-
pythonHome {
148-
includes = [".*"]
149-
excludes = []
150-
}
151-
...
152-
}
153-
```
154121
- If the **pythonResourcesDirectory** element is specified, then the given directory is used as an [external directory](#external-directory) and no Java resources are embedded.
155122
Remember to use the appropriate `GraalPyResources` API to create the Context.
156123
```

‎graalpython/com.oracle.graal.python.resources/src/com/oracle/graal/python/resources/PythonResource.java

Lines changed: 101 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright (c) 2023, 2024, Oracle and/or its affiliates. All rights reserved.
2+
* Copyright (c) 2023, 2025, Oracle and/or its affiliates. All rights reserved.
33
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
44
*
55
* The Universal Permissive License (UPL), Version 1.0
@@ -40,11 +40,17 @@
4040
*/
4141
package com.oracle.graal.python.resources;
4242

43+
import java.io.File;
4344
import java.io.IOException;
4445
import java.io.InputStream;
4546
import java.nio.file.InvalidPathException;
4647
import java.nio.file.Path;
48+
import java.util.ArrayList;
49+
import java.util.Collections;
4750
import java.util.List;
51+
import java.util.function.Predicate;
52+
import java.util.regex.Pattern;
53+
import java.util.regex.PatternSyntaxException;
4854

4955
import com.oracle.truffle.api.CompilerDirectives;
5056
import com.oracle.truffle.api.InternalResource;
@@ -100,20 +106,106 @@ public final class PythonResource implements InternalResource {
100106
public void unpackFiles(Env env, Path targetDirectory) throws IOException {
101107
OS os = env.getOS();
102108
Path osArch = Path.of(os.toString()).resolve(env.getCPUArchitecture().toString());
109+
ResourcesFilter filter = new ResourcesFilter();
103110
if (os.equals(OS.WINDOWS)) {
104-
env.unpackResourceFiles(BASE_PATH.resolve(LIBPYTHON_FILES), targetDirectory.resolve("Lib"), BASE_PATH.resolve(LIBPYTHON));
105-
env.unpackResourceFiles(BASE_PATH.resolve(LIBGRAALPY_FILES), targetDirectory.resolve("lib-graalpython"), BASE_PATH.resolve(LIBGRAALPY));
106-
env.unpackResourceFiles(BASE_PATH.resolve(INCLUDE_FILES), targetDirectory.resolve("Include"), BASE_PATH.resolve(INCLUDE));
111+
env.unpackResourceFiles(BASE_PATH.resolve(LIBPYTHON_FILES), targetDirectory.resolve("Lib"), BASE_PATH.resolve(LIBPYTHON), filter);
112+
env.unpackResourceFiles(BASE_PATH.resolve(LIBGRAALPY_FILES), targetDirectory.resolve("lib-graalpython"), BASE_PATH.resolve(LIBGRAALPY), filter);
113+
env.unpackResourceFiles(BASE_PATH.resolve(INCLUDE_FILES), targetDirectory.resolve("Include"), BASE_PATH.resolve(INCLUDE), filter);
107114
} else {
108115
String pythonMajMin = "python" + PYTHON_MAJOR + "." + PYTHON_MINOR;
109-
env.unpackResourceFiles(BASE_PATH.resolve(LIBPYTHON_FILES), targetDirectory.resolve("lib").resolve(pythonMajMin), BASE_PATH.resolve(LIBPYTHON));
110-
env.unpackResourceFiles(BASE_PATH.resolve(LIBGRAALPY_FILES), targetDirectory.resolve("lib").resolve("graalpy" + GRAALVM_MAJOR + "." + GRAALVM_MINOR), BASE_PATH.resolve(LIBGRAALPY));
111-
env.unpackResourceFiles(BASE_PATH.resolve(INCLUDE_FILES), targetDirectory.resolve("include").resolve(pythonMajMin), BASE_PATH.resolve(INCLUDE));
116+
env.unpackResourceFiles(BASE_PATH.resolve(LIBPYTHON_FILES), targetDirectory.resolve("lib").resolve(pythonMajMin), BASE_PATH.resolve(LIBPYTHON), filter);
117+
env.unpackResourceFiles(BASE_PATH.resolve(LIBGRAALPY_FILES), targetDirectory.resolve("lib").resolve("graalpy" + GRAALVM_MAJOR + "." + GRAALVM_MINOR), BASE_PATH.resolve(LIBGRAALPY),
118+
filter);
119+
env.unpackResourceFiles(BASE_PATH.resolve(INCLUDE_FILES), targetDirectory.resolve("include").resolve(pythonMajMin), BASE_PATH.resolve(INCLUDE), filter);
112120
}
113121
// ni files are in the same place on all platforms
114-
env.unpackResourceFiles(BASE_PATH.resolve(NI_FILES), targetDirectory, BASE_PATH);
122+
env.unpackResourceFiles(BASE_PATH.resolve(NI_FILES), targetDirectory, BASE_PATH, filter);
115123
// native files already have the correct structure
116-
env.unpackResourceFiles(BASE_PATH.resolve(osArch).resolve(NATIVE_FILES), targetDirectory, BASE_PATH.resolve(osArch));
124+
env.unpackResourceFiles(BASE_PATH.resolve(osArch).resolve(NATIVE_FILES), targetDirectory, BASE_PATH.resolve(osArch), filter);
125+
126+
if (filter.log) {
127+
System.out.println("unpacked python resources:");
128+
System.out.println("include pattern: '" + filter.include + "'");
129+
System.out.println("exclude pattern: '" + filter.exclude + "'");
130+
listFiles("included files:", filter.included);
131+
listFiles("excluded files:", filter.excluded);
132+
}
133+
}
134+
135+
private static void listFiles(String header, List<String> l) {
136+
if (!l.isEmpty()) {
137+
Collections.sort(l);
138+
System.out.println(header);
139+
for (String s : l) {
140+
System.out.println("\t" + s);
141+
}
142+
}
143+
}
144+
145+
private static class ResourcesFilter implements Predicate<Path> {
146+
147+
private final boolean log;
148+
private final Pattern includePattern;
149+
private final Pattern excludePattern;
150+
private final String exclude;
151+
private final String include;
152+
private final List<String> excluded;
153+
private final List<String> included;
154+
private static final String INCLUDE_PROP = "org.graalvm.python.resources.include";
155+
private static final String EXCLUDE_PROP = "org.graalvm.python.resources.exclude";
156+
157+
private static final String LOG_PROP = "org.graalvm.python.resources.exclude";
158+
159+
private ResourcesFilter() {
160+
include = getProperty(INCLUDE_PROP);
161+
exclude = getProperty(EXCLUDE_PROP);
162+
log = Boolean.parseBoolean(getProperty(LOG_PROP));
163+
includePattern = include != null ? compile(include, INCLUDE_PROP) : null;
164+
excludePattern = exclude != null ? compile(exclude, EXCLUDE_PROP) : null;
165+
included = log ? new ArrayList<>() : null;
166+
excluded = log ? new ArrayList<>() : null;
167+
}
168+
169+
private static Pattern compile(String re, String property) {
170+
try {
171+
return Pattern.compile(re);
172+
} catch (PatternSyntaxException pse) {
173+
throw new IllegalArgumentException("could not compile regex pattern '" + re + "' provided by system property '" + property + "'", pse);
174+
}
175+
}
176+
177+
@Override
178+
public boolean test(Path path) {
179+
String absolutePath = path.toAbsolutePath().toString();
180+
if (File.separator.equals("\\")) {
181+
absolutePath = absolutePath.replaceAll("\\\\", "/");
182+
}
183+
if ((includePattern != null && !includePattern.matcher(absolutePath).matches()) ||
184+
(excludePattern != null && excludePattern.matcher(absolutePath).matches())) {
185+
if (log) {
186+
excluded.add(absolutePath);
187+
}
188+
return false;
189+
} else {
190+
if (log) {
191+
included.add(absolutePath);
192+
}
193+
return true;
194+
}
195+
}
196+
197+
private static String getProperty(String prop) {
198+
String s = System.getProperty(prop);
199+
if (s != null) {
200+
if (s.isEmpty()) {
201+
s = null;
202+
} else if (s.startsWith("\"") && s.endsWith("\"")) {
203+
// native-gradle-plugin sends system properties wrapped in "
204+
s = s.substring(1, s.length() - 1);
205+
}
206+
}
207+
return s;
208+
}
117209
}
118210

119211
@Override

‎graalpython/com.oracle.graal.python.test.integration/src/org/graalvm/python/embedding/utils/test/integration/VirtualFileSystemIntegrationTest.java

Lines changed: 5 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -545,7 +545,7 @@ public void pythonPathsTest() throws IOException {
545545
defaultMountPoint = vfs.getMountPoint();
546546
}
547547

548-
String getPathsSource = "import sys; [__graalpython__.get_python_home_paths(), sys.path, sys.executable]";
548+
String getPathsSource = "import sys; [sys.path, sys.executable]";
549549
try (Context ctx = GraalPyResources.createContext()) {
550550
Value paths = ctx.eval("python", getPathsSource);
551551

@@ -563,7 +563,7 @@ public void pythonPathsTest() throws IOException {
563563
assertEquals(VFS_MOUNT_POINT, vfs.getMountPoint());
564564
try (Context ctx = GraalPyResources.contextBuilder(vfs).build()) {
565565
Value paths = ctx.eval("python", getPathsSource);
566-
checkPaths(paths.as(List.class), vfs.getMountPoint(), true);
566+
checkPaths(paths.as(List.class), vfs.getMountPoint());
567567
}
568568
Path resourcesDir = Files.createTempDirectory("python-resources");
569569

@@ -573,31 +573,12 @@ public void pythonPathsTest() throws IOException {
573573
}
574574
}
575575

576-
private static void checkPaths(List<Object> l, String pathPrefix) {
577-
checkPaths(l, pathPrefix, false);
578-
}
579-
580576
@SuppressWarnings("unchecked")
581-
private static void checkPaths(List<Object> l, String pathPrefix, boolean checkHome) {
582-
// python.PythonHome
583-
// TODO how to check python.PythonHome?
584-
// /org.graalvm.python.vfs/home has to be in fileslist.txt
585-
// - if it is in fileslist.txt, than it also has to contain the whole stdlib,
586-
// or other tests will fail when trying to load python
587-
// - the pythonHome value is implicitly covered in maven and gradle plugin tests
588-
// if (checkHome) {
589-
// assertTrue(((String) l.get(0)).contains(pathPrefix + File.separator + "home" +
590-
// File.separator +
591-
// "lib-graalpython"));
592-
// assertTrue(((String) l.get(0)).contains(pathPrefix + File.separator + "home" +
593-
// File.separator +
594-
// "lib-python" + File.separator + "3"));
595-
// }
596-
577+
private static void checkPaths(List<Object> l, String pathPrefix) {
597578
// option python.PythonPath
598-
assertTrue(((List<Object>) l.get(1)).contains(pathPrefix + File.separator + "src"));
579+
assertTrue(((List<Object>) l.get(0)).contains(pathPrefix + File.separator + "src"));
599580
// option python.Executable
600-
assertEquals(l.get(2), pathPrefix + (IS_WINDOWS ? "\\venv\\Scripts\\python.exe" : "/venv/bin/python"));
581+
assertEquals(l.get(1), pathPrefix + (IS_WINDOWS ? "\\venv\\Scripts\\python.exe" : "/venv/bin/python"));
601582
}
602583

603584
private static Builder addTestOptions(Builder builder) {

‎graalpython/com.oracle.graal.python.test/src/tests/conftest.toml

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,10 +2,19 @@
22
run_top_level_functions = true
33

44
[[test_rules]]
5-
selector = ['standalone/*']
5+
selector = [
6+
"standalone/test_jbang_integration.py",
7+
"standalone/test_standalone.py"]
68
per_test_timeout = 2400
79
partial_splits_individual_tests = true
810

11+
[[test_rules]]
12+
selector = [
13+
"standalone/test_gradle_plugin.py",
14+
"standalone/test_maven_plugin.py"]
15+
per_test_timeout = 3600
16+
partial_splits_individual_tests = true
17+
918
[[test_rules]]
1019
# Windows support is still experimental, so we exclude some unittests
1120
# on Windows for now. If you add unittests and cannot get them to work

‎graalpython/com.oracle.graal.python.test/src/tests/standalone/gradle/gradle-test-project/src/main/resources/META-INF/native-image/native-image.properties renamed to ‎graalpython/com.oracle.graal.python.test/src/tests/standalone/native-image.properties

File renamed without changes.

0 commit comments

Comments
 (0)