Skip to content

Commit b1e1896

Browse files
committed
[GR-47523] Use the truffle resource api when getLanguageHome cannot find the home
PullRequest: graalpython/2884
2 parents 17147e2 + 3df2895 commit b1e1896

File tree

18 files changed

+715
-228
lines changed

18 files changed

+715
-228
lines changed

CHANGELOG.md

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,16 @@ language runtime. The main focus is on user-observable behavior of the engine.
55

66
## Version 23.1.0
77
* Oracle GraalPy standalones (also known as GraalPy Enterprise) are now available under GFTC. The community builds published on Github have been renamed to `graalpy-community-<version>-<os>-<arch>.tar.gz`.
8-
* Support compilation and execution of C extensions using the native MSVC toolchain on Windows.
9-
* Add support for the sqlite3 module.
8+
* Add support for the sqlite3 module. This allows many packages like `coverage` or `Flask-SQLAlchemy` to work on top of this embedded database.
9+
* The GraalPy standalone tool was updated to build single-file executable Python binaries for Linux, Windows, and macOS.
10+
* The GraalPy standalone tool was updated to generate skeleton Maven projects the demonstrate polyglot embedding.
11+
* Support venv and pip installation of with pure Python wheels on Windows. This is the first preview of Windows support for this feature, and there are limitations, but pure Python packages like Pygal can be installed with `python -m pip --no-cache install pygal`.
12+
* Support compilation and execution of C extensions using the native MSVC toolchain on Windows. This feature allows building C extensions from source, installation via pip is not possible at this time.
13+
* We have [contributed](https://github.com/actions/setup-python/pull/694) support for GraalPy to GitHub's `setup-python` action. We hope it will be available in the default repository soon, until then you can try it from the fork to run your Python workloads on GraalPy.
14+
* Added support for the sqlite3 module. This allows many packages like `coverage` or `Flask-SQLAlchemy` to work on top of this embedded database.
15+
* The GraalPy standalone tool was updated. You can now build single-file executable Python binaries for Linux, Windows, and macOS as well as generate skeleton Maven projects that set up a polyglot embedding of Python packages into Java.
16+
* Support venv and pip installation of with pure Python wheels on Windows. This is the first preview of Windows support for this feature, and there are limitations, but pure Python packages like Pygal can be installed with `python -m pip --no-cache install pygal`.
17+
* Support compilation and execution of C extensions using the native MSVC toolchain on Windows. This feature allows building C extensions from source, installation via pip is not possible at this time.
1018

1119
## Version 23.0.0
1220
* Update `numpy` and `pandas` versions, add support for `scipy` and `scikit_learn` with `ginstall`. This automatically applies some fixes that make it possible to use these new versions with GraalPy.

ci.jsonnet

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1 @@
1-
{ "overlay": "76e256d73bf17fe1760a898d46482c4ed79ba96d" }
1+
{ "overlay": "f86331f835d2d903333326e664550c17bf37f896" }
Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
/*
2+
* Copyright (c) 2023, Oracle and/or its affiliates. All rights reserved.
3+
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
4+
*
5+
* The Universal Permissive License (UPL), Version 1.0
6+
*
7+
* Subject to the condition set forth below, permission is hereby granted to any
8+
* person obtaining a copy of this software, associated documentation and/or
9+
* data (collectively the "Software"), free of charge and under any and all
10+
* copyright rights in the Software, and any and all patent rights owned or
11+
* freely licensable by each licensor hereunder covering either (i) the
12+
* unmodified Software as contributed to or provided by such licensor, or (ii)
13+
* the Larger Works (as defined below), to deal in both
14+
*
15+
* (a) the Software, and
16+
*
17+
* (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
18+
* one is included with the Software each a "Larger Work" to which the Software
19+
* is contributed by such licensors),
20+
*
21+
* without restriction, including without limitation the rights to copy, create
22+
* derivative works of, display, perform, and distribute the Software and make,
23+
* use, sell, offer for sale, import, export, have made, and have sold the
24+
* Software and the Larger Work(s), and to sublicense the foregoing rights on
25+
* either these or other terms.
26+
*
27+
* This license is subject to the following condition:
28+
*
29+
* The above copyright notice and either this complete permission notice or at a
30+
* minimum a reference to the UPL must be included in all copies or substantial
31+
* portions of the Software.
32+
*
33+
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
34+
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
35+
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
36+
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
37+
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
38+
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
39+
* SOFTWARE.
40+
*/
41+
package com.oracle.graal.python.resources;
42+
43+
import java.io.IOException;
44+
import java.io.InputStream;
45+
import java.nio.file.InvalidPathException;
46+
import java.nio.file.Path;
47+
import java.util.List;
48+
49+
import com.oracle.truffle.api.CompilerDirectives;
50+
import com.oracle.truffle.api.InternalResource;
51+
import com.oracle.truffle.api.InternalResource.Id;
52+
53+
/**
54+
* This code needs to be kept in sync somehow with the suite.py code. In particular, the layouts
55+
* produced by the unpacking logic in {@link #unpackFiles} should produce the same layout as the
56+
* GRAALPYTHON_GRAALVM_SUPPORT distribution in the suite.py.
57+
*/
58+
@Id("python-home")
59+
public final class PythonResource implements InternalResource {
60+
private static final int PYTHON_MAJOR;
61+
private static final int PYTHON_MINOR;
62+
private static final int GRAALVM_MAJOR;
63+
private static final int GRAALVM_MINOR;
64+
65+
static {
66+
try (InputStream is = PythonResource.class.getResourceAsStream("/graalpy_versions")) {
67+
PYTHON_MAJOR = is.read() - ' ';
68+
PYTHON_MINOR = is.read() - ' ';
69+
is.read(); // skip python micro version
70+
GRAALVM_MAJOR = is.read() - ' ';
71+
GRAALVM_MINOR = is.read() - ' ';
72+
} catch (IOException e) {
73+
throw new RuntimeException(e);
74+
}
75+
}
76+
77+
private static final Path BASE_PATH = Path.of("META-INF", "resources");
78+
private static final String LIBGRAALPY = "libgraalpy";
79+
private static final String LIBPYTHON = "libpython";
80+
private static final String INCLUDE = "include";
81+
private static final String LIBGRAALPY_FILES = LIBGRAALPY + ".files";
82+
private static final String LIBPYTHON_FILES = LIBPYTHON + ".files";
83+
private static final String INCLUDE_FILES = "include.files";
84+
private static final String NI_FILES = "ni.files";
85+
private static final String NATIVE_FILES = "native.files";
86+
private static final Path LIBGRAALPY_SHA256 = Path.of(LIBGRAALPY + ".sha256");
87+
private static final Path LIBPYTHON_SHA256 = Path.of(LIBPYTHON + ".sha256");
88+
private static final Path INCLUDE_SHA256 = Path.of("include.sha256");
89+
private static final Path NI_SHA256 = Path.of("ni.sha256");
90+
private static final Path NATIVE_SHA256 = Path.of("native.sha256");
91+
92+
@Override
93+
public void unpackFiles(Env env, Path targetDirectory) throws IOException {
94+
OS os = env.getOS();
95+
Path osArch = Path.of(os.toString()).resolve(env.getCPUArchitecture().toString());
96+
if (os.equals(OS.WINDOWS)) {
97+
env.unpackResourceFiles(BASE_PATH.resolve(LIBPYTHON_FILES), targetDirectory.resolve("Lib"), BASE_PATH.resolve(LIBPYTHON));
98+
env.unpackResourceFiles(BASE_PATH.resolve(LIBGRAALPY_FILES), targetDirectory.resolve("lib-graalpython"), BASE_PATH.resolve(LIBGRAALPY));
99+
env.unpackResourceFiles(BASE_PATH.resolve(INCLUDE_FILES), targetDirectory.resolve("Include"), BASE_PATH.resolve(INCLUDE));
100+
} else {
101+
String pythonMajMin = "python" + PYTHON_MAJOR + "." + PYTHON_MINOR;
102+
env.unpackResourceFiles(BASE_PATH.resolve(LIBPYTHON_FILES), targetDirectory.resolve("lib").resolve(pythonMajMin), BASE_PATH.resolve(LIBPYTHON));
103+
env.unpackResourceFiles(BASE_PATH.resolve(LIBGRAALPY_FILES), targetDirectory.resolve("lib").resolve("graalpy" + GRAALVM_MAJOR + "." + GRAALVM_MINOR), BASE_PATH.resolve(LIBGRAALPY));
104+
env.unpackResourceFiles(BASE_PATH.resolve(INCLUDE_FILES), targetDirectory.resolve("include").resolve(pythonMajMin), BASE_PATH.resolve(INCLUDE));
105+
}
106+
// ni files are in the same place on all platforms
107+
env.unpackResourceFiles(BASE_PATH.resolve(NI_FILES), targetDirectory, BASE_PATH);
108+
// native files already have the correct structure
109+
env.unpackResourceFiles(BASE_PATH.resolve(osArch).resolve(NATIVE_FILES), targetDirectory, BASE_PATH.resolve(osArch));
110+
}
111+
112+
@Override
113+
public String versionHash(Env env) {
114+
StringBuilder sb = new StringBuilder();
115+
for (var s : List.of(LIBGRAALPY_SHA256, LIBPYTHON_SHA256, NI_SHA256, INCLUDE_SHA256, Path.of(env.getOS().toString()).resolve(env.getCPUArchitecture().toString()).resolve(NATIVE_SHA256))) {
116+
try {
117+
sb.append(env.readResourceLines(BASE_PATH.resolve(s)).get(0).substring(0, 8));
118+
} catch (IOException | IndexOutOfBoundsException | InvalidPathException e) {
119+
throw CompilerDirectives.shouldNotReachHere(e);
120+
}
121+
}
122+
return sb.toString();
123+
}
124+
}
Lines changed: 62 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,62 @@
1+
/*
2+
* Copyright (c) 2020, 2023, Oracle and/or its affiliates. All rights reserved.
3+
* DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
4+
*
5+
* The Universal Permissive License (UPL), Version 1.0
6+
*
7+
* Subject to the condition set forth below, permission is hereby granted to any
8+
* person obtaining a copy of this software, associated documentation and/or
9+
* data (collectively the "Software"), free of charge and under any and all
10+
* copyright rights in the Software, and any and all patent rights owned or
11+
* freely licensable by each licensor hereunder covering either (i) the
12+
* unmodified Software as contributed to or provided by such licensor, or (ii)
13+
* the Larger Works (as defined below), to deal in both
14+
*
15+
* (a) the Software, and
16+
*
17+
* (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
18+
* one is included with the Software each a "Larger Work" to which the Software
19+
* is contributed by such licensors),
20+
*
21+
* without restriction, including without limitation the rights to copy, create
22+
* derivative works of, display, perform, and distribute the Software and make,
23+
* use, sell, offer for sale, import, export, have made, and have sold the
24+
* Software and the Larger Work(s), and to sublicense the foregoing rights on
25+
* either these or other terms.
26+
*
27+
* This license is subject to the following condition:
28+
*
29+
* The above copyright notice and either this complete permission notice or at a
30+
* minimum a reference to the UPL must be included in all copies or substantial
31+
* portions of the Software.
32+
*
33+
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
34+
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
35+
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
36+
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
37+
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
38+
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
39+
* SOFTWARE.
40+
*/
41+
package com.oracle.graal.python.test.advance;
42+
43+
import static org.junit.Assert.assertTrue;
44+
45+
import java.io.File;
46+
import org.graalvm.polyglot.Context;
47+
import org.junit.Test;
48+
49+
public class ResourcesTest {
50+
@Test
51+
public void testResourcesAsHome() {
52+
try (Context context = Context.newBuilder("python").allowExperimentalOptions(true).option("python.PythonHome", "/path/that/does/not/exist").build()) {
53+
String foundHome = context.eval("python", "__graalpython__.home").asString();
54+
assertTrue(foundHome, foundHome.contains("python" + File.separator + "python-home"));
55+
}
56+
57+
try (Context context = Context.newBuilder("python").allowExperimentalOptions(true).option("python.PythonHome", "").build()) {
58+
String foundHome = context.eval("python", "__graalpython__.home").asString();
59+
assertTrue(foundHome, !foundHome.contains("graalpython"));
60+
}
61+
}
62+
}

graalpython/com.oracle.graal.python.test/src/tests/unittest_tags/test_xmlrpc.txt

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,6 @@
5959
*graalpython.lib-python.3.test.test_xmlrpc.SimpleServerTestCase.test_nonascii
6060
*graalpython.lib-python.3.test.test_xmlrpc.SimpleServerTestCase.test_nonascii_methodname
6161
*graalpython.lib-python.3.test.test_xmlrpc.SimpleServerTestCase.test_partial_post
62-
*graalpython.lib-python.3.test.test_xmlrpc.SimpleServerTestCase.test_simple1
6362
*graalpython.lib-python.3.test.test_xmlrpc.SimpleServerTestCase.test_unicode_host
6463
*graalpython.lib-python.3.test.test_xmlrpc.SimpleXMLRPCDispatcherTestCase.test_call_dispatch_func
6564
*graalpython.lib-python.3.test.test_xmlrpc.SimpleXMLRPCDispatcherTestCase.test_call_instance_func

graalpython/com.oracle.graal.python/src/com/oracle/graal/python/PythonLanguage.java

Lines changed: 46 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@
3434
import static com.oracle.graal.python.util.PythonUtils.tsLiteral;
3535

3636
import java.io.IOException;
37+
import java.io.InputStream;
3738
import java.nio.file.InvalidPathException;
3839
import java.util.ArrayList;
3940
import java.util.Arrays;
@@ -85,6 +86,7 @@
8586
import com.oracle.graal.python.pegparser.sst.ModTy;
8687
import com.oracle.graal.python.pegparser.sst.StmtTy;
8788
import com.oracle.graal.python.pegparser.tokenizer.SourceRange;
89+
import com.oracle.graal.python.resources.PythonResource;
8890
import com.oracle.graal.python.runtime.GilNode;
8991
import com.oracle.graal.python.runtime.PythonContext;
9092
import com.oracle.graal.python.runtime.PythonContext.PythonThreadState;
@@ -146,6 +148,7 @@
146148
interactive = true, internal = false, //
147149
contextPolicy = TruffleLanguage.ContextPolicy.SHARED, //
148150
fileTypeDetectors = PythonFileDetector.class, //
151+
internalResources = PythonResource.class, //
149152
website = "https://www.graalvm.org/python/")
150153
@ProvidedTags({
151154
StandardTags.CallTag.class,
@@ -178,13 +181,23 @@ public final class PythonLanguage extends TruffleLanguage<PythonContext> {
178181

179182
/**
180183
* GraalVM version. Unfortunately, we cannot just use {@link Version#getCurrent} as it relies on
181-
* a GraalVM build, but we may run from Jar files directly during development. So we hardcode
182-
* the version here. It will fail at runtime if it doesn't match the distribution layout that is
183-
* derived from the actual version.
184+
* a GraalVM build, but we may run from Jar files directly during development. We generate the
185+
* version during the build that are checked against these constants.
184186
*/
185187
public static final int GRAALVM_MAJOR = 23;
186188
public static final int GRAALVM_MINOR = 1;
187189

190+
public static final Class<PythonResource> PYTHON_RESOURCE_CLASS;
191+
192+
@SuppressWarnings("unchecked")
193+
private static Class<PythonResource> getPythonResourceClass() {
194+
try {
195+
return (Class<PythonResource>) Class.forName("com.oracle.graal.python.resources.PythonResource");
196+
} catch (ClassNotFoundException e) {
197+
return null;
198+
}
199+
}
200+
188201
static {
189202
switch (RELEASE_LEVEL) {
190203
case RELEASE_LEVEL_ALPHA:
@@ -200,6 +213,28 @@ public final class PythonLanguage extends TruffleLanguage<PythonContext> {
200213
default:
201214
RELEASE_LEVEL_STRING = tsLiteral("final");
202215
}
216+
217+
PYTHON_RESOURCE_CLASS = getPythonResourceClass();
218+
try (InputStream is = PythonLanguage.class.getResourceAsStream("/graalpy_versions")) {
219+
int ch;
220+
if (MAJOR != (ch = is.read() - ' ')) {
221+
throw new RuntimeException("suite.py version info does not match PythonLanguage#MAJOR: " + ch);
222+
}
223+
if (MINOR != (ch = is.read() - ' ')) {
224+
throw new RuntimeException("suite.py version info does not match PythonLanguage#MINOR: " + ch);
225+
}
226+
if (MICRO != (ch = is.read() - ' ')) {
227+
throw new RuntimeException("suite.py version info does not match PythonLanguage#MICRO: " + ch);
228+
}
229+
if (GRAALVM_MAJOR != (ch = is.read() - ' ')) {
230+
throw new RuntimeException("suite.py version info does not match PythonLanguage#GRAALVM_MAJOR: " + ch);
231+
}
232+
if (GRAALVM_MINOR != (ch = is.read() - ' ')) {
233+
throw new RuntimeException("suite.py version info does not match PythonLanguage#GRAALVM_MINOR: " + ch);
234+
}
235+
} catch (IOException e) {
236+
throw new RuntimeException(e);
237+
}
203238
}
204239
public static final int RELEASE_SERIAL = 0;
205240
public static final int VERSION_HEX = MAJOR << 24 |
@@ -325,7 +360,7 @@ public boolean isSingleContext() {
325360
private Shape hpySymbolCache;
326361

327362
/** For fast access to the PythonThreadState object by the owning thread. */
328-
private final ContextThreadLocal<PythonThreadState> threadState = createContextThreadLocal(PythonContext.PythonThreadState::new);
363+
private final ContextThreadLocal<PythonThreadState> threadState = locals.createContextThreadLocal(PythonContext.PythonThreadState::new);
329364

330365
public final ConcurrentHashMap<String, HiddenKey> typeHiddenKeys = new ConcurrentHashMap<>(TypeBuiltins.INITIAL_HIDDEN_TYPE_KEYS);
331366

@@ -547,7 +582,7 @@ public RootCallTarget compileForBytecodeInterpreter(PythonContext context, ModTy
547582
}
548583

549584
@TruffleBoundary
550-
public RootCallTarget compileForBytecodeInterpreter(PythonContext context, ModTy mod, Source source, boolean topLevel, int optimize, List<String> argumentNames,
585+
public RootCallTarget compileForBytecodeInterpreter(PythonContext context, ModTy modIn, Source source, boolean topLevel, int optimize, List<String> argumentNames,
551586
RaisePythonExceptionErrorCallback errorCallback, EnumSet<FutureFeature> futureFeatures) {
552587
RaisePythonExceptionErrorCallback errorCb = errorCallback;
553588
if (errorCb == null) {
@@ -556,8 +591,11 @@ public RootCallTarget compileForBytecodeInterpreter(PythonContext context, ModTy
556591
try {
557592
Compiler compiler = new Compiler(errorCb);
558593
boolean hasArguments = argumentNames != null && !argumentNames.isEmpty();
594+
final ModTy mod;
559595
if (hasArguments) {
560-
mod = transformASTForExecutionWithArguments(argumentNames, mod);
596+
mod = transformASTForExecutionWithArguments(argumentNames, modIn);
597+
} else {
598+
mod = modIn;
561599
}
562600
CompilationUnit cu = compiler.compile(mod, EnumSet.noneOf(Compiler.Flags.class), optimize, futureFeatures);
563601
CodeUnit co = cu.assemble();
@@ -656,8 +694,8 @@ public Object execute(VirtualFrame frame) {
656694
Object[] arguments = PArguments.create();
657695
// escape?
658696
PFrame pFrame = materializeFrameNode.execute(this, false, true, frame);
659-
Object locals = getFrameLocalsNode.execute(pFrame);
660-
PArguments.setSpecialArgument(arguments, locals);
697+
Object pLocals = getFrameLocalsNode.execute(pFrame);
698+
PArguments.setSpecialArgument(arguments, pLocals);
661699
PArguments.setGlobals(arguments, PArguments.getGlobals(frame));
662700
boolean wasAcquired = gilNode.acquire();
663701
try {

0 commit comments

Comments
 (0)