[GR-44758] Fixes for PyGame

PullRequest: graalpython/2676

Author: timfel

CHANGELOG.md

@@ -12,6 +12,15 @@ language runtime. The main focus is on user-observable behavior of the engine.
 * Allow excluding the Java-based SSL module that uses classes from `java.security.*`, `org.bouncycastle.*`, `javax.net.ssl.*` and any related classes from the native image by passing `-Dpython.java.ssl=false` to the native image build Java arguments.
 * Allow excluding the Java UnixSystem classes from `com.sun.security.auth.*` by passing `-Dpython.java.auth=false` to the native image build Java arguments. This makes the POSIX calls `getpwuid`, `getpwname`, and `getuid` return less precise results in the Java-based POSIX backend.
 * Allow excluding the use of `sun.misc.Signal` and `sun.misc.SignalHandler` from GraalPy by passing `-Dpython.java.signals=false` to the native image build Java arguments. This removes the `signal` module from the binary.
+* We now run an publish benchmark results from the community's [pyperformance](https://pyperformance.readthedocs.io) benchmark suite on our [website](http://graalvm.org/python). This makes it easier to compare and reproduce our results.
+* Allow building and running basic workloads on Windows. This enables Windows users to build and use GraalPy, especially for embedding into Java.
+* Implement complete support for PEP 622 pattern matching. All features of Python's structural pattern matching should now work.
+* Update the distribution layout of GraalPy to match CPython's. This reduces the number of patches we need for various build systems to discover GraalPy's library locations.
+* Add an option for embedders of GraalPy to poll asynchronous actions explicitly. This prevents GraalPy from creating system threads for collecting Python-level references and instead provides a callback that the embedder calls to do this work at regular intervals. See docs/user/PythonNativeImages.md for details.
+* Remove the intrinsified `zipimport` module in favor of the pure Python version. This fixes subtle incompatibilities with CPython when handling ZIP files.
+* Use the JDK's `MessageDigest` for hashing instead of pure Python implementations. This improves performance and compatibility in Java embeddings.
+* Add initial support for `asyncio`. While not complete, this already allows some async libraries like `aiofiles` to work.
+* Add a new implementation of our Python C API interface that uses fully native execution by default. This improves performance and compatibility with some extensions that spend a lot of time in native code, but can have negative effects in workloads that cross often from Python to native code and back. There are new options to control how extensions are built and run: `python.NativeModules` and `python.UseSystemToolchain`. The new default is to use the host system's toolchain for building extensions rather than the LLVM toolchain that ships with GraalVM, and to run all modules natively.
 
 ## Version 22.3.0
 * Rename GraalPython to GraalPy. This change also updates the launchers we ship to include symlinks from `python` and `python3` to `graalpy` for better integration with other tools.

README.md

@@ -1,24 +1,22 @@
 # GraalVM Implementation of Python
 
-This is an early-stage experimental implementation of Python. A primary goal is
-to support SciPy and its constituent libraries. GraalPy can usually execute
-pure Python code faster than CPython (but not when C extensions are
-involved). GraalPy currently aims to be compatible
-with Python 3.8, but it is a long way from there, and it is very likely that any
-Python program that uses more features of standard library modules or external
-packages will hit something unsupported. At this point, the Python
-implementation is made available for experimentation and curious end-users.
+This is GraalPy, an implementation of the Python language.
+A primary goal is to support SciPy and its constituent libraries.
+GraalPy can usually execute pure Python code faster than CPython (but not when C extensions are involved).
+GraalPy currently aims to be compatible with Python 3.10, but it is some way from there.
+While your specific workload may function, any Python program that uses external packages could hit something unsupported.
+At this point, the Python implementation is made available for experimentation and curious end-users.
 
 ### Trying It
 
-The easiest option to try GraalPy is
-[Pyenv](https://github.com/pyenv/pyenv/), the Python version manager. It allows
-you to easily install different GraalPy releases. To get version 22.3.0, for
-example, just run `pyenv install graalpy-22.3.0`.
+The easiest option to try GraalPy is [Pyenv](https://github.com/pyenv/pyenv/), the Python version manager.
+It allows you to easily install different GraalPy releases.
+To get version 22.3.0, for example, just run `pyenv install graalpy-22.3.0`.
 
-To try GraalPy with a full GraalVM, including the support for Java embedding
-and interop with other languages, you can use the bundled releases from
-[www.graalvm.org](https://www.graalvm.org/downloads/).
+To try GraalPy with a full GraalVM, including the support for Java embedding and interop with other languages, you can use the bundled releases from [www.graalvm.org](https://www.graalvm.org/downloads/).
+
+Another option is to use [Conda-Forge](https://conda-forge.org/).
+To get an environment with the latest GraalPy, use `conda create -c conda-forge -n graalpy graalpy`.
 
 ### Building from Source
 
@@ -62,30 +60,11 @@ In the venv, multiple executables are available, like `python`, `python3` and `g
 
 ### Installing Packages
 
-Currently, not enough of the standard library is implemented to run the
-standard package installers for many packages. As a convenience, we provide a
-simple module to install packages that we know to be working (including
-potential patches required for those packages). Try the following to find out
-which packages are at least partially supported and tested by us in our CI:
-```
-graalpy -m ginstall install --help
-```
-
-As a slightly exciting example, try:
-```
-graalpy -m ginstall install pandas
-```
-
-If all goes well (also consider native dependencies of NumPy), you should be
-able to `import numpy` and `import pandas` afterwards.
-
-Support for more extension modules is high priority for us. We are actively
-building out our support for the Python C API to make extensions such as NumPy,
-SciPy, Scikit-learn, Pandas, Tensorflow and the like work fully. This work means
-that some other extensions might also already work, but we're not actively
-testing other extensions right now and cannot promise anything. Note that to try
-other extensions on this implementation, you have to download, build, and
-install them manually for now.
+You should be able to use the `pip` command from a GraalPy venv to install packages.
+Our `pip` ships some patches for packages that we test internally, these will be applied automatically where necessary.
+Support for as many extension modules as possible is a high priority for us.
+We are actively building out our support for the Python C API to make extensions such as NumPy, SciPy, Scikit-learn, Pandas, Tensorflow and the like work fully.
+This means that some might already work, but we're still actively working on compatibility especially with native extensions.
 
 ### Polyglot Usage
 
@@ -96,7 +75,7 @@ cross-language interop. This will hopefully give you an idea how to use it.
 
 We are working on a mode that is "mostly compatible" with some of Jython's
 features, minus of course that Jython implements Python 2.7 and we implement
-Python 3.8+. We describe the current status of the compatibility mode
+Python 3.10+. We describe the current status of the compatibility mode
 [here](docs/user/Jython.md).
 
 ### Contributing

graalpython/com.oracle.graal.python.cext/src/capi.h

@@ -299,7 +299,7 @@ typedef struct {
     BUILTIN(PyTruffleObject_GenericGetAttr, PyObject*, PyObject*, PyObject*) \
     BUILTIN(PyTruffleObject_GenericSetAttr, int, PyObject*, PyObject*, PyObject*) \
     BUILTIN(PyTruffleObject_GetItemString, PyObject*, PyObject*, const char*) \
-    BUILTIN(PyTruffleState_FindModule, PyObject*, struct PyModuleDef*) \
+    BUILTIN(PyTruffleState_FindModule, PyObject*, Py_ssize_t) \
     BUILTIN(PyTruffleStructSequence_InitType2, int, PyTypeObject*, void*, void*, int) \
     BUILTIN(PyTruffleStructSequence_NewType, PyTypeObject*, const char*, const char*, void*, void*, int) \
     BUILTIN(PyTruffleToCharPointer, void*, PyObject*) \

graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/modules/cext/PythonCextBuiltinRegistry.java

@@ -51,7 +51,6 @@
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.Int;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.LONG_LONG;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.Long;
-import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.PYMODULEDEF_PTR;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.PY_CAPSULE_DESTRUCTOR;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.PY_COMPILER_FLAGS;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.PY_GIL_STATE_STATE;
@@ -566,7 +565,7 @@ private PythonCextBuiltinRegistry() {
                     new CApiBuiltinExecutable("PyTruffleObject_GetItemString", Ignored, PyObjectTransfer,
                                     new ArgDescriptor[]{PyObject, ConstCharPtrAsTruffleString}, 201),
                     new CApiBuiltinExecutable("PyTruffleState_FindModule", Ignored, PyObjectBorrowed,
-                                    new ArgDescriptor[]{PYMODULEDEF_PTR}, 202),
+                                    new ArgDescriptor[]{Py_ssize_t}, 202),
                     new CApiBuiltinExecutable("PyTruffleStructSequence_InitType2", Ignored, Int,
                                     new ArgDescriptor[]{PyTypeObject, Pointer, Pointer, Int}, 203),
                     new CApiBuiltinExecutable("PyTruffleStructSequence_NewType", Ignored, PyTypeObjectTransfer,

graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/modules/cext/PythonCextBuiltins.java

@@ -244,9 +244,13 @@ static PythonBuiltinObject doOther(@SuppressWarnings("unused") Object value) {
     }
 
     public static PException checkThrowableBeforeNative(Throwable t, String where1, Object where2) {
-        if (t instanceof PException) {
+        if (t instanceof PException pe) {
             // this is ok, and will be handled correctly
-            throw (PException) t;
+            throw pe;
+        }
+        if (t instanceof ThreadDeath td) {
+            // ThreadDeath subclasses are used internally by Truffle
+            throw td;
         }
         // everything else: log and convert to PException (SystemError)
         CompilerDirectives.transferToInterpreter();

graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/modules/cext/PythonCextPyStateBuiltins.java

@@ -43,10 +43,10 @@
 import static com.oracle.graal.python.builtins.modules.cext.PythonCextBuiltins.CApiCallPath.Direct;
 import static com.oracle.graal.python.builtins.modules.cext.PythonCextBuiltins.CApiCallPath.Ignored;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.Int;
-import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.PYMODULEDEF_PTR;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.PY_GIL_STATE_STATE;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.PyObjectBorrowed;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.PyThreadState;
+import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.Py_ssize_t;
 import static com.oracle.graal.python.builtins.objects.cext.capi.transitions.ArgDescriptor.Void;
 
 import com.oracle.graal.python.builtins.modules.cext.PythonCextBuiltins.CApiBuiltin;
@@ -123,7 +123,7 @@ PDict get(@Cached PythonObjectFactory factory) {
         }
     }
 
-    @CApiBuiltin(ret = PyObjectBorrowed, args = {PYMODULEDEF_PTR}, call = Ignored)
+    @CApiBuiltin(ret = PyObjectBorrowed, args = {Py_ssize_t}, call = Ignored)
     abstract static class PyTruffleState_FindModule extends CApiUnaryBuiltinNode {
 
         @Specialization

graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/objects/cext/capi/transitions/ArgDescriptor.java

@@ -313,7 +313,7 @@ public enum ArgDescriptor {
 
     func_objint("int (*)(PyObject*value)"),
     func_voidvoidptr("void (*)(void*)"),
-    func_voidvoid("void (*)(void)"),
+    func_voidvoid(ArgBehavior.Pointer, "void (*)(void)"),
     func_intvoidptr("int (*)(void*)"),
     func_objvoid("PyObject*(*)(void)"),
     func_objcharsizevoidptr("PyObject*(*)(const char*, Py_ssize_t, void*)"),

graalpython/lib-graalpython/patches/pygame/sdist/pygame-2.patch

@@ -0,0 +1,45 @@
+diff --git a/src_c/base.c b/src_c/base.c
+index e73eb001..6f1c479d 100644
+--- a/src_c/base.c
++++ b/src_c/base.c
+@@ -25,6 +25,7 @@
+ #include "pygame.h"
+ 
+ #include <signal.h>
++#include <string.h>
+ #include "doc/pygame_doc.h"
+ #include "pgarrinter.h"
+ #include "pgcompat.h"
+@@ -2022,18 +2023,27 @@ pg_install_parachute(void)
+ {
+ #ifdef HAVE_SIGNAL_H
+     int i;
+-    void (*ohandler)(int);
++    struct sigaction act;
+ 
+     if (parachute_installed) {
+         return;
+     }
+     parachute_installed = 1;
+ 
+-    /* Set a handler for any fatal signal not already handled */
++    /* Set a handler for any fatal signal not already handled.
++       Use sigaction to retrieve the current handler first, so
++       we don't mess with signal handlers in multi-threaded
++       runtimes without a GIL. This is still racy, but better
++       than using the signal function. */
+     for (i = 0; fatal_signals[i]; ++i) {
+-        ohandler = (void (*)(int))signal(fatal_signals[i], pygame_parachute);
+-        if (ohandler != SIG_DFL) {
+-            signal(fatal_signals[i], ohandler);
++        sigaction(fatal_signals[i], NULL, &act);
++        if (act.sa_handler == SIG_DFL) {
++            memset(&act, 0, sizeof(struct sigaction));
++            sigemptyset(&act.sa_mask);
++            sigaddset(&act.sa_mask, fatal_signals[i]);
++            act.sa_handler = pygame_parachute;
++            act.sa_flags = SA_RESTART;
++            sigaction(fatal_signals[i], &act, NULL);
+         }
+     }
+ 

mx.graalpython/mx_graalpython.py

@@ -1843,6 +1843,7 @@ def _python_checkpatchfiles():
             "MIT", "BSD", "BSD-3-Clause", "BSD 3-Clause License", "BSD or Apache License, Version 2.0",
             "MIT license", "PSF", "BSD-3-Clause OR Apache-2.0", "Apache", "Apache License", "new BSD",
             "(Apache-2.0 OR BSD-3-Clause) AND PSF-2.0", "Apache 2.0", "MPL-2.0", "BSD 3-Clause",
+            "LGPL",
         ]
         for line in content.split("\n"):
             if not line or os.stat(line).st_size == 0: