Merge branch 'master'

Author: fangerer

README.md

@@ -12,7 +12,7 @@ is made available for experimentation and curious end-users.
 To try it, you can use the bundled releases from
 [www.graalvm.org](https://www.graalvm.org/downloads/). For more information and
 some examples of what you can do with it, check out the
-[reference](https://www.graalvm.org/docs/reference-manual/languages/python/).
+[reference](https://www.graalvm.org/reference-manual/python/).
 
 ### Create a virtual environment
 
@@ -80,7 +80,7 @@ cross-language interop. This will hopefully give you an idea how to use it.
 We are working on a mode that is "mostly compatible" with some of Jython's
 features, minus of course that Jython implements Python 2.7 and we implement
 Python 3.8+. We describe the current status of the compatibility mode
-[here](docs/user/JYTHON.md).
+[here](docs/user/Jython.md).
 
 ### Contributing
 

bisect-benchmark.ini

@@ -0,0 +1,35 @@
+[bisect-benchmark]
+# This is the configuration file for bisecting benchmark jobs in the CI.
+# Usage:
+# - Create a temporary branch based on master (or the bad commit)
+# - Fill in this configuration file, commit the changes and push it
+# - Execute the bisect-benchmark job for the commit. When you click the "create
+#   PR" link, you can run jobs in commits UI (Actions). You need to wait a bit
+#   for the job enumerator to populate the job list. You don't need to actually
+#   create any PR.
+# - Wait for the jobs to finish. You should get an email when it's done
+
+# The command to build particular configuration. You can copy paste this from
+# the benchmark job log. Don't worry about sforceimports, the script takes care
+# of that
+build_command = mx --dy /compiler build
+
+# The commmand to run the particular benchmark on the configuration. You can
+# copy paste this from the benchmark job log. Make sure you replace the '*'
+# wildcard with a single benchmark, the script only supports bisecting a single
+# benchmark at a time
+benchmark_command = mx --dy /compiler benchmark micro:try-except-simple
+
+# The first known "bad" merge commit for bisection. Try to use long commit
+# SHAs, the CI cache has higher probability of short SHAs being ambiguous
+bad = 1234deadbeef
+
+# The last known "good" merge commit for bisection
+good = 1234cafebabe
+
+# Whether to checkout graal-enterprise
+enterprise = true
+
+# Which result parameter should be used for comparisons, one of BEST, WORST,
+# AVG (all runs), AVG (no warmup)
+benchmark_criterion = BEST

ci.jsonnet

@@ -1 +1 @@
-{ "overlay": "09a17d4b400a70768d3123235d66f54b8ea628f2" }
+{ "overlay": "5e162687796a6e3c6274904b8d7fdec55594760a" }

docs/contributor/CONTRIBUTING.md

@@ -180,10 +180,14 @@ A tag file can be regenerated with
 
 There's also multiple other gates that may fail with changes. One of these is
 our *style* gate, which checks formatting rules and copyrights. To auto-fix most
-issues, run the following command. Anything that's reported as error after this
-command you have to fix manually.
+issues, run the following commands. Anything that's reported as error after this
+command you have to fix manually. Note that to really match what's in the gate,
+you have to set the `JDT` environment variable to the path to an Eclipse
+compiler Jar file, and the `ECLIPSE_EXE` environment variable to the path of an
+eclipse executable.
 
     mx python-style --fix
+    mx python-gate --tags style
 
 Another important gate is the gate that checks if you broke the native image
 building. To test if building a native image still works, you can use the
@@ -225,3 +229,63 @@ For additional arguments to the Python launcher, you can separate them by
 another double-dash:
 
     mx benchmark meso:nbody3 -- --python-vm=graalpython -- --python.EmulateJython -Dgraal.Dump= -Dgraal.MethodFilter=*measure*
+
+#### A note on terminology
+
+Note that there may be a little confusion about the configuration names of
+benchmarks.
+
+##### GraalVM Community and GraalVM Enterprise configurations
+
+We have benchmarks for GraalVM Community and Enterprise. For historical reasons,
+these are sometimes referred to in some config files as *CE* and *EE*; *core*
+and *enterprise*; *graalvm_ce* and *graalvm_ee*; or *graalpython_core* and
+*graalpython_enterprise*, respectively.
+
+##### Different GraalVM Python configurations
+
+There are also different options for how the Python interpreter is run, passed
+via the `--python-vm-config` parameter:
+ * `default` - run using the standard options
+ * `default-multi` - run using a shared engine, which is the mode that is
+   recommended to embedders that want to spawn multiple isolated Python contexts
+ * `native` - same as `default`, its name is due to the fact that it runs C
+   extensions using a mixture of LLVM bitcode interpreted and compiled via
+   GraalVM and real native libraries
+ * `sandboxed` - this name is historical - this configuration requires a GraalVM
+   Enterprise and runs all C extensions purely as LLVM bitcode on the GraalVM,
+   without any access to the native OS libraries, i.e., using the
+   `--llvm.managed` option for GraalVM.
+
+##### Configuration of the underlying GraalVM runtime
+
+Finally, there are the `--jvm` and `--jvm-config` configuration options for `mx
+benchmark`. By default, the commands presented above will run on the JVM in
+*server* mode, using the Graal compiler in what we call *hosted* mode. This is
+almost the same but not quite the `--jvm` mode you will get when running the
+`graalpython` executable from a full GraalVM, and usually good enough if you
+want to look at the compiler graphs or peak performance numbers. In our CI,
+however, we always build a full GraalVM and benchmark using that, since that is
+what we ship. There, we have two different configurations corresponding to the
+launcher flags available for the GraalVM `graalpython` executable: *jvm* and
+*native*. The first runs on top of Hotspot using the Graal compiler, the second
+runs the AOT compiled GraalVM native image of Python.
+
+Building a GraalVM Python configuration can be achieved for the CE version like
+so:
+
+    mx --env ../../graal/vm/mx.vm/ce --exclude-components=slgm --dynamicimports /vm graalvm-show
+    mx --env ../../graal/vm/mx.vm/ce --exclude-components=slgm --dynamicimports /vm build
+
+The first command will print some information about the GraalVM configuration
+that is about to be built, and the second will build it. **IMPORTANT:** The
+first command should tell you that the `Config name` is `ce_python`, otherwise
+the next commands will not work.
+
+To run the JVM configuration:
+
+    mx --env ../../graal/vm/mx.vm/ce --exclude-components=slgm --dynamicimports /vm benchmark meso:nbody3 -- --python-vm=graalpython --jvm=graalvm-ce-python --jvm-config=jvm --python-vm-config=default --
+
+To run the Native Image configuration:
+
+    mx --env ../../graal/vm/mx.vm/ce --exclude-components=slgm --dynamicimports /vm benchmark meso:nbody3 -- --python-vm=graalpython --jvm=graalvm-ce-python --jvm-config=native --python-vm-config=default --

docs/user/Interoperability.md

@@ -5,23 +5,15 @@ Ruby, and LLVM. GraalVM provides a Python API to interact with other languages
 available on the GraalVM. In fact, GraalVM uses this API internally to
 execute Python C extensions using the LLVM implementation in GraalVM.
 
-You can import the `polyglot` module to interact with other languages.
+You can import the `polyglot` module to interact with other languages. In order
+to use the polyglot functionality, you need to specify `--jvm --polyglot`
+arguments to the `graalpython` executable.
 
 ```python
 import polyglot
 ```
 
-You can import a global value from the entire polyglot scope:
-```python
-imported_polyglot_global = polyglot.import_value("global_name")
-```
-
-This global should then work as expected; accessing attributes assumes it reads
-from the `members` namespace; accessing items is supported both with strings and
-numbers; calling methods on the result tries to do a straight invoke and falls
-back to reading the member and trying to execute it.
-
-You can evaluate some code in another language:
+## Evaluating code in other languages
 ```python
 polyglot.eval(string="1 + 1", language="ruby")
 ```
@@ -36,6 +28,7 @@ If you pass a file, you can also try to rely on the file-based language detectio
 polyglot.eval(path="./my_ruby_file.rb")
 ```
 
+## Exporting and importing values
 To export something from Python to other Polyglot languages so they can import
 it:
 ```python
@@ -51,6 +44,19 @@ def python_method():
     return "Hello from Python!"
 ```
 
+You can import a global value from the entire polyglot scope (exported from
+another language):
+```python
+imported_polyglot_global = polyglot.import_value("global_name")
+```
+
+This global should then work as expected; accessing attributes assumes it reads
+from the `members` namespace; accessing items is supported both with strings and
+numbers; calling methods on the result tries to do a straight invoke and falls
+back to reading the member and trying to execute it.
+
+
+## Examples
 Here is an example of how to use JavaScript regular expression engine to
 match Python strings. Save this code to the `polyglot_example.py` file:
 
@@ -135,7 +141,7 @@ the `java` module:
 ```python
 import java
 BigInteger = java.type("java.math.BigInteger")
-myBigInt = BigInteger(42)
+myBigInt = BigInteger.valueOf(42)
 myBigInt.shiftLeft(128)            # public Java methods can just be called
 myBigInt["not"]()                  # Java method names that are keywords in
                                    # Python can be accessed using "[]"
@@ -144,7 +150,8 @@ print(list(byteArray))             # Java arrays can act like Python lists
 ```
 
 For packages under the `java` package, you can also use the normal Python import
-syntax:
+syntax. This syntax is limited to importing concrete classes, it doesn't work
+on packages, unless in [Jython Compatibility](Jython.md) mode.
 ```python
 import java.util.ArrayList
 from java.util import ArrayList
@@ -179,5 +186,5 @@ print(java.is_function(my_list.add))# prints True, the add method of ArrayList
 print(java.instanceof(my_list, ArrayList)) # prints True
 ```
 
-See the [Polyglot Programming](https://www.graalvm.org/docs/reference-manual/polyglot-programming/) and the [Embed Languages](https://www.graalvm.org/reference-manual/embed-languages/) reference
+See the [Polyglot Programming](https://www.graalvm.org/reference-manual/polyglot-programming/) and the [Embed Languages](https://www.graalvm.org/reference-manual/embed-languages/) reference
 for more information about interoperability with other programming languages.

docs/user/Jython.md

@@ -22,7 +22,10 @@ command line flag on Graal: `--python.EmulateJython`.
 
 ### Importing
 Import statements allow you to import Java classes, but (unlike Jython), only
-packages in the `java` namespace can be directly imported. This will work:
+packages in the `java` namespace can be directly imported. Importing classes
+from packages outside `java` namespace also requires the
+`--python.EmulateJython` option to be active.
+This will work:
 ```
 import java.lang as lang
 ```
@@ -33,7 +36,7 @@ from javax.swing import *
 ```
 Instead, you will have to import one of the classes you are interested in directly:
 ```
-import javax.swing.Window as Window
+import javax.swing.JWindow as JWindow
 ```
 
 ### Basic Object Usage
@@ -47,7 +50,7 @@ methods:
     >>> boundNextInt = rg.nextInt
     >>> rg.nextInt()
     1491444859
-    >>> boundNextInt = rg.nextInt
+    >>> boundNextInt()
     1672896916
 
 ### Java-to-Python Types: Automatic Conversion
@@ -60,18 +63,18 @@ more dynamic approach to matching — Python types emulating `int` or
 example, to use Pandas frames as `double[][]` or NumPy array elements as `int[]`
 when the elements fit into those Java primitive types.
 
-| Java type              | Python type                                                                       |
-|:-----------------------|:----------------------------------------------------------------------------------|
-| null                   | None                                                                              |
-| boolean                | bool                                                                              |
-| byte, short, int, long | int, any object that has an `__int__` method                                      |
-| float                  | float, any object that has a `__float__` method                                   |
-| char                   | str of length 1                                                                   |
-| java.lang.String       | str                                                                               |
-| byte[]                 | bytes, bytearray, wrapped Java array, Python list with only the appropriate types |
-| Java arrays            | Wrapped Java array or Python list with only the appropriate types                 |
-| Java objects           | Wrapped Java object of the appropriate type                                       |
-| java.lang.Object       | Any object                                                                        |
+| Java type                       | Python type                                                                           |
+|:--------------------------------|:--------------------------------------------------------------------------------------|
+| `null`                          | `None`                                                                                |
+| `boolean`                       | `bool`                                                                                |
+| `byte`, `short`, `int` , `long` | `int`, any object that has an `__int__` method                                        |
+| `float`, `double`               | `float`, any object that has a `__float__` method                                     |
+| `char`                          | `str` of length 1                                                                     |
+| `java.lang.String`              | `str`                                                                                 |
+| `byte[]`                        | `bytes`, `bytearray`, wrapped Java array, Python list with only the appropriate types |
+| Java arrays                     | Wrapped Java array or Python list with only the appropriate types                     |
+| Java objects                    | Wrapped Java object of the appropriate type                                           |
+| `java.lang.Object`              | Any object                                                                            |
 
 ### Special Jython Modules
 Any of the special Jython modules are not available. For example, the `jarray`
@@ -85,6 +88,8 @@ The code that only needs to pass a Java array can also use Python types. However
 implicitly, this may entail a copy of the array data, which can be deceiving when
 using Java arrays as output parameters:
 
+    >>> # This example needs the --python.EmulateJython flag for the java.io import
+    >>> import java
     >>> i = java.io.ByteArrayInputStream(b"foobar")
     >>> buf = [0, 0, 0]
     >>> i.read(buf) # buf is automatically converted to a byte[] array
@@ -115,6 +120,8 @@ There is no automatic mapping of the Python syntax for accessing dictionary
 elements to the `java.util` mapping and list classes' ` get`, `set`, or `put`
 methods. To use these mapping and list clases, you must call the Java methods:
 
+    >>> # This example needs the --python.EmulateJython flag for the java.util import
+    >>> import java
     >>> ht = java.util.Hashtable()
     >>> ht.put("foo", "bar")
     >>> ht.get("foo")
@@ -153,6 +160,7 @@ public class PythonHandler extends Handler {
 ```
 Then you can use it like this in Python:
 ```
+# This example needs the --python.EmulateJython flag for the java.util import
 from java.util.logging import LogManager, Logger
 
 class MyHandler():

graalpython/com.oracle.graal.python.annotations/src/com/oracle/graal/python/annotations/ArgumentClinic.java

@@ -127,5 +127,14 @@ enum ClinicConversion {
          * and {@link #useDefaultForNone()}.
          */
         Index,
+        /**
+         * Corresponds to CPython's {@code int(accept={str})} convertor. Supports
+         * {@link #defaultValue()}, and {@link #useDefaultForNone()}.
+         */
+        CodePoint,
+        /**
+         * Corresponds to CPython's {@code Py_buffer} convertor.
+         */
+        Buffer,
     }
 }

graalpython/com.oracle.graal.python.cext/src/bytesobject.c

@@ -46,7 +46,7 @@
 // taken from CPython "Objects/bytesobject.c"
 #define PyBytesObject_SIZE (offsetof(PyBytesObject, ob_sval) + 1)
 
-PyTypeObject PyBytes_Type = PY_TRUFFLE_TYPE("bytes", &PyType_Type, Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_BYTES_SUBCLASS, PyBytesObject_SIZE);
+PyTypeObject PyBytes_Type = PY_TRUFFLE_TYPE_WITH_ITEMSIZE("bytes", &PyType_Type, Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE | Py_TPFLAGS_BYTES_SUBCLASS, PyBytesObject_SIZE, sizeof(char));
 
 typedef PyObject* (*fromStringAndSize_fun_t)(int8_t* str, int64_t sz);
 

graalpython/com.oracle.graal.python.cext/src/capi.c

@@ -300,6 +300,11 @@ PyObject* get_tp_dict(PyTypeObject* obj) {
 	return native_to_java(obj->tp_dict);
 }
 
+/** to be used from Java code only; reads native 'tp_base' field */
+PyObject* get_tp_base(PyTypeObject* obj) {
+	return native_to_java((PyObject*) obj->tp_base);
+}
+
 /** to be used from Java code only; reads native 'tp_bases' field */
 PyObject* get_tp_bases(PyTypeObject* obj) {
 	return native_to_java(obj->tp_bases);
@@ -475,13 +480,16 @@ const char* PyTruffle_StringToCstr(void* o, int32_t strLen) {
     return str;
 }
 
+/* Use this function to decode a C char array to a Java string using the source file encoding. */
 void* PyTruffle_CstrToString(void* o) {
-    if (polyglot_fits_in_i64(o)) {
-        return polyglot_from_string((const char*)polyglot_as_i64(o), SRC_CS);
-    }
     return polyglot_from_string(o, SRC_CS);
 }
 
+/* Use this function to decode a C ASCII string to a Java string. */
+void* PyTruffle_AsciiToString(void* ptr) {
+    return polyglot_from_string(ptr, "ascii");
+}
+
 /* To be used from Java code only.
  * This function is used if a native class inherits from a managed class but uses the 'object.__new__'.
  * This function roughly corresponds to CPython's 'object_new'. */