Merge branch 'master' into feature/GR-20901

# Conflicts:
#	CHANGELOG.md
#	graalpython/com.oracle.graal.python/src/com/oracle/graal/python/builtins/modules/BuiltinFunctions.java
#	graalpython/com.oracle.graal.python/src/com/oracle/graal/python/parser/PythonParserImpl.java
#	graalpython/com.oracle.graal.python/src/com/oracle/graal/python/parser/antlr/Python3Parser.java
#	graalpython/com.oracle.graal.python/src/com/oracle/graal/python/runtime/object/PythonObjectFactory.java

Author: ppisl

.gitignore

@@ -78,3 +78,5 @@ Python3.g4.stamp
 /.*.csv*
 /graal_dumps
 *.jfr
+
+graalpython/com.oracle.graal.python/src/com/oracle/graal/python/parser/antlr/.antlr/

CHANGELOG.md

@@ -3,7 +3,9 @@
 This changelog summarizes major changes between GraalVM versions of the Python
 language runtime. The main focus is on user-observable behavior of the engine.
 
-## Version 20.1.1
+## Version 20.2
+
+* Escaping Unicode characters using the character names in strings like "\N{GREEK CAPITAL LETTER DELTA}".
 * When a `*.py` file is imported, `*.pyc` file is created. It contains binary data to speed up parsing.
 * Adding option `PyCachePrefix`, which is equivalent to PYTHONPYCACHEPREFIX environment variable, which is also accepted now. 
 * Adding optin `DontWriteBytecodeFlag`. Equivalent to the Python -B flag. Don't write bytecode files.

SECURITY.md

@@ -0,0 +1,37 @@
+# Reporting Security Vulnerabilities
+
+The GraalVM team values the independent security research community and believes
+that responsible disclosure of security vulnerabilities in GraalVM Community
+Edition as well as GraalVM Enterprise Edition helps us ensure the security and
+privacy of all our users.
+
+If you believe you have found a security vulnerability, please submit a report
+to [email protected] preferably with a proof of concept. Please refer to
+[Reporting
+Vulnerabilities](https://www.oracle.com/corporate/security-practices/assurance/vulnerability/reporting.html)
+for additional information including our public encryption key for secure
+email. We ask that you do not contact project contributors directly or through
+other channels about a report.
+
+### Security Updates, Alerts and Bulletins
+
+GraalVM Community Edition security updates will be released on a quarterly basis
+in conjunction withe GraalVM Enterprise Edition security updates that are part
+of the Oracle Critical Patch Update program. Security updates are released on
+the Tuesday closest to the 17th day of January, April, July and October. A
+pre-release announcement will be published on the Thursday preceding each
+Critical Patch Update release. For additional information including past
+advisories, please refer to [Security
+Alerts](https://www.oracle.com/security-alerts/).
+
+### Security-Related Information
+
+Please refer to the [GraalVM Security
+Guide](https://www.graalvm.org/docs/security-guide/) for security related topics
+such as how to support trusted and less trusted code execution using the Truffle
+language framework, or compiler mitigations for transitive execution
+attacks. However please note that we do not currently support the execution of
+untrusted or adversarial code. Non-vulnerability related security issues may be
+discussed on GitHub Issues or the Security channel in the [GraalVM Slack
+Workspace](https://graalvm.slack.com/)
+

THIRD_PARTY_LICENSE.txt

@@ -1 +1 @@
-{ "overlay": "342d54c7c779683771fd346ae919235334523e58" }
+{ "overlay": "c949a434d869d789978d54fd6f3e8ebe3a03c82a" }

ci.jsonnet

@@ -51,24 +51,48 @@ are:
 - `python-unittest` - Run the unittests written in Python, including those for the C extension API
 - `python-license` - Check that all files have the correct copyright headers applied to them
 
-###### Builtin modules and classes
+###### Built-In modules and classes
 
-For the most part, builtin modules and classes are implemented in the
+For the most part, built-in modules and classes are implemented in the
 `com.oracle.graal.python.builtins` package. For each module or class, there's
 Java class annoted with `@CoreFunctions`. Each function in a module or a class
 is implemented in a Node annotated with `@Builtin`. Take a look at the existing
 implementations to get a feel for how this is done. For now, when adding new
 classes or modules, they need to be added to the list in
 `com.oracle.graal.python.builtins.Python3Core`.
 
-Some builtin functions, modules, and classes are implemented in pure Python. The
+Some built-in functions, modules, and classes are implemented in pure Python. The
 files for this are in `graalpython/lib-graalpython`. These files are listed in
 the Java `com.oracle.graal.python.builtins.Python3Core` class. Take a look at
 these files to see what they do. If a file is called exactly as a built-in
 module is, it is executed in the context of that module during startup, so some
 of our modules are implemented both in Java and Python. If the name matches no
 existing module, the file is executed just for the side-effects.
 
+When implementing a new (or fixing an existing) built-in, take a look at the
+CPython source. The layout and naming of modules and types is kept similar to
+the CPython source so it should be relatively easy to find the right piece of
+code. For some special dunder methods (`__add__`, `__getitem__`,
+`__getattribute__`, ...) you may have to figure out the C API slot names for
+them to find the right piece of code (`nb_add`, `sq_item`, `tp_getattr`, ...).
+
+You will find that often there are specific C API methods that are called to
+convert or coerce arguments, to look up methods either starting on the object or
+only on the class, to call a callable object or invoke a method, and more. In
+general, most of these methods should have equivalents in our
+`PythonObjectLibrary`. See the
+[IMPLEMENTATION_DETAILS.md](./IMPLEMENTATION_DETAILS.md) file for details on
+that library. If something is missing that is commonly used, we probably have
+some Node for it, but it may be a good idea to add it to the
+`PythonObjectLibrary` for easier discovery.
+
+Sometimes, you will not easily find what exactly happens for a given piece of
+code when that involves more than just a simple built-in call. The `dis` module
+on CPython can often help get an angle on what a particular piece of code is
+doing. You can call `dis.dis` on any Python function and it will print you
+details of the bytecode and associated data, which can be a good starting point
+to browse through the CPython source.
+
 ###### Python C API
 
 The C implementation and headers for our C API are in

docs/contributor/CONTRIBUTING.md

@@ -1,6 +1,71 @@
 # Implementation Details
 
-### Python Global Thread State
+## Abstract Operations on Python Objects
+
+Many generic operations on Python objects in CPython are defined in the header
+files `object.h` and `abstract.h`. These operations are widely used and their
+interplay and intricacies are the cause for the conversion, error message, and
+control flow bugs when not mimicked correctly. Our current approach is to
+provide many of these abstract operations as part of the `PythonObjectLibrary`.
+
+### Common operations in the PythonObjectLibrary
+
+The code has evolved over time, so not all built-in nodes are prime examples of
+messages that should be used from the PythonObjectLibrary. We are refactoring
+this as we go, but here are a few examples for things you can (or should soon be
+able to) use the PythonObjectLibrary for:
+
+ - casting and coercion to `java.lang.String`, array-sized Java `int`, Python
+   index, fileno, `double`, filesystem path, iterator, and more
+ - reading the class of an object
+ - accessing the `__dict__` attribute of an object
+ - hashing objects and testing for equality
+ - testing for truthy-ness
+ - getting the length
+ - testing for abstract types such as `mapping`, `sequence`, `callable`
+ - invoking methods or executing callables
+ - access objects through the buffer protocol
+
+### PythonObjectLibrary functions with and without state
+
+Usually, there are at least two messages for each operation - one that takes a
+`ThreadState` argument, and one that doesn't. The intent is to allow passing of
+exception state and caller information similar to how we do it with the `PFrame`
+argument even across library messages, which cannot take a VirtualFrame.
+
+All nodes that are used in message implementations must allow uncached
+usage. Often (e.g. in the case of the generic `CallNode`) they offer execute
+methods with and without frames. If a `ThreadState` was passed to the message, a
+frame to pass to the node can be reconstructed using
+`PArguments.frameForCall(threadState)`. Here's an example:
+
+```java
+@ExportMessage
+long messageWithState(ThreadState state,
+        @Cached CallNode callNode) {
+    Object callable = ...
+
+    if (state != null) {
+        return callNode.execute(PArguments.frameForCall(state), callable, arguments);
+    } else {
+        return callNode.execute(callable, arguments);
+    }
+}
+```
+
+*Note*: It is **always** preferable to call an `execute` method with a
+`VirtualFrame` when both one with and without exist! The reason is that this
+avoids materialization of the frame state in more cases, as described on the
+section on Python's global thread state above.
+
+### Other libraries in the codebase
+
+Accessing hashing storages (the storage for `dict`, `set`, and `frozenset`)
+should be done via the `HashingStorageLibrary`. We are in the process of
+creating a `SequenceStorageLibrary` for sequence types (`tuple`, `list`) to
+replace the `SequenceStorageNodes` collection of classes.
+
+## Python Global Thread State
 
 In CPython, each stack frame is allocated on the heap, and there's a global
 thread state holding on to the chain of currently handled exceptions (e.g. if
@@ -21,15 +86,15 @@ be forced to the heap.
 In Graal Python, the implementation is thus a bit more involved. Here's how it
 works.
 
-#### The PFrame.Reference
+### The PFrame.Reference
 
 A `PFrame.Reference` is created when entering a Python function. By default it
 only holds on to another reference, that of the Python caller. If there are
 non-Python frames between the newly entered frame and the last Python frame,
 those are ignored - our linked list only connects Python frames. The entry point
 into the interpreter has a `PFrame.Reference` with no caller.
 
-###### ExecutionContext.CallContext and ExecutionContext.CalleeContext
+#### ExecutionContext.CallContext and ExecutionContext.CalleeContext
 
 If we're only calling between Python, we pass our `PFrame.Reference` as implicit
 argument to any callees. On entry, they will create their own `PFrame.Reference`
@@ -60,7 +125,7 @@ ExecutionContext.CalleeContext classes. These also use profiling information to
 eagerly fill in frame information if the callees actually access the stack, for
 example, so that no further stack walks need to take place.
 
-###### ExecutionContext.IndirectCallContext and ExecutionContext.IndirectCalleeContext
+#### ExecutionContext.IndirectCallContext and ExecutionContext.IndirectCalleeContext
 
 If we're mixing Python frames with non-Python frames, or if we are making calls
 to methods and cannot pass the Truffle frame, we need to store the last
@@ -72,48 +137,10 @@ caller, it initially walks the stack to find it. But it will also tell the last
 Python node that made a call to a "foreign" callee that it will have to store
 its `PFrame.Reference` globally in the future for it to be available later.
 
-#### The current PException
+### The current PException
 
 Now that we have a mechanism to lazily make available only as much frame state
 as needed, we use the same mechanism to also pass the currently handled
 exception. Unlike CPython we do not use a stack of currently handled exceptions,
 instead we utilize the call stack of Java by always passing the current exception
 and holding on to the last (if any) in a local variable.
-
-### Abstract Operations on Python Objects
-
-Many generic operations on Python objects in CPython are defined in the header
-files `abstract.c` and `abstract.h`. These operations are widely used and their
-interplay and intricacies are the cause for the conversion, error message, and
-control flow bugs when not mimicked correctly. Our current approach is to
-provide many of these abstract operations as part of the
-`PythonObjectLibrary`. Usually, this means there are at least two messages for
-each operation - one that takes a `ThreadState` argument, and one that
-doesn't. The intent is to allow passing of exception state and caller
-information similar to how we do it with the `PFrame` argument even across
-library messages, which cannot take a VirtualFrame.
-
-All nodes that are used in message implementations must allow uncached
-usage. Often (e.g. in the case of the generic `CallNode`) they offer execute
-methods with and without frames. If a `ThreadState` was passed to the message, a
-frame to pass to the node can be reconstructed using
-`PArguments.frameForCall(threadState)`. Here's an example:
-
-```java
-@ExportMessage
-long messageWithState(ThreadState state,
-        @Cached CallNode callNode) {
-    Object callable = ...
-
-    if (state != null) {
-        return callNode.execute(PArguments.frameForCall(state), callable, arguments);
-    } else {
-        return callNode.execute(callable, arguments);
-    }
-}
-```
-
-*Note*: It is **always** preferable to call an `execute` method with a
-`VirtualFrame` when both one with and without exist! The reason is that this
-avoids materialization of the frame state in more cases, as described on the
-section on Python's global thread state above.

docs/contributor/IMPLEMENTATION_DETAILS.md

@@ -0,0 +1,65 @@
+# Frequently Asked Questions
+
+### Does module/package XYZ work on GraalPython?
+
+It depends, but is currently unlikely. Our first goal with GraalPython was to
+show that we can run NumPy and related packages using the managed GraalVM LLVM
+implementation. Now that we have done so we are hard at work to improve the
+number of passing CPython unittests. We are also beginning to track our
+compatibility with popular PyPI packages and expect to increase our coverage
+there soon.
+
+### Can GraalPython replace my Jython use case?
+
+We hope it can, but there are some caveats, like Python code subclassing Java
+classes or use through the `javax.script.ScriptEngine` not being
+supported. Please see our [migration document](./JYTHON) for details.
+
+### Do I need to compile and run native modules as LLVM bitcode to use GraalPython?
+
+If you want to run C extensions or use certain built-in features, yes, you need
+to build the module with GraalPython and then it will run using the GraalVM LLVM
+runtime. However, many of the core features of Python (including e.g. large
+parts of the `os` API) are implemented in pure Java and many standard library
+modules and packages work without running any LLVM bitcode. So even though
+GraalPython depends on GraalVM LLVM, for many use cases you can disallow native
+modules entirely.
+
+### Can I use GraalVM sandboxing features with GraalPython?
+
+Yes! As an embedder, you can selectively disable features. As an example, you
+can disable native code execution or filesystem access. If you are a user of
+GraalVM Enterprise Edition, you will also find that the managed execution mode
+for LLVM fully works for running extensions such as NumPy in a safer manner.
+
+### Do all the GraalVM polyglot features work?
+
+We are doing our best to ensure the polyglot features of GraalVM work as a
+Python user would expect. There are still many cases where expectations are
+unclear or where multiple behaviors are imaginable. We are actively looking at
+use cases and are continuously evolving the implementation to provide the most
+convenient and least surprising behavior.
+
+### What is the performance I can expect from GraalPython?
+
+For pure Python code, performance after warm-up can be expected to be around 5-6
+times faster than CPython 3.8 (or 6-7x faster than Jython). For native
+extensions running as LLVM bitcode, we are currently slower than CPython - you
+can expect to see between 0.1x and 0.5x performance.
+
+### I heard languages with JIT compilers have slow startup. Is that true for GraalPython?
+
+It depends. When you use the GraalVM native image feature with GraalPython or
+use the GraalPython launcher in GraalVM its startup is competitive with
+CPython. In any case, both with native image or when running on JVM we first
+need to warm up to reach peak performance. This is a complicated story in
+itself, but in general it can take a while (a minute or two) after you have
+reached and are running your core workload. We are continuously working on
+improving this.
+
+### Can I share warmed up code between multiple Python contexts?
+
+Yes, this works, and you will find that starting up multiple contexts in the
+same engine and running the same or similar code in them will get increasingly
+faster, because the compiled code is shared across contexts. However, the peak
+performance in this setup is currently lower than in the single context case.

docs/user/FAQ.md

@@ -3,4 +3,5 @@
 A primary goal of this Python implementation is to support SciPy and its
 constituent libraries as well as work with other data science and machine
 learning libraries from the rich Python ecosystem. At this point, the Python
-implementation is made available for experimentation and curious end-users.
+implementation is made available for experimentation and curious end-users.  See
+our [FAQ](./FAQ) for commonly asked questions about this Python implementation.

docs/user/README.md

@@ -292,6 +292,17 @@ def run(self):
                 else:
                     print("### iteration=%s, name=%s, duration=%s" % (iteration, self.bench_module.__name__,
                                                                       duration_str))
+                # a bit of fuzzy logic to avoid timing out on configurations
+                # that are slow, without having to rework our logic for getting
+                # default iterations
+                if os.environ.get("CI") and iteration >= 4 and duration > 20:
+                    import statistics
+                    v = durations[-4:]
+                    if statistics.stdev(v) / min(v) < 0.03:
+                        # with less than 3 percent variance across ~20s
+                        # iterations, we can safely stop here
+                        break
+
 
         print(_HRULE)
         print("### teardown ... ")