[GR-20901] Better (de)serialization of code objects

PullRequest: graalpython/876

Author: ppisl

CHANGELOG.md

@@ -6,6 +6,10 @@ language runtime. The main focus is on user-observable behavior of the engine.
 ## Version 20.2
 
 * Escaping Unicode characters using the character names in strings like "\N{GREEK CAPITAL LETTER DELTA}".
+* When a `*.py` file is imported, `*.pyc` file is created. It contains binary data to speed up parsing.
+* Adding option `PyCachePrefix`, which is equivalent to PYTHONPYCACHEPREFIX environment variable, which is also accepted now. 
+* Adding optin `DontWriteBytecodeFlag`. Equivalent to the Python -B flag. Don't write bytecode files.
+* Command option -B works
 
 ## Version 20.1.0
 

docs/contributor/PARSER_DETAILS.md

@@ -0,0 +1,84 @@
+This document is about parsing python files in GraalPython implementation. 
+It describes way how we obtain Truffle tree from a source.
+
+Creating Truffle tree for a python source has two phases. The first one creates  
+simple syntax tree (SST) and scope tree, the second phase transforms the SST to 
+the Truffle tree and for the transformation we need scope tree. The scope tree 
+contains scope locations for variable and function definitions and information 
+about scopes. The simple syntax tree contains nodes mirroring the source. 
+Comparing SST and Truffle tree, the SST is much smaller. It contains just the nodes 
+representing the source in a simple way. One SST node is usually translated 
+to many Truffle nodes. 
+
+The simple syntax tree can be created in two ways. With ANTLR parsing 
+or deserialization from appropriate `*.pyc` file. In both cases together with 
+scope tree. If there is no appropriate `.pyc` file for a source, then the source 
+is parsed with ANTLR and result SST and scope tree is serialized to the `.pyc` file.  
+The next time, we don't have to use ANTLR parser, because the result is already 
+serialized in the `.pyc` file. So instead of parsing source file with ANTLR, 
+we just deserialized SST and scope tree from the `.pyc` file.  The deserialization 
+is much faster then source parsing with ANTLR. The deserialization needs ruffly 
+just 30% of the time  that needs ANTLR parser. Of course the first run is little 
+bit slower (we need to SST and scope tree save to the `.pyc` file). 
+
+In the folder structure it looks like this:
+
+```
+top_folder
+    __pycache__
+         sourceA.graalpython.pyc
+         sourceB.graalpython.pyc
+    sourceA.py
+    sourceB.py
+    sub_folder
+        __pycache__
+            sourceX.graalpython.pyc
+        sourceX.py
+```
+
+On the same directory level of a source code file, the `__pycache__` directory 
+is created and in this directory are stored all `.*pyc` files from the same 
+directory. There can be also files created with CPython, so user can see there 
+also files with extension `*.cpython3-6.pyc` for example. 
+
+The current implementation includes also copy of the original text into `.pyc' file. 
+The reason is that we create from this Truffle Source object with path to the 
+original source file, but we do not need to read the original `*.py` file, which 
+speed up the process obtaining Truffle tree (we read just one file). 
+
+The structure of a `.graalpython.pyc` file is this:
+
+```
+MAGIC_NUMBER
+source text
+binary data - scope tree
+binary data - simple syntax tree
+```
+
+The serialized SST and scope tree is stored in Code object as well, attribute `code`
+
+For example:
+```
+>>> def add(x, y):
+... print('Running x+y')
+... return x+y
+...
+>>> co = add.__code__
+>>> co.co_code
+b'\x01\x00\x00\x02[]K\xbf\xd1\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00 ...'
+```
+
+The creating `*.pyc` files can be canceled / allowed  in the same ways like in CPython:
+
+ * evironment variable: PYTHONDONTWRITEBYTECODE - If this is set to a non-empty string, 
+Python won’t try to write .pyc files on the import of source modules. 
+ * command line option: -B, If given, Python won’t try to write .pyc files on 
+the import of source modules.
+ * in a code: setting attribute `dont_write_bytecode` of `sys` built in module
+
+
+## Security
+The serialization of SST and scope tree is hand written and during deserialization 
+is not possible to load other classes then SSTNodes. It doesn't use Java serialization 
+or other framework to serialize Java object. The main reason was performance. 
+The performance can be maximize in this way. The next reason was the security.

graalpython/com.oracle.graal.python.shell/src/com/oracle/graal/python/shell/GraalPythonMain.java

@@ -87,6 +87,7 @@ public static void main(String[] args) {
     private List<String> relaunchArgs;
     private boolean wantsExperimental = false;
     private Map<String, String> enginePolyglotOptions;
+    private boolean dontWriteBytecode = false;
 
     @Override
     protected List<String> preprocessArguments(List<String> givenArgs, Map<String, String> polyglotOptions) {
@@ -102,6 +103,7 @@ protected List<String> preprocessArguments(List<String> givenArgs, Map<String, S
             String arg = arguments.get(i);
             switch (arg) {
                 case "-B":
+                    dontWriteBytecode = true;
                     break;
                 case "-c":
                     i += 1;
@@ -374,6 +376,11 @@ protected void launch(Builder contextBuilder) {
             noUserSite = noUserSite || System.getenv("PYTHONNOUSERSITE") != null;
             verboseFlag = verboseFlag || System.getenv("PYTHONVERBOSE") != null;
             unbufferedIO = unbufferedIO || System.getenv("PYTHONUNBUFFERED") != null;
+            dontWriteBytecode = dontWriteBytecode || System.getenv("PYTHONDONTWRITEBYTECODE") != null;
+            String cachePrefix = System.getenv("PYTHONPYCACHEPREFIX");
+            if (cachePrefix != null) {
+                contextBuilder.option("python.PyCachePrefix", cachePrefix);
+            }
         }
 
         String executable = getContextOptionIfSetViaCommandLine("python.Executable");
@@ -392,6 +399,7 @@ protected void launch(Builder contextBuilder) {
         contextBuilder.option("python.InspectFlag", Boolean.toString(inspectFlag));
         contextBuilder.option("python.VerboseFlag", Boolean.toString(verboseFlag));
         contextBuilder.option("python.IsolateFlag", Boolean.toString(isolateFlag));
+        contextBuilder.option("python.DontWriteBytecodeFlag", Boolean.toString(dontWriteBytecode));
         if (verboseFlag) {
             contextBuilder.option("log.python.level", "FINE");
         }
@@ -548,8 +556,7 @@ protected String getLanguageId() {
     protected void printHelp(OptionCategory maxCategory) {
         print("usage: python [option] ... (-c cmd | file) [arg] ...\n" +
                         "Options and arguments (and corresponding environment variables):\n" +
-                        "-B     : on CPython, this disables writing .py[co] files on import;\n" +
-                        "         GraalPython does not use bytecode, and thus this flag has no effect\n" +
+                        "-B     : this disables writing .py[co] files on import\n" +
                         "-c cmd : program passed in as string (terminates option list)\n" +
                         // "-d : debug output from parser; also PYTHONDEBUG=x\n" +
                         "-E     : ignore PYTHON* environment variables (such as PYTHONPATH)\n" +
@@ -601,6 +608,8 @@ protected void printHelp(OptionCategory maxCategory) {
                         "   as specifying the -R option: a random value is used to seed the hashes of\n" +
                         "   str, bytes and datetime objects.  It can also be set to an integer\n" +
                         "   in the range [0,4294967295] to get hash values with a predictable seed.\n" +
+                        "PYTHONPYCACHEPREFIX: if this is set, GraalPython will write .pyc files in a mirror\n" +
+                        "   directory tree at this path, instead of in __pycache__ directories within the source tree.\n" +
                         "GRAAL_PYTHON_ARGS: the value is added as arguments as if passed on the\n" +
                         "   commandline. There is one special case: any `$$' in the value is replaced\n" +
                         "   with the current process id. To pass a literal `$$', you must escape the\n" +

graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/parser/FuncDefTests.java

@@ -191,6 +191,18 @@ public void functionDef21() throws Exception {
                         "foo(1,2)\n");
     }
 
+    @Test
+    public void functionDef22() throws Exception {
+        checkScopeAndTree(
+                        "def test():\n" +
+                                        "  a = 1;\n" +
+                                        "  def fn1(): pass\n" +
+                                        "  def fn2(): pass\n" +
+                                        "  return locals()\n" +
+                                        "\n" +
+                                        "print(test())\n");
+    }
+
     @Test
     public void decorator01() throws Exception {
         checkScopeAndTree();

graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/parser/ParserTestBase.java

@@ -43,6 +43,7 @@
 import com.oracle.graal.python.PythonLanguage;
 import com.oracle.graal.python.parser.PythonParserImpl;
 import com.oracle.graal.python.parser.ScopeInfo;
+import com.oracle.graal.python.parser.sst.SSTNode;
 import com.oracle.graal.python.runtime.PythonContext;
 import com.oracle.graal.python.runtime.PythonParser;
 import com.oracle.graal.python.runtime.exception.PException;
@@ -79,6 +80,7 @@ public class ParserTestBase {
     @Rule public TestName name = new TestName();
 
     private ScopeInfo lastGlobalScope;
+    private SSTNode lastSST;
 
     public ParserTestBase() {
         PythonTests.enterContext();
@@ -106,13 +108,18 @@ public Node parse(Source source, PythonParser.ParserMode mode) {
         PythonParser parser = context.getCore().getParser();
         Node result = ((PythonParserImpl) parser).parseN(mode, context.getCore(), source, null);
         lastGlobalScope = ((PythonParserImpl) parser).getLastGlobaScope();
+        lastSST = ((PythonParserImpl) parser).getLastSST();
         return result;
     }
 
     protected ScopeInfo getLastGlobalScope() {
         return lastGlobalScope;
     }
 
+    protected SSTNode getLastSST() {
+        return lastSST;
+    }
+
     public void checkSyntaxError(String source) throws Exception {
         boolean thrown = false;
         try {
@@ -253,14 +260,14 @@ public void checkScopeResult(String source, PythonParser.ParserMode mode) throws
         assertDescriptionMatches(scopes.toString(), goldenScopeFile);
     }
 
-    private String printTreeToString(Node node) {
+    protected String printTreeToString(Node node) {
         ParserTreePrinter visitor = new ParserTreePrinter();
         visitor.printFormatStringLiteralDetail = printFormatStringLiteralValues;
         node.accept(visitor);
         return visitor.getTree();
     }
 
-    protected void assertDescriptionMatches(String actual, File goldenFile) throws IOException {
+    protected void assertDescriptionMatches(String actual, File goldenFile) throws Exception {
         if (!goldenFile.exists()) {
             if (!goldenFile.createNewFile()) {
                 assertTrue("Cannot create file " + goldenFile.getAbsolutePath(), false);
@@ -272,6 +279,10 @@ protected void assertDescriptionMatches(String actual, File goldenFile) throws I
         }
         String expected = readFile(goldenFile);
 
+        assertDescriptionMatches(actual, expected, goldenFile.getName());
+    }
+
+    protected void assertDescriptionMatches(String actual, String expected, String someName) throws Exception {
         final String expectedTrimmed = expected.trim();
         final String actualTrimmed = actual.trim();
 
@@ -290,7 +301,7 @@ protected void assertDescriptionMatches(String actual, File goldenFile) throws I
 
             // There are some diffrerences between expected and actual content --> Test failed
 
-            assertTrue("Not matching goldenfile: " + goldenFile.getName() + lineSeparator(2) + getContentDifferences(expectedUnified, actualUnified), false);
+            assertTrue("Not matching results: " + (someName == null ? "" : someName) + lineSeparator(2) + getContentDifferences(expectedUnified, actualUnified), false);
         }
     }
 

graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/parser/ParserTreePrinter.java

@@ -1,5 +1,5 @@
 /*
- * Copyright (c) 2019, Oracle and/or its affiliates. All rights reserved.
+ * Copyright (c) 2019, 2020, Oracle and/or its affiliates. All rights reserved.
  * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
  *
  * The Universal Permissive License (UPL), Version 1.0
@@ -443,6 +443,12 @@ private void addSignature(Signature signature) {
         add(signature.takesPositionalOnly());
         sb.append(", requiresKeywordArgs=");
         add(signature.takesRequiredKeywordArgs());
+        if (signature.getVarargsIdx() > -1) {
+            sb.append(", varArgsIdx=").append(signature.getVarargsIdx());
+        }
+        if (signature.getPositionalOnlyArgIndex() > -1) {
+            sb.append(", positionalOnlyIdx=").append(signature.getPositionalOnlyArgIndex());
+        }
         newLine();
         level++;
         if (signature.getParameterIds() != null && signature.getParameterIds().length > 0) {

graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/parser/SSTNodeWithScopeFinderTest.java

@@ -0,0 +1,120 @@
+/*
+ * Copyright (c) 2020, Oracle and/or its affiliates. All rights reserved.
+ * DO NOT ALTER OR REMOVE COPYRIGHT NOTICES OR THIS FILE HEADER.
+ *
+ * The Universal Permissive License (UPL), Version 1.0
+ *
+ * Subject to the condition set forth below, permission is hereby granted to any
+ * person obtaining a copy of this software, associated documentation and/or
+ * data (collectively the "Software"), free of charge and under any and all
+ * copyright rights in the Software, and any and all patent rights owned or
+ * freely licensable by each licensor hereunder covering either (i) the
+ * unmodified Software as contributed to or provided by such licensor, or (ii)
+ * the Larger Works (as defined below), to deal in both
+ *
+ * (a) the Software, and
+ *
+ * (b) any piece of software and/or hardware listed in the lrgrwrks.txt file if
+ * one is included with the Software each a "Larger Work" to which the Software
+ * is contributed by such licensors),
+ *
+ * without restriction, including without limitation the rights to copy, create
+ * derivative works of, display, perform, and distribute the Software and make,
+ * use, sell, offer for sale, import, export, have made, and have sold the
+ * Software and the Larger Work(s), and to sublicense the foregoing rights on
+ * either these or other terms.
+ *
+ * This license is subject to the following condition:
+ *
+ * The above copyright notice and either this complete permission notice or at a
+ * minimum a reference to the UPL must be included in all copies or substantial
+ * portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+ */
+package com.oracle.graal.python.test.parser;
+
+import com.oracle.graal.python.PythonLanguage;
+import com.oracle.graal.python.parser.sst.FunctionDefSSTNode;
+import com.oracle.graal.python.parser.sst.LambdaSSTNode;
+import com.oracle.graal.python.parser.sst.SSTNode;
+import com.oracle.graal.python.parser.sst.SSTNodeWithScopeFinder;
+import com.oracle.graal.python.runtime.PythonParser;
+import com.oracle.truffle.api.source.Source;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class SSTNodeWithScopeFinderTest extends ParserTestBase {
+
+    @Test
+    public void calllArgTest() throws Exception {
+        String code = "class Tests():\n" +
+                        "    def test1(self):\n" +
+                        "        rdd = self.parallelize(range(100))\n" +
+                        "        assert rdd.reduce(lambda a, b: a+b) == 4950";
+        checkFinder(code, code.indexOf("def test1"), code.indexOf("950") + 3, false);
+        checkFinder(code, code.indexOf("lambda a"), code.indexOf("a+b") + 3, true);
+    }
+
+    @Test
+    public void tryTest() throws Exception {
+        String code = "try:\n" +
+                        "    import sys\n" +
+                        "    process = None\n" +
+                        "\n" +
+                        "    def fn1():\n" +
+                        "        return 20\n" +
+                        "\n" +
+                        "except ImportError:\n" +
+                        "\n" +
+                        "    def fn2():\n" +
+                        "        return 30" +
+                        "\n";
+        checkFinder(code, code.indexOf("def fn1"), code.indexOf("except"), false);
+        checkFinder(code, code.indexOf("def fn2"), code.indexOf("30") + 3, false);
+    }
+
+    @Test
+    public void ifTest() throws Exception {
+        String code = "        if True:\n" +
+                        "            a = 1\n" +
+                        "            def fn1():\n" +
+                        "                return 10\n" +
+                        "        elif False:\n" +
+                        "            b = 2\n" +
+                        "            def fn2():\n" +
+                        "                return 20\n" +
+                        "        else:\n" +
+                        "            def fn3():\n" +
+                        "                return 30\n";
+        checkFinder(code, code.indexOf("def fn1"), code.indexOf("elif"), false);
+        checkFinder(code, code.indexOf("def fn2"), code.indexOf("else"), false);
+        checkFinder(code, code.indexOf("def fn3"), code.indexOf("30") + 3, false);
+    }
+
+    private void checkFinder(String code, int startOffset, int endOffset, boolean isLambda) {
+        SSTNode result = findNodeWithScope(code, startOffset, endOffset);
+        Assert.assertNotNull("No node with scope was found ", result);
+        if (isLambda) {
+            Assert.assertTrue("Was expected LambdaSSTNode, but " + result.getClass().getSimpleName() + " found.", result instanceof LambdaSSTNode);
+        } else {
+            Assert.assertTrue("Was expected FunctionDefSSTNode, but " + result.getClass().getSimpleName() + " found.", result instanceof FunctionDefSSTNode);
+        }
+        Assert.assertTrue("Start or end offset is not the expected one.", result.getStartOffset() == startOffset && result.getEndOffset() == endOffset);
+    }
+
+    private SSTNode findNodeWithScope(String code, int startOffset, int endOffset) {
+        Source source = Source.newBuilder(PythonLanguage.ID, code, "NodeWithScopeFinderTest").build();
+        parse(source, PythonParser.ParserMode.File);
+        SSTNode lastSST = getLastSST();
+        SSTNodeWithScopeFinder finder = new SSTNodeWithScopeFinder(startOffset, endOffset);
+        return lastSST.accept(finder);
+    }
+
+}