Merge pull request #83 from Distributive-Network/wes/pmjs-improved-parsing

pmjs improved parsing

Author: Xmader

README.md

@@ -3,17 +3,28 @@
 ![Testing Suite](https://github.com/Kings-Distributed-Systems/PythonMonkey/actions/workflows/tests.yaml/badge.svg)
 
 ## About
-PythonMonkey is a Mozilla [SpiderMonkey](https://firefox-source-docs.mozilla.org/js/index.html) JavaScript engine embedded into the Python VM,
+[PythonMonkey](https://pythonmonkey.io) is a Mozilla [SpiderMonkey](https://firefox-source-docs.mozilla.org/js/index.html) JavaScript engine embedded into the Python VM,
 using the Python engine to provide the JS host environment.
 
-This product is in an early stage, approximately 80% to MVP as of July 2023. It is under active development by Distributive Corp.,
-https://distributive.network/. External contributions and feedback are welcome and encouraged.
+This product is in an early stage, approximately 80% to MVP as of July 2023. It is under active development by [Distributive](https://distributive.network/).
+External contributions and feedback are welcome and encouraged.
 
-The goal is to make writing code in either JS or Python a developer preference, with libraries commonly used in either language
-available eveywhere, with no significant data exchange or transformation penalties. For example, it should be possible to use NumPy 
-methods from a JS library, or to refactor a slow "hot loop" written in Python to execute in JS instead, taking advantage of 
-SpiderMonkey's JIT for near-native speed, rather than writing a C-language module for Python. At Distributive, we intend to use 
-this package to execute our complex `dcp-client` library, which is written in JS and enables distributed computing on the web stack.
+### tl;dr
+```bash
+$ pip install pythonmonkey
+```
+```python
+from pythonmonkey import eval as js_eval
+
+js_eval("console.log")('hello, world')
+```
+
+### Goals
+- **Fast** and memory-efficient
+- Make writing code in either JS or Python a developer preference
+- Use JavaScript libraries from Python
+- Use Python libraries from JavaScript
+- Same process runs both JS and Python VMs - no serialization, pipes, etc
 
 ### Data Interchange
 - Strings share immutable backing stores whenever possible (when allocating engine choses UCS-2 or Latin-1 internal string representation) to keep memory consumption under control, and to make it possible to move very large strings between JS and Python library code without memory-copy overhead.
@@ -243,10 +254,71 @@ globalThis.python.exit = pm.eval("""'use strict';
 """)(sys.exit);
 ```
 
-# Troubleshooting Tips
+# pmjs
+A basic JavaScript shell, `pmjs`, ships with PythonMonkey. This shell can act as a REPL or run
+JavaScript programs; it is conceptually similar to the `node` shell which ships with Node.js.
+
+## Modules
+Pmjs starts PythonMonkey's CommonJS subsystem, which allow it to use CommonJS modules, with semantics
+that are similar to Node.js - e.g. searching module.paths, understanding package.json, index.js, and
+so on. See the [ctx-module](https://www.npmjs.com/package/ctx-module) for a full list of supported
+features.
+
+In addition to CommonJS modules written in JavaScript, PythonMonkey supports CommonJS modules written
+in Python. Simply decorate a Dict named `exports` inside a file with a `.py` extension, and it can be
+loaded by `require()` -- in either JavaScript or Python.
+
+### Program Module
+The program module, or main module, is a special module in CommonJS. In a program module,
+ - variables defined in the outermost scope are properties of `globalThis`
+ - returning from the outermost scope is a syntax error
+ - the `arguments` variable in an Array-like object which holds your program's argument vector
+   (command-line arguments)
+
+```console
+# echo "console.log('hello world')" > my-program.js
+# pmjs my-program.js
+hello world
+#
+```
 
-## REPL - pmjs 
-A basic JavaScript shell, `pmjs`, ships with PythonMonkey. This shell can also run JavaScript programs with 
+### CommonJS Module: JavaScript language
+```python
+# date-lib.js - require("./date-lib")
+const d = new Date();
+exports.today = `${d.getFullYear()}-${String(d.getMonth()).padStart(2,'0')}-${String(d.getDay()).padStart(2,'0')}`
+```
+
+### CommonJS Module: Python language
+```python
+# date-lib.py - require("./date-lib")
+from datetime import date # You can use Python libraries.
+exports['today'] = date.today()
+```
+
+# Troubleshooting Tips
 
 ## CommonJS (require)
 If you are having trouble with the CommonJS require function, set environment variable DEBUG='ctx-module*' and you can see the filenames it tries to laod.
+
+## pmjs
+- there is a `.help` menu in the REPL
+- there is a `--help` command-line option
+- the `-r` option can be used to load a module before your program or the REPL runs
+- the `-e` option can be used evaluate code -- e.g. define global variables -- before your program or the REPL runs
+- The REPL can evaluate Python expressions, storing them in variables named `$1`, `$2`, etc. ```
+# ./pmjs
+Welcome to PythonMonkey v0.2.0.
+Type ".help" for more information.
+> .python import sys
+> .python sys.path
+$1 = { '0': '/home/wes/git/pythonmonkey2',
+  '1': '/usr/lib/python310.zip',
+  '2': '/usr/lib/python3.10',
+  '3': '/usr/lib/python3.10/lib-dynload',
+  '4': '/home/wes/.cache/pypoetry/virtualenvs/pythonmonkey-StuBmUri-py3.10/lib/python3.10/site-packages',
+  '5': '/home/wes/git/pythonmonkey2/python' }
+> $1[3]
+'/usr/lib/python3.10/lib-dynload'
+> 
+```

pmjs

@@ -6,14 +6,13 @@
 import sys, os, readline, signal, getopt
 import pythonmonkey as pm
 globalThis = pm.eval("globalThis")
+evalOptions = { 'strict': False }
 
 if (os.getenv('PMJS_PATH')):
     requirePath = list(map(os.path.abspath, os.getenv('PMJS_PATH').split(':')))
 else:
     requirePath = False;
 
-globalThis = pm.eval("globalThis;")
-
 pm.eval("""'use strict';
 const cmds = {};
 
@@ -24,7 +23,7 @@ cmds.help = function help() {
 .load     Load JS from a file into the REPL session
 .save     Save all evaluated commands in this REPL session to a file
 .python   Evaluate a Python statement, returning result as global variable $n.
-          Use '.python reset' to rest back to $1.
+          Use '.python reset' to reset back to $1.
           Statement starting with 'from' or 'import' are silently executed.
 
 Press Ctrl+C to abort current expression, Ctrl+D to exit the REPL`
@@ -36,6 +35,8 @@ cmds.python = function pythonCmd(...args) {
 
   if (cmd === 'reset')
   {
+    for (let i=0; i < pythonCmd.serial; i++)
+      delete globalThis['$' + i];
     pythonCmd.serial = 0;
     return;
   }
@@ -45,8 +46,9 @@ cmds.python = function pythonCmd(...args) {
 
   const retval = python.eval(cmd);
   pythonCmd.serial = (pythonCmd.serial || 0) + 1;
-  python.print('$' + pythonCmd.serial, '=', util.inspect(retval));
   globalThis['$' + pythonCmd.serial] = retval;
+  python.stdout.write('$' + pythonCmd.serial + ' = ');
+  return util.inspect(retval);
 };
 
 /**
@@ -83,13 +85,45 @@ globalThis.replCmd = function replCmd(cmdLine)
 globalThis.replEval = function replEval(statement)
 {
   const indirectEval = eval;
+  var originalStatement = statement;
+  var result;
+  var mightBeObjectLiteral = false;
+
+  /* A statement which starts with a { and does not end with a ; is treated as an object literal, 
+   * and to get the parser in to Expression mode instead of Statement mode, we surround any expression
+   * like that which is also a valid compilation unit with parens, then if that is a syntax error, 
+   * we re-evaluate without the parens.
+   */
+  if (/^\\s*\{.*[^;\\s]\\s*$/.test(statement))
+  {
+    const testStatement = `(${statement})`;
+    if (globalThis.python.pythonMonkey.isCompilableUnit(testStatement))
+      statement = testStatement;
+  }
+
   try
   {
-    const result = indirectEval(`${statement}`);
+    try
+    {
+      result = indirectEval(statement);
+    }
+    catch(evalError)
+    {
+      /* Don't retry paren-wrapped statements which weren't syntax errors, as they might have
+       * side-effects. Never retry if we didn't paren-wrap.
+       */
+      if (!(evalError instanceof SyntaxError) || originalStatement === statement)
+        throw evalError;
+      globalThis._swallowed_error = evalError;
+      result = indirectEval(originalStatement);
+    }
+
+    globalThis._ = result;
     return util.inspect(result);
   }
   catch(error)
   {
+    globalThis._error = error;
     return util.inspect(error);
   }
 }
@@ -193,7 +227,9 @@ def repl():
             if (len(statement) == 0):
                 continue
             if (statement[0] == '.'):
-                print(globalThis.replCmd(statement[1:]))
+                cmd_output = globalThis.replCmd(statement[1:]);
+                if (cmd_output != None):
+                  print(cmd_output)
                 statement = ""
                 continue
             if (pm.isCompilableUnit(statement)):
@@ -236,6 +272,7 @@ Options:
   -p, --print [...]    evaluate script and print result
   -r, --require...     module to preload (option can be repeated)
   -v, --version        print PythonMonkey version
+  --use-strict         evaluate -e, -p, and REPL code in strict mode
   
 Environment variables:
 TZ                            specify the timezone configuration
@@ -269,7 +306,7 @@ def main():
     global requirePath
 
     try:
-        opts, args = getopt.getopt(sys.argv[1:], "hie:p:r:v", ["help", "eval=", "print=", "require=", "version"])
+        opts, args = getopt.getopt(sys.argv[1:], "hie:p:r:v", ["help", "eval=", "print=", "require=", "version", "use-strict"])
     except getopt.GetoptError as err:
         # print help information and exit:
         print(err)  # will print something like "option -a not recognized"
@@ -281,16 +318,18 @@ def main():
         if o in ("-v", "--version"):
             print(pm.__version__)
             sys.exit()
+        elif o in ("--use-strict"):
+            evalOptions['strict'] = True
         elif o in ("-h", "--help"):
             usage()
             sys.exit()
         elif o in ("-i", "--interactive"):
             forceRepl = True
         elif o in ("-e", "--eval"):
-            pm.eval(a)
+            pm.eval(a, evalOptions)
             enterRepl = False
         elif o in ("-p", "--print"):
-            print(pm.eval(a))
+            print(pm.eval(a, evalOptions))
             enterRepl = False
         elif o in ("-r", "--require"):
             globalThis.require(a)

python/pythonmonkey/require.py

@@ -48,6 +48,7 @@
 globalThis.python.pythonMonkey.dir = os.path.dirname(__file__)
 #globalThis.python.pythonMonkey.version = pm.__version__
 #globalThis.python.pythonMonkey.module = pm
+globalThis.python.pythonMonkey.isCompilableUnit = pm.isCompilableUnit;
 globalThis.python.print  = print
 globalThis.python.stdout.write = sys.stdout.write
 globalThis.python.stderr.write = sys.stderr.write

src/modules/pythonmonkey/pythonmonkey.cc

@@ -174,20 +174,17 @@ static bool getEvalOption(PyObject *evalOptions, const char *optionName, bool *b
   PyObject *value;
 
   value = PyDict_GetItemString(evalOptions, optionName);
-  if (value) {
-    if (PyLong_Check(value))
-      *b_p = PyBool_FromLong(PyLong_AsLong(value));
-    else
-      *b_p = value != Py_False;
-  }
+  if (value)
+    *b_p = PyObject_IsTrue(value) == 1 ? true : false;
   return value != NULL;
 }
 
 static PyObject *eval(PyObject *self, PyObject *args) {
+  size_t argc = PyTuple_GET_SIZE(args);
   StrType *code = new StrType(PyTuple_GetItem(args, 0));
-  PyObject *evalOptions = PyTuple_GET_SIZE(args) == 2 ? PyTuple_GetItem(args, 1) : NULL;
+  PyObject *evalOptions = argc == 2 ? PyTuple_GetItem(args, 1) : NULL;
 
-  if (!PyUnicode_Check(code->getPyObject())) {
+  if (argc == 0 || !PyUnicode_Check(code->getPyObject())) {
     PyErr_SetString(PyExc_TypeError, "pythonmonkey.eval expects a string as its first argument");
     return NULL;
   }

tests/js/pmjs-eopt.bash

@@ -27,7 +27,7 @@ do
       exit 111
       ;;
     *)
-      echo "Ignored: ${keyword} ${rest} (${loaded})"
+      echo "Ignored: ${keyword} ${rest}"
       ;;
   esac
 done