Merge pull request #216 from Distributive-Network/philippe/fix/201

proper class structure for proxies: refactoring. Missing Dict-Object …

Author: philippedistributive

LICENSE

@@ -3,10 +3,12 @@
 ![Testing Suite](https://github.com/Kings-Distributed-Systems/PythonMonkey/actions/workflows/tests.yaml/badge.svg)
 
 ## About
-[PythonMonkey](https://pythonmonkey.io) is a Mozilla [SpiderMonkey](https://firefox-source-docs.mozilla.org/js/index.html) JavaScript engine embedded into the Python VM,
-using the Python engine to provide the JS host environment.
+[PythonMonkey](https://pythonmonkey.io) is a Mozilla [SpiderMonkey](https://firefox-source-docs.mozilla.org/js/index.html) JavaScript engine embedded into the Python Runtime,
+using the Python engine to provide the Javascript host environment. 
 
-This product is in an early stage, approximately 80% to MVP as of July 2023. It is under active development by [Distributive](https://distributive.network/).
+We feature JavaScript Array and Object methods on Python List and Dictionaries using the cPython C API, and the inverse using the Mozilla Firefox Spidermonkey JavaScript C++ API. 
+
+This product is in an intermediate stage, approximately 90% to MVP as of January 2024. It is under active development by [Distributive](https://distributive.network/).
 External contributions and feedback are welcome and encouraged.
 
 ### tl;dr
@@ -24,16 +26,17 @@ js_eval("console.log")('hello, world')
 - Make writing code in either JS or Python a developer preference
 - Use JavaScript libraries from Python
 - Use Python libraries from JavaScript
-- Same process runs both JS and Python VMs - no serialization, pipes, etc
+- The same process runs both JavaScript and Python VirtualMachines - no serialization, pipes, etc
+- Python Lists and Dicts behave as Javacript Arrays and Objects, and vice-versa, fully adapting to the given context.
 
 ### Data Interchange
-- Strings share immutable backing stores whenever possible (when allocating engine choses UCS-2 or Latin-1 internal string representation) to keep memory consumption under control, and to make it possible to move very large strings between JS and Python library code without memory-copy overhead.
+- Strings share immutable backing stores whenever possible (when allocating engine choses UCS-2 or Latin-1 internal string representation) to keep memory consumption under control, and to make it possible to move very large strings between JavaScript and Python library code without memory-copy overhead.
 - TypedArrays share mutable backing stores.
-- JS objects are represented by Python dicts
+- JS objects are represented by Python dicts through a Dict subclass for optimal compatibility. Similarly for JS arrays and Python lists.
 - JS Date objects are represented by Python datetime.datetime objects
 - Intrinsics (boolean, number, null, undefined) are passed by value
 - JS Functions are automatically wrapped so that they behave like Python functions, and vice-versa
-- Python Lists are represented by JS true Arrays
+- Python Lists are represented by JS true arrays and support all Array methods through a JS API Proxy. Similarly for Python Dicts and JS objects.
 
 ### Roadmap
 - [done] JS instrinsics coerce to Python intrinsics
@@ -49,14 +52,13 @@ js_eval("console.log")('hello, world')
 - [done] Python `require` function, returns a coerced dict of module exports
 - [done] Python functions coerce to JS function wrappers
 - [done] CommonJS module system .py loader, loads Python modules for use by JS
-- JS object->Python dict coercion supports inherited-property lookup (via __getattribute__?)
 - [done] Python host environment supplies event loop, including EventEmitter, setTimeout, etc.
-- [done] Python host environment supplies XMLHttpRequest (other project?)
-- Python host environment supplies basic subsets of NodeJS's fs, path, process, etc, modules; as-needed by dcp-client (other project?)
+- [done] Python host environment supplies XMLHttpRequest
+- Python host environment supplies basic subsets of NodeJS's fs, path, process, etc, modules; as-needed by dcp-client
 - [done] Python TypedArrays coerce to JS TypeArrays
 - [done] JS TypedArrays coerce to Python TypeArrays
-- [done] JS Arrays coerce to Python Lists, providing all List methods implemented on Arrays
-- [done] Python List coerce to JS Arrays, providing all Array methods implemented on Lists
+- [done] Python lists coerce to JS Arrays
+- [done] JS arrays coerce to Python lists
 
 ## Build Instructions
 
@@ -229,6 +231,8 @@ Where shared backing store is not possible, PythonMonkey will automatically emit
 the "real" data structure as its value authority. Only immutable intrinsics are copied. This means
 that if you update an object in JavaScript, the corresponding Dict in Python will be updated, etc.
 
+JavaScript Array and Object methods are implemented on Python List and Dictionaries, and vice-versa. 
+
 | Python Type | JavaScript Type |
 |:------------|:----------------|
 | String      | string

README.md

@@ -0,0 +1,55 @@
+/**
+ * @file PyBaseProxyHandler.hh
+ * @author Caleb Aikens ([email protected]) and Philippe Laporte ([email protected])
+ * @brief Structs for creating JS proxy objects.
+ * @version 0.1
+ * @date 2023-04-20
+ *
+ * Copyright (c) 2023-2024 Distributive Corp.
+ *
+ */
+
+#ifndef PythonMonkey_PyBaseProxy_
+#define PythonMonkey_PyBaseProxy_
+
+#include <jsapi.h>
+#include <jsfriendapi.h>
+#include <js/Conversions.h>
+#include <js/Proxy.h>
+
+#include <Python.h>
+
+/**
+ * @brief base class for PyDictProxyHandler and PyListProxyHandler
+ */
+struct PyBaseProxyHandler : public js::BaseProxyHandler {
+public:
+  PyBaseProxyHandler(PyObject *pyObj, const void *family) : js::BaseProxyHandler(family), pyObject(pyObj) {};
+  PyObject *pyObject; // @TODO (Caleb Aikens) Consider putting this in a private slot
+
+  bool getPrototypeIfOrdinary(JSContext *cx, JS::HandleObject proxy, bool *isOrdinary, JS::MutableHandleObject protop) const override final;
+  bool preventExtensions(JSContext *cx, JS::HandleObject proxy, JS::ObjectOpResult &result) const override final;
+  bool isExtensible(JSContext *cx, JS::HandleObject proxy, bool *extensible) const override final;
+};
+
+enum ProxySlots {PyObjectSlot};
+
+typedef struct {
+  const char *name;      /* The name of the method */
+  JSNative call;         /* The C function that implements it */
+  uint16_t nargs;        /* The argument count for the method */
+} JSMethodDef;
+
+/**
+ * @brief Convert jsid to a PyObject to be used as dict keys
+ */
+PyObject *idToKey(JSContext *cx, JS::HandleId id);
+
+/**
+ * @brief Convert Python dict key to jsid
+ */
+bool keyToId(PyObject *key, JS::MutableHandleId idp);
+
+bool idToIndex(JSContext *cx, JS::HandleId id, Py_ssize_t *index);
+
+#endif

include/PyBaseProxyHandler.hh

@@ -1,50 +1,26 @@
 /**
- * @file PyProxyHandler.hh
+ * @file PyDictProxyHandler.hh
  * @author Caleb Aikens ([email protected]) and Philippe Laporte ([email protected])
- * @brief Structs for creating JS proxy objects. Used by DictType for object coercion and by ListType for List coercion
- * @version 0.1
+ * @brief Structs for creating JS proxy objects. Used by DictType for object coercion
  * @date 2023-04-20
  *
  * Copyright (c) 2023-2024 Distributive Corp.
  *
  */
 
-#ifndef PythonMonkey_PyProxy_
-#define PythonMonkey_PyProxy_
+#ifndef PythonMonkey_PyDictProxy_
+#define PythonMonkey_PyDictProxy_
 
-#include <jsapi.h>
-#include <js/Proxy.h>
+#include "PyBaseProxyHandler.hh"
 
-#include <Python.h>
-
-/**
- * @brief base class for PyProxyHandler and PyListProxyHandler
- */
-struct PyBaseProxyHandler : public js::BaseProxyHandler {
-public:
-  PyBaseProxyHandler(PyObject *pyObj, const void *family) : js::BaseProxyHandler(family), pyObject(pyObj) {};
-  PyObject *pyObject; // @TODO (Caleb Aikens) Consider putting this in a private slot
-
-  bool getPrototypeIfOrdinary(JSContext *cx, JS::HandleObject proxy, bool *isOrdinary, JS::MutableHandleObject protop) const override final;
-  bool preventExtensions(JSContext *cx, JS::HandleObject proxy, JS::ObjectOpResult &result) const override final;
-  bool isExtensible(JSContext *cx, JS::HandleObject proxy, bool *extensible) const override final;
-};
-
-enum ProxySlots {PyObjectSlot};
-
-typedef struct {
-  const char *name;      /* The name of the method */
-  JSNative call;         /* The C function that implements it */
-  uint16_t nargs;        /* The argument count for the method */
-} JSMethodDef;
 
 /**
  * @brief This struct is the ProxyHandler for JS Proxy Objects pythonmonkey creates to handle coercion from python dicts to JS Objects
  *
  */
-struct PyProxyHandler : public PyBaseProxyHandler {
+struct PyDictProxyHandler : public PyBaseProxyHandler {
 public:
-  PyProxyHandler(PyObject *pyObj) : PyBaseProxyHandler(pyObj, &family) {};
+  PyDictProxyHandler(PyObject *pyObj) : PyBaseProxyHandler(pyObj, &family) {};
   static const char family;
 
   /**
@@ -72,7 +48,6 @@ public:
     JS::ObjectOpResult &result) const override;
   /**
    * @brief [[HasProperty]]
-   *
    * @param cx - pointer to JSContext
    * @param proxy - The proxy object who's propery we wish to check
    * @param id - key value of the property to check
@@ -82,20 +57,6 @@ public:
    */
   bool has(JSContext *cx, JS::HandleObject proxy, JS::HandleId id,
     bool *bp) const override;
-  /**
-   * @brief [[Get]]
-   *
-   * @param cx pointer to JSContext
-   * @param proxy - The proxy object who's property we wish to check
-   * @param receiver @TODO (Caleb Aikens) read ECMAScript docs about this
-   * @param id - Key of the property we wish to get
-   * @param vp - out-paramter for the gotten property
-   * @return true - call succeeded
-   * @return false - call failed and an exception has been raised
-   */
-  bool get(JSContext *cx, JS::HandleObject proxy,
-    JS::HandleValue receiver, JS::HandleId id,
-    JS::MutableHandleValue vp) const override;
   /**
    * @brief [[Set]]
    *
@@ -116,7 +77,10 @@ public:
    *
    * @param cx - pointer to JSContext
    * @param proxy - The proxy object who's keys we output
-   * @param props - out-parameter of object IDs
+   * @param props - out-parameter of object IDsoverride;
+
+     // @TODO (Caleb Aikens) The following are Spidermonkey-unique extensions, need to read into them more
+     /**
    * @return true - call succeeded
    * @return false - call failed and an exception has been raised
    */
@@ -140,7 +104,10 @@ public:
    * @brief @TODO (Caleb Aikens) read up on what this trap does exactly
    *
    * @param cx - pointer to JSContext
-   * @param proxy - The proxy object who's keys we output
+   * @param proxy - The proxy object who's keys we outputoverride;
+
+     // @TODO (Caleb Aikens) The following are Spidermonkey-unique extensions, need to read into them more
+     /**
    * @param props - out-parameter of object IDs
    * @return true - call succeeded
    * @return false - call failed and an exception has been raised
@@ -169,47 +136,4 @@ public:
   bool getBuiltinClass(JSContext *cx, JS::HandleObject proxy, js::ESClass *cls) const override;
 };
 
-/**
- * @brief This struct is the ProxyHandler for JS Proxy Objects pythonmonkey creates
- *    to handle coercion from python lists to JS Array objects
- */
-struct PyListProxyHandler : public PyBaseProxyHandler {
-public:
-  PyListProxyHandler(PyObject *pyObj) : PyBaseProxyHandler(pyObj, &family) {};
-  static const char family;
-
-  /**
-   * @brief Handles python object reference count when JS Proxy object is finalized
-   *
-   * @param gcx pointer to JS::GCContext
-   * @param proxy the proxy object being finalized
-   */
-  void finalize(JS::GCContext *gcx, JSObject *proxy) const override;
-
-  bool getOwnPropertyDescriptor(
-    JSContext *cx, JS::HandleObject proxy, JS::HandleId id,
-    JS::MutableHandle<mozilla::Maybe<JS::PropertyDescriptor>> desc
-  ) const override;
-
-  bool defineProperty(
-    JSContext *cx, JS::HandleObject proxy, JS::HandleId id,
-    JS::Handle<JS::PropertyDescriptor> desc, JS::ObjectOpResult &result
-  ) const override;
-
-  bool ownPropertyKeys(JSContext *cx, JS::HandleObject proxy, JS::MutableHandleIdVector props) const override;
-  bool delete_(JSContext *cx, JS::HandleObject proxy, JS::HandleId id, JS::ObjectOpResult &result) const override;
-  bool isArray(JSContext *cx, JS::HandleObject proxy, JS::IsArrayAnswer *answer) const override;
-  bool getBuiltinClass(JSContext *cx, JS::HandleObject proxy, js::ESClass *cls) const override;
-};
-
-/**
- * @brief Convert jsid to a PyObject to be used as dict keys
- */
-PyObject *idToKey(JSContext *cx, JS::HandleId id);
-
-/**
- * @brief Convert Python dict key to jsid
- */
-bool keyToId(PyObject *key, JS::MutableHandleId idp);
-
-#endif
+#endif

include/PyProxyHandler.hh renamed to include/PyDictProxyHandler.hh

@@ -0,0 +1,51 @@
+/**
+ * @file PyListProxyHandler.hh
+ * @author Philippe Laporte ([email protected])
+ * @brief Structs for creating JS proxy objects. Used by ListType for List coercion
+ * @version 0.1
+ * @date 2023-12-01
+ *
+ * Copyright (c) 2023-2024 Distributive Corp.
+ *
+ */
+
+#ifndef PythonMonkey_PyListProxy_
+#define PythonMonkey_PyListProxy_
+
+#include "PyBaseProxyHandler.hh"
+
+
+/**
+ * @brief This struct is the ProxyHandler for JS Proxy Objects pythonmonkey creates
+ *    to handle coercion from python lists to JS Array objects
+ */
+struct PyListProxyHandler : public PyBaseProxyHandler {
+public:
+  PyListProxyHandler(PyObject *pyObj) : PyBaseProxyHandler(pyObj, &family) {};
+  static const char family;
+
+  /**
+   * @brief Handles python object reference count when JS Proxy object is finalized
+   *
+   * @param gcx pointer to JS::GCContext
+   * @param proxy the proxy object being finalized
+   */
+  void finalize(JS::GCContext *gcx, JSObject *proxy) const override;
+
+  bool getOwnPropertyDescriptor(
+    JSContext *cx, JS::HandleObject proxy, JS::HandleId id,
+    JS::MutableHandle<mozilla::Maybe<JS::PropertyDescriptor>> desc
+  ) const override;
+
+  bool defineProperty(
+    JSContext *cx, JS::HandleObject proxy, JS::HandleId id,
+    JS::Handle<JS::PropertyDescriptor> desc, JS::ObjectOpResult &result
+  ) const override;
+
+  bool ownPropertyKeys(JSContext *cx, JS::HandleObject proxy, JS::MutableHandleIdVector props) const override;
+  bool delete_(JSContext *cx, JS::HandleObject proxy, JS::HandleId id, JS::ObjectOpResult &result) const override;
+  bool isArray(JSContext *cx, JS::HandleObject proxy, JS::IsArrayAnswer *answer) const override;
+  bool getBuiltinClass(JSContext *cx, JS::HandleObject proxy, js::ESClass *cls) const override;
+};
+
+#endif

include/PyListProxyHandler.hh

@@ -39,9 +39,9 @@ $POETRY_BIN self add 'poetry-dynamic-versioning[plugin]'
 echo "Done installing dependencies"
 
 echo "Downloading spidermonkey source code"
-wget -c -q https://ftp.mozilla.org/pub/firefox/releases/115.1.0esr/source/firefox-115.1.0esr.source.tar.xz
+wget -c -q https://ftp.mozilla.org/pub/firefox/releases/115.7.0esr/source/firefox-115.7.0esr.source.tar.xz
 mkdir -p firefox-source
-tar xf firefox-115.1.0esr.source.tar.xz -C firefox-source --strip-components=1 # strip the root folder
+tar xf firefox-115.7.0esr.source.tar.xz -C firefox-source --strip-components=1 # strip the root folder
 echo "Done downloading spidermonkey source code"
 
 echo "Building spidermonkey"

setup.sh

@@ -17,7 +17,7 @@
 #include "include/modules/pythonmonkey/pythonmonkey.hh"
 #include "include/jsTypeFactory.hh"
 #include "include/pyTypeFactory.hh"
-#include "include/PyProxyHandler.hh"
+#include "include/PyBaseProxyHandler.hh"
 
 #include <jsapi.h>
 #include <jsfriendapi.h>

src/JSArrayProxy.cc

@@ -18,7 +18,7 @@
 #include "include/modules/pythonmonkey/pythonmonkey.hh"
 #include "include/jsTypeFactory.hh"
 #include "include/pyTypeFactory.hh"
-#include "include/PyProxyHandler.hh"
+#include "include/PyBaseProxyHandler.hh"
 
 #include <jsapi.h>
 #include <jsfriendapi.h>

src/JSObjectItemsProxy.cc

@@ -17,7 +17,7 @@
 
 #include "include/pyTypeFactory.hh"
 
-#include "include/PyProxyHandler.hh"
+#include "include/PyDictProxyHandler.hh"
 
 #include <jsapi.h>
 

src/JSObjectIterProxy.cc

@@ -18,7 +18,7 @@
 #include "include/modules/pythonmonkey/pythonmonkey.hh"
 #include "include/jsTypeFactory.hh"
 #include "include/pyTypeFactory.hh"
-#include "include/PyProxyHandler.hh"
+#include "include/PyDictProxyHandler.hh"
 
 #include <jsapi.h>
 #include <jsfriendapi.h>