Skip to content

Commit 8f609c8

Browse files
committed
Merge branch 'main' into Xmader/fix/JSObjectProxy-as-dict
2 parents 4d3acb2 + 59642c3 commit 8f609c8

36 files changed

+1202
-175
lines changed

.github/workflows/test-and-publish.yaml

Lines changed: 19 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ defaults:
2323
jobs:
2424
build-spidermonkey-unix:
2525
strategy:
26+
fail-fast: false
2627
matrix:
2728
# Use Ubuntu 20.04 / macOS 12 + Python 3.10 to build SpiderMonkey
2829
os: [ 'ubuntu-20.04', 'macos-12', 'm2ci' ]
@@ -39,8 +40,13 @@ jobs:
3940
with:
4041
path: |
4142
./_spidermonkey_install/*
42-
key: spidermonkey-${{ runner.os }}-${{ runner.arch }}
43+
key: spidermonkey102.13-${{ runner.os }}-${{ runner.arch }}
4344
lookup-only: true # skip download
45+
- name: Setup Poetry
46+
if: ${{ steps.cache-spidermonkey.outputs.cache-hit != 'true' }}
47+
uses: snok/install-poetry@v1
48+
with:
49+
version: 1.5.1
4450
- name: Build spidermonkey
4551
if: ${{ steps.cache-spidermonkey.outputs.cache-hit != 'true' }}
4652
run: ./setup.sh
@@ -54,8 +60,13 @@ jobs:
5460
with:
5561
path: |
5662
./_spidermonkey_install/*
57-
key: spidermonkey-${{ runner.os }}-${{ runner.arch }}
63+
key: spidermonkey102.13-${{ runner.os }}-${{ runner.arch }}
5864
lookup-only: true # skip download
65+
- name: Setup Poetry
66+
if: ${{ steps.cache-spidermonkey.outputs.cache-hit != 'true' }}
67+
uses: snok/install-poetry@v1
68+
with:
69+
version: 1.5.1
5970
- name: Install dependencies
6071
if: ${{ steps.cache-spidermonkey.outputs.cache-hit != 'true' }}
6172
shell: powershell
@@ -124,7 +135,7 @@ jobs:
124135
with:
125136
path: |
126137
./_spidermonkey_install/*
127-
key: spidermonkey-${{ runner.os }}-${{ runner.arch }}
138+
key: spidermonkey102.13-${{ runner.os }}-${{ runner.arch }}
128139
fail-on-cache-miss: true # SpiderMonkey is expected to be cached in its dedicated job
129140
- name: Build pminit
130141
run: |
@@ -147,6 +158,11 @@ jobs:
147158
run: |
148159
poetry run python -m pip install --force-reinstall ./dist/*
149160
poetry run python -m pytest tests/python
161+
- name: Run JS tests (peter-jr)
162+
if: ${{ runner.os != 'Windows' }} # Python on Windows doesn't have the readline library
163+
# FIXME: on macOS we must make sure to use the GNU version of wc and realpath
164+
run: |
165+
poetry run bash ./peter-jr ./tests/js/
150166
sdist:
151167
runs-on: ubuntu-20.04
152168
steps:

.gitmodules

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "tests/commonjs-official"]
2+
path = tests/commonjs-official
3+
url = https://github.com/commonjs/commonjs.git

.vscode/launch.json

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
{
2+
"version": "0.2.0",
3+
"configurations": [
4+
{
5+
"name": "(gdb) Debug",
6+
"type": "cppdbg",
7+
"request": "launch",
8+
"program": "python",
9+
"args": [
10+
"${file}"
11+
],
12+
"pipeTransport": {
13+
"pipeCwd": "${workspaceFolder}",
14+
"pipeProgram": "poetry",
15+
"quoteArgs": false,
16+
"pipeArgs": [
17+
"run"
18+
],
19+
},
20+
"preLaunchTask": "Build",
21+
"cwd": "${fileDirname}",
22+
"environment": [],
23+
"externalConsole": false,
24+
"MIMode": "gdb",
25+
"setupCommands": [
26+
{
27+
"description": "Enable pretty-printing for gdb",
28+
"text": "-enable-pretty-printing",
29+
"ignoreFailures": true
30+
},
31+
{
32+
"description": "Set Disassembly Flavor to Intel",
33+
"text": "-gdb-set disassembly-flavor intel",
34+
"ignoreFailures": true
35+
}
36+
]
37+
},
38+
]
39+
}

.vscode/tasks.json

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
{
2+
"tasks": [
3+
//
4+
// Setup
5+
//
6+
{
7+
"label": "Setup SpiderMonkey",
8+
"type": "process",
9+
"command": "./setup.sh",
10+
},
11+
//
12+
// Build
13+
//
14+
{
15+
"label": "Build",
16+
"type": "process",
17+
"command": "poetry",
18+
"args": [
19+
"install",
20+
],
21+
"problemMatcher": [
22+
"$gcc"
23+
],
24+
"group": {
25+
"kind": "build",
26+
"isDefault": true
27+
}
28+
},
29+
//
30+
// Test
31+
//
32+
{
33+
"label": "Install development dependencies",
34+
"type": "process",
35+
"command": "poetry",
36+
"args": [
37+
"install",
38+
"--no-root",
39+
"--only=dev"
40+
],
41+
},
42+
{
43+
"label": "Run pytest",
44+
"type": "process",
45+
"command": "poetry",
46+
"args": [
47+
"run",
48+
"pytest",
49+
"./tests/python"
50+
],
51+
},
52+
{
53+
"label": "Test",
54+
"dependsOrder": "sequence",
55+
"dependsOn": [
56+
"Install development dependencies",
57+
"Run pytest",
58+
],
59+
"group": {
60+
"kind": "test",
61+
"isDefault": true
62+
}
63+
},
64+
],
65+
"options": {
66+
"cwd": "${workspaceFolder}"
67+
},
68+
"problemMatcher": [],
69+
"version": "2.0.0"
70+
}

README.md

Lines changed: 146 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
PythonMonkey is a Mozilla [SpiderMonkey](https://firefox-source-docs.mozilla.org/js/index.html) JavaScript engine embedded into the Python VM,
77
using the Python engine to provide the JS host environment.
88

9-
This product is in an early stage, approximately 80% to MVP as of May 2023. It is under active development by Distributive Corp.,
9+
This product is in an early stage, approximately 80% to MVP as of July 2023. It is under active development by Distributive Corp.,
1010
https://distributive.network/. External contributions and feedback are welcome and encouraged.
1111

1212
The goal is to make writing code in either JS or Python a developer preference, with libraries commonly used in either language
@@ -26,7 +26,7 @@ this package to execute our complex `dcp-client` library, which is written in JS
2626
### Roadmap
2727
- [done] JS instrinsics coerce to Python intrinsics
2828
- [done] JS strings coerce to Python strings
29-
- JS objects coerce to Python dicts [own-properties only]
29+
- [done] JS objects coerce to Python dicts [own-properties only]
3030
- [done] JS functions coerce to Python function wrappers
3131
- [done] JS exceptions propagate to Python
3232
- [done] Implement `eval()` function in Python which accepts JS code and returns JS->Python coerced values
@@ -41,11 +41,15 @@ this package to execute our complex `dcp-client` library, which is written in JS
4141
- [done] Python host environment supplies event loop, including EventEmitter, setTimeout, etc.
4242
- Python host environment supplies XMLHttpRequest (other project?)
4343
- Python host environment supplies basic subsets of NodeJS's fs, path, process, etc, modules; as-needed by dcp-client (other project?)
44-
- Python TypedArrays coerce to JS TypeArrays
45-
- JS TypedArrays coerce to Python TypeArrays
44+
- [done] Python TypedArrays coerce to JS TypeArrays
45+
- [done] JS TypedArrays coerce to Python TypeArrays
4646

4747
## Build Instructions
48-
1. You will need the following installed (which can be done automatically by running ``./setup.sh``):
48+
49+
Read this if you want to build a local version.
50+
51+
1. You will need the following installed (which can be done automatically by running `./setup.sh`):
52+
- bash
4953
- cmake
5054
- doxygen
5155
- graphviz
@@ -57,13 +61,17 @@ this package to execute our complex `dcp-client` library, which is written in JS
5761
- [Poetry](https://python-poetry.org/docs/#installation)
5862
- [poetry-dynamic-versioning](https://github.com/mtkennerly/poetry-dynamic-versioning)
5963

60-
2. Run `poetry run pip install --verbose python/pminit ./`. This command automatically compiles the project and installs the project as well as dependencies into the poetry virtualenv.
64+
2. Run `poetry install`. This command automatically compiles the project and installs the project as well as dependencies into the poetry virtualenv.
65+
66+
If you are using VSCode, you can just press <kbd>Ctrl</kbd> + <kbd>Shift</kbd> + <kbd>B</kbd> to [run build task](https://code.visualstudio.com/docs/editor/tasks#_custom-tasks) - We have [the `tasks.json` file configured for you](.vscode/tasks.json).
6167

6268
## Running tests
6369
1. Compile the project
6470
2. Install development dependencies: `poetry install --no-root --only=dev`
6571
3. From the root directory, run `poetry run pytest ./tests/python`
6672

73+
For VSCode users, similar to the Build Task, we have a Test Task ready to use.
74+
6775
## Using the library
6876

6977
### Install from [PyPI](https://pypi.org/project/pythonmonkey/)
@@ -98,22 +106,147 @@ Type "help", "copyright", "credits" or "license" for more information.
98106

99107
Alternatively, you can build a `wheel` package by running `poetry build --format=wheel`, and install it by `pip install dist/*.whl`.
100108

109+
## Debugging Steps
110+
111+
1. [build the project locally](#build-instructions)
112+
2. To use gdb, run `poetry run gdb python`.
113+
See [Python Wiki: DebuggingWithGdb](https://wiki.python.org/moin/DebuggingWithGdb)
114+
115+
If you are using VSCode, it's more convenient to debug in [VSCode's built-in debugger](https://code.visualstudio.com/docs/editor/debugging). Simply press <kbd>F5</kbd> on an open Python to start debugging - We have [the `launch.json` file configured for you](.vscode/launch.json).
116+
101117
## Examples
102118

103119
* [examples/](examples/)
104120
* https://github.com/Distributive-Network/PythonMonkey-examples
105121
* https://github.com/Distributive-Network/PythonMonkey-Crypto-JS-Fullstack-Example
106122

123+
## API
124+
These methods are exported from the pythonmonkey module.
125+
126+
### require(moduleIdentifier)
127+
Return the exports of a CommonJS module identified by `moduleIdentifier`, using standard CommonJS
128+
semantics
129+
- modules are singletons and will never be loaded or evaluated more than once
130+
- moduleIdentifier is relative to the Python file invoking `require`
131+
- moduleIdentifier should not include a file extension
132+
- moduleIdentifiers which do not begin with ./, ../, or / are resolved by search require.path
133+
and module.paths.
134+
- Modules are evaluated immediately after loading
135+
- Modules are not loaded until they are required
136+
- The following extensions are supported:
137+
** `.js` - JavaScript module; source code decorates `exports` object
138+
** `.py` - Python module; source code decorates `exports` dict
139+
** `.json` -- JSON module; exports are the result of parsing the JSON text in the file
140+
141+
### globalThis
142+
A Python Dict which is equivalent to the globalThis object in JavaScript.
143+
144+
### createRequire(filename, extraPaths, isMain)
145+
Factory function which returns a new require function
146+
- filename: the pathname of the module that this require function could be used for
147+
- extraPaths: [optional] a list of extra paths to search to resolve non-relative and non-absolute module identifiers
148+
- isMain: [optional] True if the require function is being created for a main module
149+
150+
### runProgramModule(filename, argv, extraPaths)
151+
Load and evaluate a program (main) module. Program modules must be written in JavaScript. Program modules are not
152+
necessary unless the main entry point of your program is written in JavaScript.
153+
- filename: the location of the JavaScript source code
154+
- argv: the program's argument vector
155+
- extraPaths: [optional] a list of extra paths to search to resolve non-relative and non-absolute module identifiers
156+
157+
Care should be taken to ensure that only one program module is run per JS context.
158+
159+
## Built-In Functions
160+
- `console`
161+
- `setTimeout`
162+
- `setInterval`
163+
- `clearTimeout`
164+
- `clearInterval`
165+
166+
### CommonJS Subsystem Additions
167+
The CommonJS subsystem is activated by invoking the `require` or `createRequire` exports of the (Python)
168+
pythonmonkey module.
169+
- `require`
170+
- `exports`
171+
- `module`
172+
- `python.print` - the Python print function
173+
- `python.getenv` - the Python getenv function
174+
- `python.stdout` - an object with `read` and `write` methods, which read and write to stdout
175+
- `python.stderr` - an object with `read` and `write` methods, which read and write to stderr
176+
- `python.exec` - the Python exec function
177+
- `python.eval` - the Python eval function
178+
- `python.exit` - the Python exit function (wrapped to return BigInt in place of number)
179+
- `python.paths` - the Python sys.paths list (currently a copy; will become an Array-like reflection)
180+
181+
## Type Transfer (Coercion / Wrapping)
182+
When sending variables from Python into JavaScript, PythonMonkey will intelligently coerce or wrap your
183+
variables based on their type. PythonMonkey will share backing stores (use the same memory) for ctypes,
184+
typed arrays, and strings; moving these types across the language barrier is extremely fast because
185+
there is no copying involved.
186+
187+
*Note:* There are plans in Python 3.12 (PEP 623) to change the internal string representation so that
188+
every character in the string uses four bytes of memory. This will break fast string transfers
189+
for PythonMonkey, as it relies on the memory layout being the same in Python and JavaScript. As
190+
of this writing (July 2023), "classic" Python strings still work in the 3.12 beta releases.
191+
192+
Where shared backing store is not possible, PythonMonkey will automatically emit wrappers that use
193+
the "real" data structure as its value authority. Only immutable intrinsics are copied. This means
194+
that if you update an object in JavaScript, the corresponding Dict in Python will be updated, etc.
195+
196+
| Python Type | JavaScript Type |
197+
|:------------|:----------------|
198+
| String | string
199+
| Integer | number
200+
| Bool | boolean
201+
| Function | function
202+
| Dict | object
203+
| List | Array-like object
204+
| datetime | Date object
205+
| awaitable | Promise
206+
| Error | Error object
207+
| Buffer | ArrayBuffer
208+
209+
| JavaScript Type | Python Type |
210+
|:---------------------|:----------------|
211+
| string | String
212+
| number | Float
213+
| bigint | Integer
214+
| boolean | Bool
215+
| function | Function
216+
| object - most | JSObjectProxy which inherits from Dict
217+
| object - Date | datetime
218+
| object - Array | List
219+
| object - Promise | awaitable
220+
| object - ArrayBuffer | Buffer
221+
| object - type arrays | Buffer
222+
| object - Error | Error
223+
224+
## Tricks
225+
### Integer Type Coercion
226+
You can force a number in JavaScript to be coerced as an integer by casting it to BigInt.
227+
```javascript
228+
function myFunction(a, b) {
229+
const result = calculate(a, b);
230+
return BigInt(Math.floor(result));
231+
}
232+
```
233+
234+
### Symbol injection via cross-language IIFE
235+
You can use a JavaScript IIFE to create a scope in which you can inject Python symbols:
236+
```python
237+
globalThis.python.exit = pm.eval("""'use strict';
238+
(exit) => function pythonExitWrapper(exitCode) {
239+
if (typeof exitCode === 'number')
240+
exitCode = BigInt(Math.floor(exitCode));
241+
exit(exitCode);
242+
}
243+
""")(sys.exit);
244+
```
245+
107246
# Troubleshooting Tips
108247

109248
## REPL - pmjs
110-
A basic JavaScript shell, `pmjs`, ships with PythonMonkey.
249+
A basic JavaScript shell, `pmjs`, ships with PythonMonkey. This shell can also run JavaScript programs with
111250

112251
## CommonJS (require)
113252
If you are having trouble with the CommonJS require function, set environment variable DEBUG='ctx-module*' and you can see the filenames it tries to laod.
114-
115-
### Extra Symbols
116-
Loading the CommonJS subsystem declares some extra symbols which may be helpful in debugging -
117-
- `python.print` - the Python print function
118-
- `python.getenv` - the Python getenv function
119-

TODO.md

Lines changed: 0 additions & 2 deletions
This file was deleted.

0 commit comments

Comments
 (0)