Skip to content

Commit 59642c3

Browse files
authored
Merge pull request #73 from Distributive-Network/wes/improved-pmjs
PMJS Improvements
2 parents aa27f65 + 9d00324 commit 59642c3

26 files changed

+1018
-155
lines changed

.github/workflows/test-and-publish.yaml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,11 @@ jobs:
158158
run: |
159159
poetry run python -m pip install --force-reinstall ./dist/*
160160
poetry run python -m pytest tests/python
161+
- name: Run JS tests (peter-jr)
162+
if: ${{ runner.os != 'Windows' }} # Python on Windows doesn't have the readline library
163+
# FIXME: on macOS we must make sure to use the GNU version of wc and realpath
164+
run: |
165+
poetry run bash ./peter-jr ./tests/js/
161166
sdist:
162167
runs-on: ubuntu-20.04
163168
steps:

.gitmodules

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
[submodule "tests/commonjs-official"]
2+
path = tests/commonjs-official
3+
url = https://github.com/commonjs/commonjs.git

README.md

Lines changed: 129 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,7 @@
66
PythonMonkey is a Mozilla [SpiderMonkey](https://firefox-source-docs.mozilla.org/js/index.html) JavaScript engine embedded into the Python VM,
77
using the Python engine to provide the JS host environment.
88

9-
This product is in an early stage, approximately 80% to MVP as of May 2023. It is under active development by Distributive Corp.,
9+
This product is in an early stage, approximately 80% to MVP as of July 2023. It is under active development by Distributive Corp.,
1010
https://distributive.network/. External contributions and feedback are welcome and encouraged.
1111

1212
The goal is to make writing code in either JS or Python a developer preference, with libraries commonly used in either language
@@ -26,7 +26,7 @@ this package to execute our complex `dcp-client` library, which is written in JS
2626
### Roadmap
2727
- [done] JS instrinsics coerce to Python intrinsics
2828
- [done] JS strings coerce to Python strings
29-
- JS objects coerce to Python dicts [own-properties only]
29+
- [done] JS objects coerce to Python dicts [own-properties only]
3030
- [done] JS functions coerce to Python function wrappers
3131
- [done] JS exceptions propagate to Python
3232
- [done] Implement `eval()` function in Python which accepts JS code and returns JS->Python coerced values
@@ -41,14 +41,15 @@ this package to execute our complex `dcp-client` library, which is written in JS
4141
- [done] Python host environment supplies event loop, including EventEmitter, setTimeout, etc.
4242
- Python host environment supplies XMLHttpRequest (other project?)
4343
- Python host environment supplies basic subsets of NodeJS's fs, path, process, etc, modules; as-needed by dcp-client (other project?)
44-
- Python TypedArrays coerce to JS TypeArrays
45-
- JS TypedArrays coerce to Python TypeArrays
44+
- [done] Python TypedArrays coerce to JS TypeArrays
45+
- [done] JS TypedArrays coerce to Python TypeArrays
4646

4747
## Build Instructions
4848

4949
Read this if you want to build a local version.
5050

5151
1. You will need the following installed (which can be done automatically by running `./setup.sh`):
52+
- bash
5253
- cmake
5354
- doxygen
5455
- graphviz
@@ -119,16 +120,133 @@ If you are using VSCode, it's more convenient to debug in [VSCode's built-in deb
119120
* https://github.com/Distributive-Network/PythonMonkey-examples
120121
* https://github.com/Distributive-Network/PythonMonkey-Crypto-JS-Fullstack-Example
121122

123+
## API
124+
These methods are exported from the pythonmonkey module.
125+
126+
### require(moduleIdentifier)
127+
Return the exports of a CommonJS module identified by `moduleIdentifier`, using standard CommonJS
128+
semantics
129+
- modules are singletons and will never be loaded or evaluated more than once
130+
- moduleIdentifier is relative to the Python file invoking `require`
131+
- moduleIdentifier should not include a file extension
132+
- moduleIdentifiers which do not begin with ./, ../, or / are resolved by search require.path
133+
and module.paths.
134+
- Modules are evaluated immediately after loading
135+
- Modules are not loaded until they are required
136+
- The following extensions are supported:
137+
** `.js` - JavaScript module; source code decorates `exports` object
138+
** `.py` - Python module; source code decorates `exports` dict
139+
** `.json` -- JSON module; exports are the result of parsing the JSON text in the file
140+
141+
### globalThis
142+
A Python Dict which is equivalent to the globalThis object in JavaScript.
143+
144+
### createRequire(filename, extraPaths, isMain)
145+
Factory function which returns a new require function
146+
- filename: the pathname of the module that this require function could be used for
147+
- extraPaths: [optional] a list of extra paths to search to resolve non-relative and non-absolute module identifiers
148+
- isMain: [optional] True if the require function is being created for a main module
149+
150+
### runProgramModule(filename, argv, extraPaths)
151+
Load and evaluate a program (main) module. Program modules must be written in JavaScript. Program modules are not
152+
necessary unless the main entry point of your program is written in JavaScript.
153+
- filename: the location of the JavaScript source code
154+
- argv: the program's argument vector
155+
- extraPaths: [optional] a list of extra paths to search to resolve non-relative and non-absolute module identifiers
156+
157+
Care should be taken to ensure that only one program module is run per JS context.
158+
159+
## Built-In Functions
160+
- `console`
161+
- `setTimeout`
162+
- `setInterval`
163+
- `clearTimeout`
164+
- `clearInterval`
165+
166+
### CommonJS Subsystem Additions
167+
The CommonJS subsystem is activated by invoking the `require` or `createRequire` exports of the (Python)
168+
pythonmonkey module.
169+
- `require`
170+
- `exports`
171+
- `module`
172+
- `python.print` - the Python print function
173+
- `python.getenv` - the Python getenv function
174+
- `python.stdout` - an object with `read` and `write` methods, which read and write to stdout
175+
- `python.stderr` - an object with `read` and `write` methods, which read and write to stderr
176+
- `python.exec` - the Python exec function
177+
- `python.eval` - the Python eval function
178+
- `python.exit` - the Python exit function (wrapped to return BigInt in place of number)
179+
- `python.paths` - the Python sys.paths list (currently a copy; will become an Array-like reflection)
180+
181+
## Type Transfer (Coercion / Wrapping)
182+
When sending variables from Python into JavaScript, PythonMonkey will intelligently coerce or wrap your
183+
variables based on their type. PythonMonkey will share backing stores (use the same memory) for ctypes,
184+
typed arrays, and strings; moving these types across the language barrier is extremely fast because
185+
there is no copying involved.
186+
187+
*Note:* There are plans in Python 3.12 (PEP 623) to change the internal string representation so that
188+
every character in the string uses four bytes of memory. This will break fast string transfers
189+
for PythonMonkey, as it relies on the memory layout being the same in Python and JavaScript. As
190+
of this writing (July 2023), "classic" Python strings still work in the 3.12 beta releases.
191+
192+
Where shared backing store is not possible, PythonMonkey will automatically emit wrappers that use
193+
the "real" data structure as its value authority. Only immutable intrinsics are copied. This means
194+
that if you update an object in JavaScript, the corresponding Dict in Python will be updated, etc.
195+
196+
| Python Type | JavaScript Type |
197+
|:------------|:----------------|
198+
| String | string
199+
| Integer | number
200+
| Bool | boolean
201+
| Function | function
202+
| Dict | object
203+
| List | Array-like object
204+
| datetime | Date object
205+
| awaitable | Promise
206+
| Error | Error object
207+
| Buffer | ArrayBuffer
208+
209+
| JavaScript Type | Python Type |
210+
|:---------------------|:----------------|
211+
| string | String
212+
| number | Float
213+
| bigint | Integer
214+
| boolean | Bool
215+
| function | Function
216+
| object - most | JSObjectProxy which inherits from Dict
217+
| object - Date | datetime
218+
| object - Array | List
219+
| object - Promise | awaitable
220+
| object - ArrayBuffer | Buffer
221+
| object - type arrays | Buffer
222+
| object - Error | Error
223+
224+
## Tricks
225+
### Integer Type Coercion
226+
You can force a number in JavaScript to be coerced as an integer by casting it to BigInt.
227+
```javascript
228+
function myFunction(a, b) {
229+
const result = calculate(a, b);
230+
return BigInt(Math.floor(result));
231+
}
232+
```
233+
234+
### Symbol injection via cross-language IIFE
235+
You can use a JavaScript IIFE to create a scope in which you can inject Python symbols:
236+
```python
237+
globalThis.python.exit = pm.eval("""'use strict';
238+
(exit) => function pythonExitWrapper(exitCode) {
239+
if (typeof exitCode === 'number')
240+
exitCode = BigInt(Math.floor(exitCode));
241+
exit(exitCode);
242+
}
243+
""")(sys.exit);
244+
```
245+
122246
# Troubleshooting Tips
123247

124248
## REPL - pmjs
125-
A basic JavaScript shell, `pmjs`, ships with PythonMonkey.
249+
A basic JavaScript shell, `pmjs`, ships with PythonMonkey. This shell can also run JavaScript programs with
126250

127251
## CommonJS (require)
128252
If you are having trouble with the CommonJS require function, set environment variable DEBUG='ctx-module*' and you can see the filenames it tries to laod.
129-
130-
### Extra Symbols
131-
Loading the CommonJS subsystem declares some extra symbols which may be helpful in debugging -
132-
- `python.print` - the Python print function
133-
- `python.getenv` - the Python getenv function
134-

build.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,8 @@
1+
# @file build.py
2+
# Main PythonMonkey build automation script. Run with `poetry build`.
3+
# @author Hamada Gasmallah, [email protected]
4+
# @date April 2023
5+
#
16
import subprocess
27
import os, sys
38
import platform
@@ -48,6 +53,7 @@ def copy_artifacts():
4853
execute("cp ./_spidermonkey_install/lib/libmozjs* ./python/pythonmonkey/")
4954

5055
def build():
56+
execute("git submodule update --init --recursive")
5157
ensure_spidermonkey()
5258
run_cmake_build()
5359
copy_artifacts()

js-test-runner

Lines changed: 0 additions & 15 deletions
This file was deleted.

peter-jr

Lines changed: 65 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,20 @@ set -u
2222
set -o pipefail
2323
peter_exit_code=2
2424

25+
if [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ]; then
26+
cat <<EOF
27+
Peter-jr - A simple test framework in the spirit of Peter for testing basic
28+
PythonMonkey functionality. (Peter: npmjs.com/packages/peter)
29+
Copyright (c) 2023 Distributive. Released under the terms of the MIT License.
30+
31+
Usage: $0 [-h] [-v] [test [test...]]
32+
Where: -h or --help shows this help
33+
-v enables verbose mode (shows tests' stdout)
34+
test is the name of a test or a directory containing tests. If not
35+
specified, uses ${topDir}/tests/js.
36+
EOF
37+
exit 0
38+
fi
2539
if [ "${1:-}" = "-v" ]; then
2640
VERBOSE=1
2741
shift
@@ -32,6 +46,8 @@ fi
3246
[ "${1:-}" ] || set "${topDir}/tests/js"
3347
testLocations=("$@")
3448

49+
export PMJS="${topDir}/pmjs"
50+
3551
function panic()
3652
{
3753
echo "PANIC: $*" >&2
@@ -61,6 +77,16 @@ green()
6177
printf "\e[0;32m%s\e[0m" "$*"
6278
}
6379

80+
dred()
81+
{
82+
printf "\e[22m\e[0;31m%s\e[0m" "$*"
83+
}
84+
85+
bggreen()
86+
{
87+
printf "\e[42m%s\e[0m" "$*"
88+
}
89+
6490
yellow()
6591
{
6692
printf "\e[0;33m%s\e[0m" "$*"
@@ -71,22 +97,48 @@ grey()
7197
printf "\e[0;90m%s\e[0m" "$*"
7298
}
7399

74-
(
100+
findTests()
101+
{
75102
for loc in "${testLocations[@]}"
76103
do
77104
find $(realpath "$loc") -type f -name \*.simple
78105
find $(realpath "$loc") -type f -name \*.bash
79-
done
80-
) \
106+
find $(realpath "$loc") -type f -name \*.failing
107+
done\
108+
| while read file
109+
do
110+
realpath --relative-to="${runDir}" "${file}"
111+
done
112+
}
113+
114+
longestFilenameLen=`findTests | wc -L`
115+
116+
findTests \
117+
| (shuf 2>/dev/null || cat) \
81118
| while read file
82119
do
83-
sfile=$(realpath --relative-to="${runDir}" "${file}")
84-
printf 'Testing %-40s ... ' "${sfile}"
120+
printf "Testing %-${longestFilenameLen}s ... " "${file}"
85121
ext="${file##*.}"
122+
testName="${file%.failing}"
123+
testType="${testName##*.}"
124+
125+
thisStdout="${stdout}"
126+
thisStderr="${stderr}"
127+
128+
if [ "${ext}" = "failing" ]; then
129+
knownFail="yes"
130+
PASS="$(bggreen PASS) (unexpected)"
131+
FAIL="$(dred F̶A̶I̶L̶) (expected)"
132+
[ ! "${VERBOSE}" ] && thisStderr="/dev/null"
133+
else
134+
knownFail=""
135+
PASS="$(green PASS)"
136+
FAIL="$(red FAIL)"
137+
fi
86138
(
87-
case "$ext" in
139+
case "$testType" in
88140
"simple")
89-
"${topDir}/js-test-runner" "$file"
141+
"${PMJS}" "$file"
90142
exitCode="$?"
91143
;;
92144
"bash")
@@ -95,24 +147,24 @@ grey()
95147
;;
96148
*)
97149
echo
98-
panic "${file}: invalid extension '$ext'"
150+
panic "${file}: invalid test type '$testType'"
99151
;;
100152
esac
101-
102153
exit "$exitCode"
103-
)> $stdout 2>$stderr
154+
)> ${thisStdout} 2>${thisStderr}
104155
exitCode="$?"
105156

106157
if [ "$exitCode" = "0" ]; then
107-
echo "$(green PASS)"
158+
echo "${PASS}"
108159
[ "${peter_exit_code}" = "2" ] && peter_exit_code=0
109160
printf "\e[0;31m"
110161
[ "$VERBOSE" ] || cat "$stderr"
111162
printf "\e[0m"
112163
else
113-
echo "$(red FAIL)"
164+
echo "${FAIL}"
114165
peter_exit_code=1
115-
if [ ! "$VERBOSE" ]; then
166+
167+
if [ ! "$VERBOSE" ] && [ ! "$knownFail" ]; then
116168
echo
117169
echo "$(grey --) $(yellow ${file}) $(grey vvvvvvvvvvvvvv)"
118170
cat "$stdout" | sed 's/^/ /'

0 commit comments

Comments
 (0)