Using micropython as a scripting API

[AlexApps99]

I want to write a program that can run arbitrary user-written code using APIs exposed from the program (things that micropython code can call to interact with it).
There aren't any examples structured like that, especially since I would want the APIs to be defined at runtime rather than as compile-time modules (if possible).

For these security reasons, I'd want to run the code directly as bytecode, and only run a set number of instructions before stopping each iteration. Native code, hardware interaction libraries and other things would have to be removed.

Is this possible without significant modification to the library (as a port)?
I don't know micropython's APIs that well so I'm not sure what options there are.

(this is for a game engine)

[jimmo]

@AlexApps99 Yes this is definitely possible. Probably the only thing that isn't supported is running a fixed number of instructions, although this is something you could add.

Are you running this on a PC or on an embedded microcontroller? What language is your game engine written in?

[AlexApps99]

This is C++ on Linux (but it should be simple to implement in a cross platform way)

The main motivation I have for running a fixed number of instructions at a time, is to prevent infinite loops crashing the game, and to allow the code to run on the main thread.
This would let the program run in a deterministic way, since scripts can run in a specified order and stop at specified intervals (useful for multiplayer, for example)

[jimmo]

Basically what you want is what is being implemented in #9529 which is a way to embed MicroPython into an existing piece of C/C++ software.

We plan to add more examples and hopefully merge it soon, but there should be enough to play around with.

Regarding the fixed number of instructions, I realised you can actually do something very close to this using the VM loop hook: https://github.com/micropython/micropython/blob/master/py/mpconfig.h#L588 -- basically you can have the VM run your custom function run after jump instructions and you can implement your own counter and raise an exception as necessary. Obviously not quite the same as every instruction, but should solve the same problem?

[jimmo]

Rather than embedding MicroPython into the game, could you write the game as a MicroPython program running in the Linux (Unix) port (with appropriate libraries to access graphics/audio/input/etc written as modules), but then the thing you need is essentially a "sandboxed" eval() that can run user code?

[AlexApps99]

I plan to write the majority of the game in C++, the scripting is probably intended more of a tool for high-level gameplay stuff and modding. I could still potentially structure it like that, but it might be more awkward than embedding it as a library of some kind.

[jimmo]

At some point in the future you might want to invert this and turn it into your own C++ program that embeds micropython, but as a starting point to develop the relevant bindings to graphics/audio/input this might be a lot easier. And then you can worry about locking-down the eval function later.

[stinos]

Can you define modules at run time, rather than in the build system?

Shameless plug, but that's basically exactly what https://github.com/stinos/micropython-wrap does. Plus it takes care of conversion between a bunch of standard C++ types and MicroPython counterparts. Maybe not all needed or the way you want it, but worth a look.

[AlexApps99]

That looks like it could be useful, thanks

[cwalther]

As a side note, the “run a fixed/bounded number of instructions” thing could also be useful in the WebAssembly port, where I’m currently struggling with the lack of that. JavaScript has a tendency to insist that user code run in short chunks and return to the event loop in between, while code written for microcontrollers (in MicroPython or not) tends to be infinitely-long-running. I thought I could get around that by running MicroPython in a web worker thread, but no, web workers don't deal with infinite loops well either because the message queues needed to communicate to them also work by delivering events and cannot be polled from deep in a user-code callstack, as the VM loop hook would allow. (Emscripten has an “asyncify” mechanism to have the C code suspended deep in a call stack and still return to the event loop on the JavaScript side, but I haven’t gotten it to work reliably with the VM loop hook yet.)

[AlexApps99]

I might consider opening an issue for it in a few days (unless you want to)

[cwalther]

Go ahead – it might be hard to implement though, from my cursory looks it seems like the VM is pretty fundamentally built around the model that the C call stack corresponds to the Python call stack.

For now I’m hoping that asyncify is the way to go for WebAssembly (especially since it’s used already), and am working on fixing/extending the WebAssembly port to the point that I can cleanly demonstrate that asyncify doesn’t work properly, in hopes that someone else can help figuring out why it doesn’t work properly.