Code Execution Research

[JohnnyJayJay]

We need to figure out the best way to run user-submitted code efficiently and safely.

We've been eyeing piston so far but it remains to be seen how suitable this is for our volume of code execution. We should perform tests and rough benchmarks on locally hosted instances to get a better picture. Some issues (engineer-man/piston#616) are already visible even without any testing. We should investigate if it is feasible to tailor piston more to our needs by forking and modifying it, if necessary.

We should also look for alternative ways to execute code. Safety is the most crucial aspect – we need very good sandboxing. Secondary objectives are speed and a wide array of supported languages.

[NyCodeGHG]

Piston seems to have a very weird architecture, outdated versions and depends way to much on docker for my taste.
We should maybe actually consider writing our own service for that. Sandboxing could be done via the regular linux sandboxing apis or via WebAssembly (altough this is probably not supported by every language we want to support).

We could also try forking, but I'm unsure if that is more effort than writing our own.

[JohnnyJayJay]

I currently cannot really assess how sensible or easy it would be to write our own code execution service. In the spirit of this issue, one could try to make a prototype of some sort.

[SirYwell]

As another alternative, GraalVM offers Polyglot Sandboxing that allows for different sandboxing policies.

[NyCodeGHG]

after playing around with some stuff, i've came to the conclusion that not writing our own service is probably the better way

[JohnnyJayJay]

As another alternative, GraalVM offers Polyglot Sandboxing that allows for different sandboxing policies.

@SirYwell Does this only support JavaScript, or other languages too?

[SirYwell]

It seems like languages have to define which sandbox policies they allow (https://www.graalvm.org/truffle/javadoc/com/oracle/truffle/api/TruffleLanguage.Registration.html#sandbox--), and from a quick investigation it looks like only the JavaScript implementation currently sets it to UNTRUSTED. It also seems like this feature is new, so maybe it's rather something to keep in mind for the future.

[JohnnyJayJay]

A piston alternative worth investigating:
https://github.com/judge0

[JohnnyJayJay]

Here's the best way to proceed, in my eyes:

• do some very basic tests/benchmark (set up candidates, throw some code at them)
• for the backend MVP, we simply pick one of judge0 or piston and abstract this away behind a generic code execution interface
• If we see subpar performance, we can invest more effort into optimisation. Currently, I see a couple different angles from which we could approach:
  • Improve judge0 or piston directly (probably piston, see issue mentioned in original comment) by implementing the type of batch execution that we would like to have (1 source file, multiple inputs) with things like reusing compilation output etc. This might need bigger restructuring as compilation and execution are usually intertwined in these projects.
  • Rewrite one of the two (currently, they're written in JavaScript and Ruby respectively) to squeeze out more performance. Especially with piston I see this being an option, given that it's not too big.

As for the general comparison between judge0 and piston, benchmarks will still have to be run. From what I read, piston seems to be faster (but doesn't support any kind of batching).

[NyCodeGHG]

I found an alternative to piston which looks promising: https://github.com/Defelo/sandkasten
It explicitely mentions being an alternative to piston, because of its problems

[JohnnyJayJay]

I found an alternative to piston which looks promising: https://github.com/Defelo/sandkasten It explicitely mentions being an alternative to piston, because of its problems

This looks promising.

[JohnnyJayJay]

Just throwing this in here for good measure. Have not taken a closer look, has very strange licensing and probably isn't completely finished. Seems like they're using it for a (somewhat) similar project.
https://github.com/HelixStudio/exec