Separating compiler and executer from Micropython interpreter

[synapticonashaikh]

Hello,

I'm working on a project where memory and other resources are a bit less. Also, the management requested to implement a feature where a user can write his script outside, compile it and send the bytecode over any communication media (wired or wirelessly) to the device. Hence, the device on the field will receive the byte code and starts executing it.

So, I'm planning to separate the micropython stack into two different parts, one that interprets and compiles the script to generate the byte code. And the other one executes it. I'm attaching a few images in the description to explain in detail.

To run the Python script, we have a simple function:

#if MICROPY_ENABLE_COMPILER
void do_str(const char *src, mp_parse_input_kind_t input_kind) 
{
    nlr_buf_t nlr;
    if ( nlr_push(&nlr) == 0) 
       {
         mp_lexer_t *lex  = mp_lexer_new_from_str_len(MP_QSTR__lt_stdin_gt_, src, strlen(src), 0);
         qstr source_name = lex->source_name;
         mp_parse_tree_t parse_tree = mp_parse(lex, input_kind);
         mp_obj_t module_fun = mp_compile(&parse_tree, source_name, true);  
         mp_call_function_0(module_fun);
         nlr_pop( );
       } 
    else 
       {
         // uncaught exception
         mp_obj_print_exception(&mp_plat_print, (mp_obj_t)nlr.ret_val);
       }
}
#endif

I tried separating this function into two blocks. So the first one would be the compiler.

void FnCompileTheCode(const char *src, mp_parse_input_kind_t input_kind)
{
  mp_lexer_t *lex  = mp_lexer_new_from_str_len(MP_QSTR__lt_stdin_gt_, src, strlen(src), 0);
  qstr source_name = lex->source_name;
  mp_parse_tree_t parse_tree = mp_parse(lex, input_kind);
  mp_obj_t module_fun = mp_compile(&parse_tree, source_name, true);    
}

But as in the output of the compiled script, module_fun is a void *, so I don't know the length and type of data it is pointing to. Therefore I don't know how to transfer this data over wired or wireless communication. And then, the other device receives this data and starts executing.

The receiver function could be like this:

void FnExecuteTheScript(mp_obj_t module_fun)
{
    nlr_buf_t nlr;
    if ( nlr_push(&nlr) == 0) 
       {
           mp_call_function_0(module_fun);
           nlr_pop( );
        }
}

So if anyone knows how to achieve this or could give me valuable suggestions, it would be highly appreciated.

[dlech]

one that interprets and compiles the script to generate the byte code.

This is what mpy-cross does.

And the other one executes it.

It is already possible to disable the compiler and leave only the VM to execute bytecodes by setting MICROPY_ENABLE_COMPILER to 0.

we have a simple function:

To run MPY bytecodes directly, this is what we are using in combination with #8381

        mp_reader_t reader;
        mp_vfs_map_minimal_t data;
        mp_vfs_map_minimal_new_reader(&reader, &data, mpy_buffer, mpy_buffer_size);
        mp_module_context_t *context = m_new_obj(mp_module_context_t);
        context->module.globals = mp_globals_get();
        mp_compiled_module_t compiled_module;
        compiled_module.context = context;
        mp_raw_code_load(&reader, &compiled_module);
        mp_obj_t module_fun = mp_make_function_from_raw_code(compiled_module.rc, context, NULL);

        // Run the script while letting CTRL-C interrupt it.
        mp_hal_set_interrupt_char(CHAR_CTRL_C);
        mp_call_function_0(module_fun);
        mp_hal_set_interrupt_char(-1);

Hopefully that is enough to get you pointed in the right direction.

[synapticonashaikh]

Thanks a lot! I can generate the mpy file, but I still have not found a suitable way to execute the bytecode file because I'm unable to find the libraries which contain all the definitions you mentioned above in your code snippet (Sorry, I'm still a noob).
For example, I have compiled the script (pycode.py) using mpy-cross.

from gpio import *
from delay import *
while True:
	PortWrite(PORT4C,0x0F)
	delaySec(1)
	PortWrite(PORT4C,0x00)
	delaySec(1)

Now, I have a mpy file, pycode.mpy.
So, I can copy and paste the data of the bytecode file into an array to execute it.

uint8_t ByteCode [] = 
{0x4d,0x06,0x00,0x1f,0x08,0x00,0x12,0x70,0x79,0x63,0x6f,0x64,0x65,0x2e,0x70,0x79,
0x00,0x0f,0x0b,0x08,0x67,0x70,0x69,0x6f,0x00,0x0a,0x64,0x65,0x6c,0x61,0x79,0x00,
0x12,0x50,0x6f,0x72,0x74,0x57,0x72,0x69,0x74,0x65,0x00,0x0c,0x50,0x4f,0x52,0x54,
0x34,0x43,0x00,0x10,0x64,0x65,0x6c,0x61,0x79,0x53,0x65,0x63,0x00,0x83,0x48,0x10,
0x0e,0x01,0x28,0x48,0x20,0x28,0x26,0x28,0x80,0x10,0x02,0x2a,0x01,0x1b,0x03,0x69,
0x80,0x10,0x02,0x2a,0x01,0x1b,0x04,0x69,0x11,0x05,0x11,0x06,0x8f,0x34,0x02,0x59,
0x11,0x07,0x81,0x34,0x01,0x59,0x11,0x05,0x11,0x06,0x80,0x34,0x02,0x59,0x11,0x07,
0x81,0x34,0x01,0x59,0x42,0x22,0x51,0x63};

I replaced the mpy_buffer with the following array. But I'm not able to find the definition of mp_vfs_map_minimal_t and mp_vfs_map_minimal_new_reader.
Error:

PythonRun.c:229:5: error: use of undeclared identifier 'mp_vfs_map_minimal_t'
    mp_vfs_map_minimal_t data;
    ^
PythonRun.c:230:5: warning: implicit declaration of function 'mp_vfs_map_minimal_new_reader' is invalid in C99 [-Wimplicit-function-declaration]
    mp_vfs_map_minimal_new_reader(&reader, &data, ByteCode, sizeof(ByteCode));
    ^
PythonRun.c:230:45: error: use of undeclared identifier 'data'
    mp_vfs_map_minimal_new_reader(&reader, &data, ByteCode, sizeof(ByteCode));
                                            ^
1 warning and 2 errors generated.
make: *** [Makefile:37: output.xe] Error 1

[dlech]

I'm not able to find the definition

These come from the pull request I linked and haven't been merged into the main branch yet. Specifically, we are using these two patches on top of MicroPython v1.20:

• pybricks@d00f8e0
• pybricks@ff9987f

Before that, we were doing something like:

            // run user .mpy file without erasing it
            mp_reader_t reader;
            mp_reader_new_mem(&reader, buf, len, 0);
            mp_module_context_t *context = m_new_obj(mp_module_context_t);
            context->module.globals = mp_globals_get();
            mp_compiled_module_t compiled_module = mp_raw_code_load(&reader, context);
            mp_obj_t module_fun = mp_make_function_from_raw_code(compiled_module.rc, context, MP_OBJ_NULL);
            mp_hal_set_interrupt_char(CHAR_CTRL_C); // allow ctrl-C to interrupt us
            mp_call_function_0(module_fun);
            mp_hal_set_interrupt_char(-1); // disable interrupt
            mp_handle_pending(true); // handle any pending exceptions (and any callbacks)

[synapticonashaikh]

As you mentioned I downloaded the codes:

1. https://github.com/pybricks/micropython/tree/d00f8e0b8a52905fdb7da04495ee68c56f2c74af 
2. https://github.com/pybricks/micropython/tree/ff9987f3638ed08a6df6f44f0c9aa8b9dbfe5c34

But I'm still not able to find these two definitions:

mp_vfs_map_minimal_t data;
mp_vfs_map_minimal_new_reader(&reader, &data, mpy_buffer, mpy_buffer_size);

Also, I tried using the old syntax you mentioned above, but the code is getting stuck at mp_reader_new_mem(&reader, buf, len, 0);.
I'm passing the byte code stored in an array generated by mpy-cross for my script.

from gpio import *
from delay import *
while True:
	PortWrite(PORT4C,0x0F)
	delaySec(1)
	PortWrite(PORT4C,0x00)
	delaySec(1)

uint8_t ByteCode [] = 
{0x4d,0x06,0x00,0x1f,0x08,0x00,0x12,0x70,0x79,0x63,0x6f,0x64,0x65,0x2e,0x70,0x79,
0x00,0x0f,0x0b,0x08,0x67,0x70,0x69,0x6f,0x00,0x0a,0x64,0x65,0x6c,0x61,0x79,0x00,
0x12,0x50,0x6f,0x72,0x74,0x57,0x72,0x69,0x74,0x65,0x00,0x0c,0x50,0x4f,0x52,0x54,
0x34,0x43,0x00,0x10,0x64,0x65,0x6c,0x61,0x79,0x53,0x65,0x63,0x00,0x83,0x48,0x10,
0x0e,0x01,0x28,0x48,0x20,0x28,0x26,0x28,0x80,0x10,0x02,0x2a,0x01,0x1b,0x03,0x69,
0x80,0x10,0x02,0x2a,0x01,0x1b,0x04,0x69,0x11,0x05,0x11,0x06,0x8f,0x34,0x02,0x59,
0x11,0x07,0x81,0x34,0x01,0x59,0x11,0x05,0x11,0x06,0x80,0x34,0x02,0x59,0x11,0x07,
0x81,0x34,0x01,0x59,0x42,0x22,0x51,0x63};

[ctimmer]

This is a really interesting concept. Wish I had the MP internal skills to help.

Another way to look at this is to implement the compiler as an application server. The MP device would call the compiler server with the script name (and MP version and device type) and is returned the compiled byte code for execution.

Hope you get this running,
Curt

[synapticonashaikh]

Hello @dlech, Thanks for your support!
I combined your old and new sample codes, and now I'm able to get it run on my device. AS I disabled the compiler, I could free up to 30KB of space. Also, as I'm working on a unique multi-core controller by XMOS, I'm wondering if I could contribute to this port so that in the future other people could benefit from it.

#if MICROPY_ENABLE_GC
    gc_init(heap, heap + sizeof(heap));
#endif
/*init the mp stack*/
    mp_init( );

mp_reader_t reader;
mp_reader_new_mem(&reader, ByteCode, sizeof(ByteCode), 0);
mp_module_context_t *context = m_new_obj(mp_module_context_t);
context->module.globals = mp_globals_get();
mp_compiled_module_t compiled_module;
compiled_module.context = context;
mp_raw_code_load(&reader, &compiled_module);
mp_obj_t module_fun = mp_make_function_from_raw_code(compiled_module.rc, context, NULL);
mp_call_function_0(module_fun);