Compilation of .py to .mpy files on the device?

[Vellug]

Hello everyone,

I was wondering if it is possible to compile a .py to .mpy file on my device?
The reason for my question is, that the .mpy files compiled with mpy-cross have to be the same version as the mpy-version on the device. So if I could compile the files directly on the device there is no need to care about versions and get a faster execution time.

Since MicroPython compiles .py files to bytecode before execution I think it should be possible, but I haven't found something about that.

Best regards

[IHOXOHI]

Hi,
My opinion: '.mpy' is a type of file created to preserve a production, so it's aim is to not be uncompiled... maybe if you know how compile...

[Josverl]

The aim of .mpy is to

• skip the compilation step, and thus
  • reduce time to load,
  • and reduce memory fragmentation that occurs as a side effect of that compilation
• .mpy files can be added to the firmware, effectively moving the program from ram to flash memory, and allowing that RAM to be used for other things.

.mpy can be 'decompiled' with ease , so it's no real security on your code

[IHOXOHI]

thanks

[rkompass]

I'm not an expert here, but this is a very interesting question.
In the Digi XBee MP port there is an uos.compile() function that does that.
I did not find a public repo of their port. Perhaps someone could ask for the code.

In principle the code (including the bytecode) is there: If you do e.g. a
cfunc = compile("def r1():\n return 1", "_", 'exec') in ordinary MP (also not the compile from uos module but from builtins) then you get a object in cfunc. You may execute it by passing it to exec().
Would be interesting to know how to extract the bytecode from that. An/or how to manipulate it.
With the right knowledge one should be able to write a function that translates this into an .mpy file
(study the code of compile, and of mpy-cross).

Also there is the sys.settrace function which you may enable by compiling with the mpconfig.h settings

// To activate sys.settrace(tracefunc)
#define MICROPY_PY_SYS_SETTRACE       (1) // To activate sys.settrace(tracefunc)
#define MICROPY_COMP_CONST            (0) // required for MICROPY_PY_SYS_SETTRACE
#define MICROPY_PERSISTENT_CODE_SAVE  (1) // required for MICROPY_PY_SYS_SETTRACE

and in this older discussion Jimmo told how to use that to get the bytecode of a function. Although it seems overkill for just the purpose of getting bytecode from within a running MP .?.

[Vellug]

Thanke you for your response.

Yeah it seems like an overkill, but here's a bit more of context what I'm working on.
I'm currently working on my bachelorthesis, with the topic how to get some software functions to an embedded controller system after compilation. One of the requirements is that the costumers should be able to load that code from something like an app store in the future. So not to bother them with a lot of software versions, the py file would be prefered.
But now I measured the execution time of a file and realised that the most time (of course depending of the size of the file) is used to get the file from the filesystem and to compile it.
So my idea to speed that a bit up was to compile them to .mpy on the device. My thoughts was since there is already a lexer and parser on board and the translation to bytecode already happens before executing a py file, it shoudn't be that hard to do that.

But I only got 2 weeks left, so there is not enough time to check everything you mentioned and I'm fine with the answer that it isn't possbile (maybe yet).

[rkompass]

One of the requirements is that the costumers should be able to load that code from something like an app store in the future. So not to bother them with a lot of software versions, the py file would be prefered.

In that case I would suggest to set up a system where the MP running machine sends it's requirements to the server, i.e. MP version, and other config details that are relevant for the MPY version. After receiving that the server may use mpy-cross to generate a matching .mpy - file that the MP running machine may then get and put onto it's file system.
Or: There is also the possibility to receive the .mpy - file and load it into ram and modify it so it can be executed immediately. This is discussed (incl. a solution) here.

[Vellug]

In my thesis I just worked on "setting up" the architecture on the controller to be able to load code to the device. So there is no server software yet, I just mentioned that to explain how application will look like in the future.
But I think this is a good alternative to solve that "problem", so thank you.

[daleka]

Hi!

I'm working on a MPyTools extension for VS Code that automates .py to .mpy compilation and file upload to the device. It's still a bit raw, but overall, it works.

The extension automatically detects modified files, compiles them, and immediately uploads them to the device. You can also choose the optimization method during compilation. Everything runs through mpremote, making the process as simple as possible.

🔗 Marketplace: MPyTools

[daleka]

Hi!

At the moment, my extension only provides optimization level selection and compiles files for the latest versions of MicroPython.

However, I'm currently working on expanding mpy-cross support. My goal is to provide full compatibility by automatically retrieving:

The MicroPython version from the device,
The bytecode version (-b option) using sys.implementation._mpy.
For other options, I plan to implement manual configuration, including:

-msmall-int-bits
emit={bytecode, native, viper}
-march=
I could automate -march detection as well, but I would need a detailed table mapping CPU cores to their corresponding -march values. As far as I know, no such table exists, so this option will likely require manual selection.

Let me know if you have any thoughts or suggestions!

[Vellug]

Sounds great, thank you for your response.

[rkompass]

I could automate -march detection as well, but I would need a detailed table mapping CPU cores to their corresponding -march values. As far as I know, no such table exists, so this option will likely require manual selection.

Hi @daleka,

how about:

    print("CPU")
    sys_mpy = sys.implementation._mpy
    try:
        arch = [None, 'x86', 'x64', 'armv6', 'armv6m', 'armv7m', 'armv7em', 
            'armv7emsp', 'armv7emdp', 'xtensa', 'xtensawin', 'rv32imc'][sys_mpy >> 10]
        print("   arch........", arch)
    except:
        print('   architecture unknown')

[daleka]

Hi @rkompass,

Thanks a lot for your suggestion! You’re awesome!
Your approach inspired me to dig deeper, and I was able to retrieve the -march value directly from the MicroPython device.

Here’s the command I used with mpremote:

mpremote connect COM4 exec "import sys; print(sys.implementation._mpy >> 10)"

This successfully returned the architecture ID, which I can now map to mpy-cross -march values!

Again, I really appreciate your input—it helped a lot! 🤘

[rkompass]

You are welcome. Actually this is in the official docs here, but I didn't find it in the afternoon, so I had to look up my own code.