docs: exhaustive documentation of mpy file format?

[massimiliano-mantione]

Hello,
I really looked for it, but I could not find it: is there a complete documentation of the mpy file format?

The idea is hinted at in the project docs, with something about the header and sections, but I did not find a complete specification of all possible sections, their format and, most importantly each and every bytecode operator.

I am toying with the idea of creating a separated compiler that generates mpy files but not starting from python, but without that kind of documentation I cannot even evaluate the general feasibility of the idea...
And I'd prefer not to reverse-engineer mpy-cross :-D

And sorry if the documentation is there but I did not find it!

[jimmo]

hi @massimiliano-mantione No there isn't such documentation at the moment. There are two parts -- one is the structure of a .mpy file, the second is the actual bytecode. Both of these things are not fixed, they do occasionally change between micropython versions, so they're seen as somewhat of an implementation detail. The bytecode tends to get quite small changes, but e.g. the .mpy format had quite a major overhaul last year.

Probably your best bet is to look at mpy-tool.py which is the most self contained tool that can work with both .mpy files and bytecode. It now includes a disassembler.

It's also useful to turn on verbosity output on the unix port (via the -v -v -v flags) which will show a lot of detail about the generated bytecode.

[dlech]

I am toying with the idea of creating a separated compiler that generates mpy files but not starting from python

Probably easier to write something that compiles from not-python to python then use mpy-cross to get bytecodes.

[jimmo]

Yep I think this is very likely to be the case. The bytecode is very closely aligned to what you can be written in Python.

[massimiliano-mantione]

Thanks! The pointer to "mpy-tool.py" is perfect, understanding the file contents from that tool seems reasonably easy 👍

I also understand the suggestion to transpile to python first and I will keep that in mind. However what I really have in mind is a tool to generate mpy modules that also contain native code (or even mostly native code), but in an ergonomic way. I'll see how it is feasible.

In that spirit, I will probably do not need to emit every possible micropython bytecode instruction, just a subset of them.

But, is there a central place to understand the meaning (semantics) of bytecode instructions?
Something like "mpy-tool.py", but for execution?
Or is it the interpreter itself?

[massimiliano-mantione]

BTW, @dlech , you are omnipresent! 😄
I see you at ev3dev, pybricks, here, and always helping... thanks!!!

[dlech]

mpy modules that also contain native code (or even mostly native code), but in an ergonomic way

So not like this? https://docs.micropython.org/en/latest/develop/natmod.html

[jimmo]

But, is there a central place to understand the meaning (semantics) of bytecode instructions? Something like "mpy-tool.py", but for execution? Or is it the interpreter itself?

I don't think this is fully documented either. vm.c is my reference for the exact meaning.

And yeah, like @dlech said, it really sounds like you're describing the dynamic native modules feature?

[massimiliano-mantione]

BTW, related: I keep stumbling upon "viper" code next to "native" code.
Where is that described?

[cwalther]

https://docs.micropython.org/en/latest/reference/speed_python.html#the-viper-code-emitter