mpremote + mip: does not create MPY files

[awachtler]

Hello community,

at first I want to express that I appreciate the recently started mpremote/mip concept.
This will shorten the time for publishing, deploying and testing of code amazingly. (My recent approach of running upip on the target was working but was always a bit fragile, due to establishing Wifi connection on almost empty ESP and messing around with credentials).

Here is now my question:

Just with my first trial I found that no MPY Files are created. I have done this steps:

• create a package.json and started a local fileserver in the directory of the package file: python3 -m http.server 8080
• run mpremote u0 mip install http://localhost:8080 in a second terminal.

That json-file looks ok, because the module get installed correctly, but as source code not as mpy-files.

{
  "urls": [
   ["iot_utils/bme280.py", "http://localhost:8080/iot_utils/bme280.py"],
   ["iot_utils/hcsr04.py", "http://localhost:8080/iot_utils/hcsr04.py"],
   ["iot_utils/hx711.py", "http://localhost:8080/iot_utils/hx711.py"],
   ["iot_utils/__init__.py", "http://localhost:8080/iot_utils/__init__.py"],
   ["iot_utils/pinmappings.py", "http://localhost:8080/iot_utils/pinmappings.py"],
   ["iot_utils/uping.py", "http://localhost:8080/iot_utils/uping.py"],
   ["iot_utils/wlan.py", "http://localhost:8080/iot_utils/wlan.py"]
  ],
  "deps": [
  ],
  "version": "0.2"
}

The logging module I used for testing was installed as expected as mpy file.

Does It mean that I have to provide the mpy-files on the server?
I was hoping that the mip compiles the PY files directly while downloading.

Any explanation is appreciated, thanks, Axel.

[awachtler]

Just had a look in the mpremote code, it does just download files, so that means mpy files have to be hosted on the web site.

Since each micropython version comes with its own mpy-cross (because bytecode seems to change from time-to-time) - how is that handled?

How about adding on the fly compile to mpremote/mip version?

[ricksorensen]

mpy-cross is built and runs on the host, while mip is run on the target (that pulls objects from network connection). I believe to accomplish on the fly .mpy creation would require a utility service on the host which would have to deal with the versioning. A general fallback would be to use the modules source.

[jimmo]

@awachtler The generation of .mpy files is independent of the mechanism mip (and mpremote mip) uses to fetch. Basically mip just gets a package.json and downloads exactly what it tells it to.

For the specific case of requesting a named package from micropython-lib, the URL it uses to get the package.json indicates the bytecode version supported by the device. e.g. https://micropython.org/pi/v2/package/6/html/latest.json (for bytecode v6) or https://micropython.org/pi/v2/package/py/html/latest.json (to explicitly get the uncompiled version).

So if you want to host your own package.json you need to pre-compile the files yourself with mpy-cross.

[awachtler]

@jimmo, @ricksorensen thanks for your reply. I did afraid so. Anyway since the target generates the bytecode in RAM, I wonder if there could be a method to get the bytecode buffer and write it to a file on the target.

Another idea is to extend the mpremote-mip by a "download" option, then run mpy-cross, then use mpremote-push to install the files.

Hosting the mpy-files on the server is i.m.h.o sub-optimal, since one have to keep a bunch of precompiled modules dependent on the version of the bytecode for each version of the source code.

I will dig in the code of mpremote-mip first, since this my preferred use case and will comeback to here if I have a working solution.

[jimmo]

@awachtler Adding support for on-the-fly pre-compiling of .py files into mpremote is definitely something we're keen to add. I recently started the process to make mpy-cross a standalone Python package, so that mpremote can use it.

Note that pre-compiling the .py files to .mpy with mpy-cross has only a small benefit for RAM usage. When loading a .mpy file off the filesystem, the bytecode is still copied to RAM.

The only way right now to pre-compile such that the bytecode executes from ROM is to freeze into the firmware. (See https://docs.micropython.org/en/latest/reference/manifest.html). There is work in progress to dynamically freeze modules, see #8381.

[dlech]

I recently started the process to make mpy-cross a standalone Python package, so that mpremote can use it.

FYI, there is already https://gitlab.com/alelec/mpy_cross and https://github.com/pybricks/python-mpy-cross