ESP32 Frozen Byte code updates

[Tom-sog]

I'm building frozen bytecode by placing my .py files in the modules directory. Something I noticed, when I build for the esp8266 it determines which files have changed, and will just update the frozen bytecode only and create a new firmware file. Which takes a few seconds... example is below.

When building for the ESP32, make just calls 'idf.py -D MICROPY_BOARD=GENERIC -B build-GENERIC build'
But it doesn't update the frozen bytecode. I need to do a make clean, then make and rebuild the entire firmware from scratch. Which isn't particularly fast on the RPI 4.

I'm wondering if there an option to just update the frozen bytecode and rebuild the firmware for the esp32?

Example esp8266:

pi@rpi-3:~/git/micropython/micropython_v1.19.1/ports/esp8266 $  make
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
MPY my_base_mqtt.py
GEN build-GENERIC/frozen_content.c
CC build-GENERIC/frozen_content.c
LINK build-GENERIC/firmware.elf
   text    data     bss     dec     hex filename
 715056     996   66376  782428   bf05c build-GENERIC/firmware.elf
Create build-GENERIC/firmware-combined.bin
esptool.py v3.1-dev
Creating image for ESP8266...
flash     32832
 .text    30720 at 0x40100000
 .data    996 at 0x3ffe8000
 .rodata  1080 at 0x3ffe83f0
padding   4032
irom0text 683256
total     720120
md5       d31586695108fb7b5ab77d31466bb3d4

[jimmo]

I'm building frozen bytecode by placing my .py files in the modules directory.

This works, but it's kind of an implementation detail of the build system. i.e. these are the modules required for MicroPython to function. Your own additional modules should be added via a manifest -- see https://docs.micropython.org/en/latest/reference/manifest.html

I'm wondering if there an option to just update the frozen bytecode and rebuild the firmware for the esp32?

There's definitely a bug here with detecting frozen content changes in the modules directory. The ESP32 build is really complicated due to the way it uses CMake with the IDF, but we need to fix this.

In the meantime, the way to force it to update just the frozen content is to do

rm build-GENERIC/frozen_content.c build-GENERIC/frozen_mpy/*.mpy
make

[Tom-sog]

Tried and tested - works a treat, thank you.

[puppet13th]

@Tom-sog how long it takes to build mpy using rpi3?

from my experience(building esp32 firmware):
i5 gen3 + hdd = around 25 minute per build
i5 gen3 + sdd = around 1 or 2 minute per build

i'm using debian(wsl2) on windows 10.

[Tom-sog]

I have a RPI 4B with 2GB RAM on debian bullseye, drive is a Samsung EVO 870 SATA III drive - 256GB connected via USB3.2 controller. I don't observe the disk as a bottle neck.

Although the wait is really painful, it's not just the time. I also use the RPI as a database server, dhcp & dnsserver (pihole), mqtt server and a bunch of custom python based apps. ESP32 build will drive all 4 cores to 100%, with load up to ~6, this causes timeout's from the other apps during the compile of the ESP32 firmware. As reference, normal load is ~0.3.

ESP32 build stats:

$make clean
$time make
idf.py -D MICROPY_BOARD=GENERIC -B build-GENERIC  build
....
real    6m15.264s
user    15m52.771s
sys     3m24.418s

$rm build-GENERIC/frozen_content.c build-GENERIC/frozen_mpy/*.mpy
$time make
idf.py -D MICROPY_BOARD=GENERIC -B build-GENERIC  build
....
real    0m59.038s
user    0m55.513s
sys     0m3.555s

ESP8266 build times for comparison:

$ time make
....
real    1m17.976s
user    1m8.163s
sys     0m15.282s

update module water.py
$ time make
Use make V=1 or set BUILD_VERBOSE in your environment to increase build verbosity.
MPY water.py
GEN build-GENERIC/frozen_content.c
CC build-GENERIC/frozen_content.c
LINK build-GENERIC/firmware.elf
   text    data     bss     dec     hex filename
 715096     996   66376  782468   bf084 build-GENERIC/firmware.elf
Create build-GENERIC/firmware-combined.bin
...
real    0m4.805s
user    0m4.147s
sys     0m0.678s

Drive write stats:

dd if=/dev/zero of=/tmp/test1.img bs=1G count=1 oflag=dsync
1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 6.35619 s, 169 MB/s

[puppet13th]

remove these file/folder from build_dir(eg. micropython/ports/esp32/build-GENERIC):
frozen_content.c
frozen_mpy

then call make to rebuild firmware with updated modules. #9332