Two new tools for performing OTA firmware updates and manipulating esp32 firmware image files.

[glenn20]

As some of you may have seen me mention already, I have posted a few new tools which might be useful for ESP32 users.

One runs on your esp32 board to execute Over-The-Air (OTA) firmware updates and the other runs on your computer to manipulate micropython firmware images and devices.

I have found these to be very useful for my use cases, maybe they will be for you too. See a brief summary below. For all the details checkout the README.md in each repo.

micropython-esp32-ota: supports esp32 Over The Air firmware updates and rollback control:

• Write a new micropython app image firmware (.app-bin) to the next available OTA partition and set it as the boot partition:

  import ota.update
  ota.update.from_file("http://nas.local/micropython.app-bin", reboot=True)

• You can also update from a json file which contains the url of the firmware, it's expected length and sha256 checksum. The downloaded
  firmware will be checked against these:

  ota.update.from_json("http://nas.local/micropython.json", reboot=True)

• Control OTA app rollback with ota.rollback.cancel().
• Install: mpremote mip install github:glenn20/micropython-esp32-ota/mip/ota/mpy

NOTE: You must be using an OTA-enabled firmware for this to work, which brings me to the next tool:

mp-image-tool-esp32: manipulates esp32 firmware files and firmware installed on esp32 devices:

Typical use cases include:

• Convert firmware for a 4MB device to a 16MB device: mp-image-tool-esp32 filename --resize-flash 16M
• Convert a defaut micropython firmware to an OTA firmware: --table ota
• Install any other partition table you want: --table nvs=6B,phy_init=1B,factory=0x1f0B,vfs=0
• Delete, resize or add partitions: --resize factory=0x2M,vfs=0x400K, --add vfs2=fat:2M:1M, --delete phy_init
• Read and write data from partitions on the flash storage: --read factory=micropython.app-bin,nvs=nvs.bin, --write factory=micropython.app-bin
• Erase partitions: --erase nvs,otadata, --erase-fs vfs micropython will automatically build a fresh filesystem or 'nvs' partition
  on the next boot
• Perform an OTA firmware update over the serial/USB connection: --ota-update micropython.app-bin

BONUS: Did you know that micropython does not use the phy_init partition which is installed by default in all micropython esp32 firmware?

• You can reclaim that space and add it to your nvs partition with: --delete phy_init --resize nvs.
  • NOTE: this will erase your nvs partition and micropython will automatically, on the next boot, initialise a new nvs store using the extra space.

Install and use it with (will print a summary of the partition tables in the firmware and on the device):

• If you use a python virtualenv, activate it first.

git clone https://github.com/glenn20/mp-image-tool-esp32
cd mp-image-tool-esp32
pip install -r requirements.txt
./mp-image-tool-esp32 ~/Downloads/ESP32_GENERIC-20231005-v1.21.0.bin
./mp-image-tool-esp32 /dev/ttyACM0

If you have any questions, checkout the README.md file in each repository or feel free to ask here. If you find any problems, or have suggestions, please post an issue on the github repositories of each tool.

[shariltumin]

Thanks!

[bixb922]

Thank you! I have used the mp-image-tool-esp32. Worked very well. It saved the time to install the tool chain and recompile Micropython for a new flash size.

[gitcnd]

Do you think it would be possible to...

1. store a new firmware on the file system of a running micropython device, where this firmware has a tiny "flash stub" on the end.
2. "defrag" the above, so it's contiguous (see next step)
3. halt everything and transfer control to the "Flash stub", which which writes the new firmware over the top of the one which was just running, and resets.

(All without having any dedicated OTA partition of course)

I am trying to think of some way to have updatable devices in-the-field, without having to waste 50% of the storage on OTA partitions...

[glenn20]

This is something I have been thinking about for a while, but not sure i'll ever get around to implementing.

[gitcnd]

I took a poke around at that stuff - turns out the flash-writing code is already in the stub itself (of course - so it won't get overwritten during the re-flash), and I know the fat FS format well, so a basic defrag is almost a no-brainer (to make the new firmware contiguous - I'd probably arrange it such that the very last filesystem block is blank, then the second last block is the last block of the new firmware, and so on, and use that last block to tag the start address and any other reflash related housekeeping that might be needed).

"lose the ability to rollback on a failed update" - true, but, crazy-idea: it's an extremely minor few extra bytes of code to swap the new firmware with the old one (so the file "update.bin" gets written to the firmware, while, the original contents of the firmware are written back over the top of update.bin at the same time).

with some suitable minor adjustment to the booting stub, a failed update could easily revert itself by simply re-running the original update (which swaps back the original firmware).

gets a bit messy when new and old firmware are different sizes, but that can be accommodated.

doesn't solve the problem of "main.py" having a nasty bug in it (since the FS itself stays the same), but that's a different problem which is also fixable at the same time (e.g. some kind of watchdog startup plan)

Bonus: there are a lot of really great benefits that come from putting the new firmware on the file-system first - like - you no longer have to care about how the file got there (web pull? sure. bluetooth? ok. custom espnow socket? fine. lora? yep. incoming HTTP POST - easy ...)

[glenn20]

I can see that would work...but, to me it feels like more work and more fragile than leveraging the existing OTA mechanisms and espressif bootloader logic to achieve virtually the same end goal. But, I am very aware that might be due to my limited understanding of the flash mechanism.

Note: micropython uses the littlefs2 filesystem by default (but can also use FAT).

[glenn20]

"lose the ability to rollback on a failed update" - true, but, crazy-idea: it's an extremely minor few extra bytes of code to swap the new firmware with the old one (so the file "update.bin" gets written to the firmware, while, the original contents of the firmware are written back over the top of update.bin at the same time).

In this scheme you have to ensure there is enough space in the filesystem to hold the new/old firmware, which in practice means you have to reserve space for a future firmware update (~1.5-1.7MB). This doesn't save you much over reserving that space in a dedicated OTA partition.

Bonus: there are a lot of really great benefits that come from putting the new firmware on the file-system first - like - you no longer have to care about how the file got there (web pull? sure. bluetooth? ok. custom espnow socket? fine. lora? yep. incoming HTTP POST - easy ...)

I can certainly see a convenience here, rather than streaming the firmware directly into the OTA partition, but it does mean that for some period of time you still have to have two copies of the firmware on your flash - and the space you need to reserve for that. I can see that for some needs, clearing out space for a firmware update may be fine, but in a lot of cases this will just mean you have to plan to keep ~1.5MB of flash permanently free - which doesn't gain you much over the current OTA mechanism.

[gitcnd]

littlefs2 is FAT (not to mention, since FAT formatted SD cards must be supported, it makes no sense to add support for anything else too). You do not need to "reserve space" for firmware updates (that's the original point), you simply need to make space available when the time comes (e.g. by erasing logs or removing other files which can later be restored). Half the problem with OTA is the lack of standards and interoperability: Arduino's mDNS system is different to other people's web-pull, is different to bluetooth, ... and the list goes on. Decoupling the OTA step from the "how the OTA file gets there" step would be a huge improvement (and the only way to allow more than one method, which seems increasingly important as security-conscious people take their IoT's off the internet)

rudimentary compression would be supportable if necessary too, which currently saves 51% of the size of the firmware (so you'd only need
1160549 bytes space for the latest release presently)

The difference is pretty huge - you go from ~ 339,552 bytes usable space in the current setup, to 2,436,704 (a huge 7 times more), or if you optionally decide to reserve space upfront for a compressed new image, you get 1,276,155 (which is still 3.75 times more space)... while adding a bunch more useful features for accommodating OTA methods.

I think though, best of all, doing it this way allows everyone to have OTA if they need it later, without anyone having to decide up front or sacrifice space.