Deploying Frozen Files in a Micropython UF2 Build in a Production Environment

[BryceTuppurainen]

Hello,

If anyone has any advice, suggestions, or resources to point me to in regards to this project please let me know 😄

I'm currently working on a companion desktop app to allow for end users of my devices to perform the automated retrieval and flashing of a pre-built UF2 to chipsets that support Micropython such as the RP2040 (for example on a Raspberry Pi Pico)

I've experimented with and researched some popular methods of creating custom Micropython builds with frozen files, and it appears that the most robust method of doing so is to:

1. Create a fork of a preferred Micropython release (such as the current v1.19 release)

2. Add the files I wish to freeze into the build such as a custom main.py, modules, and configuration files (such as data stored in an XML, JSON, etc.) into the micropython/ports/rp2/modules/ folder

3. Use cmake to compile and build a firmware.uf2 file

4. My companion desktop app (which customers of the device will eventually use), will then copy the new UF2 file over into a board booted in DFU mode, which will cause it to update its flash with the new firmware

The commands I've ran in order to do so (currently running in Ubuntu 22.04.1 LTS with all of the appropriate dependencies installed) were:

# OR - clone another version
$ git clone https://github.com/micropython/micropython

$ cd micropython/mpy-cross
$ make
$ ..
$ git submodule update --init lib/pico-sdk lib/tinyusb
$ cd ports/rp2
$ make BOARD=PICO submodules

# At which point, from /rp2 I can now run:

$ make

# Which leaves the uf2 of the build (and various other files) in the subfolder ./build-PICO

Does this seem like an appropriate method to generate a UF2 with frozen firmware to flash to boards in a production environment (5000+ devices)?

Thank you,
Bryce

[peterhinch]

If you prefer you can locate your files for freezing elsewhere. For example my build script has

MANIFEST='/mnt/qnap2/Scripts/manifests/rp2_manifest.py'

make submodules
make clean
if make -j 8 BOARD=PICO FROZEN_MANIFEST=$MANIFEST

with the contents of rp2_manifest.py being

include("$(MPY_DIR)/ports/rp2/boards/manifest.py")
freeze('/mnt/qnap2/Scripts/modules/rp2_modules')

This adds a set of extra modules to those provided by default. The benefits are that the source tree is unmodified and it readily facilitates freezing different sets of modules.

[jimmo]

I've experimented with and researched some popular methods of creating custom Micropython builds with frozen files, and it appears that the most robust method of doing so is to:

This is not the case. The way you've described is fragile, relies on internal implementation details (e.g. the modules directory which is for system modules), and modifying the micropython repo.

The best way to do this is described here: https://docs.micropython.org/en/latest/reference/manifest.html

[jimmo]

(Unfortunately i'm aware that a lot of existing third-party documentation and forum posts do describe this way of doing things, but yeah, the manifest way is the way to go, and in particular writing your own custom board definition containing your manifest.py is an even better option -- that way you can just build your custom board and it automatically pulls in the manifest).

[BryceTuppurainen]

Thank you for the responses and suggestions. I've adjusted the method I use to build a UF2 to instead point to an external folder with a custom manifest file including instructions to freeze the appropriate sub-directories and modules required for the firmware update.

Something else that I've been experimenting with is a method of copying other plain-text or similar non-python files to an RP2040 while it is in Device Firmware Update mode.

As far as I can tell from the micropython docs it isn't possible, but if anyone knows of a way to either bake-in non-.mpy/.py content into the UF2 or has suggestions for a better way to do so please let me know.

---

Currently, my idea is to append a function to the beginning of the main.py that gets uploaded to the board as frozen firmware and then have it listen for a standard serial connection for a few seconds after the board starts up.

If a serial connection is established in this time and the incoming buffer contains a particular 'signature' requesting a file write, then the raw file data is transferred via serial and subsequently written to a file on the board from within this function (which I can then verify with some parity checks).

Although, I have a few concerns with this method's reliability, and being able to transfer this content while the board is still in DFU mode would be preferable to minimise the number of steps to transfer all of the updated firmware and config.

[jimmo]

Something else that I've been experimenting with is a method of copying other plain-text or similar non-python files to an RP2040 while it is in Device Firmware Update mode.

I'm guessing from the second part of your message that you mean your own implementation of a Device Firmware Update functionality -- the term DFU has a fairly specific meaning. Also assuming you don't mean the rp2 .uf2 bootloader.

Your approach does sound fairly reasonable. I wonder if it might be easier just to use the mechanism used by pyboard.py (and mpremote) to copy files via scripting the REPL? (In a nutshell, pyboard.py sends Ctrl-C to stop the current program, then scripts the REPL to do file write commands, then triggers a soft reset to resume the program).

To your question about putting content into the .uf2 file, there are two main options here:

• Have the .uf2 file also include the filesystem. (The .uf2 files currently only write the firmware, leaving the rest of the flash containing the fileystem untouched). You can also partition the fileystem and have one partition left alone, and the other written by the .uf2. We do not have any automated tooling around this at the moment, but it is something that is often requested. (And I know people have done it with .dfu and .bin images... in summary you build a local LFS/FAT image, then concatenate it to the firmware, then build that in to the .dfu/.uf2/.hex).

• Have a way to include non-filesystem "resources" in the firmware. Roughly the simplest version of this would be to able to add "foo.bin" to the firmware, and you'd end up being able to write import foo" and then foo.data` would be a read-only, in ROM, bytestring containing that data. This would allow you to memoryview it and access slices of it with no RAM used.
  @peterhinch has implemented this as a tool that produces a .py file from an arbritrary file that can be frozen. See https://github.com/peterhinch/micropython_data_to_py

For both of those two options, my ideal is to extend the manifest system to support both. i.e. adding filesystem() and resource() directives to the manifest file.

[EliasKotlyar]

Hello,

If someone is reading this, and want to have the first approach done(bundling the application into the uf2 file), please have a look at following project:
https://github.com/Gadgetoid/dir2uf2

It basically does exactly that. I have tested it and its working perfect out of the box for a PicoW Board.