How to freeze non-code/assets files into firmware and access later.

[sethstenzel]

I'm looking for help, or a pointer to some documentation on how to "freeze" non-python files in firmware such as images, sounds, or other compressed assets into the firmware.

I currently have a built toolchain setup in WSL, and can build custom images for my board. I can create .mpy, or add .py files to be frozen into firmware but I don't see documentation on how to freeze other things like graphics, or sounds into firmware, or how those assets would be accessed. I also can't seem to find how I would modify the default file system with additional files, like settings files along side the current defaults of main.py and boot.py.

Is there a way to add additional default files to the file system, like boot.py and main.py are added automatically now?

I'm dreading the idea that I might have to convert all my assets to some sort of .py file containing binary data.

Has anyone done this, knows how to do this, or can you point me to documentation specifically on these questions?

Thanks all!

[Josverl]

Credits to Peter

https://github.com/peterhinch/micropython_data_to_py

There are more gems where this came from.

[sethstenzel]

I alluded to my distaste for doing precisely that in my post, encoding the data in python scripts, similar to what I'd need to do with arduino/c/c++ code. But thanks for the link; easier than having to write something similar myself. Does that mean there is no easy way to include that data otherwise? Also, my second question on modifying the default file system to include some setting files (.json files) is that something possible, or would I have to do something like convert using that tool and then on startup, check the file system and write that data out to the file system if I wanted it there?

[sethstenzel]

I guess I just wanted to know what was available before I spent my time writing a library to do the same as Peter. @Josverl, thanks again for this link; most of the leg work is done for the creation, and I think I can extend this to behave a bit more like a file system being able to keep track of original/relative virtual file paths as well as allowing the file data to be flagged and compressed when building, and then decompressed with zlib to save some space if desired.

I assume there would be a need to check and modify the partition sizes if the firmware size grew too large due to the added static data. I'm not sure if there is a hard cap on the firmware size.

[sethstenzel]

In truth, I don't know how important handling decompression would need to be as far as the flash file system goes. My gut tells me I want it for large amounts of image and sound data. The device I'm currently working on has a TFT display on it of a reasonable size and there was 6 seconds of branding information at the start, which was animated .bmp files. I started doing tests on ways to decrease that payload as pushing that much data (say 120-180 frames at 60k each) over the raw REPL was painfully slow. I started looking at stripping unimportant information in the .bmp header, converting to lower bit depths, and finally, compression. At one point, I did a test, and if memory serves (which it may not, it was months ago), loading the full uncompressed image into a buffer took something like 8ms, while loading a compressed image and decompressing took something like 10ms. But the compressed file was only ~27% of the original, so, a good trade-off, I think, and for me, even after pushing to the display, well under the 33ms required to achieve 30fps (though I later reduced this to 15fps since my eyes couldn't tell the difference with the animations I was using anyway perhaps rendering the work moot. My codebase is also somewhat larger, around 260k (of .py files, much smaller when converted to .mpy), but only 79k compressed. I don't have numbers on it since it was some late night testing, but it was a reasonably good upload speed performance increase to compress the .py files upload them over the slow raw repl, and then decompress and have the esp32 rewrite them. Later I abandoned that technique and just wrote a script to convert everything to .mpy that I don't freeze with every build and a utility that lets me upload them faster than pymkr plugin or ampy allowed me to. This utility also does some hard resetting after the device is updated since I noticed what I think may be a bug, where running a ble stack in a separate _thread appears to cause ctrl+d soft resets to not function correctly and put the board into a locked up state after trying to soft reset.

There was also a question of customers possibly needing to reflash the devices in the field. Because of the slow baud rate of the raw repl, building the file system as we are now, with a large number of images, is painfully slow via rawl repl. So compressing them into flash would improve that experience. Also, I started wanting to do something with some audio clips in a personal project, .wav files can be relatively large. But if the decompression overhead is acceptable memory and time wise, which for me it has been, then compressing image and sound data into the flash could be useful for allowing much more content. With these large files stored in flash, you also gain the benefit of avoiding the raw repl slow baud rate since most of the boards I deal with can easily do 10 x 115200 speeds when flashing, significantly reducing a manual file system restore on a device and improving a reflash experience.

For the record, I do realize all my attempts to do things differently mentioned above, are probably akin to a blind man finding his way out of a corn/hedge maze :)

[peterhinch]

You'll find that WAV files are not very compressible without special audio algorithms. It is possible to achieve 50% lossless compression with FLAC, and higher lossy compression with MP3. However decompressing these formats on MP currently requires external hardware.

If you didn't need compression one approach would be to look for a PC utility that maps a directory tree onto a filesystem located in a binary file. For MP compatibility the filesystem would need to be FAT or Littlefs. Using data_to_py techniques the file could be built into Flash. The resultant data could be mounted using code based on the official docs link I provided.

I think it's highly likely that such a utility exists, so that solution would require minimal coding.

Linux has a means of mounting a filesystem contained in a binary file. I can't recall how to go about this or if you can define the FS as FAT, but if so it could be the answer to the PC side.

[sethstenzel]

Yeah, you're probably right. I can always store a compressed file anyway if needed. Be, better not to complicate it, especially since it's me, and more complications probably mean less chance for completing it fully as it is so tempting to just get it to a good enough state and just use it with caveats instead of fully finishing and releasing to the community.

[russhughes]

You can download a copy of a fat filesystem and modify it using Linux or Windows's Linux subsystem, then flash the modified image back onto the device. Of course, this does not save flashing time since you have to reflash the entire filesystem, but you can replace all your files at once, making it easier to update multiple devices. See https://penfold.owt.com/turtleplotbot2/software.html#customizing-the-vfat-image for an example of how I did this for one of my projects.

There are tools for modifying other types of filesystems, but I have not used them.

[peterhinch]

Ingenious. Presumably you have a MicroPython device driver that enables the Flash based filesystem to be mounted. I imagine this is pretty straightforward.

[sethstenzel]

There are more gems where this came from.

@Josverl you have my attention 😄
Do you have an index of these gems, or a blog?

[peterhinch]

@Josverl was perhaps referring to my other libraries. There is an index here. The resource listing a wide range of user contributions is awesome micropython.

[mattytrentini]

You may want to read up on mapfs in PR #8381. It's had a lot of effort and looks very promising - but probably won't land on mainline for a little while as it's quite a large feature.

[JohannesMaierhofer]

Seems that PR #8381 made progress. I do not completely understand how vfs-map could help me in practice to bundle also asset files into one single firmware.bin that I can deploy?

[bixb922]

Until #8381 is ready, take a look at https://github.com/bixb922/freezefs

That utility archives files and folders into a .py file. You freeze that file into a MicroPython image. When you import that file, the files/folders get mounted as a readonly vfs file system with little RAM overhead.