Read and write ZIP files using micropython-zipfile

[jonnor]

Ever wanted to read or write .zip archive files in MicroPython? Now there is a module for that: https://github.com/jonnor/micropython-zipfile
It supports the most common subsets of ZIP archives, including DEFLATE compression (when available in the MicroPython firmware build). I ported this from CPython 3.12, and used their quite comprehensive test suite (around 160 test-cases) to guide the development. I removed/skipped out some of the more niche tests though, to focus on core functionality relevant for a MicroPython usecase.

It is the first time I ported non-trivial (but reasonably stand-alone) CPython code to MicroPython, and it was interesting to see what kind of incompatibilities that exist. Many are rather minor, but it all adds up, taking several hours to address them all. Those considering to port other Python code to MicroPython (while still being runnable under CPython), might find some of the strategies interesting. See especially the top of zipfile.py, where I try to handle all the differences.

I initially created this because I wanted to support Numpy .npz files (which are ZIP archives). This format is popular for Data Science / Digital Signal Processing / Machine Learning / etc. And I needed it for https://github.com/emlearn/emlearn-micropython/ - a MicroPython library for Machine Learning.
But zip files are used for many purposes, so I hope this can be useful for someone else :)

[andrewleech]

Nice one, thanks for sharing and the extra detail and the efforts required. It really can be hit and miss which libraries port trivially, and which ones are harder than expected. It's hard to know / estimate how much effort it'll be until you're in the deep end!

I recommend submitting your link to https://awesome-micropython.com/ if you haven't already, and/or a PR to micropython-lib if it's pretty much stable already :-D

[jonnor]

Thank you Andrew! Yeah, it was a nice learning excercise regarding porting and CPython compatibility. Overall, I actually found it very promising/rewarding - turns out that yes, it is feasible to port general Python code where no-one has thought about MicroPython or embedded restrictions at all. Not only that, but I was able to maintain compatibility with CPython, so code can run under either interpreter. Sure it took some time, but thats software engineering for you :)

I recently also ported some more arithmetic oriented code (https://github.com/emlearn/emlearn-micropython/blob/master/examples/har_trees/timebased.py). There I had very little MicroPython to CPython incompatiblities. Instead, the porting effort was mostly re-architecting from processing N samples at a time to 1 sample at a time, and getting rid of a lot of string handling and mutating lists, which slowed things down too much when running real-time on a microcontroller.

[jonnor]

micropython-zipfile is already on awesome-micropython :D The MR was merged yesterday!

[andrewleech]

getting rid of a lot of string handling and mutating lists, which slowed things down too much when running real-time on a microcontroller.

These things do slow down on the desktop too! Writing things for compatibility on both can help you think about efficient practices automatically once you've been working with micropython for a while. 😄

[jonnor]

Yeah it was not particularly fast there either. But a few seconds for a whole dataset, no-one noticed or cared. But yes, writing MicroPython is good way of learning some Python optimization techniques. Though on desktop one should really try to leverage numpy/pandas/polars/etc, to really go fast