Is it possible to have two different sections of frozen bytecode?

[victorallume]

Is it possible/feasible to have two different sections of flash containing frozen bytecode, but one is loaded at a later point (i.e. at some point after boot.py has been run)?

[jimmo]

@victorallume A mostly-accurate way to think of frozen bytecode is as a special filesystem containing .mpy files where the loader knows that certain data structures are (a) already laid out exactly as the runtime expects, and (b) directly memory mappable.

Compare to loading a .mpy file from the filesystem, we need to do more translation of the data structures (because the .mpy format is generic whereas the in-memory structure the runtime expects depends on the architecture), although as of the v1.19 there is much less translation. But from the filesystem you cannot do memory mapping because the files are not necessarily contiguous.

Anyway, the point is that none of this happens until you import a given module. So unless I've misunderstood the question, I don't think you need to worry about this?

[victorallume]

The problem is not about cost of loading the second stage modules, and more about verifying the second stage, and allowing the second stage to be updated independently of the first.

Regarding verification: this checksums the flash region for the relevant stage, compares against a known checksum, and if it fails, goes into a 'safe mode' and attempts an OTA.

[peterhinch]

I'm pretty sure the flashing process verifies the outcome so runtime checks are not necessary.

[victorallume]

Thanks @peterhinch - we're flashing via an OTA (i.e. from micropython code), so would like an extra level of verification; we also perform this on boot as an extra check. But if we put that aside, the other part is to be able to update the two sections of bytecode independently.

[jimmo]

@victorallume I think the thing you want is #8381 :D

(This feature is going to be a pretty big deal for many people I think, because it gives you almost all the benefits of frozen code but with all the advantages of not having to build and flash the firmware. It's also really great for development -- see #8426 (comment) -- because you can do a full deploy of your entire codebase much much faster than copying .py/.mpy files to the regular filesystem).

But in terms of what you can do now.... I think I understand your original question now. It's not so much that you want two different groups of frozen code to be loaded independently (they are anyway), rather that you plan to load them independently, so you'd also like them to be physically laid out independently on the flash (e.g. so you could update just one region via OTA or something).

In theory you can use the linker script to put particular symbols into different regions, however there is a bunch of stuff in the frozen code section that is shared (for example the QSTR table), and we definitely don't have a way to safely only update a subsection of it.

[victorallume]

Thanks @jimmo - that PR does indeed look like what we're after.