How to reduce memory usage when importing mpy files

[ghost]

Hey, I have a huge .py file (70k bytes) with a lot of logic, and after adding some more code to it, it starts crashing with memory allocation errors. I'm currently compiling the .py to a .mpy before importing it.

Doing some tests I noticed that if I import the file, then run the garbage collector, and then check the allocated memory (before start running the code) it takes like 70k memory.

So my question is where should I direct my efforts to reduce the memory allocation when importing the file. Should I reduce global variables? Shouold I try to change the lists by tuples? Should I try to reduce characters on the names of functions and variables? Should I try to simply write more efficient logic thus reducing lines of code?

There are a lot of things I can think of that may reduce memory usage, but none seems too important, so, how should I face this? Or what is the best way to investigate what are the things that consumes the most memory when compiled and imported?

Note: I'm using micropython 20 with a generic esp32 board like the one below:

Edit: I want to specifically lower the amount of memory consumed when importing the file. I want to understand why it consumes so much memory, the .mpy file is half the size of the memory allocated when the file is imported, and variables and buffers created are not so many

[glenn20]

One suggestion that has worked for me in the past is to break up into multiple smaller modules with judicious use of gc.collect(). Others will have deeper insights into the kinds of code changes that can reduce memory.

[ghost]

Thanks for the answer, but the problem is not about the memory consumed by the code running, or even fragmentation, I particulary want to lower memory consumed when importing the file.

[amplatzer]

I read somewhere, if you pack your code into the firmware, it will reduce the memory used.

[ghost]

Yeah, that would probably work, by I have some limitations, I need to use and import the mpy

[robert-hh]

So--called frozen bytecode moves the code from RAM to flash. It requires building the firmware yourself, but saves space and code startup time.

[ghost]

Yeah, that would probably work, by I have some limitations, I need to use and import the mpy

[bixb922]

See here for details;
https://docs.micropython.org/en/latest/reference/manifest.html#manifest

The actual compilation instructions depend on what microcontroller you use: RP2040, ESP32, etc

[ghost]

I updated the question adding those details, its a generic esp32 board, thanks for the suggestion.
And regarding the limitations, basically I need to be able to update the code using OTA, and that is working great sending the .mpy in chunks. Also, I need the process to be easy, and simply copying the mpy to the filesystem and importing it is much easier than upload the firmware.

[amplatzer]

I have exactly the same situation as you, by downloading the new mpy files to OTA. and my mpy files was like 80k(loads of dicts) , and the kernel crashed / throw out OSError xxx (OOM). I end up by split the big mpy file into several small ones(16k max). and load each one when needed (I dont need to load all of them at the same time). I did do some research by looking into the meminfo(), but get nothing. I guess I will follow this thread again.

[glenn20]

You can also update the micropython firmware by OTA (which can include new frozen bytecode). I understand it may not be as suitable for rapid prototyping, but is a pretty easy way to update code on a device.

[glenn20]

Depending on your other needs, you can also obviate the need to mount a filesystem from flash at all.

[kjm1102]

Interesting that you settled on 16k .mpy file size. I've found that no matter how much free ram a system reports via gc.mem_free() I can never write anything bigger than ~15.5k into ram https://github.com/orgs/micropython/discussions/14032 without getting a memory allocation error.

[amplatzer]

Interesting that you settled on 16k .mpy file size. I've found that no matter how much free ram a system reports via gc.mem_free() I can never write anything bigger than ~15.5k into ram https://github.com/orgs/micropython/discussions/14032 without getting a memory allocation error.

oh, sorry , all I mentioned was refer to the size of .py files.

[amplatzer]

You can also update the micropython firmware by OTA (which can include new frozen bytecode). I understand it may not be as suitable for rapid prototyping, but is a pretty easy way to update code on a device.

@glenn20 Can you do the native OTA on a 4MB device? (with Micropython firmware of cause). can't really find a full example for that case.

[glenn20]

Yes, you can. Take a look at https://github.com/glenn20/micropython-esp32-ota for an example of a tool that writes micropython firmware updates OTA. It is pretty easy to use.

You need to start with an OTA enabled firmware on the device (eg. the firmware with OTA support from https://micropython.org/download/ESP32_GENERIC/) OR you can use another tool to convert an existing non-OTA device to an OTA-ready device (https://github.com/glenn20/mp-image-tool-esp32).

There is more detail about what it means for a firmware to be OTA-ready in the READMEs of the tools.

[glenn20]

A few tips for reliable OTA update usage:

• use the esp32 rollback support
• reboot micropython to initiate the OTA update (eliminates any risk of memory or other resource contention issues with a running app)

The last point is probably not necessary in almost all cases. eg. you might just have the option to reboot into the OTA update if an OTA attempt fails or fails repeatedly.

When you perform an update, you import ota.update. This is written to be flexible, thorough and check everything. It is not optimised for size, but since the very next thing you will do is to reboot, I figure that is a good approach. The other modules are much more memory efficient (ota.status and ota.rollback).

[MRIIOT]

Here is my OTA flow. The trickiest part was managing configuration files living in filesystem, that also might need to be upgraded, separately from the ESP OTA process.

https://github.com/SharcTech/sharc-support/blob/main/firmware/sharc_ota_diagram.pdf

[glenn20]

This looks interesting and robust. One thing to note is that you may consider cancelling the rollback only after you have tested the app and configuration to mitgate any risk that your new app fails with the new firmware or the new configuration. However, then you need a mechanism for rolling back the configuration as well or keeping the configuration and the firmware in sync.

[MRIIOT]

This has been a chicken-egg type of thing for sure.

[ned-pcs]

Another suggestion is to make sure that you're not doing unnecessary object creation in your imported module. Import the .mpy file then call constructors, etc. later.

[ghost]

That is a good thing to have in mind, thanks

[cboostwvanooijen]

Do you really need all the code? In a similar situation I used on-demand loading of module attributes, and even class attributes. But trhis doesn't help (and may actually cost more RAM) if you need everything.

Are there parts of your code that are so unlikely to change that you could freeze them into the image? That really makes a difference! I have a docker to automate this.