How to upload (.mpy) files to ROM

[sir-user]

Hi,
I am currently develeoping a project on custom ESP32 board using micropython in Thonny IDE. I am facing memory allocation error. My python files are large (it includes some wave files). I used gc.collect() and converted all my files to mpy but I still face memory allocation error. I decided to convert my files into frozen bytecode to save RAM space. I have following questions.

1. I came across following link https://github.com/micropython/micropython-example-boards/tree/main to generate frozen bytecode(?). I use machine functions on ESP32 in my py files by micropython that I set up firmware bin file using this website https://micropython.org/download/ESP32_GENERIC/. Can I still use machine functions after implementing generate frozen bytecode using given github link.
2. Is there any complete guideline on how to build frozen bytecode using Micropython. I was not able to build frozen bytecode using the documentation in Micropython webpage.

Thank you in advance for your help,
Volkan

[jimmo]

@sir-user The documentation is here
http://docs.micropython.org/en/latest/reference/manifest.html

You need to make your own board definition (start by copying ESP32_GENERIC), and then you can adapt the manifest file as described there.

[shariltumin]

This is a little write-up about custom-building MicroPython firmware.
This is how I do it on a Linux Mint PC. It may not be the right or best way to do it, but it works for me.

1. $ cd micropython/
   Create a new directory to store your modules
   $ mkdir extra
2. $ cd micropython/ports/esp32/board/
   Create a new board
   $ cp -r ESP32_GENERIC ESP32
   Edits:
   1. mpconfigboard.cmake
   2. mpconfigboard.h
   3. sdkconfig.board
   4. manifest.py
3. $ cd micropython/ports/esp32/
   Customise your build. Save original and edited files:
   1. $ cp mpconfigport.h mpconfigport.h-ORIG
   2. $ cp mpconfigport.h mpconfigport.h-STD
   3. $ vi mpconfigport.h-STD
   4. $ cp esp32_common.cmake esp32_common.ORIG
   5. $ cp esp32_common.cmake esp32_common.STD
   6. $ vi esp32_common.STD
4. Make build script
   $ vi mpb_esp32_std.sh
5. Run make
   $ chmod u+x mpb_esp32_std.sh
   $ ./mpb_esp32_std.sh

These are my files:

mpconfigboard.cmake

set(IDF_TARGET esp32)

set(SDKCONFIG_DEFAULTS
    boards/sdkconfig.base
boards/sdkconfig.ble
${MICROPY_BOARD_DIR}/sdkconfig.board
)

set(MICROPY_FROZEN_MANIFEST ${MICROPY_BOARD_DIR}/manifest.py)

mpconfigboard.h

#define MICROPY_HW_BOARD_NAME "ESP32 STD/BLE KAKI5"
#define MICROPY_HW_MCU_NAME "ESP32"

sdkconfig.board

# MicroPython on ESP32, ESP IDF configuration with 240MHz CPU
# Lower speed save power
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_40=
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_80=
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_160=
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ_240=y
CONFIG_ESP_DEFAULT_CPU_FREQ_MHZ=240

# DEBUGGING
# -CONFIG_LOG_DEFAULT_LEVEL_VERBOSE=y
CONFIG_LOG_DEFAULT_LEVEL_DEBUG=y
# -CONFIG_LOG_DEFAULT_LEVEL_INFO=y
# -CONFIG_LOG_DEFAULT_LEVEL_WARN=y
# -CONFIG_LOG_DEFAULT_LEVEL_ERROR=y

# Flash PARTITION
CONFIG_ESPTOOLPY_FLASHSIZE_4MB=y
CONFIG_ESPTOOLPY_FLASHSIZE_8MB=
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions-4MiB.csv"

# Hostname
CONFIG_LWIP_LOCAL_HOSTNAME="Kaki5"

manifest.py

# -freeze("$(PORT_DIR)/modules")
freeze("$(PORT_DIR)/modules", ("_boot.py", "espnow.py", "flashbdev.py", "inisetup.py"))
# -include("$(MPY_DIR)/extmod/asyncio")

# board specific modules
# -freeze("modules", 'app.py')

# Require some micropython-lib modules.
require("dht")
require("ds18x20")
require("neopixel")
require("onewire")
require("umqtt.robust")
require("umqtt.simple")
require("upysh")
require("mip")
require("ntptime")
require("urequests")
# -require("webrepl")

# my extra modules
freeze('$(MPY_DIR)/extra', 'Caesar.py')
freeze('$(MPY_DIR)/extra', 'CryptoXo.py')
# -freeze('$(MPY_DIR)/extra', 'PickleDB.py')
# -freeze('$(MPY_DIR)/extra', 'Tasks.py')
# -freeze('$(MPY_DIR)/extra', 'Guard.py')
freeze('$(MPY_DIR)/extra', 'sdcard.py')
freeze('$(MPY_DIR)/extra', 'worker.py')
freeze('$(MPY_DIR)/extra', 'pipe.py')
freeze('$(MPY_DIR)/extra', 'Telnet.py')
freeze('$(MPY_DIR)/extra', 'Ping.py')
freeze('$(MPY_DIR)/extra', 'Ramdisk.py')

To save memory you may need to disable some standard Python libraries and C modules. Here I choose to disable "asyncio". I also exclude "help", "webrepl", "usb", "lan", etc. by modifying "mpconfigport.h" and "esp32_common.cmake".

The build script is simple, and saves you having to write cmds on the terminal lines

#!/bin/bash

export PATH="$HOME/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"

echo "=========================================================="
echo "Setting up for esp32 build"
echo "Using esp-idf-511"
export IDF_PATH="$HOME/disk/esp/esp-idf-511"
export IDF_TARGET="esp32"
source $IDF_PATH/export.sh

# ESP32 
cp mpconfigport.h-STD mpconfigport.h
cp esp32_common.cmake-STD esp32_common.cmake
make V=1 BOARD=ESP32 MICROPY_VFS_LFS2=1

echo "=========================================================="

As you can see, I'm compiling my firmware with ESP-IDF V5.1.1, and it seems to build and run OK.

The MicroPython build process and flags are constantly changing. You may need to modify the above files as they become necessary.

Hope this helps and good luck!

[sir-user]

Thank you a lot, it really helped. I did not know anything and I did not exactly do what you share with me but what you mentioned guided me quite well.