Custom firmware

[Alain-Godo]

First of all, sorry if my English is not the best. I have two main questions, both in relation. How can I modify the MicroPython firmware before uploading it to the device? For example, I have an ESP-32 and my code is created in .py and .mpy files, but I don't need to use the typical structure of boot.py/main.py, and more important: I want to integrate the code with the firmware to just upload all the code as a single piece one time.
The other question is, how can I protect the code to avoid it reading at least in a simple way like we now can do in an IDE with the .py files?

[ned-pcs]

First answer: read the documents about making custom firmware. For instance: https://github.com/micropython/micropython-example-boards

You can configure the various pieces that are included in the build using cmake, make, and/or C .h files.

Second answer: if you have .mpy files as part of your build, these can't be read directly as Python sources. However, they can be read and decompiled with the proper tools. For instance, there is a decompiler here (though I have not tried it): https://github.com/gpdaniels/spike-prime

[Alain-Godo]

Infinite thanks, based on your answer I only need to allow the code to be “frozen” into the firmware...But still is so confusing for me, a lot of concepts to deal with, Can you show me an overview of the process? Or share a link to a post or video with the process step-by-step up-to-date? Again, Thanks !!!...

[sethstenzel]

Greetings,

I had some similar questions a few months back. Currently, what I do is convert all .py to .mpy.
I'm on Windows and have Ubuntu via WSL2. I have a batch file in my primary coding directory that calls my WSL2 instance to convert all my .py to .mpy files.

This means I have the build environment setup there, and I call into with a batch script, which runs a shell script.

I also build a custom firmware for my board, and in my build structure since I'm targeting ESP32's I have this folder:
\\wsl.localhost\Ubuntu\home\sstenzel\micropython\ports\esp32\modules
Modules can also be added here to freeze in.

There is a file there called inisetup.py that appears to build the initial file system .py scripts on the device. I modify that to have my own bootstrap files written after a new flash. I have no idea if this is the "right" way to do this, but it works for me. So for instance my script in the setup sections reads:

def setup():
    check_bootsec()
    print("Performing initial setup")
    uos.VfsLfs2.mkfs(bdev)
    vfs = uos.VfsLfs2(bdev)
    uos.mount(vfs, "/")
    with open("boot.py", "w") as f:
        f.write(
            """
from machine import reset as rst
from machine import reset as reset



try:
    import utilities.firmware_update
    from utilities.unix_commands import *
    from utilities.sys_info import *
except:
    ...

"""
        )
    with open("firmware_update.py", "w") as f:
        f.write(
            """
# <code I can't share>


        )

Here is the batch snippet:

@echo off
pushd "%~dp0"

echo Building .mpy files with WSL
echo wsl.exe ~/build_epx_client_mpys.sh
wsl.exe ~/build_epx_client_mpys.sh
echo.

Here is the bash snippet:

#!/bin/bash
cd /mnt/c/Users/SStenzel/Desktop/name/esp32/client
for file in *.py; do ~/micropython/mpy-cross/build/mpy-cross $file; done
cd /mnt/c/Users/SStenzel/Desktop/name/esp32/client/classes
for file in *.py; do ~/micropython/mpy-cross/build/mpy-cross $file; done

This does mean I have a reasonably long bash file that adds my different directories, which I'm sure I could do better with some clever bash if only I knew some...

This post where I'm asking how to freeze assets into the firmware may also be useful to you:
https://github.com/orgs/micropython/discussions/10707

[sethstenzel]

I should have been clearer above, so I solved this in two steps.

1. Convert .py to .mpy files
2. Build a custom firmware that has a custom inistup.py which builds my initial file system how I want. It is important to point out though, that doing it this way, the initial boot.py and firmware_update.py are not .mpy files and therefore could be read, but since all they do is entry point the app code, I don't care.

[sethstenzel]

You may find this useful. I keep meaning to write my "1st" blog post with it, (well I mean that is mostly written, but my sites SSL cert expired and I have not had time to fix) but that hasn't happened, lol.
Keep in mind this is building micropython 1.19 for my esp32. I would assume many steps would be the same for other things, infact I think I build a .uf2 for my rpi2040 with this once.

Setting up a MicroPython firmware build environment on (WSL & Debian)

These instructions should work for both Windows Subsystem for Linux and Debian Linux distributions. Minor changes would need to be made for other *nix distributions and macOS. While it is theoretically possible to setup the toolchains in native windows, it is much more difficult and error prone to do so.
This tutorial assumes a fresh install of WSL or Debian based OS is being used.
When building custom MicroPython firmware and .mpy files, the ESP-IDF is a required dependency. The current version of ESP-IDF is in the 5.x version family. This is not currently compatible with the MicroPython build toolchain. When downloading the ESP-IDF be sure to use a 4.x version. Some MicroPython instructions may show downloading the most current ESP-IDF but this is not compatible currently.
Setup for building / compiling custom firmware.

1. Install required MicroPython dependencies
   Install the required dependencies for Linux:
   sudo apt-get install build-essential libffi-dev git pkg-config
2. Install required ESP-IDF dependencies
   Install the required dependencies for Linux:
   sudo apt-get install git wget flex bison gperf python3 python3-venv cmake ninja-build ccache libffi-dev libssl-dev dfu-util libusb-1.0-0 python-is-python3 pip
3. Get the ESP-IDF codebase:
   mkdir -p ~/esp
cd ~/esp
git clone -b v4.4 --recursive https://github.com/espressif/esp-idf.git
4. Setup the ESP-IDF tools:
   cd ~/esp/esp-idf
./install.sh esp32,esp32s2,esp32s3
5. Add an alias to your .bashrc file:
   echo "alias get_idf='. $HOME/esp/esp-idf/export.sh'" >> ~/.bashrc
echo "alias python='python3'" >> ~/.bashrc
6. Reload your session with .bashrc changes.
   cd ~/
source .bashrc
7. Check that the alias is working and you can activate the ESP-IDF session exports.
   This will need to be done any time you will build. You can also simply add . $HOME/esp/esp-idf/export.sh to your .bashrc file to always run.
   Test the alias is working:
   get_idf
8. Get the MicroPython codebase
   cd ~/
git clone https://github.com/micropython/micropython.git
9. Building the MicroPython cross-compiler
   Almost all ports require building mpy-cross first to perform pre-compilation of Python code that will be included in the port firmware:
   cd ~/micropython
make -C mpy-cross
   If you are only needing to convert .py files to .mpy files and don’t plan to build custom firmware you can stop here. Find the instructions for converting .py to .mpy files _HERE (I never ever made the post, will post below.)_
10. Test firmware build setup
    This will build a generic MicroPython firmware, this is to test and ensure the build toolchain is set up correctly.
    cd ~/micropython/ports/esp32
make submodules
make
    Once complete the firmware image should be located in: ~/micropython/ports/esp32/build-GENERIC/firmware.bin
11. Add new board definition for our ESP32 board.
    We will add a custom board definition for building custom images. To start we will copy the GENERIC_SPIRAM board definition and modify it.
    cd ~/micropython/ports/esp32/boards
cp -r GENERIC_SPIRAM/ LOLIN_D32_PRO_V2_SPIRAM/
12. Modify the sdkconfig.board file.
    Here we will add board specific options for our boards including the flash size, and the custom partition file to use.
    Open ~/micropython/ports/esp32/boards/LOLIN_D32_PRO_v2_SPIRAM/sdkconfig.board
    nano ~/micropython/ports/esp32/boards/LOLIN_D32_PRO_V2_SPIRAM/sdkconfig.board
    Under CONFIG_COMPILER_OPTIMIZATION_SIZE=y add the following and save:
    CONFIG_FLASHMODE_QIO=y
CONFIG_ESPTOOLPY_FLASHFREQ_80M=y
CONFIG_ESPTOOLPY_FLASHSIZE_DETECT=y
CONFIG_ESPTOOLPY_AFTER_NORESET=y
CONFIG_SPIRAM_MEMTEST=
CONFIG_ESPTOOLPY_FLASHSIZE_4MB=
CONFIG_ESPTOOLPY_FLASHSIZE_8MB=
CONFIG_ESPTOOLPY_FLASHSIZE_16MB=y
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions-16MiB.csv"
13. Modify the mpconfigboard.cmake file.
    Open ~/micropython/ports/esp32/boards/LOLIN_D32_PRO_v2_SPIRAM/mpconfigboard.cmake
    nano ~/micropython/ports/esp32/boards/LOLIN_D32_PRO_V2_SPIRAM/mpconfigboard.cmake
    Add the following two lines under boards/sdkconfig.spiram:
    boards/LOLIN_D32_PRO_V2_SPIRAM/sdkconfig.board
boards/sdkconfig.240mhz
14. Build the firmware
    cd ~/micropython/ports/esp32/
make submodules
make BOARD=LOLIN_D32_PRO_V2_SPIRAM
15. Move and rename firmware.
    mkdir ~/firmware
mv ~/micropython/ports/esp32/build-LOLIN_D32_PRO_V2_SPIRAM/firmware.bin ~/firmware/esp32spiram-1.19.1-custom-v1.bin
16. Clean the build environment for future use.
    Failing to do this step may cause future builds to fail.
    make clean BOARD=LOLIN_D32_PRO_V2_SPIRAM

References

https://github.com/micropython/micropython/blob/master/ports/esp32/README.md
https://blog.espressif.com/how-to-use-custom-partition-tables-on-esp32-69c0f3fa89c8
https://www.jarutex.com/index.php/2022/01/03/9190/

[sethstenzel]

The references sections may be the most useful to you.

[sethstenzel]

Converting .py files to .mpy files with mpy-cross

MicroPython defines the concept of an .mpy file which is a binary container file format that holds precompiled code, and which can be imported like a normal .py module. The file foo.mpy can be imported via import foo
These .mpy files can contain bytecode which is usually generated from Python source files (.py files) via the mpy-cross program. For some architectures, an .mpy file can also contain native machine code, which can be generated in a variety of ways, most notably from C source code.
Converting .py to .mpy is fairly strait forward. Any .py file can be compiled using the mpy-cross utility.
You will need a working MicroPython firmware build environment. Instructions for setting up a MicroPython build environment can be found Dead link reference post above...
Converting .py to .mpy is fairly straightforward. However, they can be compatibility issues with some .mpy files on certain systems. This means not all .mpy files you find online might be compatible with the board you are using. Review the MicroPython documentation on .mpy compatibility HERE for more information.

1. Create a directory for converted files.
   mkdir ~/py-to-mpy
2. Run the mpy-cross utility.
   cd ~/micropython/mpy-cross/build/
./mpy-cross ~/py-to-mpy/<script-name>.py
   A compiled .mpy file will be created in the same directory as the given .py file.

References

https://docs.micropython.org/en/latest/reference/mpyfiles.html

[Alain-Godo]

What a lot of information !!!! Woooow !!! It's going to be hard !!! But I hope not impossible jjjj I have a doubt here, the .mpy where are finally put? And I see that u modify the boot.py, but this firmware_update.py overwrites the default main.py?

[sethstenzel]

The inclusion of firmware_update.py may have been a mistake here as it could be misleading. That file has nothing to do with the microphone at all. It is my own application's auto-update feature. I wanted to show that you could write more than one file than just the boot.py.

Right now, I flash a bootstrap firmware with that update script and which downloads a compressed file containing all of my .mpy files and then uncompresses them into the correct file system folders instead of trying to add everything as frozen modules.

Something like the worst implementation of an OTA update I would imagine lol. But it works.

[Alain-Godo]

It's something like I'm building !!! In my case, an MQTT OTA updater from scratch.

[sethstenzel]

That is what I'm doing to. I'm having some trouble because none of the MQTT libs I've seen seem to support MQTT5.0 features which I need so I'm trying to re-write pieces of the paho-mqtt library for micropython. It's slow going.
But I'll be pushing updates out over MQTT as well. My compression/decompression utility might be useful to you if that is what you are doing. It takes about 100K of .mpy files and compresses down to about 74K. But my .py files are like 300K so just converting them over to .mpy saves a huge amount of space and speeds up loading.

[dhalbert]

For instructions on how to build CircuitPython with variations, we keep an Adafruit Learn Guide, with a lot of details, that requires frequent updating: https://learn.adafruit.com/building-circuitpython.

Given that MicroPython does not have a Learn Guide infrastructure, perhaps turning on the GitHub Wiki might create a place for this kind of thing?

[Alain-Godo]

Yes, I think it's time for an up-to-date guide step-by-step to create a custom MicroPython firmware. There is no clear information out there about the topic...In English, there is null good information about it....imagine in my native language: Spanish.

[jimmo]

@Alain-Godo the high level-answer is that any sort of customisation of the firmware build involves writing what we call a "board definition". As @ned-pcs linked to, the reference for this is https://github.com/micropython/micropython-example-boards but yes it definitely needs a lot more work. (And we need to link to this from the main documentation).

I do hope to be able to spend some time on this as part of the Google Season of Docs sponsored work in the next few months.

In your case you board definition needs to define frozen modules/packages by specifying them in the board's manifest.
https://docs.micropython.org/en/latest/reference/packages.html#freezing-packages

As already pointed out, even compiling to .mpy (and freezing) does not prevent someone getting access to your code, it just changes the level of difficulty. If all you want to do is make it more difficult then just disabling USB filesystem access might be enough.

@dhalbert We do have the GitHub wiki enabled -- https://github.com/micropython/micropython/wiki . Anyone with a GitHub account should be able to create/modify pages there.

[KeyTetra]

I learned a lot from this. Thank you.

[micropyt]

Hello, I have a problem to build firmware in ubuntu for esp32, especially with missing idf.py.

I download esp-idf
then /install.sh
then source export.h
then build idf.py in esp32 folder

I got this message:

`/mnt/c/Users/mkned/micropython/ports/esp32$ idf.py build

Executing action: all (aliases: build)
Running cmake in directory /mnt/c/Users/mkned/micropython/ports/esp32/build
Executing "cmake -G Ninja -DPYTHON_DEPS_CHECKED=1 -DPYTHON=/home/mknedlik/.espressif/python_env/idf5.2_py3.10_env/bin/python -DESP_PLATFORM=1 -DCCACHE_ENABLE=0 /mnt/c/Users/mkned/micropython/ports/esp32"...
-- IDF_TARGET not set, using default target: esp32
-- Found Git: /usr/bin/git (found version "2.34.1")
CMake Warning at esp-idf/tools/cmake/git_submodules.cmake:52 (message):
Git submodule
components/bootloader/subproject/components/micro-ecc/micro-ecc is out of
date. Run 'git submodule update --init --recursive' to fix.
Call Stack (most recent call first):
esp-idf/tools/cmake/build.cmake:80 (git_submodule_check)
esp-idf/tools/cmake/build.cmake:242 (__build_get_idf_git_revision)
esp-idf/tools/cmake/idf.cmake:53 (__build_init)
esp-idf/tools/cmake/project.cmake:29 (include)
CMakeLists.txt:22 (include)

CMake Warning at esp-idf/tools/cmake/git_submodules.cmake:52 (message):
Git submodule components/bt/controller/lib_esp32c6/esp32c6-bt-lib is out of
date. Run 'git submodule update --init --recursive' to fix.
Call Stack (most recent call first):
esp-idf/tools/cmake/build.cmake:80 (git_submodule_check)
esp-idf/tools/cmake/build.cmake:242 (__build_get_idf_git_revision)
esp-idf/tools/cmake/idf.cmake:53 (__build_init)
esp-idf/tools/cmake/project.cmake:29 (include)
CMakeLists.txt:22 (include)

-- The C compiler identification is GNU 12.2.0
-- The CXX compiler identification is GNU 12.2.0
-- The ASM compiler identification is GNU
-- Found assembler: /home/mknedlik/.espressif/tools/xtensa-esp32-elf/esp-12.2.0_20230208/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /home/mknedlik/.espressif/tools/xtensa-esp32-elf/esp-12.2.0_20230208/xtensa-esp32-elf/bin/xtensa-esp32-elf-gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /home/mknedlik/.espressif/tools/xtensa-esp32-elf/esp-12.2.0_20230208/xtensa-esp32-elf/bin/xtensa-esp32-elf-g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building ESP-IDF components for target esp32
CMake Error at esp-idf/tools/cmake/build.cmake:268 (message):
Failed to resolve component 'esp_adc_cal'.
Call Stack (most recent call first):
esp-idf/tools/cmake/build.cmake:304 (__build_resolve_and_add_req)
esp-idf/tools/cmake/build.cmake:599 (__build_expand_requirements)
esp-idf/tools/cmake/project.cmake:547 (idf_build_process)
CMakeLists.txt:58 (project)

-- Configuring incomplete, errors occurred!
See also "/mnt/c/Users/mkned/micropython/ports/esp32/build/CMakeFiles/CMakeOutput.log".
HINT: The component esp_adc_cal could not be found. This could be because: component name was misspelled, the component was not added to the build, the component has been moved to the IDF component manager or has been removed and refactored into some other component.
Please look out for component in 'https://components.espressif.com' and add using 'idf.py add-dependency' command.
Refer to the migration guide for more details about moved components.
Refer to the build-system guide for more details about how components are found and included in the build.
cmake failed with exit code 1, output of the command is in the /mnt/c/Users/mkned/micropython/ports/esp32/build/log/idf_py_stderr_output_4584 and /mnt/c/Users/mkned/micropython/ports/esp32/build/log/idf_py_stdout_output_4584`

Do you have any advice?

[ned-pcs]

Did you do what the warnings said to do?

git submodule update --init --recursive

[micropyt]

yes, I did this

[leosok]

There used to be a docker container for building custom firmware. I build my own on my other computer. Is there need for something like this?

[leosok]

Here you go: https://github.com/leosok/mp_builder/
May this be a good starting point. As said in the readme, I do not have time right now to test if everything still works. But the general idea did work and should be a good starting point for those not wanting to install everything.
In the long run, we could make a web-app from this :-)

[hemakumarm72]

https://www.horan.hk/blog/micropythonesp32/

try this 100 % work

[Alain-Godo]

Hi everyone !!! I come back with a new question related to the topic, this time a little more ambitious and really really interesting!!! Is possible to allow coexist MicroPython and a firmware (.bin) written in C/C++ on the same board? I'm talking about something like using the boot.py and then running a separate segment of memory with the Arduino compiled program. Is there someone with any insight about this? Because this opens a lot of doors!!

[hemakumarm72]

Yes, possible
https://youtu.be/437CZBnK8vI