Custom firmware ESP32 WROVER

[JessicaMBG]

So, I'm building a custom firmware for an ESP32 WROVER - E N8R8 following the steps that a user posted as an answer for this discussion https://github.com/orgs/micropython/discussions/11612
But when I try to build It as a "GENERIC-SPIRAM" the memory free I get using gc.mem_free is only 2164784, this is half I get when I use the downloaded firmware from the micropython.org
When I try to build as "ESP32_S2_WROVER" the firmware is not recognized as an ESP32 firmware and if I force it to flash, It doesn't boot
Is there any additional config that must do? Or am I doing something wrong?
I would appreciate any help :)

[bidrohini]

It seems like you are facing some issues while building a custom firmware for your ESP32 WROVER-E N8R8 board using Micropython. The difference in available memory and the firmware not being recognized as an ESP32 firmware might be due to configuration or compilation issues.

To help you troubleshoot and get a better understanding of the problem, let's go through some steps and possible solutions:

Double-check the configuration settings when building the firmware. Make sure that you have selected the correct target configuration for your ESP32 WROVER-E N8R8 board. The target configuration should match the specific variant of the ESP32 chip you are using.

Ensure that you are using the correct version of ESP-IDF (Espressif IoT Development Framework) corresponding to the Micropython version you are building. Different versions of ESP-IDF may have different memory configurations and compatibility with specific ESP32 chip variants.

Memory allocation and partitioning can significantly affect the available RAM. Check the memory partition configuration in your custom firmware build and compare it to the downloaded firmware from micropython.org. Make sure the memory partitioning matches the requirements of your board.

4. Check Build Flags:
   Verify if there are any specific build flags or options set in the downloaded firmware from micropython.org that you might be missing in your custom firmware build. These build flags could impact the memory configuration and behavior.

   Examine the build output logs for any warning or error messages. These logs can provide valuable information about potential issues during the build process.

If you are manually flashing the firmware, ensure that you are using the correct flashing procedure for your board. Double-check the flash size and the memory addresses used for flashing.

Enable debugging options during the firmware build to get more insights into the boot process. This can help identify the point at which the firmware fails to boot.

Reach out to the Micropython community or the specific ESP32 development community for support. The members in these communities might have experience with similar issues or provide guidance for custom firmware builds.

Remember that building custom firmware for embedded platforms like ESP32 requires attention to detail, and small configuration errors can lead to unexpected behavior. Be cautious when making changes to the firmware, as incorrect configurations can result in bricked devices.

Before flashing any firmware to your ESP32 board, always ensure you have a backup of the original firmware, and proceed with caution when experimenting with custom builds.

[JessicaMBG]

I still can't figure it out, I download the firmware from the micropython org and everything goes fine, but when I try to build the same one I get the same result as before, not get the total spi ram memory
I am flashing both firmwares the same way using esptool.py manually

[scruss]

please ignore bidrohini's answer: they've been warned about using ChatGPT to provide answers here before.

The ESP32_S2_WROVER firmware is for the ESP32-S2: a different chip from the one on your ESP32 WROVER. So it's not surprising it won't load

[JessicaMBG]

Yeah, I know that now lol, but I was using the GENERIC-SPIRAM, only tried the ESP32_S2_WROVER firmware because nothing was working :/

[scruss]

Which version of IDF are you using? It seems that with v.4.x, some builds only get access to half of the resources.

The latest ESP32 nightlies are built with IDF v.5.x

[JessicaMBG]

I'm using ESP-IDF v5.0.2

[shariltumin]

There was nothing wrong with your build. ESP32 can only access 4MB SPIRAM!
If you build firmware with CONFIG_LOG_DEFAULT_LEVEL_INFO=y, you will see these boot messages:

...
I (622) quad_psram: This chip is ESP32-D0WD
I (623) esp_psram: Found 8MB PSRAM device
I (623) esp_psram: Speed: 40MHz
I (624) esp_psram: PSRAM initialized, cache is in low/high (2-core) mode.
W (632) esp_psram: Virtual address not enough for PSRAM, map as much as we can. 4MB is mapped
...

In MP, the rest of the 4MB is not addressable (not used - wasted). There is nothing MicroPython can do, this is a hardware limitation of the ESP32. You can use the hmem API to address different banks of RAM, but this is not part of MicroPython.

$ strings firmware.bin | grep IDF
[0;32mI (%lu) %s: ESP-IDF:          %s
MicroPython-1.20.0-xtensa-IDFv5.0.2-with-newlib4.1.0

Anyway, this is what I have of RAM

MicroPython v1.20.0-328-g01c758e26-dirty on 2023-08-01; ESP32-WROVER with ESP32
>>> gc.mem_free()
2164944
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(4194304, 1974956, 1966080, 1974956), (240, 4, 0, 4), (7288, 4, 0, 4), (16648, 6180, 4096, 2256), (86960, 66880, 65536, 66880), (15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968)]
>>> esp32.idf_heap_info(esp32.HEAP_EXEC)
[(15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968), (276, 4, 0, 4)]
>>>

I managed to use all available SPIRAM on the ESP32-S3. If your projects require a lot of RAM, it is best to migrate to the ESP32-S3.

[JessicaMBG]

The thing is that when I use the downloaded firmware I get a higher Ram value
That's why I thought something was wrong

[jimmo]

See #11853 for more about SPIRAM on the latest builds and #12141 for the (in-progress) work to address it.

[shariltumin]

All MicroPython firmwares lower than or equal to v1.20.0-206 were built with ESP-IDF-4.x.x and sdkconfig.spiram uses CONFIG_SPIRAM_USE_MEMMAP=y. Newer firmwares are only built with ESP-IDF-5.0.2 and sdkconfig.spiram uses CONFIG_SPIRAM_USE_CAPS_ALLOC=y.

The esp32-wrover-n4r8 board has 8MB of PSRAM, but only 4MB is usable in the MicroPython esp32 port. See execution results below:

MicroPython v1.18 on 2022-01-17; ESP32 module (spiram) with ESP32
Type "help()" for more information.
>>> gc.mem_free()
4097024
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 0, 0, 0)]
>>> esp32.idf_heap_info(esp32.HEAP_EXEC)
[]

---

MicroPython v1.19.1 on 2022-06-18; ESP32 module (spiram) with ESP32
Type "help()" for more information.
>>> gc.mem_free()
4096848
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 0, 0, 0)]
>>> esp32.idf_heap_info(esp32.HEAP_EXEC)
[]

---

MicroPython v1.20.0 on 2023-04-26; ESP32 module (spiram) with ESP32
Type "help()" for more information.
>>> gc.mem_free()
4096848
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(240, 0, 0, 0)]
>>> esp32.idf_heap_info(esp32.HEAP_EXEC)
[]

---

MicroPython v1.20.0-327-gd14ddcbdb on 2023-07-27; ESP32 module (spiram) with ESP32
Type "help()" for more information.
>>> gc.mem_free()
2164928
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(4194304, 1974956, 1966080, 1974956), (240, 4, 0, 4), (7288, 20, 12, 4), (16648, 3344, 2688, 3184), (86960, 69700, 69632, 69700), (15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968)]
>>> esp32.idf_heap_info(esp32.HEAP_EXEC)
[(15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968), (664, 368, 352, 352)]

---

So for ESP32, only 4MB of PSRAM will be used, no matter how the MicroPython heap is allocated and how much more physical PSRAM a board has. It seems that to use the rest of the physical PSRAM, the himem API must be used.

[JessicaMBG]

This answer helped me understand the situation, thank you!
I created a new machine with ESP-IDF v4.4 and got the result of the downloaded firmware
But now I have a new question, I did lose some memory resources building with ESP-IDF v5.x, is that gonna be standard from now on? And is there a way to unlock It?

This is what I got with ESP-IDF v4.4:

MicroPython v1.20.0-329-g70c564324-dirty on 2023-08-03; ESP32 module (spiram) with ESP32
Type "help()" for more information.

gc.mem_free()
4096864
a = bytearray(310241024)
gc.mem_free()
950448

MPY: soft reboot
MicroPython v1.20.0-329-g70c564324-dirty on 2023-08-03; ESP32 module (spiram) with ESP32
Type "help()" for more information.

a = bytearray(410001000)
gc.mem_free()
96368

This is what I got with ESP-IDF v5.0.2:

MicroPython v1.20.0-329-g70c564324-dirty on 2023-08-03; ESP32 module (spiram) with ESP32
Type "help()" for more information.

gc.mem_free()
2164480
a = bytearray(310241024)
Traceback (most recent call last):
File "", line 1, in
MemoryError: memory allocation failed, allocating 3145728 bytes

[shariltumin]

Newer versions of MicroPython will be based on ESP-IDF-5.X.X only. The ESP-IDF-4.X.X will not be supported anymore. Supporting both ESP-IDF versions is too much work for the MP-Core developers and maintainers.

It is not quite correct to say that you have lost RAM with the new build. The main difference with the new firmware is that the ratio of RAM between the MicroPython heap and the IDF heap is 50/50. This is a fair assumption for the default distribution.

Now some users may want to have more RAM allocated to the MicroPython heap. Like you for example, your want as much SPIRAM (4MB) allocated to MicroPython heap. Some users, like me, want to have more SPIRAM reserved for the IDF heap because I need RAM for the camera framebuffer. Therefore, there is an ongoing effort on gc to support a dynamic heap allocation mechanism.

The memory mappings presented by Espressif are confusing. For an ESP32 board with SPIRAM, we have TotalRAM = PSRAM + SRAM, where PSRAM is the external SPIRAM and SRAM is the internal RAM. For an ESP32 board without SPIRAM, of course, TotalRAM = SRAM. Now 'full' MicroPython will run on ESP32 without SPIRAM, but with limited MicroPython heap size. So it is possible to allocate almost all SPIRAM to the MicroPython heap for SPIRAM-enhanced ESP32.

Espressif also mentioned DRAM and IRAM, Data RAM and Instruction RAM, respectively. The important thing to remember about RAM is not its total size, but the largest continuously free allocatable block.

MicroPython is an application built on top of the ESP-IDF architecture. It is just like any other application. Custom MicroPython builders are free to allocate as much or as little RAM to the application as necessary for the smooth running of their system. Keep in mind that WiFi, SSL, Bluetooth also need RAM.

I manage to build a firmware that give these results:

>>> gc.mem_free()
4032784
>>> import esp32
>>> esp32.idf_heap_info(esp32.HEAP_DATA)
[(32767, 32031, 31744, 32031), (4194304, 63364, 62464, 63364), (240, 4, 0, 4), (7288, 4, 0, 4), (16648, 2308, 1600, 2116), (86960, 36928, 36864, 36928), (15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968)]
>>> esp32.idf_heap_info(esp32.HEAP_EXEC)
[(15072, 14656, 14336, 14656), (113840, 112968, 110592, 112968), (264, 40, 32, 24)]

I need to modiify the main.c file:

    // size_t mp_task_heap_size = MIN(heap_caps_get_largest_free_block(caps), heap_total / 2);
    size_t mp_task_heap_size = MIN(heap_caps_get_largest_free_block(caps), heap_total);

Now 4MB of SPIRAM is still wasted due to the 4MB mapping limitation of ESP32. Unless we use the himem API. Using this API I managed to include a C module named himem to reclame 62321024 bytes out of the 4MB.

>>> import himem
>>> p = himem.pipe(62)
>>> p.size()
2031616
>>> 62*32*1024
2031616
>>> gc.mem_free()
4030576
>>> p.put('Hello')
5
>>> p.get(p.any())
b'Hello'
>>> p.
any             get             free            put
sget            size

Below is the full listing of my sdkconfig.board file.

CONFIG_IDF_TARGET="esp32"
CONFIG_IDF_TARGET_ESP32=y

# MicroPython on ESP32, ESP IDF configuration with 240MHz CPU
CONFIG_ESP32_DEFAULT_CPU_FREQ_80=
CONFIG_ESP32_DEFAULT_CPU_FREQ_160=
CONFIG_ESP32_DEFAULT_CPU_FREQ_240=y
CONFIG_ESP32_DEFAULT_CPU_FREQ_MHZ=240

# SPI RAM config
#
CONFIG_SPIRAM=y
CONFIG_SPIRAM_BOOT_INIT=y
CONFIG_SPIRAM_SIZE=-1
CONFIG_SPIRAM_IGNORE_NOTFOUND=n
CONFIG_SPIRAM_USE_MALLOC=y
CONFIG_SPIRAM_TYPE_AUTO=y
CONFIG_SPIRAM_SPEED_40M=y
CONFIG_SPIRAM_MEMTEST=y
CONFIG_SPIRAM_CACHE_WORKAROUND=y
# These are for himem API
CONFIG_SPIRAM_BANKSWITCH_ENABLE=y
# 1 - 62  ( default = 8 ) we have 4MB, set to max
CONFIG_SPIRAM_BANKSWITCH_RESERVE=62

# DEBUGGING
# -CONFIG_LOG_DEFAULT_LEVEL_VERBOSE=y
# -CONFIG_LOG_DEFAULT_LEVEL_DEBUG=y
CONFIG_LOG_DEFAULT_LEVEL_INFO=y
# -CONFIG_LOG_DEFAULT_LEVEL_WARN=y

#
# PARTITION
#
CONFIG_PARTITION_TABLE_CUSTOM=y
CONFIG_PARTITION_TABLE_CUSTOM_FILENAME="partitions.csv"

# SPIRAM increases the size of the firmware, use -Os to reduce it again to fit in iram
CONFIG_COMPILER_OPTIMIZATION_SIZE=y
CONFIG_FREERTOS_PLACE_FUNCTIONS_INTO_FLASH=y

[JessicaMBG]

Thank you so much, that was so enlightening! Could finally solve what I thought was my problem and learned a lot more
May I buy you a panettone for Christmas? (ɔ◔‿◔)ɔ ♥

[SL2021-Dev]

C module named himem

Where is this file and how to Incorporated into the build.