esp-hal binaries (partition-table/application) don't seem to be compatible to new version of esp-idf bootloader.

[yanshay]

I want to flash my application not through esp-flash, but using another web based flashing tool that is easier to use for end users, and that is very commonly used.

I created the partition-table.bin, app-firmware.bin and bootloader.bin (modified for flash size and SPI mode based on espflash files for the relevant chip) and flashed them using (espflash / esptool.py) to the proper addresses and the application worked except for some minor issue of the bootloader stating on boot (because I didn't update the SHA).
But the key is that it worked.

Now, the bootloader files in espflash are pretty old - ESP-IDF v5.1-beta1-378-gea5e0ff298-dirt 2nd stage bootloader in the logs and I want to check if that other flashing need a more up to date version of boot loader, so I used latest esp-idf to create bootloader.bin and verified it is working on my device with an esp-idf application.

When I use that bootloader with the partition-table.bin and app-firmware.bin just like before, boot fails and keeps rebooting with the logs below (I generated bootloader with full debug logs), with key errors:

E (389) boot_comm: Image requires efuse blk rev >= v140.8, but chip is v1.3
E (397) boot: Factory app partition is not bootable
D (403) boot: Can't boot from zero-length partition
E (408) boot: No bootable app partitions in the partition table

So next I also used esp-idf to generate the partition-table.bin file to make sure it is compatible with esp-idf. The issues stayed the same.

So it seems to me like the application binary created by espflash is not compatible with new version of esp-idf (ESP-IDF v5.5-dev-1050-gb5ac4fbdf9 2nd stage bootloader). Because it work with the old esp-idf bootloader but doesn't with the new. Unless there is some parameter I need to set to make the new bootloader compatible with the application binary.

Should it be this way? Or am I limited to using the old esp-idf bootloader with esp-hal applications?

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x15 (USB_UART_CHIP_RESET),boot:0xa (SPI_FAST_FLASH_BOOT)
Saved PC:0x40379f96
SPIWP:0xee
mode:DIO, clock div:2
load:0x3fce2820,len:0x2738
load:0x403c8700,len:0x1018
load:0x403cb700,len:0x3b38
entry 0x403c8a20
I (34) boot: ESP-IDF v5.5-dev-1050-gb5ac4fbdf9 2nd stage bootloader
I (34) boot: compile time Jan 16 2025 12:57:33
I (34) boot: Multicore bootloader
D (39) bootloader_flash: non-XMC chip detected by SFDP Read (00), skip.
D (45) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (52) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (59) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (65) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (72) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (79) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
I (86) boot: chip revision: v0.2
I (90) boot: efuse block revision: v1.3
D (94) boot.esp32s3: magic e9
D (97) boot.esp32s3: segments 03
D (100) boot.esp32s3: spi_mode 02
D (104) boot.esp32s3: spi_speed 00
D (107) boot.esp32s3: spi_size 03
I (111) boot.esp32s3: Boot SPI Speed : 40MHz
I (116) boot.esp32s3: SPI Mode       : DIO
I (120) boot.esp32s3: SPI Flash Size : 8MB
D (125) boot: Enabling RTCWDT(9000 ms)
I (129) boot: Enabling RNG early entropy source...
D (135) bootloader_flash: rodata starts from paddr=0x00008000, size=0xc00, will be mapped to vaddr=0x3c000000
V (144) bootloader_flash: after mapping, starting from paddr=0x00000000 and vaddr=0x3c000000, 0x10000 bytes are mapped
D (155) boot: mapped partition table 0x8000 at 0x3c008000
D (161) flash_parts: partition table verified, 6 entries
I (166) boot: Partition Table:
I (170) boot: ## Label            Usage          Type ST Offset   Length
D (177) boot: load partition table entry 0x3c008000
D (182) boot: type=40 subtype=1
I (185) boot:  0 lic              unknown          40 01 00009000 00001000
D (193) boot: load partition table entry 0x3c008020
D (198) boot: type=40 subtype=1
I (201) boot:  1 map              unknown          40 01 0000a000 00010000
D (209) boot: load partition table entry 0x3c008040
D (214) boot: type=1 subtype=2
I (217) boot:  2 nvs              WiFi data        01 02 0001a000 00006000
D (224) boot: load partition table entry 0x3c008060
D (229) boot: type=1 subtype=1
I (232) boot:  3 phy_init         RF data          01 01 00020000 00001000
D (240) boot: load partition table entry 0x3c008080
D (245) boot: type=0 subtype=0
I (248) boot:  4 factory          factory app      00 00 00030000 00500000
I (256) boot: End of partition table
D (260) boot: Trying partition index -1 offs 0x30000 size 0x500000
D (266) esp_image: reading image header @ 0x30000
D (271) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (278) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (284) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (291) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (298) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (305) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (312) esp_image: image header: 0xe9 0x06 0x00 0x03 403788e0
V (318) esp_image: loading segment header 0 at offset 0x30018
D (324) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (330) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
V (337) esp_image: segment data length 0x754e8 data starts 0x30020
V (343) esp_image: MMU page size 0x10000
V (347) esp_image: segment 0 map_segment 1 segment_data_offs 0x30020 load_addr 0x3c1e0020
I (356) esp_image: segment 0: paddr=00030020 vaddr=3c1e0020 size=754e8h (480488) map
D (364) esp_image: free data page_count 0x00000200
D (369) bootloader_flash: rodata starts from paddr=0x00030020, size=0x754e8, will be mapped to vaddr=0x3c000000
V (379) bootloader_flash: after mapping, starting from paddr=0x00030000 and vaddr=0x3c000000, 0x80000 bytes are mapped
E (390) boot_comm: Image requires efuse blk rev >= v140.8, but chip is v1.3
E (398) boot: Factory app partition is not bootable
D (403) boot: Can't boot from zero-length partition
E (408) boot: No bootable app partitions in the partition table
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x3 (RTC_SW_SYS_RST),boot:0xa (SPI_FAST_FLASH_BOOT)
Saved PC:0x403cd505
SPIWP:0xee
mode:DIO, clock div:2
load:0x3fce2820,len:0x2738
load:0x403c8700,len:0x1018
load:0x403cb700,len:0x3b38
entry 0x403c8a20
I (33) boot: ESP-IDF v5.5-dev-1050-gb5ac4fbdf9 2nd stage bootloader
I (33) boot: compile time Jan 16 2025 12:57:33
I (34) boot: Multicore bootloader
D (38) bootloader_flash: non-XMC chip detected by SFDP Read (00), skip.
D (45) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (51) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (58) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (65) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (72) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (78) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
I (85) boot: chip revision: v0.2
I (89) boot: efuse block revision: v1.3
D (94) boot.esp32s3: magic e9
D (97) boot.esp32s3: segments 03
D (100) boot.esp32s3: spi_mode 02
D (103) boot.esp32s3: spi_speed 00
D (107) boot.esp32s3: spi_size 03
I (110) boot.esp32s3: Boot SPI Speed : 40MHz
I (115) boot.esp32s3: SPI Mode       : DIO
I (120) boot.esp32s3: SPI Flash Size : 8MB
D (125) boot: Enabling RTCWDT(9000 ms)
I (129) boot: Enabling RNG early entropy source...
D (134) bootloader_flash: rodata starts from paddr=0x00008000, size=0xc00, will be mapped to vaddr=0x3c000000
V (144) bootloader_flash: after mapping, starting from paddr=0x00000000 and vaddr=0x3c000000, 0x10000 bytes are mapped
D (155) boot: mapped partition table 0x8000 at 0x3c008000
D (160) flash_parts: partition table verified, 6 entries
I (166) boot: Partition Table:
I (169) boot: ## Label            Usage          Type ST Offset   Length
D (177) boot: load partition table entry 0x3c008000
D (182) boot: type=40 subtype=1
I (185) boot:  0 lic              unknown          40 01 00009000 00001000
D (192) boot: load partition table entry 0x3c008020
D (197) boot: type=40 subtype=1
I (201) boot:  1 map              unknown          40 01 0000a000 00010000
D (208) boot: load partition table entry 0x3c008040
D (213) boot: type=1 subtype=2
I (216) boot:  2 nvs              WiFi data        01 02 0001a000 00006000
D (224) boot: load partition table entry 0x3c008060
D (229) boot: type=1 subtype=1
I (232) boot:  3 phy_init         RF data          01 01 00020000 00001000
D (239) boot: load partition table entry 0x3c008080
D (244) boot: type=0 subtype=0
I (247) boot:  4 factory          factory app      00 00 00030000 00500000
I (255) boot: End of partition table
D (259) boot: Trying partition index -1 offs 0x30000 size 0x500000
D (265) esp_image: reading image header @ 0x30000
D (270) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (277) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (284) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (291) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (298) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (305) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (311) esp_image: image header: 0xe9 0x06 0x00 0x03 403788e0
V (317) esp_image: loading segment header 0 at offset 0x30018
D (323) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (330) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
V (337) esp_image: segment data length 0x754e8 data starts 0x30020
V (343) esp_image: MMU page size 0x10000
V (347) esp_image: segment 0 map_segment 1 segment_data_offs 0x30020 load_addr 0x3c1e0020
I (355) esp_image: segment 0: paddr=00030020 vaddr=3c1e0020 size=754e8h (480488) map
D (364) esp_image: free data page_count 0x00000200
D (368) bootloader_flash: rodata starts from paddr=0x00030020, size=0x754e8, will be mapped to vaddr=0x3c000000
V (379) bootloader_flash: after mapping, starting from paddr=0x00030000 and vaddr=0x3c000000, 0x80000 bytes are mapped
E (389) boot_comm: Image requires efuse blk rev >= v140.8, but chip is v1.3
E (397) boot: Factory app partition is not bootable
D (403) boot: Can't boot from zero-length partition
E (408) boot: No bootable app partitions in the partition table
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x3 (RTC_SW_SYS_RST),boot:0xa (SPI_FAST_FLASH_BOOT)
Saved PC:0x403cd505
SPIWP:0xee
mode:DIO, clock div:2
load:0x3fce2820,len:0x2738
load:0x403c8700,len:0x1018
load:0x403cb700,len:0x3b38
entry 0x403c8a20
I (33) boot: ESP-IDF v5.5-dev-1050-gb5ac4fbdf9 2nd stage bootloader
I (33) boot: compile time Jan 16 2025 12:57:33
I (34) boot: Multicore bootloader
D (38) bootloader_flash: non-XMC chip detected by SFDP Read (00), skip.
D (45) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (51) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (58) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (65) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (72) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (78) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
I (85) boot: chip revision: v0.2
I (89) boot: efuse block revision: v1.3
D (94) boot.esp32s3: magic e9
D (97) boot.esp32s3: segments 03
D (100) boot.esp32s3: spi_mode 02
D (103) boot.esp32s3: spi_speed 00
D (107) boot.esp32s3: spi_size 03
I (110) boot.esp32s3: Boot SPI Speed : 40MHz
I (115) boot.esp32s3: SPI Mode       : DIO
I (120) boot.esp32s3: SPI Flash Size : 8MB
D (125) boot: Enabling RTCWDT(9000 ms)
I (129) boot: Enabling RNG early entropy source...
D (134) bootloader_flash: rodata starts from paddr=0x00008000, size=0xc00, will be mapped to vaddr=0x3c000000
V (144) bootloader_flash: after mapping, starting from paddr=0x00000000 and vaddr=0x3c000000, 0x10000 bytes are mapped
D (155) boot: mapped partition table 0x8000 at 0x3c008000
D (160) flash_parts: partition table verified, 6 entries
I (166) boot: Partition Table:
I (169) boot: ## Label            Usage          Type ST Offset   Length
D (177) boot: load partition table entry 0x3c008000
D (182) boot: type=40 subtype=1
I (185) boot:  0 lic              unknown          40 01 00009000 00001000
D (192) boot: load partition table entry 0x3c008020
D (197) boot: type=40 subtype=1
I (201) boot:  1 map              unknown          40 01 0000a000 00010000
D (208) boot: load partition table entry 0x3c008040
D (213) boot: type=1 subtype=2
I (216) boot:  2 nvs              WiFi data        01 02 0001a000 00006000
D (224) boot: load partition table entry 0x3c008060
D (229) boot: type=1 subtype=1
I (232) boot:  3 phy_init         RF data          01 01 00020000 00001000
D (239) boot: load partition table entry 0x3c008080
D (244) boot: type=0 subtype=0
I (247) boot:  4 factory          factory app      00 00 00030000 00500000
I (255) boot: End of partition table
D (259) boot: Trying partition index -1 offs 0x30000 size 0x500000
D (265) esp_image: reading image header @ 0x30000
D (270) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (277) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (284) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (291) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (298) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (305) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (311) esp_image: image header: 0xe9 0x06 0x00 0x03 403788e0
V (317) esp_image: loading segment header 0 at offset 0x30018
D (323) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
D (330) bootloader_flash: mmu set block paddr=0x00030000 (was 0xffffffff)
V (337) esp_image: segment data length 0x754e8 data starts 0x30020
V (343) esp_image: MMU page size 0x10000
V (347) esp_image: segment 0 map_segment 1 segment_data_offs 0x30020 load_addr 0x3c1e0020
I (355) esp_image: segment 0: paddr=00030020 vaddr=3c1e0020 size=754e8h (480488) map
D (364) esp_image: free data page_count 0x00000200
D (368) bootloader_flash: rodata starts from paddr=0x00030020, size=0x754e8, will be mapped to vaddr=0x3c000000
V (379) bootloader_flash: after mapping, starting from paddr=0x00030000 and vaddr=0x3c000000, 0x80000 bytes are mapped
E (389) boot_comm: Image requires efuse blk rev >= v140.8, but chip is v1.3
E (397) boot: Factory app partition is not bootable
D (403) boot: Can't boot from zero-length partition
E (408) boot: No bootable app partitions in the partition table
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x3 (RTC_SW_SYS_RST),boot:0xa (SPI_FAST_FLASH_BOOT)
Saved PC:0x403cd505
SPIWP:0xee
mode:DIO, clock div:2
load:0x3fce2820,len:0x2738
load:0x403c8700,len:0x1018
load:0x403cb700,len:0x3b38
entry 0x403c8a20
I (33) boot: ESP-IDF v5.5-dev-1050-gb5ac4fbdf9 2nd stage bootloader
I (33) boot: compile time Jan 16 2025 12:57:33
I (34) boot: Multicore bootloader
D (38) bootloader_flash: non-XMC chip detected by SFDP Read (00), skip.
D (45) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (51) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (58) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (65) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (72) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
D (78) bootloader_flash: mmu set block paddr=0x00000000 (was 0xffffffff)
I (85) boot: chip revision: v0.2
I (89) boot: efuse block revision: v1.3
D (94) boot.esp32s3: magic e9
D (97) boot.esp32s3: segments 03
D (100) boot.esp32s3: spi_mode 02
D (103) boot.esp32s3: spi_speed 00
D (107) boot.esp32s3: spi_size 03
I (110) boot.esp32s3: Boot SPI Speed : 40MHz
I (115) boot.esp32s3: SPI Mode       : DIO
I (120) boot.esp32s3: SPI Flash Size : 8MB
D (125) boot: Enabling RTCWDT(9000 ms)
I (129) boot: Enabling RNG early entropy source...

[yanshay]

I managed to workaround the web flashing issue so this is not critical anymore, but in case of interest it still seems like some compatibility issue.