How can I use all 16MB of my C5 flash for filesystem ?

[ullix]

I have an ESP32-C5-WROOM-1 with 16 MB flash. But default installation allows only 960kB for fs. How to use all of it?

I uploaded tasmota32c5.factory.bin. So far, all is working (except PWM, reported elsewhere #24396)

To increase file system to max size i installed Partition Wizard. This allowed me to increase the fs from the original 320kB to 960kB, but it does not allow anything beyond that. I had no such difficulties with my multiple ESP32-WROOM32E; their Flash of 4, 8, 16 MB was always fully available. All installations were always done with esptool.py.

[sfromis]

Did you verify that it actually has 16 MB of flash? It happens that sellers are not very conscientious about their product pages or what they ship.

Usually, Tasmota will automatically allocate the rest of available flash to the file system, like when I have an ESP32-C6 with 8 MB of flash. Did nothing extra to get a file system of 4416 KB, even with the default app0 partition size of 2880 KB. And the boards files for ESP32-C5 and ESP32-C6 have the same settings, except that ESP32-C5 has a higher CPU frequency.

You could check with tools like:
https://github.com/thelastoutpostworkshop/ESPConnect#readme
https://jason2866.github.io/esp32tool/

[ullix]

This is what Tasmota tells me:

[sfromis]

Then try Partition Manager instead of Partition Wizard. The latter was created for the more complex operation of upgrading setups from before the safeboot scheme, and the former only has an action button for "Resize filesystem". I'm not 100% sure if or how much difference it makes, but still worth a shot. If nothing else, it is a step in debugging what happens.

[ullix]

Where do I find it? It is not among the extensions?

[sfromis]

I used the UrlFetch command like this:

UrlFetch https://github.com/arendst/Tasmota/raw/development/tasmota/berry/modules/Partition_Manager.tapp

I was using an URL on my own local server, but when verifying with the URL above, Tasmota crashed. Not sure why that happened, but you can also download via that URL, and upload to the Tasmota file system.

Reboot is needed to have it start.

[ullix]

Nope. Same Non-performance as with the wizard; 960kB max

[s-hadinger]

Could you enter the following command in the Berry console and report back the result:

import partition_core print(partition_core.Partition())

[ullix]

Done:

import partition_core print(partition_core.Partition())
<instance: Partition([
<instance: Partition_info(1 (data),2 (nvs),0x00009000,0x00005000,'nvs',0x0)>
<instance: Partition_info(1 (data),0 (otadata),0x0000E000,0x00002000,'otadata',0x0)>
<instance: Partition_info(0 (app),0 (factory),0x00010000,0x000D0000,'safeboot',0x0)>
<instance: Partition_info(0 (app),16 (ota_0),0x000E0000,0x00230000,'app0',0x0)>
<instance: Partition_info(1 (data),130 (spiffs),0x00310000,0x000F0000,'spiffs',0x0)>
],
<instance: Partition_otadata(ota_active:ota_0, ota_seq=[1,0], ota_max=0)>

[sfromis]

Outside of the resized app0 and spiffs partitions, that's the same as I'm seeing on my "normal" ESP32-C6, except for me having ota_seq=[1,] (without the 0) at the bottom.

[Jason2866]

Dont have a 16MB C5. For the 8MB version it does work as expected

[ullix]

You seem to have a devel version (15.2.0.4)? I have the release (15.2.0). Could this account for anything?

[Jason2866]

Probably your device has not 16MB :-) Take a look in info page or if you dont trust Tasmota use esptool to see your real! flash size.

[sfromis]

I also though that, but a screenshot posted a couple of days ago says "Flash Size 16384 KB". If the chip does not have 16 MB "for real", that would be an issue in the Tasmota "Information" page.

[ullix]

... and esptool agrees:

:~$ esptool flash-id
esptool v5.1.0
Found 35 serial ports...
...
Connected to ESP32-C5 on /dev/ttyACM0:
Chip type:          ESP32-C5 (revision v1.0)
Features:           Wi-Fi 6 (dual-band), BT 5 (LE), IEEE802.15.4, Single Core + LP Core, 240MHz
Crystal frequency:  48MHz
MAC:                d0:cf:13:ff:fe:ed:a3:ac
BASE MAC:           d0:cf:13:ed:a3:ac
MAC_EXT:            ff:fe

Stub flasher running.

Flash Memory Information:
=========================
Manufacturer: 46
Device: 4018
Detected flash size: 16MB

Hard resetting via RTS pin...

By the way, WiFi-6 on this RISC V is a real value: I easily get more than twice the download speed of the classic's WiFi ! Will be very relevant once I get the full 16 MB.

[Jason2866]

Erase flash and try latest development. Cant explain why it should not work for 16MB when 8MB works correctly.

[ullix]

Which one?

[sfromis]

You can find latest development here https://ota.tasmota.com/tasmota32/ including .factory binaries for initial flashing.

[ullix]

This is now 'Tasmota 15.2.0.4 (7169312-tasmota32) by Theo Arends'. Same as before, no 16MB

[ullix]

I uploaded the latest devel Program Version 15.2.0.4 (16ec520-tasmota32) , and on my 16MB C5 it performes even worse:

• the Partition Manager does not allow any change of file system size beyond the default 320kB
• the Partition Wizard does not allow any change of file system size beyond 960kB
• and worse, when you select 960 kB and click ResizeFS the ESP C5 crashes and cannot be recovered by anything but a full esptool flash!

Strange for me: after reflashing my fs is still present, including all files I had stored there, and the ESP C5 boots right into my home WiFi

[sfromis]

Old fs and settings still present is normal, unless you erase all of flash before flashing a new binary. I'd try if doing this just might benefit the issue of getting all of the 16 MB available.

[ullix]

Nope.

Did a esptool erase-flash, then esptool'ed 15.2.0.4. After re-configuring home-wifi I had access to the C5. FS was 320kB, while info still showed that flash is 16MB.

Up-loaded Partition Manager-tapp. This would again NOT allow to set anything higher than the aleady set 320kB.

Then installed Partition-Wizard as extension. It allowed to raise setting from 320kB to 960kB, but nothing beyond. Semi-Good news: Clicking Resize FS did not crash! Fs is now 960kB and useable.

I see no option to reach 16 MB ( or 8, or 4, or 2)!

[sfromis]

Since the Partition Wizard downscaled the app0 partition size to the minimum to hold your binary, that seems to be where your ceiling of 960 KB comes from. As the numbers add up to 4 MB, this suggests that the tool does not recognize the additional flash. When the Partition Manager did not even allow to step above 320 KB, this is because this tool cannot downsize app0, only take advantage of available unallocated flash size to expand the file system.

If there's free space to spare, the Partition Manager will show lines like this, and suggest to resize the file system:

Partition size: 4096 KB
Max size: 12608 KB
Unallocated: 8512 KB

And the Partition Wizard will have a line like <free>: 8512 KB and have a button to "Resize FS to max".

Looks like the ESP32-C5 has "some difference" in flash handling leading those tools to only recognize 4 MB of flash, even if the "Information" page does report the full size 16 MB, and even if @Jason2866 found it to be working for an ESP32-C5 with 8 MB of flash.

On my ESP32-C6, tasmota.memory() shows this:
tasmota.memory() {'stack_size': 8, 'program_free': 720, 'program': 2159, 'flash_real': 8192, 'gc_time': 1, 'stack_low': 3.05078, 'iram_free': 0, 'flash': 8192, 'frag': 9, 'gc_heap': 245, 'heap_free': 231}
What does it say for you, especially values of flash and flash_real?

Those numbers are based on getting ESP chip info:
https://github.com/arendst/Tasmota/blob/2744a11c8a82207ae7f089c8495005cd80e02c98/tasmota/tasmota_xdrv_driver/xdrv_52_3_berry_tasmota.ino#L220C1-L221C72

Added: I observe a potentially important detail, on my working-as-expected ESP32-C6, the "Information" page reports 8192 KB for both Flash Size, and Program Flash Size, where these numbers do not match in your screenshot. Thus, it could be detection of that Program Flash Size value which is the key.

[ullix]

my ESP32-C5:

tasmota.memory()
{'stack_size': 8, 'program_free': 22, 'program': 2217, 'flash_real': 16384, 'gc_time': 2, 'stack_low': 3.36328, 'iram_free': 0, 'flash': 4096, 'frag': 69, 'gc_heap': 44, 'heap_free': 44}

[ullix]

The ESP32-C5 really has a big advantage in that it supports 2.4 and 5 GHz dual-band Wi-Fi 6 MCU!

While in the "classic" ESP32 I can transfer data by WiFi with up to 200 kB/s, I can more than double that on the C5. For big memory, like 16 MB - let alone that Espressif is working on 128MB - this is a very important speed advantage!

[Jason2866]

@ullix What flash size is shown when using this web browser tool?

[ullix]

Failure to run.

Your tool does not run on Firefox. But it does on my Chromium. But it failed:

There is a little complication: My C5 has 2 USB-C ports, and I am not sure what exactly they do. But I tried both; the first 2 in the screenshot are from one, the other 2 from the other. One of the two does NEVER show up among the drivers ls /dev/tty*, the other only as ls /dev/ttyACM0, never as ls /dev/ttyUSB*. Would your tool cover the ACM*?

[Jason2866]

As the log tells not allowed to access the serial port. You have to allow the browser to use the USB / Serial ports

[ullix]

And how?

[Jason2866]

You are using Linux :-) I am using a Mac. Google for udev rules

[sfromis]

On Linux, I'm doing this to grant serial access to myself, including when running Chrome:
sudo usermod -a -G dialout $USER

[ullix]

This is the port:

~$ ls -al /dev/ttyA*
crw-rw---- 1 root dialout 166, 0 Feb 11 09:28 /dev/ttyACM0

and I am already member of the dialout group.

~$ groups ullix
ullix : ullix adm dialout  ...

Yet access remains denied.

[sfromis]

Is your Chromium installed as a snap? In that case you may need to connect like this:
snap connect chromium:raw-usb
(I asked Gemini and got this suggestion. Sound plausible to me, but haven't tested)

Another suggestion was to stop the service ModemManager (or have a udev rule to have it ignore specific devices). In case of using Flatpak, the tool Flatseal should provide a toggle for "All system files" or "Devices". As already mentioned by arendst, udev rules is also a possibility.

[ullix]

That 's the trick! (Nice tool, I had no idea this is possible in a browser!)

It now shows:

ESP32Tool loaded.
Platform: Desktop (UA: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36...)
Using: Web Serial
USB-Serial: CH343 (VID: 0x1a86, PID: 0x55d3)
Max baudrate: 6000000
Connected USB Serial successfully with UnixTight reset.
Detected chip via IMAGE_CHIP_ID: 23 (ESP32-C5)
Not Espressif VID - external serial chip
USB connection type: External Serial Chip
Connected to ESP32-C5
Bootloader flash offset: 0x2000
MAC Address: D0:CF:13:ED:A3:AC
Loading stub for ESP32-C5, revision: null
Uploading stub...
Stub is now running...
Detecting Flash Size
FlashId: 0x184046
Flash Manufacturer: 46
Flash Device: 4018
Auto-detected Flash size: 16MB
Reader released in readLoop cleanup
Finished read loop
Serial buffers flushed
Changed baud rate to 153600

[sfromis]

And what do you see after clicking "Read Partition Table"?

[ullix]

Read Partition Table:

Reading partition table from 0x8000...
Serial buffers flushed
Reading 4096 bytes from flash at address 0x8000...
Reading chunk at 0x8000, size: 0x1000
SLIP read error at 0 bytes: Timed out waiting for packet content
Sent abort frame to stub
Drained 112 bytes from input buffer
Timed out waiting for packet content at 0x8000. Draining buffer and retrying (attempt 1/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
Invalid head of packet (0xFF) at 0x8000. Draining buffer and retrying (attempt 2/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
Invalid head of packet (0xFF) at 0x8000. Draining buffer and retrying (attempt 3/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
Invalid head of packet (0xFF) at 0x8000. Draining buffer and retrying (attempt 4/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
Invalid head of packet (0xFF) at 0x8000. Draining buffer and retrying (attempt 5/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
All retries exhausted at 0x8000. Attempting recovery (close and reopen port)...
Reconnecting serial port...
Reader released in readLoop cleanup
Finished read loop
Port closed
Opening port...
Classic reset.
Serial buffers flushed
Reconnect complete (chip: ESP32-C5)
Loading stub for ESP32-C5, revision: null
Uploading stub...
Stub is now running...
Stub loaded
Reader released in readLoop cleanup
Finished read loop
Serial buffers flushed
Changed baud rate to 153600
Reconnection successful
Deep recovery successful. Resuming read from current position...
Reading chunk at 0x8000, size: 0x1000
SLIP read error at 0 bytes: Timed out waiting for packet content
Sent abort frame to stub
Drained 112 bytes from input buffer
Timed out waiting for packet content at 0x8000. Draining buffer and retrying (attempt 1/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
Invalid head of packet (0xFF) at 0x8000. Draining buffer and retrying (attempt 2/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
Invalid head of packet (0xFF) at 0x8000. Draining buffer and retrying (attempt 3/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
Invalid head of packet (0xFF) at 0x8000. Draining buffer and retrying (attempt 4/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
Invalid head of packet (0xFF) at 0x8000. Draining buffer and retrying (attempt 5/5)...
Drained 112 bytes from input buffer
Serial buffers flushed
Error: Failed to read partition table: Error: Failed to read chunk at 0x8000 after 5 retries and recovery attempt

[ullix]

More complications: I had mentioned that my ESP32-C5 breakout board (https://www.amazon.de/dp/B0G34WYL8M) has 2 USB-C ports. One is labelled "USB", the other is labelled "UART". These are detected properties, when either one is connected:

"USB"   lsusb               Bus 003 Device 038: ID 303a:1001 Espressif USB JTAG/serial debug unit
        ls -al /dev/ttyA*   crw-rw---- 1 root dialout 166, 0 Feb 11 10:30 /dev/ttyACM0
        esptool             Does NOT find a port!

"UART"  lsusb               Bus 003 Device 039: ID 1a86:55d3 QinHeng Electronics USB Single Serial
        ls -al /dev/ttyA*   crw-rw---- 1 root dialout 166, 0 Feb 11 10:54 /dev/ttyACM0
        esptool             DOES find a port and allows flashing and else

Using @Jason2866 's web tool i get:

"USB":
Platform: Desktop (UA: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36...)
Using: Web Serial
USB-Serial: ESP32 Native USB (VID: 0x303a, PID: 0x1001)
Max baudrate: 2000000
Connected CDC/JTAG successfully with USB-JTAG/Serial reset.
Detected chip via IMAGE_CHIP_ID: 23 (ESP32-C5)
USB-JTAG/OTG detection: YES (PID=0x1001)
USB connection type: USB-JTAG/OTG
Connected to ESP32-C5
Bootloader flash offset: 0x2000
MAC Address: D0:CF:13:ED:A3:AC
Loading stub for ESP32-C5, revision: null
Uploading stub...
Stub is now running...
Detecting Flash Size
FlashId: 0x184046
Flash Manufacturer: 46
Flash Device: 4018
Auto-detected Flash size: 16MB
Reader released in readLoop cleanup
Finished read loop
Serial buffers flushed
Changed baud rate to 153600
Reading partition table from 0x8000...
Serial buffers flushed
Reading 4096 bytes from flash at address 0x8000...
Reading chunk at 0x8000, size: 0x1000
Total progress: 0x1000 from 0x1000 bytes
Found 5 partition(s)

and a very nice table:

while on "UART" it shows:

"UART":
Reader released in readLoop cleanup
Finished read loop
Port closed successfully
All cached data cleared
Platform: Desktop (UA: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36...)
Using: Web Serial
USB-Serial: CH343 (VID: 0x1a86, PID: 0x55d3)
Max baudrate: 6000000
Connected USB Serial successfully with UnixTight reset.
Detected chip via IMAGE_CHIP_ID: 23 (ESP32-C5)
Not Espressif VID - external serial chip
USB connection type: External Serial Chip
Connected to ESP32-C5
Bootloader flash offset: 0x2000
MAC Address: D0:CF:13:ED:A3:AC
Loading stub for ESP32-C5, revision: null
Uploading stub...
Stub is now running...
Detecting Flash Size
FlashId: 0x184046
Flash Manufacturer: 46
Flash Device: 4018
Auto-detected Flash size: 16MB
Reader released in readLoop cleanup
Finished read loop
Serial buffers flushed
Changed baud rate to 153600

I hope I am wrong, but I see that the bottom 4MB is detected - just as Tasmota does - but the remaining 12 MB remain hidden, with no option to activate?

[sfromis]

With the newer ESP32 variants on dev boards having a single USB, that corresponds to the one you have labeled "USB", and is the preferable one to use in most cases, like when it works better here.

The other "UART" is when having an additional separate USB/serial adapter (in this case CH343) chip to handle USB outside of the ESP32-C5, to instead have the extra adapter chip connect to serial pins on the MCU.

Still, it does look like esp32tool could do with a fix to better handle the edge case of going via UART instead of built-in USB support.

[ullix]

It's worse: I power my C5 via a USB-Brick, to avoid any computer to/from ESP-C5 interactions.

When the power is connected to "UART" all Tasmota stuff with my Berry-Code works as expected.
However, when connected to "USB", the C5 fails to boot properly, and never executes my Berry-Code.

Really troubling.

[sfromis]

What I meant by "preferable" was when wanting to have data connection. With power-only, do whatever works.

Still odd with it failing with power-only on "USB", but as Jason2866 noted, failing to boot does not look like being related to Tasmota. As you're not even having a data connection, you do not get much in details for actually debugging what happens.

[Jason2866]

The check with esp32tool shows the device has 16MB. No doubts. So the issue is somewhere in Tasmota code.

[Jason2866]

However, when connected to "USB", the C5 fails to boot properly, and never executes my Berry-Code.

Either bug in C5 hardware or in Arduino code. Both nothing Tasmota is responsible for

[ullix]

My guess: the JTAG/debug interferes?

"USB" lsusb Bus 003 Device 038: ID 303a:1001 Espressif USB JTAG/serial debug unit

The C5 is the first of Espressif's chips to support 5 GHz WiFi. This is very important once the 16MB - and higher - is available! 64 MB is already available, though AFAIK only on the classic ESP32
https://german.alibaba.com/product-detail/original-ESP32-WROOM-32E-ble-64Mb-1601111887587.html

[ullix]

Most Tasmota devices no need neither a big (flash-) memory, nor a fast data transfer speed. A Tasmota Geiger Counter is quite different. It wants big memory, and this must go hand-in-hand with big transfer rates.

The C% is well positioned. Once the big memory is accessible! But speed is already very pleasing. A comparison of the "classic" ESP32 with the C5 in my Tasmota Geiger Counter code:

### Download via WiFi
### ESP32-D0WD-V3 v3.0
#     Downloaded 857,624 Bytes in 4.3 sec  (199.9 kBytes/sec)
#     Downloaded 857,427 Bytes in 4.3 sec  (200.4 kBytes/sec)
#     Downloaded 857,626 Bytes in 4.3 sec  (200.9 kBytes/sec)
#     Downloaded 860,706 Bytes in 4.3 sec  (200.5 kBytes/sec)
#     Downloaded 566,832 Bytes in 2.9 sec  (193.8 kBytes/sec)

### ESP32-C5 v1.0
#     Downloaded 152,571 Bytes in 0.3 sec  (578.9 kBytes/sec)
#     Downloaded 152,558 Bytes in 0.3 sec  (572.6 kBytes/sec)
#     Downloaded 100,555 Bytes in 0.2 sec  (570.3 kBytes/sec)
#     Downloaded 152,568 Bytes in 0.3 sec  (565.0 kBytes/sec)
#     Downloaded 152,562 Bytes in 0.3 sec  (564.7 kBytes/sec)

500+ kB/s versus 200 kB/s! This is quite something!