Filesystem Corruption Error - Ways to Prevent Loss of Data?

[botos-e]

Dear community,

I wanted to discuss what the best solution is to deal with a file system corruption error and how to prevent data loss.

The following situation brought me to this topic:
I built a statemachine based uPy custom-firmware for an ESP32-D0WDQ6 based hardware that does some measurements and sends them via mqtt to a broker. For this, it needs to connect WLAN, of course. The WLAN credentials are entered by the user at first use and get stored to the internal filesystem. This is important data I can't afford to lose. I also store measurements in case if temporarily no network is available and do extensive logging with log rotation for debugging purposes. This is less important data which I can afford to lose sometimes. Currently I am testing with 30 boards. Everything works great so far - except that I get filesystem corruption errors once in a while - at roughly 30% of the boards after around 30k cycles.

The error is:

The filesystem appears to be corrupted. If you had important data there, you
may want to make a flash snapshot to try to recover it. Otherwise, perform
factory reprogramming of MicroPython firmware (completely erase flash, followed
by firmware programming).

So I was thinking about the following solutions:

1. I write the WIFI credentials to flash outside of the filesystem. In case of a filesystem corruption error I simply recreate the fs with

import os, machine
from flashbdev import bdev
os.VfsLfs2.mkfs(bdev)
os.mount(bdev, "/")

and accept to lose some measurement data and the logs. That is perfectly acceptable in my use case.
Question 1: How do I know where to write to the flash? And how do I write strings to the flash?

2. I create two partitions and use the first for the important, rarely changing data, e.g. the wifi credentials and use the second partition for the more frequent writes (data and logs). I assume that the first partition won't get corrupted and the second could simply be recreated with the code example from above. In this case I again accept to lose some measurement data and the logs.
Question 2: How do I create multiple partitions instead of one? Also how do I know how many free space is available for my partitions?

3. I create two partitions and use one as a backup partition and the other as "main storage". In this case I have good chances to recover all of my data.
Question 3: How do I check if/which partition is corrupted?

4. I try to recover the data.
Question 4: Is it possible and how is it possible to recover data from a corrupted fs on a running system?

I am looking forward for your answers and/or suggestions.

[andrewleech]

I think each board on esp port can have a partitions CSV that lists everything built internally.
You could split the main filesystem one for your board into two, compile and flash, then use https://docs.micropython.org/en/latest/library/esp32.html#esp32.Partition.find
to find and configure the second one

[botos-e]

Dear @andrewleech , thanks for your suggestion. Indeed, I found the corresponding csv that lists my partition table (found flash size with esptool.py --chip esp32 --port /dev/ttyUSB0 flash_id = 4MB, and used micropython/ports/esp32/partitions-4MiB.csv).

I altered the table from:

# Notes: the offset of the partition table itself is set in
# $IDF_PATH/components/partition_table/Kconfig.projbuild.
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x1F0000,
vfs,      data, fat,     0x200000, 0x200000,

to:

# Notes: the offset of the partition table itself is set in
# $IDF_PATH/components/partition_table/Kconfig.projbuild.
# Name,   Type, SubType, Offset,  Size, Flags
nvs,      data, nvs,     0x9000,  0x6000,
phy_init, data, phy,     0xf000,  0x1000,
factory,  app,  factory, 0x10000, 0x1F0000,
vfs,      data, fat,     0x200000, 0x100000,
vfs2,      data, fat,     0x300000, 0x100000,

After compiling and flashing I tried:

>>> import os
>>> fs_stat = os.statvfs("/")
>>> free_space = fs_stat[0] * fs_stat[4]
>>> free_space/1024
1012.0

So it looks like the size change worked. Then I tried:

>>> import esp32
>>> esp32.Partition.find(label=None)  # label=None matches any label
[<Partition type=0, subtype=0, address=65536, size=2031616, label=factory, encrypted=0>]

I got stuck here. Could you please give me hint how to proceed to find vfs2? Or did I do someting wrong?

[botos-e]

@davefes thanks for your input. A ramfs is a neat solution! Unfortunately my firmware goes to deepsleep at the end of every cycle. So as far as I understand, the contents of the ramfs would have to be written to the filesystem prior sleeping - otherwise the data would get lost, right? Therefore the ramfs solution does not fit to my finite statemachine. But I hope some day someone else will find this discussion and your approach will be useful for him or her.

Edit: Yes, I am aware of the limitation of the flash - sectors get faulty over time, so it is not suitable for frequent write operations. Hence a ramfs is the right solution for these cases. In my usecase I just want to be able to do the extensive logging in debug/test mode and switch off deug outputs later. That reduces write cycles to zero if WLAN is available. So I take the risk of a flash fault and my approach is to deal with the consequences instead of avoiding it.

[davefes]

Correct. Every hour I save to flash so at the least I only lose an hours data and save on flash writes

[andrewleech]

Could you please give me hint how to proceed to find vfs2

I haven't actually used partitions on esp myself, but I think it might work with esp32.Partition.find(label="vfs2")

[botos-e]

I've tried label="vfs2" - it returned an empty array.

esp32.Partition.find(type=esp32.Partition.TYPE_DATA)

was the right approach as suggested by @glenn20 below. That works.

[shariltumin]

I would have a try with [esp32.NVS] (https://docs.micropython.org/en/latest/library/esp32.html#esp32.NVS).

>>> import esp32
>>> nvs = esp32.NVS('A')
>>> nvs.set_blob('a', b'This is a test')
>>> nvs.set_blob('b', b'We will save this')
>>> nvs.commit()
>>> import machine
>>> machine.reset()

-----------

>>> import esp32
>>> nvs =esp32.NVS('A')
>>> nvs.get_blob('a', buf)
14
>>> buf[:14].decode()
'This is a test'
>>> l = nvs.get_blob('b', buf)
>>> buf[:l].decode()
'We will save this'

[botos-e]

I was not aware of the NVS, thats a great hint! It perfectly fits solution 1. So that would solve my problem with the WLAN credentials at the esp32 platform. Thank you for the code example, too.

[glenn20]

So it looks like the size change worked. Then I tried:

>>> import esp32
>>> esp32.Partition.find(label=None)  # label=None matches any label
[<Partition type=0, subtype=0, address=65536, size=2031616, label=factory, encrypted=0>]

esp32.Partition.find(), by default, searches only for app partitions. Use esp32.Partition.find(type=esp32.Partition.TYPE_DATA) to find your new data partitions. See https://docs.micropython.org/en/latest/library/esp32.html#esp32.Partition.find

[botos-e]

With esp32.Partition.TYPE_DATA it works, like you described! Thanks!

[mendenm]

@botos-e Have you considered an external fram for the cycling data? fram has a lifetime in the 10^10 - 10^12 cycle range, so it lives a lot longer than flash. It's much more expensive, but for short-term buffering you might not need a lot.

[peterhinch]

Device drivers for memory chips of various technologies (including FRAM) may be found here. You can trade endurance, cost, speed and capacity to find a solution for most applications.

[botos-e]

I am not able to change the board's design anymore. So Fram is no solution for me. Also I might just postpone the failure to a later time and still have the same problem with loss of the WIFI credentials data in future. Therefore using esp32.NVS or working with two partitions is a better alternative in my case

[karfas]

You can use the ESP32 RTC SRAM to save small amounts (up to ~3kB) of data.
An external SRAM might also an option (and is most likely cheaper than the FRAM recommended above).

I consider it bad practice to use the flash for (relative) frequent writes, as it is rated for 100k writes only (see the ESP32 datasheets), even when the LFS distributes writes over the whole partition.
However, it might also be better (for the filesystem) to write into a pre-allocated file with fixed size (so creating a kind of database).

[botos-e]

This is similar to esp32.NVS, therefore also a good approach. However, Ido't see the implementation of the RTC SRAM access in Micropython in the docs.

In general you are right, it is bad practice to use the flash for frequent writes because of the wear. In my case I need to do persistent logging for debugging purposes and using/wearing the flash seems an acceptable and straightforward option.

In which way would a pre-allocated file with fixed size be better for the flash? I did try to distribute writing to files by implementing the rotation of two files. I thought that would constantly change the used sectors. But didn't check what's happening in reality.

[karfas]

You are right. The machine.RTC.memory() docs still slumber in years old PRs #5199 and #7133, but you can access the RTC memory by other means (see e.g. https://github.com/karfas/upy-esp32-lib.git).

I can't prove "better" behaviour with the fixed-length file. This is only an idea coming from 30-years old handling of files in DOS.
Appending a file requires at least two writes (one to the file itself, one to the directory), so the impact on the flash might be minimized with the fixed-length file.
On the other hand, I have no idea whether LFS handles directory writes better than I imagine.