Due to block size mismatch of NATIVE_BLOCK_SIZE_BYTES: Files on FAT formatted internal flash by Pycom ESP32 port have wrong filenames and cannot be opened under Micropython's port

[x4t-de]

Dear community,

following up the issue #11226, which I was able to solve, I now have the following problem: when I format the internal flash with FATFS (no problems with LittleFS) and create files under Pycom's ESP32 port (micropython sigfox 1.20.2r6), I am able to mount the filesystem, but filenames are all scrambled up and cannot be opened. The filenames have text from file content of one of the files, e.g. one file is named "-----BEG.IN", which is part of a certificate file placed in the flash. I can open a file with fileHandle = open("-----BEG.IN"), but on reading from the file, I get OSError: [Errno 2] ENOENT.
I compared both oofatfs libs, the source code is identical between Pycom and Micropython. All of the FATFS related configurations are also identical, despite MICROPY_FATFS_REENTRANT, which is disabled in Micropython. This might be an issue in the future, but currently I am testing on a single threaded environment in Micropython.

So it seems to me that some kind of mapping in the internal filesystem tables is not correct. Which additional confguration should I look for, that may has influence on how the lib interprets and reads data from the flash? Could the flash driver maybe have something to do with it?

[Josverl]

What are you trying to achieve?.
A possible alternative:
Can you copy the files off with the Picom firmware, reflash, and then re-upload to micropythons filesystem?

[x4t-de]

Hi Josverl,

thank you for your reply and sorry for my late one, I can only work part time on this project.
And also apologies for the missing goal in my question. The scenario is as follows: there are several devices in use, with important configurations saved on the internal flash. The devices shall get an OTA with a new firmware based on official/original Micropython ESP32 port from this Git repo. Some of them do not necessarily have an SD card as a temp buffer for internal data available. That's why moving data to intermediate memory and copy back unfortunately is currently not an option.

Because both ports use the exact same oofatfs lib, I can only imagine that there must be some kind of different configuration that is not just in ff.h or ffconf.h, because they are now also configured identically. But its hard to find other differences, because VFS implementation already is quite different. If you have any suggestions for files where configurations regarding FATFS are defined and where I can check differences, I would be most grateful.
Currently I am investigating differences in the filesystems basic config bytes stored at the beginning of the flash, in the hope that the name of such a config byte leads me to a configuration in the source code.

[x4t-de]

OK, I think I found it. Long story short: the diskio_ctl function returns a sector size, that is fixed and thus not necessarily the one configured in the file system, which was it in my case.

The long story:
In the provided inisetup.py, there is series of function to check the fs integrity. I used this function to check differences between a fs created with Micropython and Pycom, and discovered the first configuration issue with the different block size, see #11226 . Now I tried to find other differences and compared the config bytes of the FATFS in the first block of the flash. At first, I didn't wonder why

buf = bytearray(bdev.ioctl(5, 0))  # 5 is SEC_SIZE
bdev.readblocks(0, buf)

returned so many bytes. Today I checked the actual size of buf, which returned 4096. The block size for the FATFS in ffconf.h is configured to be 512. There is a #define NATIVE_BLOCK_SIZE_BYTES (4096) setting in esp32_partition.c that is used as a default if Partition.find(Partition.TYPE_DATA, label="vfs") is called without a provided block size. That will be the block_size (bdev.ioctl(5, 0) will return.

When I call it like this Partition.find(Partition.TYPE_DATA, label="vfs", block_size=512), reading from the flash works correctly. File names are correctly read (checked with os.listdir) and the content is also correct.

Could someone maybe please clarify if this is intended behaviour, and am I supposed to provide the block_size by myself, or could this be done automatically by the partition object? Using a fixed block size on a file system that may has a different or varying block size seems unexpected to me, but maybe I am missing something here? In theory, the block size can be read from the flash, but admittedly the file system must be known in advance to know where the setting can be read from.

Pycom's implementation also uses a fixed block size that is returned in the diskio_ctl function used by ff.c