Reading start of flash: Pyb is different?

[rkompass]

In an attempt to query the kind of filesystem on a board I use:

buf = bytearray(16)
and one of

bdev.readblocks(0, buf)     # esp8266 or esp32
rp2.Flash().readblocks(0, buf)
mimxrt.Flash().readblocks(0, buf)
w600.Flash().readblocks(0, buf)
pyb.Flash().readblocks(0, buf)

after importing rp2, mimxrt, w600 or pyb.

As a result I get buf=bytearray(b'\x01\x00\x00\x00\xf0\x0f\xff\xf7littlefs').
This works with all but the pyb versions (pyboard and Blackpill).
Even if those have perhaps FAT filesystem I would expect a signature different from
bytearray(b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00') there.
The pyb.Flash().readblocks(0, buf) also returns a 0 instead of nothing (the other boards), perhaps indicating that nothing was read into the buffer.
What is it that I am missing?

[robert-hh]

You may have a look at the fsinfo script by Damien, which he published a while ago. I did not find the link, but here is the code:

"""
Dump info about a filesystem.
"""

import sys, struct

def pr(addr, label, data):
    if type(data) == bytes or type(data) == bytearray:
        data = ':'.join('%02x' % b for b in data)
    print('%04x  %-32s %s' % (addr, label, data))

def decode_bootsec_fat(b):
    print('FAT filesystem')
    pr(0, 'jump', b[0:3])
    pr(3, 'OEM name', str(b[3:11], 'ascii'))
    pr(11, 'sector size (bytes)', struct.unpack_from('<H', b, 11)[0])
    pr(13, 'cluster size (sectors)', struct.unpack_from('<B', b, 13)[0])
    pr(14, 'reserved area (sectors)', struct.unpack_from('<H', b, 14)[0])
    pr(16, 'number of FATs', struct.unpack_from('<B', b, 16)[0])
    pr(17, 'size of root dir area', struct.unpack_from('<H', b, 17)[0])
    pr(19, 'volume size (sectors)', struct.unpack_from('<H', b, 19)[0])
    pr(21, 'media descsriptor', hex(struct.unpack_from('<B', b, 21)[0]))
    pr(22, 'FAT size (sectors)', struct.unpack_from('<H', b, 22)[0])
    pr(24, 'track size (sectors)', struct.unpack_from('<H', b, 24)[0])
    pr(26, 'number of heads', struct.unpack_from('<H', b, 26)[0])
    pr(28, 'volume offset (sectors)', struct.unpack_from('<L', b, 28)[0])
    pr(32, 'volume size (32-bit) (sectors)', struct.unpack_from('<I', b, 32)[0])
    pr(36, 'physical drive number', struct.unpack_from('<B', b, 36)[0])
    pr(37, 'error flag', hex(struct.unpack_from('<B', b, 37)[0]))
    pr(38, 'extended boot signature', hex(struct.unpack_from('<B', b, 38)[0]))
    pr(39, 'volume serial number', hex(struct.unpack_from('<L', b, 39)[0]))
    pr(43, 'volume label', str(b[43:51], 'ascii'))
    pr(54, 'filesystem type', str(b[54:62], 'ascii'))
    pr(510, 'signature', hex(struct.unpack_from('<H', b, 510)[0]))


def decode_bootsec_lfs1(b):
    print("Littlefs v1 filesystem")
    pr(24, "type", struct.unpack_from("<I", b, 24)[0])
    pr(25, "elen", struct.unpack_from("<I", b, 25)[0])
    pr(26, "alen", struct.unpack_from("<I", b, 26)[0])
    pr(27, "nlen", struct.unpack_from("<I", b, 27)[0])
    pr(28, "block_size", struct.unpack_from("<I", b, 28)[0])
    pr(32, "block_count", struct.unpack_from("<I", b, 32)[0])
    pr(36, "version", hex(struct.unpack_from("<I", b, 36)[0]))
    pr(40, "magic", str(b[40:40 + 8], 'ascii'))


def decode_bootsec_lfs2(b):
    print("Littlefs v2 filesystem")
    pr(8, "magic", str(b[8:8 + 8], "ascii"))
    pr(20, "version", hex(struct.unpack_from("<I", b, 20)[0]))
    pr(24, "block_size", struct.unpack_from("<I", b, 24)[0])
    pr(28, "block_count", struct.unpack_from("<I", b, 28)[0])
    pr(32, "name_max", struct.unpack_from("<I", b, 32)[0])
    pr(36, "file_max", struct.unpack_from("<I", b, 36)[0])
    pr(40, "attr_max", struct.unpack_from("<I", b, 40)[0])


def decode_bootsec(b):
    if b[40:48] == b"littlefs":
        decode_bootsec_lfs1(b)
    elif b[8:16] == b"littlefs":
        decode_bootsec_lfs2(b)
    else:
        decode_bootsec_fat(b)

def main():
    if sys.platform == 'pyboard':
        bootsec = bytearray(512)
        import pyb
        #sd = pyb.SDCard()
        sd = pyb.Flash(start=0)
        sd.readblocks(0, bootsec)
    elif sys.platform in ('esp8266', 'esp32'):
        import esp
        bootsec = bytearray(512)
        esp.flash_read(esp.flash_user_start(), bootsec)
    elif sys.platform == 'mimxrt':
        import mimxrt
        bootsec = bytearray(512)
        mimxrt.Flash().readblocks(0, bootsec)
    else:
        with open(sys.argv[1], 'rb') as f:
            bootsec = f.read(512)
    decode_bootsec(bootsec)

main()

[rkompass]

Thats perfect. Thank you.

I took rp2 and w600 into the board selection and tried the report on the different boards:
Everything is fine except for rp2, which reports

Littlefs v2 filesystem
0008  magic                            littlefs
0014  version                          0x6e656770
0018  block_size                       1869181023
001c  block_count                      544829486
0020  name_max                         3355639856
0024  file_max                         2030043136
0028  attr_max                         268435466

instead of

Littlefs v2 filesystem
0008  magic                            littlefs
0014  version                          0x20001
0018  block_size                       4096
001c  block_count                      74
0020  name_max                         255
0024  file_max                         2147483647
0028  attr_max                         1022

Mhmm??

Another closely related question:
Assume you want to have a function that reports about the different mounted file systems like df in unix does.
How can one get the info about the different mounted filesystems from within MP?

For example in case of the pyboard if there is the SD card you can still access flash, can't you? And they might have different file system types mounted in certain directories. How to get the relevant info about that?

[robert-hh]

Adding rp2 here yields:

Littlefs v2 filesystem
0008  magic                            littlefs
0014  version                          0x20000
0018  block_size                       4096
001c  block_count                      352
0020  name_max                         255
0024  file_max                         2147483647
0028  attr_max                         1022

The change for rp2 is:

    elif sys.platform == 'rp2':
        import rp2
        bootsec = bytearray(512)
        rp2.Flash().readblocks(0, bootsec)

[robert-hh]

Edit: But at a 2nd rp2 it completely fails, detecting a LFS file system as FAT. When dumping the content, it's neither LFS nor FAT. So it may be that when a LFS file system is in use since a long time, and wear leveling took place, that the Superblock pair (0, 1) has moved and the first sector has a different content.
I can replicate that as well for e.g. a SAMD board set, where the rarely used boards show fine content, and heavily ones fail.

[rkompass]

pyb.Flash().readblocks(0, buf) does not read correctly if the length of bytearray buf is less than 512. This was the problem.
Is this a bug? It at least is not documented.
The other filesystem info needed I can get with
stat = os.statvfs('/flash')
which should be the official way to query it.

[jimmo]

@rkompass

The requirement for the start kwarg for pyb.Flash is documented here: https://docs.micropython.org/en/latest/reference/filesystem.html#stm32-pyboard
(This is a historical quirk with the way USB MSC works on stm32. Unifying the way flash and filesystems are accessed is one of the major things planned for v2.0, it's been difficult to do so far because it's a breaking change)

The requirement that the buffer size is a multiple of the block size for readblocks is documented here: https://docs.micropython.org/en/latest/library/os.html#os.AbstractBlockDev.readblocks