SD card LittleFS driver for micropython

[jornamon]

Although LittleFS may not be the best choice for SD cards, I wanted to give it a try. I've adapted the existing SD card driver so it can be used with LFS (or any other file system that could depend on the block device driver extended interface).

You can check the driver out here: https://github.com/jornamon/mp-sdcard-littleFS

LFS is better suited for devices that can be addressed at byte level, not block level (like SD-cards), so it's slower than FAT in general, but you might still be interested in using it to take advantage of some of its features, like better reliability under power loss.

Without a cache, the performance is terrible, so I had to implement a simple block cache. Even though LFS is slower than FAT with SD cards, with the cache is quite usable. Additionally, by tweaking the cache parameters, you might even see a performance boost while using FAT in certain scenarios. You can see some test results here: https://github.com/jornamon/mp-sdcard-littleFS/blob/main/TEST_RESULTS.md

Feedback and contributions are welcome!

[rkompass]

Hello @jornamon,

did you notice this PR?
Is it relevant for you?
Your graphics are nice, I recommend to add units (MB/s for speed? KB for cache size?) I had to guess these.

[jornamon]

Hi @mendenm, thanks for the clarification. I had observed that multi-block operations were generally faster than a series of single-block operations, but I hadn't specifically tested whether multi-block commands for single-block operations were as efficient as dedicated single-block commands. It's enlightening to know this.

Your decision to eliminate single-block operations makes sense, especially in the context of reducing redundant code. However, in my experience, most filesystem requests, particularly with the FAT implementation used in MicroPython, involve single-block reads (all of them) and a significant portion of single-block writes—approximately 50% of the bytes transferred in some workloads, with the remainder being multi-block (8-block/4KB) writes. Without a caching mechanism, the advantages of multi-block commands seem minimal. This is even more pronounced with LFS, where there are no multi-block operation requests originating from the filesystem. That's one of the reason I implemented a cache layer for the LFS driver.

I'm keenly following your PR and plan to integrate your changes into this driver as soon as they are finalized.

Regarding the CRC routines, your optimization using viper and asm_thumb is impressive. It raises a question for me: did you choose viper from the start, or was it a response to a performance degradation observed after introducing CRC checks, aiming to minimize the impact? I'm curious because in my own experiments with enhancing cache policies, I've noticed that performance on these small MCUs can be significantly affected.

[mendenm]

@jornamon All the directory operations for LFS are, indeed, small transfers, but what I observed was that actual data transfer using write() and readinto() were multi-block transfers, if the buffers were bigger than one block (at least for FAT... I didn't try LFS on SD). That was the case I was trying to do best on.

The CRC discussion got so far out of hand it was fairly comical. The whole speed thing really was a side issue, with a number of us just poking around to see how fast we could make it. Yes, I did viper very early on, since the overall read speed was degraded more than a factor of two (maybe even a factor of 10, I forget) with pure-python CRCs. In the end, I note that most people do not seem to use the CRCs on data (only commands) so data CRCs were factored out, and the command CRCs left in pure python (since they are a tiny amount of effort).

[jornamon]

@mendenm It's insightful to hear about the dramatic performance drop with pure python CRC implementations. Deviating slightly from the LFS topic, I'm curious about the future plans for CRC handling, especially considering the challenges with pure Python performance and the absence of mpy code distribution in micropython-lib. How do you foresee addressing this issue?

To shed some light on filesystem usage patterns, I've compiled stats from a mixed read/write workload, highlighting the prevalence of single block calls.

readblocks and writeblocks call stats for FAT. Mixed workload:

------ Usage Stats ------
Key                                  Value
sdcard/rb                             1360 (single readblocks calls, -all-)
sdcard/wb                             1076 (total writeblocks calls)
sdcard/wb/multi                         35 (multi block calls)
sdcard/wb/multi/avg (bytes)           4081 (average bytes per multi block call)
sdcard/wb/single                      1041 (single writeblocks calls)

For LFS, under the same workload, the trend towards single block operations is even more dominant:

------ Usage Stats ------
Key                                  Value
sdcard/rb                            20251 (all single block reads)
sdcard/wb                             3940 (all single block writes)
sdcard/wb/single/avg (bytes)           126 (average bytes per single block write)

[mendenm]

@jornamon The future for CRC handling is that the CRC7 will remain in pure python, and if anyone want to use the CRC16 on data, they can provide a callback to do it. There were many options for the callbacks provided. Unfortunately, it is hard to put them in with the SDCard code, since the presence of any of the optimized version (viper more or put asm-thumb) break the way the automated build process works, so it cannot be done (or so I am told). There may be a future examples directory with a whole pile of the various CRC16 implementations.

In the end, it seems that one should really have the SD wired carefully enough that data CRCs just should never come up wrong, and this is probably why I don't see strong indications that they are widely used. The CRC16 issue is probably a secondary priority on this project.

[rkompass]

I found that with the ability to set the 16 32-bit words defining the CRC in crc32 of uzlib one can process all CRCs (both left- and right-shift) up to 32 bits with very little overhead. The processing speed would roughly be 60 % of maximum assembler speed due to given processing of 4-bit chunks instead of 8-bit chunks.
I will prepare a PR for this. With that one can implement library functions for the different CRC variants by computing and setting these 16 words and after-processing the resulting checksums. This is all due to symmetries in the CRC computations.