Add FatFS and FatFSUSB - Wear-Leveled FTL based FAT filesystem for onboard flash (#2028)

* Add FatFS for onboard flash use/sharing of FAT

* Move all to "fatfs" namespace

The FatFS library defines commonly used names like WORD which could conflict
with user code otherwise.

* Restyle

* FTL-based, wear-leveling FatFS with USB export

Allow using FAT filesystem with onboard flash in a safer, wear-leveled
way and to export the onboard flash to the host as a USB drive for easy
data transfer.

* Update documentation

* Fix submodule reference

* Don't spellehcek ChaN FatFS files

* Disable FTL debugging

* More codespell skips

* Move to latest SPIFTL library

* Allow using raw flash instead of FTL

* Remove unneeded static FIL 4k allocation

* Expose FAT FS format configuration options

* Update documentation

* Remove USB partial flash rewrites

* Remove unneeded dups of FatFS sources

Leave the LICENSE.md and README.md to point to upstream.

* Clean up comments

Author: earlephilhower

.github/workflows/pull-request.yml

@@ -24,7 +24,7 @@ jobs:
     - name: Run codespell
       uses: codespell-project/actions-codespell@master
       with:
-        skip: ./ArduinoCore-API,./libraries/ESP8266SdFat,./libraries/Adafruit_TinyUSB_Arduino,./libraries/LittleFS/lib,./tools/pyserial,./pico-sdk,./.github,./docs/i2s.rst,./cores/rp2040/api,./libraries/FreeRTOS,./tools/libbearssl/bearssl,./include,./libraries/WiFi/examples/BearSSL_Server,./ota/uzlib,./libraries/http-parser/lib,./libraries/WebServer/examples/HelloServerBearSSL/HelloServerBearSSL.ino,./libraries/HTTPUpdateServer/examples/SecureBearSSLUpdater/SecureBearSSLUpdater.ino,./.git
+        skip: ./ArduinoCore-API,./libraries/ESP8266SdFat,./libraries/Adafruit_TinyUSB_Arduino,./libraries/LittleFS/lib,./tools/pyserial,./pico-sdk,./.github,./docs/i2s.rst,./cores/rp2040/api,./libraries/FreeRTOS,./tools/libbearssl/bearssl,./include,./libraries/WiFi/examples/BearSSL_Server,./ota/uzlib,./libraries/http-parser/lib,./libraries/WebServer/examples/HelloServerBearSSL/HelloServerBearSSL.ino,./libraries/HTTPUpdateServer/examples/SecureBearSSLUpdater/SecureBearSSLUpdater.ino,./.git,./libraries/FatFS/lib/fatfs,./libraries/FatFS/src/diskio.h,./libraries/FatFS/src/ff.cpp,./libraries/FatFS/src/ffconf.h,./libraries/FatFS/src/ffsystem.cpp,./libraries/FatFS/src/ff.h
         ignore_words_list: ser,dout
 
 # Consistent style

.gitmodules

@@ -37,3 +37,6 @@
 [submodule "libraries/http_parser/lib/http-parser"]
 	path = libraries/http-parser/lib/http-parser
 	url = https://github.com/nodejs/http-parser.git
+[submodule "libraries/FatFS/lib/SPIFTL"]
+	path = libraries/FatFS/lib/SPIFTL
+	url = https://github.com/earlephilhower/SPIFTL.git

README.md

@@ -253,6 +253,8 @@ The installed tools include a version of OpenOCD (in the pqt-openocd directory)
 * [http-parser](https://github.com/nodejs/http-parser) is copyright Joyent, Inc. and other Node contributors.
 * WebServer code modified from the [ESP32 WebServer](https://github.com/espressif/arduino-esp32/tree/master/libraries/WebServer) and is copyright (c) 2015 Ivan Grokhotkov and others.
 * [Xoshiro-cpp](https://github.com/Reputeless/Xoshiro-cpp) is copyright (c) 2020 Ryo Suzuki and distributed under the MIT license.
+* [FatFS low-level filesystem](http://elm-chan.org/fsw/ff/) code is Copyright (C) 2024, ChaN, all rights reserved.
+
 
 -Earle F. Philhower, III  
  [email protected]

docs/fatfsusb.rst

@@ -0,0 +1,42 @@
+FatFSUSB
+========
+
+When the onboard flash memory is used as a ``FatFS`` filesystem, the
+``FatFSUSB`` can be used to allow exporting it to a PC as a standard
+memory stick.  The PC can then access, add, and remove files as if the
+Pico was a USB memory stick, and upon ejection the Pico can access
+any new files just as if it made them itself.
+
+(Note, if you are using LittleFS then you need to use ``SingleFileDrive``
+to export a single file, not this class, because the PC does not
+understand the LittleFS disk format.)
+
+Callbacks, Interrupt Safety, and File Operations
+------------------------------------------------
+
+The ``FatFSUSB`` library allows your application to get a callback
+when a PC attempts to mount or unmount the Pico as a FAT drive.
+
+When the drive is being used by the Pico (i.e. any ``File`` is open for
+read or write, the ``FatFS`` is not ``end()`` -ed and still mounted,
+etc.) the host may not access it.  Conversely, while the host PC is
+connected to the drive no ``FatFS`` access by the Pico is allowed.
+
+Your ``driveReady`` callback will be called when the PC attempts to mount
+the drive.  If you have any files open, then this callback can report back
+that the drive is not yet ready.  When you complete file processing, the PC
+can re-attempt to mount the drive and your callback can return ``true`` .
+
+The ``onPlug`` callback will generally ``FatFS.end()`` and set a
+global flag letting your application know not to touch the filesystem until
+the flag is cleared by the ``onUnplug`` callback (which will also do a
+``FatFS.begin()`` call).
+
+Failing to close all files **and** ``FatFS.end()`` before granting the
+PC access to flash memory will result in corruption.  FAT does not allow multiple
+writers to access the same drive.  Even mounting and only reading files from
+the PC may cause hidden writes (things like access time, etc.) which would
+also cause corruption.
+
+See the included ``Listfiles-USB`` sketch for an example of working with
+these limitations.

docs/fs.rst

@@ -33,6 +33,7 @@ following include to the sketch:
     #include "LittleFS.h" // LittleFS is declared
     // #include <SDFS.h>
     // #include <SD.h>
+    // #include <FatFS.h>
 
 
 Compatible Filesystem APIs
@@ -48,10 +49,45 @@ SD card reader.
 SD is the Arduino-supported, somewhat old and limited SD card filesystem.
 It is recommended to use SDFS for new applications instead of SD.
 
-All three of these filesystems can open and manipulate ``File`` and ``Dir``
+FatFS implements a wear-leveled, FTL-backed FAT filesystem in the onboard
+flash which can be easily accessed over USB as a standard memory stick
+via FatFSUSB.
+
+All of these filesystems can open and manipulate ``File`` and ``Dir``
 objects with the same code because the implement a common end-user
 filesystem API.
 
+FatFS File System Caveats and Warnings
+--------------------------------------
+
+The FAT filesystem is ubiquitous, but it is also around 50 years old and ill
+suited to SPI flash memory due to having "hot spots" like the FAT copies that
+are rewritten many times over.  SPI flash allows a high, but limited, number
+of writes before losing the ability to write safely.  Applications like
+data loggers where many writes occur could end up wearing out the SPI flash
+sector that holds the FAT **years** before coming close to the write limits of
+the data sectors.
+
+To circumvent this issue, the FatFS implementation here uses a flash translation
+layer (FTL) developed for SPI flash on embedded systems.  This allows for the
+same LBA to be written over and over by the FAT filesystem, but use different
+flash locations.  For more information see
+[SPIFTL](https://github.com/earlephilhower/SPIFTL).  In this mode the Pico
+flash appears as a normal, 512-byte sector drive to the FAT.
+
+What this means, practically, is that about 5KB of RAM per megabyte of flash
+is required for housekeeping.  Writes can also become very slow if most of the
+flash LBA range is used (i.e. if the FAT drive is 99% full) due to the need
+for garbage collection processes to move data around and preserve the flash
+lifetime.
+
+Alternatively, if an FTL is not desired or memory is tight, FatFS can use the
+raw flash directly.  In this mode sectors are 4K in size and flash is mapped
+1:1 to sectors, so things like the FAT table updates will all use the same
+physical flash bits.  For low-utilization operations this may be fine, but if
+significant writes are done (from the Pico or the PC host) this may wear out
+portions of flash very quickly , rendering it unusable.
+
 LittleFS File System Limitations
 --------------------------------
 
@@ -115,8 +151,8 @@ second SPI port, ``SPI1``.  Just use the following call in place of
     SD.begin(cspin, SPI1);
 
 
-File system object (LittleFS/SD/SDFS)
--------------------------------------
+File system object (LittleFS/SD/SDFS/FatFS)
+-------------------------------------------
 
 setConfig
 ~~~~~~~~~
@@ -131,14 +167,22 @@ setConfig
     c2.setCSPin(12);
     SDFS.setConfig(c2);
 
+    FatFSConfig c3;
+    c3.setUseFTL(false); // Directly access flash memory
+    c3.setDirEntries(256); // We need 256 root directory entries on a format()
+    c3.setFATCopies(1); // Only 1 FAT to save 4K of space and extra writes
+    FatFS.setConfig(c3);
+    FatFS.format(); // Format using these settings, erasing everything
+
 This method allows you to configure the parameters of a filesystem
 before mounting.  All filesystems have their own ``*Config`` (i.e.
 ``SDFSConfig`` or ``LittleFSConfig`` with their custom set of options.
 All filesystems allow explicitly enabling/disabling formatting when
 mounts fail.  If you do not call this ``setConfig`` method before
 perforing ``begin()``, you will get the filesystem's default
-behavior and configuration. By default, LittleFS will autoformat the
-filesystem if it cannot mount it, while SDFS will not.
+behavior and configuration. By default, LittleFS and FatFS will autoformat the
+filesystem if it cannot mount it, while SDFS will not.  FatFS will also use
+the built-in FTL to support 512 byte sectors and higher write lifetime.
 
 begin
 ~~~~~

keywords.txt

@@ -8,13 +8,19 @@ Arduino	KEYWORD3		RESERVED_WORD
 # Datatypes (KEYWORD1)
 #######################################
 
+Dir	KEYWORD1
+File	KEYWORD1
 timeval	KEYWORD1
 time_t	KEYWORD1
 
 #######################################
 # Methods and Functions (KEYWORD2)
 #######################################
 
+openDir	KEYWORD2
+setConfig	KEYWORD2
+fileCreationTime	KEYWORD2
+
 setup1	KEYWORD2
 loop1	KEYWORD2
 
@@ -64,7 +70,6 @@ digitalReadFast	KEYWORD2
 
 enableDoubleResetBootloader	KEYWORD2
 
-Dir	KEYWORD2
 openDir	KEYWORD2
 next	KEYWORD2
 getLastWrite	KEYWORD2

libraries/FatFS/keywords.txt

@@ -0,0 +1,24 @@
+#######################################
+# Syntax Coloring Map
+#######################################
+
+#######################################
+# Datatypes (KEYWORD1)
+#######################################
+
+FatFS	KEYWORD1
+
+#######################################
+# Methods and Functions (KEYWORD2)
+#######################################
+
+format	KEYWORD2
+FatFSConfig	KEYWORD2
+setUseFTL	KEYWORD2
+setDirEntries	KEYWORD2
+setFATCopies	KEYWORD2
+
+#######################################
+# Constants (LITERAL1)
+#######################################
+

libraries/FatFS/lib/SPIFTL

@@ -0,0 +1,24 @@
+FatFs License
+
+FatFs has being developped as a personal project of the author, ChaN. It is free from the code anyone else wrote at current release. Following code block shows a copy of the FatFs license document that heading the source files.
+
+/*----------------------------------------------------------------------------/
+/  FatFs - Generic FAT Filesystem Module  Rx.xx                               /
+/-----------------------------------------------------------------------------/
+/
+/ Copyright (C) 20xx, ChaN, all right reserved.
+/
+/ FatFs module is an open source software. Redistribution and use of FatFs in
+/ source and binary forms, with or without modification, are permitted provided
+/ that the following condition is met:
+/
+/ 1. Redistributions of source code must retain the above copyright notice,
+/    this condition and the following disclaimer.
+/
+/ This software is provided by the copyright holder and contributors "AS IS"
+/ and any warranties related to this software are DISCLAIMED.
+/ The copyright owner or contributors be NOT LIABLE for any damages caused
+/ by use of this software.
+/----------------------------------------------------------------------------*/
+
+Therefore FatFs license is one of the BSD-style licenses, but there is a significant feature. FatFs is mainly intended for embedded systems. In order to extend the usability for commercial products, the redistributions of FatFs in binary form, such as embedded code, binary library and any forms without source code, do not need to include about FatFs in the documentations. This is equivalent to the 1-clause BSD license. Of course FatFs is compatible with the most of open source software licenses include GNU GPL. When you redistribute the FatFs source code with changes or create a fork, the license can also be changed to GNU GPL, BSD-style license or any open source software license that not conflict with FatFs license.

libraries/FatFS/lib/fatfs/LICENSE.txt

@@ -0,0 +1 @@
+FatFS home at http://elm-chan.org/fsw/ff/