[QUESTION] What is the file state upon power down?

[amgross]

Hi,
I understand that power down in middle of file write not corrupts the file system, and that every file that syncd/closed will be exist after the power down.

From my experience, if I open file which creates it and write to it and then power down happens - the file will be exist but without any data.
Is this really the case and always? (full revert of any writes done without explicit call to sync, but created files exist although wasn't synced/closed. not matter how much was written and if inline etc.).

If indeed, it is surprising because it mean the file metadata is written twice - upon creation and upon close. And also it is very not convenient as if I see file after power down I need to check if it has content (can't assume that if it exist then its data written already).

The only documentation I found is:

littlefs/README.md

Lines 14 to 16 in adad0fb

	**Power-loss resilience** - littlefs is designed to handle random power
	failures. All file operations have strong copy-on-write guarantees and if
	power is lost the filesystem will fall back to the last known good state.

The term 'last known good state' is not clear enough to me

[Ictogan1]

See https://github.com/littlefs-project/littlefs/blob/master/lfs.h#L619

Takes a buffer and size indicating the data to write. The file will not actually be updated on the storage until either sync or close is called.

[amgross]

Hi @Ictogan1
According to my checks the creation of file does occur although there is no sync/close.
Am I missing something?

[geky]

Hi @amgross, this is intentional behavior, but you're not the first to be confused by it (#908). It's also not well documented, you're right this could be better.

What @Ictogan1 commented is arguably correct, writes do not update the file until sync/close. But lfs_file_open creates an initially zero-length file immediately. This results in zero-length files if you lose power before sync/close, which is not ideal.

---

The problem is interleaved code (more often when multithreaded):

lfs_file_t file;
lfs_file_open(&lfs, &file, "a", LFS_O_RDWR | LFS_O_CREAT | LFS_O_EXCL) => 0;
lfs_stat(&lfs, "a", &info) => ?;
lfs_mkdir(&lfs, "a") => ?;

lfs_file_write(&lfs, &file, "blablabla", 9) => 9;
lfs_file_close(&lfs, &file) => 0;

You could argue the file shouldn't exist yet, but then LFS_O_EXCL is more-or-less useless:

lfs_file_t file1;
lfs_file_open(&lfs, &file1, "a", LFS_O_RDWR | LFS_O_CREAT | LFS_O_EXCL) => 0;
lfs_file_t file2;
lfs_file_open(&lfs, &file2, "a", LFS_O_RDWR | LFS_O_CREAT | LFS_O_EXCL) => ?;

Creating zero-length files is not the greatest solution, but it at least mostly works.

The good news is there is an attempt to improve this in littlefs3 (#1111, unstable) via a new file type: LFS_TYPE_STICKYNOTE. Stickynotes behave like regular files, but are implicitly deleted if you lose power before sync/close.

---

If indeed, it is surprising because it mean the file metadata is written twice - upon creation and upon close.

This is true. Though littlefs breaks file metadata up into small parts that can be separately committed to a shared log. Each commit is a prog_size at minimum, which can get quite large on NAND flash, but on NOR flash it's incredibly cheap.

The first commit is also necessary to store the file name somewhere. Otherwise we'd need to ask the user to keep the string allocated until sync/close, which would be, uh, a hell of a library requirement to document.

littlefs3 is also adding lfs3_set, which can create the file in a single commit if your file fits in RAM.

I've also considered also adding something like LFS3_O_SET, which atomically creates a file based on what's in filecfg.buffer (which is how lfs3_set works internally), but decided against it as it risks being confusing with how it interacts with filecfg, and if you keep a new file open you're probably going to write+close later anyways.

[amgross]

Thank you @geky
Just sharing here my thoughts how to cope with such power down:

1. Adding attribute that sign the file is in middle of creation is not relevant, as the file created without attributes (they are written just upon sync/close). (This is similar to what will done in littlefs3...)
2. Having for each file attribute that sign it as finished and going over all the file system each mount, searching for files without this attribute and deleting it.
3. In case no file with 0 size expected, can go over the entire file system each mount and delete all 0 size files.
4. Give temp name/location for newly created files and move/rename them after writing finished. (after mount can delete the temp files exist)

@geky do you have other ideas before I am closing this issue?

[geky]

4. Give temp name/location for newly created files and move/rename them after writing finished. (after mount can delete the temp files exist)

This is likely the best solution at the moment, as it avoids needing to scan for incomplete files. If you have a bounded number of files, ignoring 0-length files at the application-level may be easier, but it depends on the application.

It's interesting to note in other filesystems rename is often the only guaranteed atomic operation, which is why some editors like vim rely on renaming temporary files.

1. Adding attribute that sign the file is in middle of creation is not relevant, as the file created without attributes (they are written just upon sync/close). (This is similar to what will done in littlefs3...)

This is a good point, atomically creating a file with an attr is not possible in littlefs's current API (without rename).

Even in littlefs3, stickynotes exist without attrs until first call to sync/close. Though it's unclear if this is something worth providing an API for (there is some discussion on replacing the current attr system with a RAM-backed lfs3_file_setattr/getattr, which would make atomic attr creation trickier).

[amgross]

Thank you very much @geky
You probably can close this issue, unless you want to leave it till documentation will get updated

[amgross]

A valid workaround to identify such power failures is to always create file with some attribute, and if the attribute not exist it means had power down upon creation.
It should be mentioned that reading attribute with opencfg not return any indication that attribute not exist, so need to write some wrong value to the attribute buffer before opencfg and check if it changed afterwards. (In different than lfs_getattr which return error if attribute not exist)