Best practice for persistent storage?

[owenfromcanada]

I've gotten a feature request that would require storing some persistent data.

It looks like most of that is done through the Settings controller. The data I'm looking to store is minimal (a single int), but it's not really a setting. It looks like the Settings controller is using a file read/write under the hood--should I be implementing this separately, or use the Settings controller, or is there another solution already in place? Or do we generally discourage this sort of thing?

[mark9064]

I'm assuming you need long term storage i.e. across reboots, and that this is for an app. If you need storage just while the app is closed, then you can implement a small controller that stores the value in memory. Maybe #2141 would be useful as a reference?

You can store data in your own file on the flash. Have a look at what the alarm app does for example (alarm has a controller as well as the alarm runs in the background and therefore needs some storage in memory too, but if you don't need to run in the background no controller is needed). Try to avoid modifying the file too often, as the flash memory does not have infinite durability and the filesystem can cause quite a bit of read/write amplification

[owenfromcanada]

Thanks for this! Yeah, writing to flash repeatedly is what I figured we'd want to avoid. It looks like these parts can handle around 10k erase cycles?

I also noticed that #2314 is somewhat related to this. Would it maybe make sense to implement a layer between the fs controller and the applications, essentially doing what the settings controller does, but abstractly for any app to use? Or rather, moving the file system aspects of settings into a generic controller that would decide if/when to write to flash? There could be an explicit write command as well as a "soft" write command (where the controller could decide when to write them, such as on a soft reboot or flash, or after a certain delay).

If that's something that the team is open to, I can try putting some work into it.

[mark9064]

There's also #2171 which sounds like it's along the lines of what you're thinking?

Yeah I wouldn't worry about durability too much, the flash is rated for 100K erases IIRC and the filesystem is wear levelling. So I think writing a file every time an app exits would be generally OK

An abstraction layer sounds really nice. I'm not quite sure what it would look like but it sounds like you have some good ideas :) I think for a lot of apps memory persistence would be fine e.g. it would be nice for twos to have memory persistence, but flash persistence is probably unnecessary. Still, having the option for filesystem persistence would be great too

[owenfromcanada]

Yeah, #2171 is along those lines, though I was thinking of a slightly different implementation.

I was thinking of implementing a pattern I see in a lot of other systems, specifically database abstraction layers and how Android handles simple persistent storage (called SharedPreferences). The app calls the storage controller with a string parameter for the group of data (the string parameter could end up being the file name), then data is read or written in key-value pairs. When the data is being modified, the app calls one of two methods to finalize it - one of those methods writes the file immediately, the other writes it whenever the storage controller deems appropriate (to avoid flash fatigue). So if the data is super important, the app can write it immediately (and ensure it's not writing too often). If the data is less important (for example, a high score for a game), it can write it as often as it wants without worrying about it.

Under the hood, the data is stored in JSON or something similar. This is nice because the data can still be read even if your data structures change. But it also takes up more space than a binary format, and I'm not sure how concerned we are with data structure changes (at least until we implement sideloading).

But that might all be overkill. It might make sense to go with something more like #2171 and change it later if we find it's not flexible enough. I'll take a look at that other pull request and see what they did.