November 2021: a bit of go-ipcmsg, a bit of go-privsep and a ton of plakar · poolp.org

[poolpOrg]

It's 3:29 AM, I'm done writing this article that I started at 10 PM.

[Glandos]

Regarding SQLite safety, you should have a look at https://corecursive.com/066-sqlite-with-richard-hipp/ Especially https://corecursive.com/066-sqlite-with-richard-hipp/#billions-of-tests

Of course SQLite does write on top on your filesystem, so you'll have to rely on both of them. But my own professional experience with storing video surveillance images (~100 millions of them) told me that SQLite is very reliable. And since there are less files, the filesystem is much more reliable.

Anyway : great work on plakar!

[poolpOrg]

I initially had an irrational fear of storing backups in a single structured file until I had to admit that... it's what I already do with tarballs.

My other concern was that, years ago, SQLite didn't support concurrent writers and would lock the entire database when a process opened it for writing. This was ok for programs that do a lot of small writes without maintaining the lock too long, but not so much for a program like plakar which establishes a transaction and retains the lock until the entire snapshot is written... as it would prevent another snapshot from being performed until a pending one is finished... BUT this is what SQLite did years ago and I was stuck in the past: they introduced a WAL a ... 11 years ago.

Thanks for sharing your experience and feedback, it's going to be very helpful as even the SQLite code written in plakar for the local cache assumed this old behaviour and will be simplified 🖖🏻