Synchronous IDBMirrorVFS implementation

[Spxg]

There is also a related discussion in the previous post: #250.

Inspired by IDBMirrorVFS, I implemented a synchronous version of IDBMirrorVFS in rust. https://github.com/Spxg/sqlite-wasm-rs/blob/master/sqlite-wasm-rs/src/vfs/relaxed_idb.rs.

The principle is to preload the db into memory before xOpen, and then all operations are synchronous. When sqlite calls sync, it asynchronously writes the changed blocks to the indexed db through the indexed transaction. The difference from IDBMirrorVFS is that it does not support pragma synchronous=full at all;

As for performance, since both reading and writing are done in memory, the performance is good. However, we need to pay attention to the performance of preload the database, because the database is divided into multiple blocks and stored in the indexed db, and it takes some time to read all of them into memory.

I tested the preloading time of 4k, 8k, and 64k on a 1G file (In Macbook Pro m1max). The results are as follows (Different language implementations may differ):

It can be seen that as the size of the block increases, the reading time of each block increases, but as the number of blocks decreases, the overall time also decreases.

[rhashimoto]

Nice! Congratulations on getting that working, especially with the language barrier.

I don't know anything about Rust, but it looks like you're requesting all the blocks from IndexedDB at once to preload memory. That could be the fastest way because it makes the fewest IndexedDB calls, but I'm wondering if it might be faster to fetch a subset of the blocks at a time so you can overlap inserting blocks with fetching blocks (instead of fetching all the blocks before inserting). I don't think it can be dramatically faster because I expect insertion time to be much less than fetch time, but maybe there's a sweet spot where it would be a bit faster.

[Spxg]

you can overlap inserting blocks with fetching blocks (instead of fetching all the blocks before inserting).

Forgive me for not quite understanding what this means. In a synchronous environment, is there a way to not load all of the db into memory and still ensure that the correct data can be obtained when xRead is called? After all, xRead is also synchronous, and there is no way to load the block from idb at this time.

[rhashimoto]

I'm talking about reducing the preload time. As I understand it, this line will first fetch all the blocks from IndexedDB and after that will insert them into the data structure. If the insertion time is non-negligible (e.g. if it involves copying data) then you may be able to hide most of it by fetching the blocks in chunks and overlap insertion with fetching the next chunk.

[Spxg]

Thank you for your patient reply. I understand what you mean. After a simple test, the time taken to insert data is acceptable, 2µs per record on average.

You may have also noticed that I added a base test to insert 1g of data into the indexed db as one record and read it out. Without a doubt, this is the fastest.