Skip to content

Conversation

rkistner
Copy link

sqlite3.dart has two main VFS implementations apart from the OPFS-based ones:

  1. InMemoryFileSystem
  2. IndexedDbFileSystem

The IndexedDbFileSystem loads the files into memory, and is then just an InMemoryFileSystem + background write support.

The InMemoryFileSystem keeps a Uint8Array buffer per file. When writing to the file, it grows the buffer by creating a new Uint8Array and copying the data over. This gets really slow when doing that one page at a time with a 10MB+ database size.

This adds a new GrowingByteBuffer implementation, that grows the size by a factor of 2 each time. The significantly reduces the overhead of resizing the buffer, at the cost of using more memory.

In my tests of syncing a 30MB / 50k rows database, this reduced the sync time from 120s to 10s with IndexedDbFileSystem, and 5.5s with InMemoryFileSystem. This is compared to around 5s with a :memory: database (not sure if there is any significant difference between that and InMemoryFileSystem after this change).

While it worked in my tests, this still has to be cleaned up and tested properly, before making an upstream PR.

@rkistner
Copy link
Author

rkistner commented Oct 21, 2024

Refactored and merged upstream in simolus3#258; will likely be released in v2.4.7.

@rkistner rkistner closed this Oct 21, 2024
@rkistner rkistner deleted the bytebuffer branch October 21, 2024 10:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant