Disk I/O errors with OPFSCoopSyncVFS

[stevensJourney]

One of our PowerSync users recently reported disk I/O error errors when applying a relatively large dataset to a SQLite DB using the OPFSCoopSyncVFS VFS. The write operation is currently being executed by our PowerSync SDK while syncing rows to a client, but the error seems to be thrown from within the OPFSCoopSyncVFS jWrite method.

In this scenario the reported number of bytes written is much larger than the buffer provided to be written. From testing the reported number of bytes seems to consistently be reported as 4294967288 (when the error occurs), while the buffer size is fixed at 4096.

The PowerSync SDK eventually retries this operation and does eventually succeed to write the data to the VFS.

The reported user managed to create a relatively simple reproducible example here. The issue is not reproducible 100% of the time. I have only been able to intermittently reproduce, when using a fresh DB between retries.

I was able to capture the following logs when running the reproducible example in Chrome (running under Playwright)

Chromium | 139.0.7258.5 (Official Build) (arm64)
-- | --
Revision | fbf9452ec20f078528f7056a9b48d1f97a37fbd6-refs/branch-heads/7258@{#13}
OS | macOS Version 15.6 (Build 24G84)

I added some basic logs to the OPFSCoopSyncVFS jWrite method.

  jWrite(fileId, pData, iOffset) {
    try {
      const file = this.mapIdToFile.get(fileId);

      // On Chrome (at least), passing pData to accessHandle.write() is
      // an error because pData is a Proxy of a Uint8Array. Calling
      // subarray() produces a real Uint8Array and that works.
      const accessHandle = file.accessHandle || file.persistentFile.accessHandle;
      const nBytes = accessHandle.write(pData.subarray(), { at: iOffset });
      if (nBytes !== pData.byteLength) {
        debugger
        console.log(`${nBytes} !== ${pData.byteLength}`)
        console.log(iOffset)
        console.log(accessHandle)
        console.log(file)
        throw new Error('short write')
      };
      return VFS.SQLITE_OK;
    } catch (e) {
      this.lastError = e;
      return VFS.SQLITE_IOERR_WRITE;
    }
  }

I'm not quite sure if this might be a browser related issue, or under which circumstances the short write Error condition could be reached - so I thought I'd start this discussion as a starting point. Has anyone perhaps seen this before?

[rhashimoto]

You're using your own fork, right? Can you tell me whether you have the commits from #285 in your fork? Whatever version you have, can you try to reproduce the problem the other way - i.e. if you don't have those commits then add them, and if you have those commits then revert them.

That PR changes the wrapper for the jWrite buffer argument, so it's possible a bug was added or unknowingly fixed.

[stevensJourney]

Thanks for the quick response.

Correct, we do use our own fork in order to link our PowerSync SQLite extension.

We did not have the commits from #285 included in our fork yet. Thanks for recommending that. I've merged in those changes and tested a development @journeyapps/[email protected] NPM package in the linked reproducible repo.

The error still persisted with the upstream latest changes. It looks like the error is still the same as before.
(I did confirm that the latest code is being used, FacadeVFS.js contains DataViewProxy and Uint8ArrayProxy)

[rhashimoto]

Thanks for the info. So if this is a bug in the Uint8Array wrapper, then it's a bug that I preserved when I reimplemented it. Not ruling that out, but I think your result makes it less likely.

You mention:

I'm not quite sure if this might be a browser related issue, or under which circumstances the short write Error condition could be reached - so I thought I'd start this discussion as a starting point.

What platforms are you seeing this on? Does the test case launch a headless browser, or is it using a nodejs context? I haven't managed to run the test case yet - I develop inside a Docker container for security so I'll need to spin up an alternative run-time environment to make it work.

A quick look through the code hasn't given me any immediate thoughts on how a byteLength of 2^32 - 8 might be produced. I'll need to investigate further. But since it looks like nBytes is reasonable, you could try this patch as a temporary workaround. This might just briefly postpone failure but there's also a chance it could sidestep the problem until we can figure it out.

[stevensJourney]

Thanks for the useful feedback!

I've currently only tested on MacOS. The test case spawns and runs inside a "non-headless" (actual window) of Chromium via Vitest's Browser Mode. I can't think of any differences that might have to an actual browser instance.

I tested the patch linked above, and was still able to reproduce the issue.

I'm not sure if this is related, but might be worth mentioning - I was able to run the test 10 times successfully in a row with a clean browser state (by starting the test suite from scratch by exiting and invoking Vitest (and thus Chromium) between runs). I could relatively frequently reproduce the issue if I used a new SQLite DB between tests without restarting the browser (also without clearing the old databases before using a new one). I did check the navigator storage estimates when that failed, it didn't seem like the qouta was filled.

[rhashimoto]

A quick look through the code hasn't given me any immediate thoughts on how a byteLength of 2^32 - 8 might be produced.

I had this backwards. It is nBytes, the return value from FileSystemSyncAccessHandle.write(), that is 2^32 - 8. And that I have seen before here. It is a browser bug that Chrome is returning an error code cast to a 64-bit integer when it should be throwing some sort of exception, but the underlying cause is exceeding available disk space (either physical space or quota). I think your issue is related to this, and may be exactly this.

Is it possible that all these failures are happening in incognito mode? Note that the storage quota reported will not reflect the incognito limit of about 100 MB (Chrome intentionally lies to prevent easy incognito mode detection). That could also explain your clean runs when starting with fresh browser state.

It looks like vitest is using playwright. AI tells me that playwright launches the browser in incognito mode by default but it is possible to override that (I know nothing about playwright).

[stevensJourney]

I initially didn't think that the tests were running in incognito mode. Mostly because I'm used to the You’ve gone Incognito message when opening a blank tab in Chrome, but it seems like the Chromium version used by Vitest only displays New incognito tab new tab's name.

It does look like the tab is opened in incognito mode and it does seem like this is being caused by the storage limit mentioned in the issue above.

Thank you for all the insight and help!