[POC] Dynamic Extension Loading

[stevensJourney]

Overview

This is a (rough) POC for loading our Rust SQLite extension dynamically instead of statically linking via a Git submodule.

This compiles WA-SQLite as a MAIN_MODULE which can dynamically link to SIDE_MODULEs. JS functions to enable loading of extensions and load extension binaries are added. Currently this modifies the main dist exports to support side modules. It should be possible to add an additional dist export, along side the existing exports, which enables modules.

This has been tested with the in synchronous build and in-memory filesystem. Currently the async build fails to bind to the sqlite_load_extension call. This needs additional investigation.

The call to sqlite3_load_extension currently makes a network request for the extension binary. This binary has been placed in the dist folder where it can be served for demos. This behaviour might change when other VFS's are used.

Our Rust SQLite extension was compiled by running

 RUSTFLAGS="--emit=llvm-bc -C linker=/usr/bin/true" cargo build -p powersync_loadable --profile wasm --no-default-features --features "powersync_core/static powersync_core/omit_load_extension sqlite_nostd/static sqlite_nostd/omit_load_extension" -Z build-std=panic_abort,core,alloc --target wasm32-unknown-emscripten

Then in the /target/wasm32-unknown-emscripten/release/deps directory the bytecode was linked to a binary with

emcc *.bc [path_to_shim].o  -o side_module.wasm -s SIDE_MODULE 

A shim was included to introduce functionality for

int __rust_no_alloc_shim_is_unstable = 0;

Since this seems to not be resolveable. Improving this requires additional investigation.

The demo can be started with yarn start. See the screenshot below for results:

[rkistner]

I got asyncify builds working using this command:

rm -f "target/wasm32-unknown-emscripten/wasm/deps/*" && RUSTFLAGS="--emit=llvm-bc -C linker=/usr/bin/true" cargo build -p powersync_loadable --profile wasm --no-default-features --features "powersync_core/static powersync_core/omit_load_extension sqlite_nostd/omit_load_extension" -Z build-std=panic_abort,core,alloc --target wasm32-unknown-emscripten && sh -c "cd target/wasm32-unknown-emscripten/wasm/deps && emcc -g *.bc ../../../../shim.bc -o ~/src/wa-sqlite/debug/side_module_async.wasm -sSIDE_MODULE -sASYNCIFY -sJSPI_IMPORTS=@../../../../asyncify_imports.json"

asyncify_imports.json:

[
  "sqlite3_close",
  "sqlite3_finalize",
  "sqlite3_open_v2",
  "sqlite3_prepare",
  "sqlite3_prepare16",
  "sqlite3_prepare_v2",
  "sqlite3_prepare16_v2",
  "sqlite3_prepare_v3",
  "sqlite3_prepare16_v3",
  "sqlite3_reset",
  "sqlite3_step",
  "sqlite3_exec"
]

[stevensJourney]

Superseded by #19