Postgres Bucket Storage

[stevensJourney]

Overview

In some circumstances configuring a Postgres server for bucket storage might be simpler than a MongoDB replica set.

This PR introduces a new @powersync/service-module-postgres-storage module which registers the ability to use a Postgres database for sync bucket storage.

See the readme for some additional context.

Testing

The Postgres storage implementation currently passes all unit tests. Tests include the shared storage unit tests from @powersync/service-core-tests and the replicator tests for all the current replication modules where Postgres is used in addition to MongoDB as a storage provider.

This has also been tested in an application demo environment in the new Postgres Bucket storage self host demo from powersync-ja/self-host-demo#45.

Basic performance comparisons between MongoDB were made for testing purposes only. Running locally on an M1 Mac, an initial replication of 2 million rows was timed. Initial replication took 4 minutes and 20 seconds for Postgres and 3 minutes 59 seconds for MongoDB.

Memory allocations were monitored using a debugger while doing initial replication and streaming data to clients. No noticeable memory leaks were found.

Future work

Some of the storage implementation classes duplicate minor implementation between them. These could be improved by using abstract classes at some stage.

There is room for performance optimisations for both reads and writes. E.g. when using the test-client to fetch-operations for 2.5 million rows of a single global bucket on an M3 Macbook it takes about 25 seconds for MongoDB to respond with all the data, while Postgres takes 75 seconds.

[changeset-bot]

🦋 Changeset detected

Latest commit: 780770b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 13 packages

Name	Type
@powersync/service-module-mongodb-storage	Minor
@powersync/service-core-tests	Minor
@powersync/service-core	Minor
@powersync/service-module-postgres-storage	Minor
@powersync/service-module-postgres	Minor
@powersync/lib-service-postgres	Minor
@powersync/lib-services-framework	Minor
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-image	Patch
test-client	Patch
@powersync/service-rsocket-router	Patch
@powersync/lib-service-mongodb	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR