[Postgres] Optimize transaction replication throughput

[rkistner]

Previously, if we got 100k transactions each with a single operation, we'd commit/flush that 100k times to the bucket storage. Flushing is slow - in the case of a storage cluster under load, this can take 100-500ms per flush, resulting in throughput of less than 10 transactions per second.

Lucky for us, Postgres already chunks messages together in the replication stream. So now we look ahead in the current chunk to see if there are any more commit messages. If there are, we only flush/commit on the last one. This means if we get many transactions in a single chunk, they are all batched together in a single flush to the bucket storage.

For reference, we already have similar behavior in our MongoDB replication implementation.

In theory we could also batch transactions across multiple replication chunks, but I don't expect significant further gains from that, and it could increase complexity and memory usage.

This also adds Postgres 17 to the test matrix (released Sept 2024).

[changeset-bot]

🦋 Changeset detected

Latest commit: 2856a48

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
@powersync/service-module-postgres	Minor
@powersync/service-core	Minor
@powersync/service-image	Minor
@powersync/service-core-tests	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[Copilot]

Pull Request Overview

This PR optimizes transaction replication throughput by batching flushes, thereby reducing the performance overhead when handling many small transactions.

• The changeset file now reflects the new flush behavior for Postgres replication.
• The replication stream logic in WalStream.ts has been modified to batch multiple transactions by only flushing on the last commit message in a chunk.
• The test workflow has been updated to include Postgres version 17.

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
.changeset/chilly-flowers-fix.md	Updates changeset metadata to document the flush optimization change.
modules/module-postgres/src/replication/WalStream.ts	Refactors transaction handling to batch commit flushes per chunk.
.github/workflows/test.yml	Adds Postgres 17 to the testing matrix.

Comments suppressed due to low confidence (1)

modules/module-postgres/src/replication/WalStream.ts:707

• [nitpick] Consider renaming the variable 'skipKeepalive' to a name that more clearly reflects its purpose, such as 'inTransaction', for improved clarity when handling keepalive messages during transactions.

skipKeepalive = true;

[stevensJourney]

Looks like a nice optimisation to me :)