Postgres Compact Fix

[stevensJourney]

Overview

A bug was recently identified in the Postgres compacting logic. Compacting can fail due to an "unexpected PUT operation" error.

The Postgres compacting operations logic contained an error in the bucket operation processing logic.

The current Postgres windowing logic functions as follows:

• Bucket operations are processed in a moving window of batched operations.
• Each batch queries over all buckets.
• On the second batch, the window is still over all buckets, only limiting the last op_id.

Suppose the scenario:

• There are two buckets: bucket1 and bucket2, each with 8k ops.
• bucket1 is ops [1:8_000], bucket2is ops[8_001:16_000]`.
• Batch 1 returns 8k ops from bucket1, and 2k ops [14_001:16_000] from bucket2.
• For batch 2, the compactor queries for op_id < 14_001. It gets the 8k ops from bucket1 again. But since it was busy with bucket2, and now gets data for bucket1, it thinks bucket2 is done. The compactor does clearBucket('bucket2', 14123), even though there are more PUT operations in bucket2.

The updated Postgres compactor is now better aligned with the MongoDB implementation. The bucket data operations are sorted in descending order of bucket_name and op_id. The last operation of the batch is now correctly used as an upper limit when querying the next batches.

A unit test has been added to ensure correct compaction in the scenario above.

The compacting tests were also improved to use varying compacting options.

[changeset-bot]

🦋 Changeset detected

Latest commit: 801ea77

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 9 packages

Name	Type
@powersync/service-module-postgres-storage	Patch
@powersync/service-core-tests	Minor
@powersync/service-module-mongodb-storage	Patch
@powersync/service-core	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

Happy with the fix