[MongoDB Storage] Only compact changed buckets / Indexed bucket_state

[rkistner]

#343 introduced pre-computing of bucket checksums as the final step of initial replication.

The issue is that the query for buckets that need checksum calculations (bucket_state collection) was not indexed (and could not be effectively indexed), causing timeouts when the instance has a large number of buckets. We've seen a case of 100 million buckets, where this step fails due to the timeout.

This tweaks the query to use a new partial index on bucket_state, which fixes these timeouts.

This also changes the normal compact process to only compact buckets with changes since the last compact. Compacting 100k+ modified buckets this way is still slow since we have around 3 mongodb queries/operations per bucket, but it should not be slower than what it was, and ignoring unmodified buckets should make a significant difference here.

[changeset-bot]

🦋 Changeset detected

Latest commit: 29817ec

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 11 packages

Name	Type
@powersync/service-module-mongodb-storage	Patch
@powersync/service-core	Patch
@powersync/service-image	Patch
@powersync/service-schema	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres	Patch
@powersync/service-core-tests	Patch
@powersync/service-module-core	Patch
@powersync/service-module-postgres-storage	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR