Bucket priorities

[simolus3]

This implements the bucket priorities proposal. When buckets with different priorities are involved:

1. We only guarantee consistency within each priority (clients see changes from higher-priority buckets before changes from lower-priority buckets, even if there is a causal relationship between them).
2. (not a concern for the sync service): For the highes priority, clients are allowed to upload local changes before they have received a complete checkpoint.

Priorities are currently declared with a static literal column named _priority in a data query, e.g.

bucket_definitions:
  global_todos:
    parameters: SELECT 0 as _priority;
    data:
      - SELECT * FROM todos
  global_lists: # implicit default priority 1
    data:
      - SELECT * FROM lists

At the moment, the implementation simply groups buckets into their priorities and then synchronizes each priority in a batch (instead of using a batch for all buckets like before).
An interesting improvement might be to interrupt lower-priority sync work when higher-priorities have new data. For instance, in a flow like:

1. New checkpoint sent to client.
2. No changed todos to send to client, immediately send partial complete message for priority 0.
3. Start syncing a large amount of lists rows.
4. Before being done with step 3, a new checkpoint with new todos comes in.

Here, we might want to stop sending rows for 4 (it's not that they're lost, the client has received the operations and we can later resume from that state) so that we can send the new checkpoint first.

[changeset-bot]

🦋 Changeset detected

Latest commit: b683952

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 10 packages

Name	Type
@powersync/service-core	Patch
@powersync/service-sync-rules	Minor
@powersync/service-core-tests	Patch
@powersync/service-module-mongodb-storage	Patch
@powersync/service-module-mongodb	Patch
@powersync/service-module-mysql	Patch
@powersync/service-module-postgres-storage	Patch
@powersync/service-module-postgres	Patch
@powersync/service-image	Patch
test-client	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[rkistner]

Looks good overall! Added some minor comments on the implementation.

If I understand the implementation correctly, it will effectively change nothing if bucket priorities are not specified, which is great.

For specifying bucket priorities, could you also add support for this syntax?

bucket_definitions:
  bulk_data:
    priority: 2
    parameters: select id as project_id from projects where ...

I think this would be simpler for the majority of cases, leaving the as _priority form only for cases where developers need more control.

[rkistner]

After looking at the consistency implications in the client implementation, I think it would be better to make the default priority 3 instead of 1.

If the client gets a partial checkpoint, we only sync PUT operations, ignoring REMOVEs, which changes the consistency properties a little. I think that's a decent trade-off in many cases, but the developer needs to be aware of it.

Now imagine a developer starts with only having a priority 1 bucket (the default). Now they want to sync some bulk data as well, so they add a new bucket with priority 2. The issue is that this now affected the consistency properties of bucket 1, despite not touching it in the sync rules.

While if we instead have the default as priority 3 (the lowest), it means the developer will have to explicitly modify the original bucket to assign a higher priority, making the implications more obvious.

Thoughts?

[simolus3]

Now imagine a developer starts with only having a priority 1 bucket (the default). Now they want to sync some bulk data as well, so they add a new bucket with priority 2. The issue is that this now affected the consistency properties of bucket 1, despite not touching it in the sync rules.

I agree that making the lowest priority the default is a good choice to make sure changing the priority of some buckets doesn't affect unrelated buckets 👍

[rkistner]

Happy to merge this.

As mentioned offline, there is a fix in f60c705 that should be included before releasing this.