Bucket priorities #192
Bucket priorities #192
Conversation
🦋 Changeset detectedLatest commit: b683952 The changes in this PR will be included in the next version bump. This PR includes changesets to release 10 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
There was a problem hiding this comment.
Looks good overall! Added some minor comments on the implementation.
If I understand the implementation correctly, it will effectively change nothing if bucket priorities are not specified, which is great.
For specifying bucket priorities, could you also add support for this syntax?
bucket_definitions:
bulk_data:
priority: 2
parameters: select id as project_id from projects where ...I think this would be simpler for the majority of cases, leaving the as _priority form only for cases where developers need more control.
|
After looking at the consistency implications in the client implementation, I think it would be better to make the default priority 3 instead of 1. If the client gets a partial checkpoint, we only sync PUT operations, ignoring REMOVEs, which changes the consistency properties a little. I think that's a decent trade-off in many cases, but the developer needs to be aware of it. Now imagine a developer starts with only having a priority 1 bucket (the default). Now they want to sync some bulk data as well, so they add a new bucket with priority 2. The issue is that this now affected the consistency properties of bucket 1, despite not touching it in the sync rules. While if we instead have the default as priority 3 (the lowest), it means the developer will have to explicitly modify the original bucket to assign a higher priority, making the implications more obvious. Thoughts? |
I agree that making the lowest priority the default is a good choice to make sure changing the priority of some buckets doesn't affect unrelated buckets 👍 |
simolus3
force-pushed
the
feat/bucket-priorities
branch
from
b8b7836 to
7b1fcde
Compare
2 participants
This implements the bucket priorities proposal. When buckets with different priorities are involved:
Priorities are currently declared with a static literal column named
_priorityin a data query, e.g.At the moment, the implementation simply groups buckets into their priorities and then synchronizes each priority in a batch (instead of using a batch for all buckets like before).
An interesting improvement might be to interrupt lower-priority sync work when higher-priorities have new data. For instance, in a flow like:
todosto send to client, immediately send partial complete message for priority 0.listsrows.todoscomes in.Here, we might want to stop sending rows for 4 (it's not that they're lost, the client has received the operations and we can later resume from that state) so that we can send the new checkpoint first.