Skip to content

Conversation

simolus3
Copy link
Contributor

This alters the ps_buckets table to add two new columns:

  • count_at_last: The amount of operations expected in a bucket, as received from the sync service. Set by client SDKs when completing a checkpoint.
  • count_since_last: The amount of operations downloaded since the last full checkpoint. This is set by the core extension when receiving an update batch.

Having this information available allows client SDKs to calculate the progress we've made (we can display the total target as new_checkpoint.$bucket.count - ps_buckets.$bucket.count_at_last and the current progress as ps_buckets.$bucket.count_since_last).

I have considered setting count_at_last in sync_local (by just adding count_since_last to it, and then resetting count_since_last). However, I didn't like the possibility of count_at_last ever going out of sync with the count value the sync service gives us for buckets. That shouldn't be possible because we only sync local data when we have received all operations, but applying the value straight from the source still sounds more reliable to me.

Since I sometimes loose sight of which table does what in the core extension, I've also started a document in docs/ that's supposed to explain the role of local data eventually. It only mentions parts of ps_buckets for now, maybe we can expand on that further in the future.

@simolus3 simolus3 requested a review from rkistner March 26, 2025 13:44
@simolus3 simolus3 merged commit 17b7f46 into main Apr 24, 2025
21 checks passed
@simolus3 simolus3 deleted the track-opcounts branch July 3, 2025 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants