Sync bucket priorities

migration-guides/mongodb-atlas.mdx

@@ -45,7 +45,7 @@
 Here is a quick overview of the resulting PowerSync architecture:
 * **PowerSync Service** is available as a cloud-hosted service ([PowerSync Cloud](https://powersync.com/pricing)), or you can self-host using our Open Edition.
 * **Authentication**: PowerSync piggybacks off your app’s existing authentication, and JWTs are used to authenticate between clients and the PowerSync Service. If you are using Atlas Device SDKs for authentication, you will need to implement an authentication provider.
 * **PowerSync Client SDKs** use **SQLite** under the hood. Even though MongoDB is a “NoSQL” document database, PowerSync’s use of SQLite works well with MongoDB, since the [PowerSync protocol](/architecture/powersync-protocol) is schemaless (it syncs schemaless JSON data) and we dynamically apply a [client-side schema](/installation/client-side-setup/define-your-schema) to the data in SQLite using SQLite views. Client-side queries can be written in SQL or you can make use of an ORM (we provide a few [ORM integrations](https://www.powersync.com/blog/using-orms-with-powersync))
 * **Reads vs Writes**: PowerSync handles syncing of reads differently from writes.
    * **Reads**: The PowerSync Service connects to your MongoDB database and replicates data in real-time to PowerSync clients. Reads are configured using PowerSync’s [“Sync Rules”](/usage/sync-rules/). Sync Rules are more flexible than MongoDB Realm Flexible Sync, but are defined on the server-side, not on the client-side.
    * **Writes**: The client-side application can perform writes directly on the local SQLite database. The writes are also automatically placed into an upload queue by the PowerSync Client SDK. The SDK then uses a developer-defined `uploadData()` function to manage the uploading of those writes sequentially to the backend.
@@ -54,7 +54,7 @@
    * **Writes**: The backend controls authorization for how users can modify data.
 * **Backend**: PowerSync requires a backend API interface to upload writes to MongoDB. There are currently two options:
    * **Custom self-hosted backend**: If you already have a backend application as part of your stack, you should use your existing backend. If you don’t yet have one: We have Node.js, Django and Rails [example implementations](/resources/demo-apps-example-projects#custom-backend-examples) available.
    * **Serverless cloud functions (hosted/managed)**: An alternative option is to use CloudCode, a serverless cloud functions environment provided by PowerSync. We have a template available that you can use as a turnkey starting point.
 
 
 ## Migration Steps
@@ -83,7 +83,7 @@
 
 If you have a PowerSync Service instance set up and connected, open the `sync-rules.yaml` file associated with your PowerSync project and edit the SQL-like queries based on your database schema. Below is a simple Sync Rules example using a simple database schema. Sync Rules involve organizing data into ["buckets"](/usage/sync-rules/organize-data-into-buckets) (a bucket is a grouping of data). The example below uses a ["global bucket"](/usage/sync-rules/example-global-data) as a simple starting point — data in a "global bucket" will be synced to all users.
 
 <Info>Note that MongoDB uses “_id” as the name of the ID field in collections whereas PowerSync uses “id” in its client-side database. This is why `SELECT _id as id` should always be used in the data queries when pairing PowerSync with MongoDB.</Info>
 
 ```yaml
 bucket_definitions:
@@ -443,16 +443,16 @@
 
 MongoDB Atlas Device Sync provides built-in writes/uploads to the backend MongoDB database.
 
-PowerSync offers full custommizability regarding how writes are applied. This gives you control to apply your own business logic, data validations, authorization and conflict resolution logic.
+PowerSync offers full customizability regarding how writes are applied. This gives you control to apply your own business logic, data validations, authorization and conflict resolution logic.
 
 There are two options:
 
 * **Serverless cloud functions (hosted/managed)**: PowerSync offers serverless cloud functions hosted on the same infrastructure as PowerSync Cloud which can used for the needed backend functionality needed. We provide a MongoDB-specific template for this which can be used as a turnkey solution.
 * **Custom self-hosted backend**: Alternatively, writes can be processed through your own backend.
 
 #### Using PowerSync’s serverless cloud functions
 
 PowerSync provides serverless cloud functions for backend functionality, with a template available for MongoDB. See the [step-by-step instructions](/usage/tools/cloudcode) on how to use the template. The template can be customized, or it can be used as-is.
 
 The template provides [turnkey conflict resolution](https://www.powersync.com/blog/turnkey-backend-functionality-conflict-resolution-for-powersync#turnkey-conflict-resolution) which roughly matches the built-in conflict resolution behavior provided by MongoDB Atlas Device Sync.
 

usage/use-case-examples/prioritized-sync.mdx

@@ -3,48 +3,164 @@
 description: "In some scenarios, you may want to sync tables using different priorities. For example, you may want to sync a subset of all tables first to log a user in as fast as possible, then sync the remaining tables in the background."
 ---
 
-<Info>
-  Note that this strategy is specifically to prioritize data on initial sync, and cannot be used for incremental sync after that.
-</Info>
+# Overview
 
-## Overview
+PowerSync supports defining [Sync Bucket](/usage/sync-rules/organize-data-into-buckets) Priorities, which allows you to control the sync order for different data sets. This is particularly useful when certain data should be available sooner than others.
 
-The general approach is as follows:
+# Why Use Sync Bucket Priorities?
 
-1. Define how many priority types you want - typically only two are needed: "high priority" and "the rest"
+PowerSync's standard sync protocol ensures that:
+- The local data view is only updated when a fully consistent checkpoint is available.
+- All pending local changes must be uploaded, acknowledged, and synced back before new data is applied.
 
-2. Create a sync bucket for each priority type
+While this guarantees consistency, it can lead to delays, especially for large datasets or continuous client-side updates. Sync Bucket Priorities provide a way to speed up syncing of high-priority data while still maintaining overall integrity.
 
-3. Use [client parameters](/usage/sync-rules/advanced-topics/client-parameters) to control which priorities you want the client to sync
+# How It Works
 
-## Example
+Each sync bucket is assigned a priority value between 0 and 3, where:
 
-Suppose we have two tables: `lists` and `todos` (as per the standard todolist demo app [schema](/integration-guides/supabase-+-powersync#create-the-demo-database-schema)). Further, suppose we want the sync priority to behave as follows:
+- 0 is the highest priority and has special behavior (detailed below).
+- 3 is the default and lowest priority.
+- Lower numbers indicate higher priority.
 
-1. First, sync all the user's lists, enabling us to render the initial screen in the app
+Buckets with higher priorities sync first, and lower-priority buckets sync later. It's worth noting that if you only use a single priority, there is no difference between priorities 1-3. The difference only comes in if you use multiple different priorities.
 
-2. Then, sync the user's todos
+# Syntax and Configuration
 
-Below are the sync rules that will enable this:
+Priorities can be defined for a bucket using the `priority` YAML key, or with the `_priority` attribute inside parameter queries:
 
 ```yaml
 bucket_definitions:
-  # always sync high priority tables (first), in this case the user's lists
-  high_priority:
-    parameters: select id as list_id from lists where owner_id = token_parameters.user_id
-    data:
-        - select * from lists where id = bucket.list_id
-  # sync any remaining tables, in this case todo items
-  remaining:
-    parameters: select id as list_id from lists where owner_id = token_parameters.user_id and (request.parameters() ->> 'remaining_tables' = true)
+  # Using the `priority` YAML key
+  user_data:
+    priority: 1
+    parameters: SELECT request.user_id() as id where...;
+    data: 
+      # ...
+
+  # Using the `_priority` attribute
+  project_data:
+    parameters: select id as project_id, 2 as _priority from projects where ...; # This approach is useful when you have multiple parameter queries with different priorities.
     data: 
-        - select * from todos where list_id = bucket.list_id
+      # ...
+```    
+
+Note:
+- If multiple parameter queries specify different priorities for the same bucket, the highest priority (lowest number) is used.
+- Priorities must be static and cannot depend on row values within a parameter query.
+
+# Example: Syncing Lists Before Todos
+
+Consider a scenario where you want to display lists immediately while loading todos in the background. This approach allows users to view and interact with lists right away without waiting for todos to sync. Here's how to configure sync priorities in your Sync Rules to achieve this:
+
+```yaml
+bucket_definitions:
+  user_lists:
+    # Sync the user's lists with a higher priority
+    priority: 1
+    parameters: select id as list_id from lists where user_id = request.user_id()
+    data:
+      - select * from lists where id = bucket.list_id
+
+  user_todos:
+    # Sync the user's todos with a lower priority
+    priority: 3
+    parameters: select id as todo_id from todos where list_id in (select id from lists where user_id = request.user_id())
+    data:
+      - select * from todos where list_id = bucket.todo_id
 ```
 
-It is recommended to set Client Parameters in the [Diagnostics App](https://github.com/powersync-ja/powersync-js/tree/main/tools/diagnostics-app) to verify functionality at this point:
+In this configuration:
+
+The `lists` bucket has the default priority of 1, meaning it syncs first.
+
+The `todos` bucket is assigned a priority of 2, meaning it may sync only after the lists have been synced.
+
+
+# Behavioral Considerations
+
+- **Interruption for Higher Priority Data**: Syncing lower-priority buckets _may_ be interrupted if new data for higher-priority buckets arrives.
+- **Local Changes & Consistency**: If local writes fail due to validation or permission issues, they are only reverted after _all_ buckets sync.
+- **Deleted Data**: Deleted data may only be removed after _all_ buckets have synced. Future updates may improve this behavior.
+- **Data Ordering**: Data in lower-priority buckets will never appear before higher-priority data.
+
+## Special Case: Priority 0
+
+Priority 0 buckets sync regardless of pending uploads.
 
-<Frame>
-  <img src="/images/usage/use-case-prioritized.png" />
-</Frame>
+For example, in a collaborative document editing app (e.g., using Yjs), each change is stored as a separate row. Since out-of-order updates don’t affect document integrity, Priority 0 can ensure immediate availability of updates.
+
+Caution: If misused, Priority 0 may cause flickering or inconsistencies, as updates could arrive out of order.
+
+# Consistency Considerations
+
+PowerSync's full consistency guarantees only apply once all buckets have completed syncing.
+
+When higher-priority buckets are synced, all inserts and updates within the buckets for the specific priority will be consistent. However, deletes are only applied when the full sync completes, so you may still have some stale data within those buckets.
+
+Consider the following example:
+
+Example: Todo List Syncing
+
+Imagine a task management app where users create lists and todos. Some users have millions of todos. To improve first-load speed:
+
+- Lists are assigned Priority 1, syncing first to allow UI rendering.
+- Todos are assigned Priority 2, loading in the background.
+
+Now, if another user adds new todos, it’s possible for the list count (synced at Priority 1) to temporarily not match the actual todos (synced at Priority 2). If real-time accuracy is required, both lists and todos should use the same priority.
+
+# Client-Side Considerations
+
+PowerSync's client SDKs provide APIs to allow applications to track sync status at different priority levels. Developers can leverage these to ensure critical data is available before proceeding with UI updates or background processing. This includes:
+
+1. `waitForFirstSync(priority: int)`. When passing the optional `priority` parameter to this method, it will wait for specific priority level to complete syncing. 
+2. `SyncStatus.priorityStatusEntries()` A list containing sync information for each priority that was seen by the PowerSync Service.
+3. `SyncStatus.statusForPriority(priority: int)` This method takes a fixed priority and returns the sync state for that priority by looking it up in `priorityStatusEntries`.
+
+## Example
+Using the above we can render a lists component only once the user's lists (with priority 1) have completed syncing, else display a message indicating that the sync is still in progress:
+
+```dart
+  // Define the priority level of the lists bucket
+  static final _listsPriority = BucketPriority(1);
+
+  @override
+  Widget build(BuildContext context) {
+    // Use FutureBuilder to wait for the first sync of the specified priority to complete
+    return FutureBuilder(
+      future: db.waitForFirstSync(priority: _listsPriority),
+      builder: (context, snapshot) {
+        if (snapshot.connectionState == ConnectionState.done) {
+          // Use StreamBuilder to render the lists once the sync completes
+          return StreamBuilder(
+            stream: TodoList.watchListsWithStats(),
+            builder: (context, snapshot) {
+              if (snapshot.data case final todoLists?) {
+                return ListView(
+                  padding: const EdgeInsets.symmetric(vertical: 8.0),
+                  children: todoLists.map((list) {
+                    return ListItemWidget(list: list);
+                  }).toList(),
+                );
+              } else {
+                return const CircularProgressIndicator();
+              }
+            },
+          );
+        } else {
+          return const Text('Busy with sync...');
+        }
+      },
+    );
+  }
+
+```
 
-If everything checks out, you can then proceed to implement the client parameter switching accordingly in your app.
+Example implementations of prioritized sync are also available in the following apps:
+- Flutter: [Supabase To-Do List](https://github.com/powersync-ja/powersync.dart/tree/main/demos/supabase-todolist)
+- React Native: Coming soon.
+- JavaScript/Web: Coming soon.
+- Kotlin Multiplatform: 
+  - [Supabase To-Do List (KMP)](https://github.com/powersync-ja/powersync-kotlin/blob/main/demos/supabase-todolist/shared/src/commonMain/kotlin/com/powersync/demos/App.kt#L46)
+  - [Supabase To-Do List (Android)](https://github.com/powersync-ja/powersync-kotlin/blob/main/demos/android-supabase-todolist/app/src/main/java/com/powersync/androidexample/screens/HomeScreen.kt#L69)
+- Swift: Coming soon.