Experimental: Trigger Based Table Diff Tracking

[stevensJourney]

Overview

We currently expose watched query functionality which allows watching changes of an arbitrary SQLite query's result set. This PR adds methods which can be used to track and process row level changes to individual SQLite tables.

Temporary SQLite triggers and tables are used to populate a set of table change operations. Records for table row INSERT, UPDATEandDELETE` operations are created, and persisted in a temporary SQLite table, as changes occur.

Diff Differences

This API provides a similar result to the diff from WatchedQuerys created with differentialWatch(), but there are a few key differences to consider.

• WatchedQuerys can track the diff of arbitrary query results, while triggers are limited to row level changes of a single table.
• The trigger based diffs record the changes to temporary tables as they are made. This adds some overhead to the WRITE operations. WatchedQuerys run off read-connections which have the ability to run concurrently in some platforms.
• Trigger based diffs store the diff as rows in SQLite, allowing SQLite queries to use the diff as an input. WatchedQuery diffs expose the diff in JS as a JS objects/arrays.
• WatchedQuerys query the SQLite DB on any change to the query's dependent tables, the changes are filtered after querying SQLite. Trigger based diffs can filter/skip storing diff records inside the SQLite trigger, which prevents emissions on a lower level.
• To avoid race conditions, the trigger based diff tracking requires a writeLock during processing (we only allow a single concurrent writeLock).

This PR currently adds 2 new SQLite trigger based methods.

Creating a Diff Trigger

createDiffTrigger Configures temporary SQLite triggers which populate a temporary SQLite table with operations as they occur.

const dispose = await database.triggers.createDiffTrigger({
  source: 'lists',
  destination: tempTable,
  columns: ['name'],
  when: { 
     [DiffTriggerOperation.INSERT]: `TRUE`
     [DiffTriggerOperation.UPDATE]: `TRUE`
  },
});

// logic performs operations on the lists table

// The changes can be fetched in a writeLock

await database.writeLock(async (tx) => {
  const changes = await tx.getAll<TriggerDiffRecord>(/* sql */ `
    SELECT
      *
    FROM
      ${tempTable}
  `);

  // Process changes

  // Clear the temp table after processing
  await tx.execute(/* sql */ ` DELETE FROM ${tempTable}; `);
});

Processing Change Operations

A more convenient trackTableDiff method uses createDiffTrigger to get table changes, but adds additional logic to consume the changes as they occur.

The example below uses triggers to process todo records as they are inserted/created.

const dispose = await database.triggers.trackTableDiff({
  source: 'todos',
  columns: ['list_id'],
  when: { [DiffTriggerOperation.INSERT]: `json_extract(NEW.data, '$.list_id') = '${firstList.id}'` },
  onChange: async (context) => {
    // Fetches the todo records that were inserted during this diff
    const newTodos = await context.getAll<Database['todos']>(/* sql */ `
      SELECT
        todos.*
      FROM
        DIFF
        JOIN todos ON DIFF.id = todos.id
    `);

    // Process newly added TODOs
  }
});

Docs

For more info see the unit tests and API doc comments.

[changeset-bot]

🦋 Changeset detected

Latest commit: 1501c24

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 8 packages

Name	Type
@powersync/common	Minor
@powersync/node	Minor
@powersync/react-native	Minor
@powersync/web	Minor
@powersync/adapter-sql-js	Patch
@powersync/op-sqlite	Patch
@powersync/tanstack-react-query	Patch
@powersync/diagnostics-app	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[simolus3]

I think it's great to have this functionality! I have some general API notes:

• I see how trackTableDiff neatly builds ontop of createDiffTrigger, but I'm not sure if the latter needs to be a public API. It sounds to me like it would be easy to misuse by e.g. forgtting to clean up the pending changes after processing them. I can't think of scenarios where the additional flexibility is required, do you have some?
• I'm not that familiar with the JS libraries, but I think we should check if it's possible to use Drizzle / Kysely with this API. I don't think we need dedicated APIs in our packages for those, but it could be worth checking whether we can e.g. build the WHEN clause with Kysely APIs. Similarly, can we run the inner query in trackTableDiff with those query builders or do they need access to some connection context that would be hard to pass down?

[stevensJourney]

I see how trackTableDiff neatly builds ontop of createDiffTrigger, but I'm not sure if the latter needs to be a public API. It sounds to me like it would be easy to misuse by e.g. forgtting to clean up the pending changes after processing them. I can't think of scenarios where the additional flexibility is required, do you have some?

That's a fair point. I currently view the trackTableDiff method as an automatic "watch" which allows users to process the diffs reactively as they occur. While I can imagine there might be a use-case where the diff should not be processed in "real-time". One example might be a view/form which initiates diff tracking on open, allows for changes to be performed, and only processes the changes when a user-action (like pressing a button) is performed. One could buffer the changes from trackTableDiff in memory, but you could also avoid obtaining a writeLock during that period by handling it manually.

I'm not that familiar with the JS libraries, but I think we should check if it's possible to use Drizzle / Kysely with this API. I don't think we need dedicated APIs in our packages for those, but it could be worth checking whether we can e.g. build the WHEN clause with Kysely APIs. Similarly, can we run the inner query in trackTableDiff with those query builders or do they need access to some connection context that would be hard to pass down?

I took a brief look-in to this. I think building the WHEN clauses with the JS "ORMs" could be tricky due to the JSON structure of the data column which the triggers are based off. The ORMS generally are typed according to the views we expose. Currently, the ORM packages we expose "wrap" the PS JS client. It seems like we could expose additional utilities in those ORMs which expose the schema of the diff tables - this could help for querying against tables tracked with createDiffTrigger. It doesn't seem like it would be easy to directly use a ORM query inside the onChange callback of trackTableDiff, but I think we might be able to wrap this method inside ORM versions of this API which could be compatible with their query builders. Overall I get the feeling we could release ORM support for triggers as a later release.

[Copilot]

Pull Request Overview

This PR introduces SQLite trigger-based table change tracking functionality to PowerSync. It allows developers to monitor and process changes to individual SQLite tables using temporary triggers and tables.

• Adds two new methods: createDiffTrigger for basic trigger setup and trackTableDiff for automated change processing
• Implements a comprehensive trigger management system with support for INSERT, UPDATE, and DELETE operations
• Provides filtering capabilities through columns and WHEN conditions with built-in sanitization helpers

Reviewed Changes

Copilot reviewed 10 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
packages/common/src/client/triggers/TriggerManager.ts	Defines core interfaces and types for trigger management
packages/common/src/client/triggers/TriggerManagerImpl.ts	Implements the trigger management functionality
packages/common/src/client/triggers/whenClause.ts	Provides SQL sanitization utilities for WHEN clauses
packages/common/src/client/AbstractPowerSyncDatabase.ts	Integrates trigger manager into main database class
packages/common/src/index.ts	Exports new trigger functionality
packages/node/tests/trigger.test.ts	Comprehensive test suite for trigger functionality
packages/node/tests/tsconfig.json	TypeScript configuration for tests

Files not reviewed (1)

• pnpm-lock.yaml: Language not supported

[simolus3]

The implementation and tests look good to me 👍

I have one suggestion to possibly simplify the API, but I don't think it's that important.

[simolus3]

Happy with these changes, I think this is a very useful addition 👍