[Experimental] Add include_old and include_metadata options

[rkistner]

The goal is to allow more fine-grained tracking of updates to rows, especially when the rows contain nested json data.

One example is having an array column, and the developer adds or removes individual items to the array. Currently we record the entire new array, but that does not provide the individual updates to the developer in the uploadData function.

Right now the best workaround is to create a new insert_only table, that you use to associate custom metadata per transaction.

This PR experiments with adding two new options that can be configured per table, which should simplify the process of custom metadata for a lot of use cases.

Specific use case examples:

• Discord thread 1
• Discord thread 2

include_old

Instead of just including updated fields in data in ps_crud, this additionally includes all previous values in old. This option can be extended to support an array of columns to include, instead of all columns.

-- setup
PRAGMA trusted_schema=1;
SELECT powersync_replace_schema('{"tables": [{"name": "test", "columns": [{"name": "name", "type": "text"}], "include_old": {}}]}');
delete from test;
insert into test(id, name) values('1', 'foo'), ('2', 'bar');
delete from ps_crud;

-- the demo
update test set name = name || '.';
select data from ps_crud;

{"op":"PATCH","type":"test","id":"1","data":{"name":"foo."},"old":{"name":"foo"}}
{"op":"PATCH","type":"test","id":"2","data":{"name":"bar."},"old":{"name":"bar"}}

The old data could be used to:

1. Keep track of additional id fields / other fields needed in the update (some use cases use a compound primary keys, which may not be fully represented in the id column).
2. Do a custom diff on fields when uploading data.

TODO:

• Support include_old for delete operations.
• Support a column list for include_old.
• Support only tracking old value of a column if it changed

include_metadata

This adds a _metadata column to the view. This is always null when reading, but can be set when inserting or updating. Unfortunately, it is not possible to set this when deleting rows. We could theoretically work around this by having a virtual UPDATE myview SET _delete = true, _metadata = ... type update, but leaving that for later.

-- setup
PRAGMA trusted_schema=1;
SELECT powersync_replace_schema('{"tables": [{"name": "test", "columns": [{"name": "name", "type": "text"}], "include_metadata": true}]}');
delete from test;
insert into test(id, name) values('1', 'foo'), ('2', 'bar');
delete from ps_crud;

-- the demo
update test set name = name || '.', _metadata = 'this is an update to ' || id;
select data from ps_crud;

{"op":"PATCH","type":"test","id":"1","data":{"name":"foo."},"metadata":"this is an update to 1"}
{"op":"PATCH","type":"test","id":"2","data":{"name":"bar."},"metadata":"this is an update to 2"}

delete from ps_crud;
update test set name = name || '.', _metadata = json_object('old_name', name);
select data from ps_crud;

{"op":"PATCH","type":"test","id":"1","data":{"name":"foo.."},"metadata":{"old_name":"foo."}}
{"op":"PATCH","type":"test","id":"2","data":{"name":"bar.."},"metadata":{"old_name":"bar."}}

You can store any custom metadata here - it could include the old value of columns or custom values provided in code. It could be a number, plain text, or dynamically constructed json.

The metadata can be used for any custom metadata required for uploading operations. We do not impose any restriction or meaning on the structure here, although we could support some structured operations in a higher-level update.

This can also be combined with include_old.

TODO:

• Support include_metadata for insert operations.
• Investigate UPDATE ... SET _deleted = true, _metadata = ...