[docs] Synchronization guides

Author: jamesgpearce

site/guides/04_persistence/1_an_intro_to_persistence.md

@@ -1,6 +1,6 @@
 # An Intro To Persistence
 
-The persister module lets you save and load Store data to and from different
+The persister module framework lets you save and load Store data to and from different
 locations, or underlying storage types.
 
 Remember that TinyBase Stores are in-memory data structures, so you will

site/guides/05_synchronization/1_an_intro_to_synchronization.md

@@ -0,0 +1,138 @@
+# Using A MergeableStore
+
+The basic building block of TinyBase's synchronization system is the
+MergeableStore interface.
+
+## The Anatomy Of A MergeableStore
+
+The MergeableStore interface is a sub-type of the regular Store - and it shares
+its underlying implementation.
+
+This means that if you want to add synchronization to your app, all of your
+existing calls to the Store methods will be unchanged - you just need to use the
+createMergeableStore function to instantiate it, instead of the classic
+createStore function.
+
+```js
+import {createMergeableStore} from 'tinybase';
+
+const store1 = createMergeableStore('store1'); // !resetHlc
+store1.setCell('pets', 'fido', 'species', 'dog');
+
+console.log(store1.getContent());
+// -> [{pets: {fido: {species: 'dog'}}}, {}]
+```
+
+The difference, though, is that a MergeableStore records additional metadata as
+the data is changed so that potential conflicts between it and another instance
+can be reconciled. This metadata is intended to be opaque, but you can see it if
+you call the getMergeableContent method:
+
+```js
+console.log(store1.getMergeableContent());
+// ->
+[
+  [
+    {
+      pets: [
+        {
+          fido: [
+            {species: ['dog', 'Nn1JUF-----FnHIC', 290599168]},
+            '',
+            2682656941,
+          ],
+        },
+        '',
+        2102515304,
+      ],
+    },
+    '',
+    3506229770,
+  ],
+  [{}, '', 0],
+];
+```
+
+Without going into the detail of this, the main point to understand is that each
+update gets a timestamp, based on a hybrid logical clock (HLC), and a hash. As a
+result, TinyBase is able to understand which parts of the data have changed, and
+which changes are the most recent. The resulting 'last write wins' (LWW)
+approach allows the MergeableStore to act as a Conflict-Free Replicated Data
+Type (CRDT).
+
+(Notice we provided an explicit `uniqueId` when we initialized the
+MergeableStore: this is not normally required, but here it just ensures the
+hashes in the example are deterministic).
+
+We can of course, create a second MergeableStore with different data:
+
+```js
+const store2 = createMergeableStore();
+store2.setCell('pets', 'felix', 'species', 'cat');
+```
+
+And now merge them together with the convenient merge method:
+
+```js
+store1.merge(store2);
+
+console.log(store1.getContent());
+// -> [{pets: {felix: {species: 'cat'}, fido: {species: 'dog'}}}, {}]
+
+console.log(store2.getContent());
+// -> [{pets: {felix: {species: 'cat'}, fido: {species: 'dog'}}}, {}]
+```
+
+Magic!
+
+This all said, it's very unlikely you will need to use the numerous extra
+methods available on a MergeableStore (compared to a Store) since most of them
+exist to support synchronization behind the scenes.
+
+In general, you'll just use a MergeableStore in the same was as you would have
+used a Store, and instead rely on the more approachable Synchronizer API for
+synchronization. We'll discuss this next in the Using A Synchronizer guide.
+
+# Persisting A MergeableStore
+
+Once important thing that you need to be aware of is that a MergeableStore
+cannot currently be persisted by every type of Persister available to a regular
+Store. This is partly because some are already designed to work with alternative
+third-party CRDT systems (like the YjsPersister and AutomergePersister), and
+partly because this extra metadata cannot be easily stored in a plain SQLite
+database.
+
+The following Persister types _can_ be used to persist a MergeableStore:
+
+| Persister        | Storage                     |
+| ---------------- | --------------------------- |
+| SessionPersister | Browser session storage     |
+| LocalPersister   | Browser local storage       |
+| FilePersister    | Local file (where possible) |
+
+The following database-oriented Persister types can be used to persist a
+MergeableStore, but _only_ in the 'JSON-serialization' mode:
+
+| Persister           | Storage                                                                                                |
+| ------------------- | ------------------------------------------------------------------------------------------------------ |
+| Sqlite3Persister    | SQLite in Node, via [sqlite3](https://github.com/TryGhost/node-sqlite3)                                |
+| SqliteWasmPersister | SQLite in a browser, via [sqlite-wasm](https://github.com/tomayac/sqlite-wasm)                         |
+| ExpoSqlitePersister | SQLite in React Native, via [expo-sqlite](https://github.com/expo/expo/tree/main/packages/expo-sqlite) |
+
+The following database-oriented Persister types _cannot_ currently be used to
+persist a MergeableStore:
+
+| Persister             | Storage                                                                                  |
+| --------------------- | ---------------------------------------------------------------------------------------- |
+| IndexedDbPersister    | Browser IndexedDB                                                                        |
+| RemotePersister       | Remote server                                                                            |
+| CrSqliteWasmPersister | SQLite CRDTs, via [cr-sqlite-wasm](https://github.com/vlcn-io/cr-sqlite)                 |
+| ElectricSqlPersister  | Electric SQL, via [electric-sql](https://github.com/electric-sql/electric)               |
+| LibSqlPersister       | LibSQL for Turso, via [libsql-client](https://github.com/tursodatabase/libsql-client-ts) |
+| PowerSyncPersister    | PowerSync, via [powersync-sdk](https://github.com/powersync-ja/powersync-js)             |
+| YjsPersister          | Yjs CRDTs, via [yjs](https://github.com/yjs/yjs)                                         |
+| AutomergePersister    | Automerge CRDTs, via [automerge-repo](https://github.com/automerge/automerge-repo)       |
+| PartyKitPersister     | [PartyKit](https://www.partykit.io/), via the persister-partykit-server module           |
+
+Next, let's see how to synchronize MergeableStore objects together with the
+synchronizers module. Please continue on to the Using A Synchronizer guide.

site/guides/05_synchronization/1_using_a_mergeablestore.md

@@ -0,0 +1,204 @@
+# Using A Synchronizer
+
+The synchronizer module framework lets you synchronize MergeableStore data
+between different devices, systems, or subsystems.
+
+It contains the Synchronizer
+interface, describing objects which can be used to synchronize a MergeableStore.
+
+Under the covers, a Synchronizer is actually a very specialized type of
+Persister that _only_ supports MergeableStore objects, and which has a startSync
+method and a stopSync method.
+
+## Types Of Synchronizer
+
+In TinyBase v5.0, there are three types of Synchronizer:
+
+- The WsSynchronizer uses WebSockets to communicate between different systems.
+- The BroadcastChannelSynchronizer uses the browser's BroadcastChannel API to
+  communicate between different tabs and workers.
+- The LocalSynchronizer demonstrates synchronization in memory on a single local
+  system.
+
+Of course it is also possible to create custom Synchronizer objects if you have
+a transmission medium that allows the synchronization messages to be sent
+reliably between clients.
+
+## Synchronizing with WebSockets
+
+A common pattern for synchronizing over the web is to use WebSockets. This
+allows multiple clients to pass lightweight messages to each other, facilitating
+efficient synchronization.
+
+One thing to understand is that this set up will typically require a server.
+This can be a relatively 'thin server' - it does not need to store data of its
+own - but is needed to keep a list of clients that are being synchronized
+together, and route and broadcast messages between the clients.
+
+TinyBase includes a simple implementation of such a server. You simply need to
+create it, instantiated with a configured WebSocketServer object from the `ws`
+package:
+
+```js
+// On a server machine:
+import {WebSocketServer} from 'ws';
+import {createWsServer} from 'tinybase/synchronizers/synchronizer-ws-server';
+const server = createWsServer(new WebSocketServer({port: 8048}));
+```
+
+This sets up a WsServer object, listening on port 8048.
+
+Each client then needs to create a WsSynchronizer object, instantiated with the
+MergeableStore being synchronized, and a WebSocket configured to connect to the
+aforementioned server:
+
+```js
+// On the first client machine:
+import {WebSocket} from 'ws';
+import {createMergeableStore} from 'tinybase';
+import {createWsSynchronizer} from 'tinybase/synchronizers/synchronizer-ws-client';
+
+const clientStore1 = createMergeableStore();
+const clientSynchronizer1 = await createWsSynchronizer(
+  clientStore1,
+  new WebSocket('ws://localhost:8048'),
+);
+```
+
+This WsSynchronizer can then be started, and data manipulated as normal:
+
+```js
+await clientSynchronizer1.startSync();
+clientStore1.setCell('pets', 'fido', 'species', 'dog');
+// ...
+```
+
+Meanwhile, on another client, an empty MergeableStore and another WsSynchronizer
+can be created and started, connecting to the same server.
+
+```js
+// On the second client machine:
+const clientStore2 = createMergeableStore();
+const clientSynchronizer2 = await createWsSynchronizer(
+  clientStore2,
+  new WebSocket('ws://localhost:8048'),
+);
+await clientSynchronizer2.startSync();
+```
+
+Once the synchronization is started, the server will broker the messages being
+passed back and forward between the two clients, and the data will be
+synchronized. The empty second MergeableStore will be populated with the data
+from the first:
+
+```js
+// ...
+console.log(clientStore2.getTables());
+// -> {pets: {fido: {species: 'dog'}}}
+```
+
+And of course the synchronization is bi-directional:
+
+```js
+clientStore2.setCell('pets', 'felix', 'species', 'cat');
+console.log(clientStore2.getTables());
+// -> {pets: {fido: {species: 'dog'}, felix: {species: 'cat'}}}
+```
+
+```js
+// ...
+console.log(clientStore1.getTables());
+// -> {pets: {fido: {species: 'dog'}, felix: {species: 'cat'}}}
+```
+
+When done, it's important to destroy a WsSynchronizer to close and tidy up the
+client WebSockets:
+
+```js
+clientSynchronizer1.destroy();
+```
+
+```js
+clientSynchronizer2.destroy();
+```
+
+And, if shut down, the WsServer should also be explicitly destroyed to close its
+listeners:
+
+```js
+server.destroy();
+```
+
+## Synchronizing over the browser BroadcastChannel
+
+There may be situations where you need to synchronize data between different
+parts of a browser. For example, you might have a transient in-memory
+MergeableStore driving your UI, but then another instance in a Service Worker
+that can be persisted to (say) IndexedDB or another medium.
+
+To facilitate keeping these in sync, the BroadcastChannelSynchronizer lets you
+synchronize over the browser's BroadcastChannel API, common to each browser
+sub-system. You simply need to provide a distinguishing channel name that can be
+used to identify what the two parts should be using to send and receive
+messages.
+
+For example, in the UI part of your app:
+
+```js
+import {createBroadcastChannelSynchronizer} from 'tinybase/synchronizers/synchronizer-broadcast-channel';
+
+const frontStore = createMergeableStore();
+const frontSynchronizer = createBroadcastChannelSynchronizer(
+  frontStore,
+  'syncChannel',
+);
+await frontSynchronizer.startSync();
+```
+
+And then in the service worker:
+
+```js
+const backStore = createMergeableStore();
+const backSynchronizer = createBroadcastChannelSynchronizer(
+  backStore,
+  'syncChannel',
+);
+await backSynchronizer.startSync();
+```
+
+Since they both share the `syncChannel` channel name, the data of the two is now
+synchronized:
+
+```js
+frontStore.setCell('pets', 'fido', 'species', 'dog');
+```
+
+```js
+// ...
+console.log(backStore.getTables());
+// -> {pets: {fido: {species: 'dog'}}}
+```
+
+And so on!
+
+When finished, these synchronizers should also be explicitly destroyed to ensure
+the channel listeners are cleaned up:
+
+```js
+frontSynchronizer.destroy();
+```
+
+```js
+backSynchronizer.destroy();
+```
+
+## Wrapping Up
+
+The Synchronizer interface provides an easy way to keep multiple TinyBase
+MergeableStores in sync. The WebSocket and BroadcastChannel options above allow
+for numerous interesting and powerful app architectures - and they are not
+sufficient, consider exploring the createCustomSynchronizer function to develop
+your own!
+
+We now move on to ways in which TinyBase can be used like a database, starting
+with the Using Metrics guide.

site/guides/05_synchronization/2_using_a_mergeablestore.md

@@ -3,4 +3,22 @@
 These guides discuss how to merge and synchronize data in MergeableStore
 instances using synchronization techniques.
 
-See also the Todo App v6 (collaboration) demo.
+The basic building block of TinyBase's synchronization system is the
+MergeableStore interface. This is a sub-type of the regular Store - so all your
+existing calls to the Store methods will be unchanged - but it records
+additional metadata as the data is changed so that potential conflicts can be
+reconciled. See the Using A MergeableStore guide for more details.
+
+On top of this, the synchronizer module framework uses this metadata to let you
+synchronize MergeableStore data between different devices, systems, or
+subsystems. Synchronization can take place over WebSockets, the browser's
+BroadcastChannel API, or other custom media. See the Using A Synchronizer guide
+for more details.
+
+It's possible - and in fact recommended! - to use both persistence and
+synchronization at the same time. You will often want to persist changes to your
+TinyBase data between browser reloads even when offline, for example, and then
+synchronize to other devices or a server once the device comes back online.
+
+See also the Todo App v6 (collaboration) demo for a simple example of
+adding synchronization between clients to an app.