|
1 | | -# Async Sync Engine |
2 | | -[Source](https://x.com/i/grok?conversation=1893597036278100311) |
3 | | - |
4 | | -1. Init: Store clientId locally. |
5 | | -2. Create: Owner creates workshop, gets masterToken, stores it. |
6 | | -3. Share: Owner issues accessToken for collaborator, shares manually. |
7 | | -4. Access: Collaborator inputs workshopId and accessToken, downloads data. |
8 | | -5. Sync: Clients edit locally, sync with server using their token. |
9 | | -6. Revoke: Owner revokes access, collaborator’s token fails. |
10 | | -7. Expiry: Client handles token expiration gracefully. |
11 | | - |
12 | | -> Allow user to check all changes (syncs) by selecting history by `workspaceId` (`"read"` scope?) |
| 1 | +# Sync Engine Web |
| 2 | + |
| 3 | +A local-first, offline-capable web sync engine implementation with CRDT-based synchronization. The project provides a complete solution for data synchronization between multiple devices while maintaining data consistency and offline capabilities. |
| 4 | + |
| 5 | +## Architecture |
| 6 | + |
| 7 | +The project is structured as a monorepo with three main components: |
| 8 | + |
| 9 | +1. **Client** |
| 10 | + - Web-based UI implementation |
| 11 | + - Local-first storage using IndexedDB ([`dexie`](https://dexie.org/)) |
| 12 | + - Background sync using Web Workers |
| 13 | + - Client-side schema migrations |
| 14 | + - Offline-first data management |
| 15 | + |
| 16 | +2. **Server** |
| 17 | + - REST API for data synchronization |
| 18 | + - JWT-based authentication and authorization |
| 19 | + - Byte-level data storage (client-agnostic) |
| 20 | + - Workspace management and backup |
| 21 | + - Token-based access control |
| 22 | + |
| 23 | +3. **Shared Packages** |
| 24 | + - Schema definitions |
| 25 | + - Type definitions |
| 26 | + - Shared utilities |
| 27 | + - Migration utilities |
| 28 | + |
| 29 | +## Key Features |
| 30 | + |
| 31 | +- **Local-First Architecture**: Data is primarily stored and managed locally, with server acting as sync + backup |
| 32 | +- **CRDT-Based Sync**: Uses [`loro-crdt`](https://loro.dev/) for conflict-free data synchronization |
| 33 | +- **Offline Support**: Full offline capability with background sync |
| 34 | +- **Multi-Device Access**: Share workspaces across devices using access tokens |
| 35 | +- **Type-Safe Migrations**: Version-controlled schema migrations |
| 36 | +- **Secure Authentication**: JWT-based auth with master/access token system |
| 37 | + |
| 38 | +## Implementation Details |
| 39 | + |
| 40 | +- Client stores and syncs data using CRDT (Conflict-free Replicated Data Type) |
| 41 | +- Server is client-agnostic, storing only byte-level data |
| 42 | +- Master client controls workspace access through token generation |
| 43 | +- Schema migrations are performed client-side |
| 44 | +- Background syncing handled by Web Workers |
| 45 | +- No server-side querying - all data operations happen client-side |
| 46 | + |
| 47 | +## Authentication Flow |
| 48 | + |
| 49 | +1. Client creates local workspace |
| 50 | +2. Server stores workspace and designates client as "master" |
| 51 | +3. Master client receives master token |
| 52 | +4. Master client generates access tokens for other devices |
| 53 | +5. Access tokens can be shared via links |
| 54 | +6. Server handles token verification and authorization |
| 55 | + |
| 56 | +## Data Flow |
| 57 | + |
| 58 | +``` |
| 59 | +Client (Local Storage) <-> Web Worker (Background Sync) <-> Server (Byte Storage) |
| 60 | +``` |
| 61 | + |
| 62 | +- UI reads directly from local storage |
| 63 | +- Sync operations happen in background |
| 64 | +- Server stores encoded CRDT data as bytes |
| 65 | +- Migrations happen during client initialization |
| 66 | + |
| 67 | +## Technologies |
| 68 | + |
| 69 | +- CRDT: `loro-crdt` for data synchronization |
| 70 | +- Storage: IndexedDB (client), Database (server) |
| 71 | +- Authentication: JWT tokens |
| 72 | +- Background Processing: Web Workers |
| 73 | + |
| 74 | +## Dependencies |
| 75 | + |
| 76 | +### Client |
| 77 | +- **UI & Routing** |
| 78 | + - `react` - UI framework |
| 79 | + - `@tanstack/react-router` - Type-safe routing |
| 80 | +- **Storage & Sync** |
| 81 | + - `loro-crdt` - CRDT implementation |
| 82 | + - `dexie` - IndexedDB wrapper |
| 83 | + - `dexie-react-hooks` - React hooks for Dexie |
| 84 | +- **Core** |
| 85 | + - `effect` - Functional programming toolkit |
| 86 | + - `@effect/platform-browser` - Browser-specific Effect utilities |
| 87 | + |
| 88 | +### Server |
| 89 | +- **Database** |
| 90 | + - `drizzle-orm` - TypeScript ORM |
| 91 | + - `@effect/sql` - SQL integration for Effect |
| 92 | + - `@effect/sql-pg` - PostgreSQL driver |
| 93 | + - `pg` - PostgreSQL client |
| 94 | +- **Authentication** |
| 95 | + - `jsonwebtoken` - JWT implementation |
| 96 | +- **Core** |
| 97 | + - `effect` - Functional programming toolkit |
| 98 | + - `@effect/platform-node` - Node.js-specific Effect utilities |
| 99 | + |
| 100 | +### Build Tools |
| 101 | +- `turbo` - Monorepo build system |
| 102 | +- `vite` - Frontend build tool |
| 103 | +- `typescript` - Type system |
| 104 | + |
| 105 | +## Project Status |
| 106 | + |
| 107 | +This is a functional implementation of a web sync engine, optimized for single-user scenarios (not real-time collaboration). The focus is on providing reliable data synchronization while maintaining offline capabilities. |
0 commit comments