Skip to content

Commit f78b2cf

Browse files
Update README with comprehensive sync engine architecture and design details
1 parent cba5cff commit f78b2cf

File tree

1 file changed

+107
-12
lines changed

1 file changed

+107
-12
lines changed

README.md

Lines changed: 107 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,107 @@
1-
# Async Sync Engine
2-
[Source](https://x.com/i/grok?conversation=1893597036278100311)
3-
4-
1. Init: Store clientId locally.
5-
2. Create: Owner creates workshop, gets masterToken, stores it.
6-
3. Share: Owner issues accessToken for collaborator, shares manually.
7-
4. Access: Collaborator inputs workshopId and accessToken, downloads data.
8-
5. Sync: Clients edit locally, sync with server using their token.
9-
6. Revoke: Owner revokes access, collaborator’s token fails.
10-
7. Expiry: Client handles token expiration gracefully.
11-
12-
> Allow user to check all changes (syncs) by selecting history by `workspaceId` (`"read"` scope?)
1+
# Sync Engine Web
2+
3+
A local-first, offline-capable web sync engine implementation with CRDT-based synchronization. The project provides a complete solution for data synchronization between multiple devices while maintaining data consistency and offline capabilities.
4+
5+
## Architecture
6+
7+
The project is structured as a monorepo with three main components:
8+
9+
1. **Client**
10+
- Web-based UI implementation
11+
- Local-first storage using IndexedDB ([`dexie`](https://dexie.org/))
12+
- Background sync using Web Workers
13+
- Client-side schema migrations
14+
- Offline-first data management
15+
16+
2. **Server**
17+
- REST API for data synchronization
18+
- JWT-based authentication and authorization
19+
- Byte-level data storage (client-agnostic)
20+
- Workspace management and backup
21+
- Token-based access control
22+
23+
3. **Shared Packages**
24+
- Schema definitions
25+
- Type definitions
26+
- Shared utilities
27+
- Migration utilities
28+
29+
## Key Features
30+
31+
- **Local-First Architecture**: Data is primarily stored and managed locally, with server acting as sync + backup
32+
- **CRDT-Based Sync**: Uses [`loro-crdt`](https://loro.dev/) for conflict-free data synchronization
33+
- **Offline Support**: Full offline capability with background sync
34+
- **Multi-Device Access**: Share workspaces across devices using access tokens
35+
- **Type-Safe Migrations**: Version-controlled schema migrations
36+
- **Secure Authentication**: JWT-based auth with master/access token system
37+
38+
## Implementation Details
39+
40+
- Client stores and syncs data using CRDT (Conflict-free Replicated Data Type)
41+
- Server is client-agnostic, storing only byte-level data
42+
- Master client controls workspace access through token generation
43+
- Schema migrations are performed client-side
44+
- Background syncing handled by Web Workers
45+
- No server-side querying - all data operations happen client-side
46+
47+
## Authentication Flow
48+
49+
1. Client creates local workspace
50+
2. Server stores workspace and designates client as "master"
51+
3. Master client receives master token
52+
4. Master client generates access tokens for other devices
53+
5. Access tokens can be shared via links
54+
6. Server handles token verification and authorization
55+
56+
## Data Flow
57+
58+
```
59+
Client (Local Storage) <-> Web Worker (Background Sync) <-> Server (Byte Storage)
60+
```
61+
62+
- UI reads directly from local storage
63+
- Sync operations happen in background
64+
- Server stores encoded CRDT data as bytes
65+
- Migrations happen during client initialization
66+
67+
## Technologies
68+
69+
- CRDT: `loro-crdt` for data synchronization
70+
- Storage: IndexedDB (client), Database (server)
71+
- Authentication: JWT tokens
72+
- Background Processing: Web Workers
73+
74+
## Dependencies
75+
76+
### Client
77+
- **UI & Routing**
78+
- `react` - UI framework
79+
- `@tanstack/react-router` - Type-safe routing
80+
- **Storage & Sync**
81+
- `loro-crdt` - CRDT implementation
82+
- `dexie` - IndexedDB wrapper
83+
- `dexie-react-hooks` - React hooks for Dexie
84+
- **Core**
85+
- `effect` - Functional programming toolkit
86+
- `@effect/platform-browser` - Browser-specific Effect utilities
87+
88+
### Server
89+
- **Database**
90+
- `drizzle-orm` - TypeScript ORM
91+
- `@effect/sql` - SQL integration for Effect
92+
- `@effect/sql-pg` - PostgreSQL driver
93+
- `pg` - PostgreSQL client
94+
- **Authentication**
95+
- `jsonwebtoken` - JWT implementation
96+
- **Core**
97+
- `effect` - Functional programming toolkit
98+
- `@effect/platform-node` - Node.js-specific Effect utilities
99+
100+
### Build Tools
101+
- `turbo` - Monorepo build system
102+
- `vite` - Frontend build tool
103+
- `typescript` - Type system
104+
105+
## Project Status
106+
107+
This is a functional implementation of a web sync engine, optimized for single-user scenarios (not real-time collaboration). The focus is on providing reliable data synchronization while maintaining offline capabilities.

0 commit comments

Comments
 (0)