sync files to hyper cloud

[ComputelessComputer]

one-liner

we need our version of "obsidian sync"

context

hyprnote is local-first because we respect two things:

1. data layer that respects filesystem like obsidian
2. on-device transcription

food for thought

should all notes be local-only?
and have notes shared only when there are participants in it?
and should it be only shared to people in the user's org?

[thecodeblood]

Hi @ComputelessComputer, I want to work on this feature.
Can you assign this me.

[yujonglee]

Hi @thecodeblood. This is big change across many part of our application. So we can not just assign someone.
Feel free to leave a comment on overall design, if you are familiar with Obsidian-like sync.

[yujonglee]

File Types

• Text (.md .json etc)
• Attachments (.wav, .png etc)

Auth

• Device registration can be helpful, but skip for now for simplicity
• Use existing crates/supabase-auth infra.

Storage

• Should work for any S3-compat storage
• We will use Supabase Storage for dev for now

Server

• Axum, just like apps/ai.

[thecodeblood]

Hi @yujonglee ,
I am working on the design only also writing down implementation details.
Will share with complete docs in one days after that we can look what modification we can do before implementing our solution.

[thecodeblood]

Hyprnote Sync Service - Comprehensive Implementation Plan

1. System Overview

The Hyprnote Sync Service provides a private, secure, and local-first synchronization engine for Hyprnote clients. It follows the principles of "bring your own storage" (S3) and "filesystem as truth" (Obsidian-like).

1.1 Architecture Diagram

1.2 Key Design Decisions

1. Vault-Centric: The top-level unit of synchronization is a "Vault" (a root directory).
2. State-Based Sync: We sync the state of files, not the operations (like OT/CRDTs). This is simpler and fits the "file is truth" model.
3. Last-Write-Wins (LWW) with Safety: We use a modified_at timestamp and content checksum to detect drift. If a client tries to overwrite a file without having the latest version, the server rejects it (Optimistic Concurrency Control).
4. Metadata vs Content:
   • Metadata (Registry of what files exist, their versions, and checksums) lives in PostgreSQL.
   • Content (The actual .md, .wav bits) lives in S3.

---

2. Database Design (PostgreSQL)

We will use PostgreSQL to track the state of every file in every vault to enable quick "What changed?" queries.

-- 2.1 Vaults Registry
-- A Vault maps to a UUID. It can be owned by a user or an organization.
CREATE TABLE vaults (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    owner_id UUID NOT NULL REFERENCES auth.users(id),
    name TEXT NOT NULL,
    created_at TIMESTAMPTZ DEFAULT NOW(),
    updated_at TIMESTAMPTZ DEFAULT NOW()
);

-- 2.2 Permissions (Sharing)
-- Controls who can access a vault.
CREATE TABLE vault_access (
    vault_id UUID REFERENCES vaults(id) ON DELETE CASCADE,
    user_id UUID REFERENCES auth.users(id) ON DELETE CASCADE,
    role TEXT CHECK (role IN ('viewer', 'editor', 'admin')),
    created_at TIMESTAMPTZ DEFAULT NOW(),
    PRIMARY KEY (vault_id, user_id)
);

-- 2.3 File Registry ( The "Source of Truth" for State)
-- Tracks the latest known version of every file.
CREATE TABLE file_metadata (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    vault_id UUID REFERENCES vaults(id) ON DELETE CASCADE,
    path TEXT NOT NULL,          -- Relative path: "Meetings/Morning Sync.md"
    checksum TEXT NOT NULL,      -- SHA256 of the content
    size BIGINT NOT NULL,        -- File size in bytes
    modified_at TIMESTAMPTZ NOT NULL, -- Client-reported mtime
    uploaded_at TIMESTAMPTZ DEFAULT NOW(), -- Server receive time
    last_modified_by UUID REFERENCES auth.users(id),
    is_deleted BOOLEAN DEFAULT FALSE,
    version BIGINT DEFAULT 1,    -- Monotonic version counter
    
    UNIQUE(vault_id, path)
);

-- Index for fast sync queries
CREATE INDEX idx_files_vault_sync ON file_metadata(vault_id, updated_at);

---

3. Sync Protocol & Flow

The sync process is a loop that runs periodically (e.g., every 30s) or on file change events.

3.1 The "Handshake" (State Reconciliation)

Before sending files, the client and server agree on what is different.

3.2 Flows

A. Upload (Client Push)

1. Client receives to_upload list.
2. For each file:
   • Client PUT /files/:vault_id/:path.
   • Server validates stream, calculates checksum.
   • Server Uploads to S3.
   • Server Updates file_metadata (Increment version).

B. Download (Client Pull)

1. Client receives to_download list.
2. For each file:
   • Client GET /files/:vault_id/:path.
   • Server fetches from S3 (presigned URL or proxy stream).
   • Client writes to disk, updating local DB state with new checksum.
   • Client writes to disk, updating local DB state with new checksum.

---

4. API Specification (Rust/Axum)

The server will be structured as a new monorepo app: apps/sync.

4.1 Endpoints

Method	Path	Description	Payload
GET	/health	Health Check	-
POST	/v1/vaults	Create a new Cloud Vault	{ name: "My Notes" }
GET	/v1/vaults	List user's vaults	-
POST	/v1/sync/state	Core Sync Handshake	{ vault_id, changes: [FileState] }
PUT	/v1/files/:vault_id/*path	Upload File Content	Binary Body
GET	/v1/files/:vault_id/*path	Download File Content	-

4.2 Code Structure (apps/sync)

apps/sync/
├── Cargo.toml
├── src/
│   ├── main.rs              # App entry, Router setup
│   ├── config.rs            # Env vars (S3_BUCKET, DB_URL)
│   ├── api/
│   │   ├── mod.rs
│   │   ├── auth.rs          # Auth Middleware (Supabase)
│   │   ├── handlers.rs      # Route logic
│   │   └── dto.rs           # JSON Structs
│   ├── domain/
│   │   ├── models.rs        # Database Structs
│   │   └── services.rs      # Business Logic (Diff Algorithm)
│   └── infra/
│       ├── db.rs            # Postgres Connection (sqlx)
│       └── storage.rs       # S3 Client (rust-s3)

---

5. Implementation Roadmap

Phase 1: Foundation

• Initialize apps/sync Rust project.
• Setup axum, tokio, tracing.
• Integrate crates/supabase-auth.
• Setup Postgres connection (sqlx) and migrations.
• Setup S3 client (rust-s3 or aws-sdk-s3).

Phase 2: The Core Loop

• Implement POST /v1/sync/state (The logic brain).
• Implement File Upload/Download (The I/O muscle).
• Write integration tests for the sync flow.

Phase 3: Desktop Integration (Later)

• Update plugins/fs-db to calculate checksums.
• Create a BackgroundWorker in apps/desktop to run the sync loop.

6. Security Considerations

1. Transport: All traffic over HTTPS (TLS).
2. Auth: standard JWT Bearer tokens from Supabase.
3. Access Control: Middleware checks vault_access table before allowing ANY operation on a vault.
4. Path Traversal Prevention: Strict validation of path params to prevent writing outside the bucket prefix.

7. Local File Structure (Context)

Based on codebase analysis (plugins/fs-db), the local client stores data in:
$DATA/sessions/<session_id>/

• memo.md
• transcript.json

The Sync Service will map this transparently:
VaultID = User Root
Path = sessions/<session_id>/memo.md

[thecodeblood]

Hi @yujonglee,

Attached is the system design for the sync engine. It follows a local-first, vault-centric model using Rust/Axum.

Could you take a look at the Sync Handshake logic and the Optimistic Concurrency Control section? Once approved, I’ll begin the foundation work in apps/sync.

[thecodeblood]

Hi @yujonglee,

Just following up on the sync engine design doc I shared earlier.

Please let me know if you’ve had a chance to review the Sync Handshake and Optimistic Concurrency Control sections. If it looks good, I’d like to start implementing it in my fork so we can iterate there and later merge the changes into the main repo after your review.

Thanks again for your time and feedback!