perf(submit): diff-based incremental submission optimization

[junhoyeo]

Summary

Optimizes tokscale submit by implementing diff-based incremental uploads. Instead of sending the entire contribution history on every submit, the CLI now:

1. Fetches per-date per-source checksums from the server
2. Computes local checksums using the same algorithm
3. Only uploads changed or new (date, source) pairs

This significantly reduces payload size and server processing time for repeat submissions.

Changes

New API Endpoint

• GET /api/submit/checksum - Returns checksums for user's existing daily breakdown data
  • Format: { checksums: { "2024-12-01": { "claude": "abc123", "opencode": "def456" }, ... } }
  • Uses Bearer token authentication (same as submit)

CLI Changes

• Added hashSourceBreakdown() - djb2 hash function for deterministic source data hashing
• Added fetchServerChecksums() - fetches existing checksums from server
• Added computeLocalChecksums() - computes hashes for local contribution data
• Added computeDiff() - filters to only changed (date, source) pairs
• Modified submit() - now uses diff-based flow by default
• Added --full flag - bypasses diff optimization (for troubleshooting)

Helper Function

• Added hashSourceBreakdown() to packages/frontend/src/lib/db/helpers.ts
  • Deterministic djb2 hash for source breakdown data
  • Used by both server (checksum endpoint) and CLI (local checksums)

New Submit Flow

┌─────────────────────────────────────────────────────────────┐
│                          CLI                                │
├─────────────────────────────────────────────────────────────┤
│ 1. Scan local session data (same as before)                │
│ 2. GET /api/submit/checksum → server checksums             │
│ 3. Compute local checksums (djb2)                          │
│ 4. Diff: find changed (date, source) pairs                 │
│ 5. POST /api/submit with only changed data                 │
└─────────────────────────────────────────────────────────────┘

CLI Output Examples

When changes detected:

  Tokscale - Submit Usage Data

  Scanning local session data...
  Local data scanned:
    Date range: 2024-01-01 to 2024-12-23
    Active days: 180
    Total tokens: 45,000,000
    ...

  Fetching server checksums for diff...
  Changes detected:
    Days with changes: 3
    Tokens in diff: 45,000
    Cost in diff: $2.50

  Submitting changes...
  Successfully submitted!
    Mode: incremental (diff-based)

When already up to date:

  Fetching server checksums for diff...

  Already up to date! No changes to submit.

  View your profile: https://tokscale.ai/u/junhoyeo

Flags

Flag	Description
--full	Skip diff optimization, upload all data

Edge Cases Handled

• First submission: Falls back to full upload (no server checksums exist)
• Server unreachable: Falls back to full upload
• Source filter (--claude, --opencode, etc.): Only diffs for specified sources

Testing

• TypeScript compilation passes (CLI + Frontend)
• Manual testing with actual submit flow
• Verify checksum consistency between CLI and server

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Review	Updated (UTC)
tokscale	Ready	Preview, Comment	Dec 23, 2025 5:51am

[junhoyeo]

Closing since performance issues have been resolved in #129