Skip to content

Commit 6dcf3b5

Browse files
EconoBenwesmclaude
authored
feat: TUI remote support (#130) (#138)
## The Problem msgvault already supports remote CLI commands (`stats`, `search`, `show-message`) against a NAS server. But the TUI—the primary way to explore your archive—only works locally. If your archive lives on a NAS, you can't browse it from your laptop. ## What This PR Does Enables `msgvault tui` to work transparently against a remote server. When `[remote].url` is configured, the TUI connects to your NAS and gives you the full browsing experience: aggregate views, drill-down navigation, search, filtering—everything works identically to local mode. ```bash # In config.toml [remote] url = "https://nas:8080" api_key = "your-key" # Then just run TUI as normal msgvault tui # Connects to NAS automatically msgvault tui --local # Force local if needed ``` ## Implementation **Server side:** 6 new API endpoints that expose the `query.Engine` capabilities needed by the TUI: - `/api/v1/aggregates` – sender/domain/label/time aggregations - `/api/v1/aggregates/sub` – drill-down into sub-aggregations - `/api/v1/messages/filter` – filtered message lists with pagination - `/api/v1/stats/total` – stats with filter context - `/api/v1/search/fast` – metadata search (subject, sender, recipient) - `/api/v1/search/deep` – full-text body search via FTS5 **Client side:** `remote.Engine` implements the full `query.Engine` interface over HTTP, so the TUI code doesn't change at all—it just gets a different engine. **Safety:** Deletion staging and attachment export are disabled in remote mode. These require local file access and are potentially destructive, so they're local-only operations. ## Testing - [x] All handler tests pass (10 new tests for aggregate endpoints) - [x] Remote engine unit tests pass - [x] TUI builds with remote support - [ ] Manual end-to-end testing against NAS --- Closes #130 --------- Co-authored-by: Wes McKinney <wesmckinn+git@gmail.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
1 parent bc9e377 commit 6dcf3b5

File tree

22 files changed

+2867
-171
lines changed

22 files changed

+2867
-171
lines changed

.roborev.toml

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,12 @@ HTTP remote defaults, plaintext key display in interactive CLI,
1414
enabled=true override on account creation, and page-aligned pagination
1515
are documented design decisions — see code comments at each site.
1616
17+
Remote engine query string reconstruction in buildSearchQueryString is
18+
intentionally simplified — phrase quoting edge cases are acceptable since
19+
the search parser on the server re-parses the query. Empty search queries
20+
sending q= is expected; the server returns empty results gracefully.
21+
TimeGranularity defaults to "month" when unspecified, which is correct.
22+
1723
This is a single-user personal tool with no privilege separation, no
1824
setuid, no shared directories, and no multi-tenant access. Do not flag
1925
symlink-following, local file overwrites, or similar CWE patterns that

CLAUDE.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,7 @@ make lint # Run linter
5353
# TUI and analytics
5454
./msgvault tui # Launch TUI
5555
./msgvault tui --account you@gmail.com # Filter by account
56+
./msgvault tui --local # Force local (override remote config)
5657
./msgvault build-cache # Build Parquet cache
5758
./msgvault build-cache --full-rebuild # Full rebuild
5859
./msgvault stats # Show archive stats
@@ -63,6 +64,9 @@ make lint # Run linter
6364
./msgvault import-emlx --account me@gmail.com # Specific account(s)
6465
./msgvault import-emlx /path/to/dir --identifier me@gmail.com # Manual fallback
6566

67+
# Daemon mode (NAS/server deployment)
68+
./msgvault serve # Start HTTP API + scheduled syncs
69+
6670
# Maintenance
6771
./msgvault repair-encoding # Fix UTF-8 encoding issues
6872
```

README.md

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,15 +81,20 @@ msgvault tui
8181
| `add-account EMAIL` | Authorize a Gmail account (use `--headless` for servers) |
8282
| `sync-full EMAIL` | Full sync (`--limit N`, `--after`/`--before` for date ranges) |
8383
| `sync EMAIL` | Sync only new/changed messages |
84-
| `tui` | Launch the interactive TUI (`--account` to filter) |
84+
| `tui` | Launch the interactive TUI (`--account` to filter, `--local` to force local) |
8585
| `search QUERY` | Search messages (`--json` for machine output) |
86+
| `show-message ID` | View full message details (`--json` for machine output) |
8687
| `mcp` | Start the MCP server for AI assistant integration |
88+
| `serve` | Run daemon with scheduled sync and HTTP API for remote TUI |
8789
| `stats` | Show archive statistics |
90+
| `list-accounts` | List synced email accounts |
8891
| `verify EMAIL` | Verify archive integrity against Gmail |
8992
| `export-eml` | Export a message as `.eml` |
9093
| `import-mbox` | Import email from an MBOX export or `.zip` of MBOX files |
9194
| `import-emlx` | Import email from an Apple Mail directory tree |
9295
| `build-cache` | Rebuild the Parquet analytics cache |
96+
| `update` | Update msgvault to the latest version |
97+
| `setup` | Interactive first-run configuration wizard |
9398
| `repair-encoding` | Fix UTF-8 encoding issues |
9499
| `list-senders` / `list-domains` / `list-labels` | Explore metadata |
95100

@@ -125,6 +130,30 @@ See the [Configuration Guide](https://msgvault.io/configuration/) for all option
125130

126131
msgvault includes an MCP server that lets AI assistants search, analyze, and read your archived messages. Connect it to Claude Desktop or any MCP-capable agent and query your full message history conversationally. See the [MCP documentation](https://msgvault.io/usage/chat/) for setup instructions.
127132

133+
## Daemon Mode (NAS/Server)
134+
135+
Run msgvault as a long-running daemon for scheduled syncs and remote access:
136+
137+
```bash
138+
msgvault serve
139+
```
140+
141+
Configure scheduled syncs in `config.toml`:
142+
143+
```toml
144+
[[accounts]]
145+
email = "you@gmail.com"
146+
schedule = "0 2 * * *" # 2am daily (cron)
147+
enabled = true
148+
149+
[server]
150+
api_port = 8080
151+
bind_addr = "0.0.0.0"
152+
api_key = "your-secret-key"
153+
```
154+
155+
The TUI can connect to a remote server by configuring `[remote].url`. Use `--local` to force local database when remote is configured. See [docs/api.md](docs/api.md) for the HTTP API reference.
156+
128157
## Documentation
129158

130159
- [Setup Guide](https://msgvault.io/guides/oauth-setup/): OAuth, first sync, headless servers

cmd/msgvault/cmd/build_cache.go

Lines changed: 43 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -27,10 +27,11 @@ var fullRebuild bool
2727
// files (_last_sync.json, parquet directories) can corrupt the cache.
2828
var buildCacheMu sync.Mutex
2929

30-
// syncState tracks the last exported message ID for incremental updates.
30+
// syncState tracks the message and sync-run watermarks covered by the cache.
3131
type syncState struct {
32-
LastMessageID int64 `json:"last_message_id"`
33-
LastSyncAt time.Time `json:"last_sync_at"`
32+
LastMessageID int64 `json:"last_message_id"`
33+
LastSyncAt time.Time `json:"last_sync_at"`
34+
LastCompletedSyncRunID int64 `json:"last_completed_sync_run_id,omitempty"`
3435
}
3536

3637
var buildCacheCmd = &cobra.Command{
@@ -116,13 +117,39 @@ func buildCache(dbPath, analyticsDir string, fullRebuild bool) (*buildResult, er
116117
}
117118

118119
var maxMessageID sql.NullInt64
120+
var lastCompletedSyncRunID int64
119121
// Use indexed query: id is PRIMARY KEY, sent_at has an index
120122
maxIDQuery := `SELECT MAX(id) FROM messages WHERE sent_at IS NOT NULL`
121123
if err := sqliteDB.QueryRow(maxIDQuery).Scan(&maxMessageID); err != nil {
122-
sqliteDB.Close()
124+
if closeErr := sqliteDB.Close(); closeErr != nil {
125+
return nil, fmt.Errorf("get max message id: %w; close sqlite: %v", err, closeErr)
126+
}
123127
return nil, fmt.Errorf("get max message id: %w", err)
124128
}
125-
sqliteDB.Close()
129+
var hasSyncRunsTable int
130+
if err := sqliteDB.QueryRow(`
131+
SELECT COUNT(*) FROM sqlite_master
132+
WHERE type = 'table' AND name = 'sync_runs'
133+
`).Scan(&hasSyncRunsTable); err != nil {
134+
if closeErr := sqliteDB.Close(); closeErr != nil {
135+
return nil, fmt.Errorf("check sync_runs table: %w; close sqlite: %v", err, closeErr)
136+
}
137+
return nil, fmt.Errorf("check sync_runs table: %w", err)
138+
}
139+
if hasSyncRunsTable > 0 {
140+
if err := sqliteDB.QueryRow(`
141+
SELECT COALESCE(MAX(id), 0) FROM sync_runs
142+
WHERE status = 'completed' AND completed_at IS NOT NULL
143+
`).Scan(&lastCompletedSyncRunID); err != nil {
144+
if closeErr := sqliteDB.Close(); closeErr != nil {
145+
return nil, fmt.Errorf("get last completed sync run id: %w; close sqlite: %v", err, closeErr)
146+
}
147+
return nil, fmt.Errorf("get last completed sync run id: %w", err)
148+
}
149+
}
150+
if err := sqliteDB.Close(); err != nil {
151+
return nil, fmt.Errorf("close sqlite after metadata check: %w", err)
152+
}
126153

127154
maxID := int64(0)
128155
if maxMessageID.Valid {
@@ -180,6 +207,12 @@ func buildCache(dbPath, analyticsDir string, fullRebuild bool) (*buildResult, er
180207
fmt.Println("Building cache...")
181208
buildStart := time.Now()
182209

210+
// Capture deletion watermark before export starts. Any deletion
211+
// with deleted_from_source_at after this timestamp may not be
212+
// reflected in the exported Parquet data and will trigger a
213+
// cache rebuild on the next freshness check.
214+
cacheWatermark := time.Now().UTC().Truncate(time.Second)
215+
183216
// Build WHERE clause for incremental exports
184217
idFilter := ""
185218
if !fullRebuild && lastMessageID > 0 {
@@ -391,10 +424,12 @@ func buildCache(dbPath, analyticsDir string, fullRebuild bool) (*buildResult, er
391424
exportedCount = 0
392425
}
393426

394-
// Save sync state
427+
// Save sync state using the pre-export watermark so any deletion
428+
// that occurs during or after the build is detected as stale.
395429
state := syncState{
396-
LastMessageID: maxID,
397-
LastSyncAt: time.Now(),
430+
LastMessageID: maxID,
431+
LastSyncAt: cacheWatermark,
432+
LastCompletedSyncRunID: lastCompletedSyncRunID,
398433
}
399434
stateData, _ := json.Marshal(state)
400435
if err := os.WriteFile(stateFile, stateData, 0644); err != nil {

0 commit comments

Comments
 (0)