Rewrite client-side GraceStatus storage to use SQLite#61
Conversation
…k to SQLite. Improvements to Aspire setup. Doc updates. More tests.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8a4973bda2
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| directoryCommand.CommandText <- | ||
| "SELECT relative_path, parent_path, directory_version_id, sha256_hash, size_bytes, created_at_unix_ticks, last_write_time_utc_ticks FROM status_directories;" | ||
|
|
||
| use directoryReader = directoryCommand.ExecuteReader() |
There was a problem hiding this comment.
readStatusSnapshot issues separate SELECT statements for status_directories and status_files without an explicit read transaction, so each query can observe a different committed version when another command writes concurrently. In WAL mode this means a writer can commit between the two reads, producing a mixed snapshot (e.g., files referencing directory IDs not present in the directory set), which can silently drop entries when rebuilding GraceStatus and lead to incorrect diffs/saves.
Useful? React with 👍 / 👎.
| let schemaExists (connection: SqliteConnection) = | ||
| use cmd = connection.CreateCommand() | ||
| cmd.CommandText <- "SELECT 1 FROM sqlite_master WHERE type = 'table' AND name = 'meta' LIMIT 1;" | ||
| use reader = cmd.ExecuteReader() |
There was a problem hiding this comment.
Initialization currently treats the DB as schema-ready as soon as the meta table exists, but partial/aborted schema creation can leave meta present while tables like status_meta are absent. In that state ensureDbInitialized skips runSchema and later fails on insertStatusMetaIfMissing (no such table), causing commands to fail instead of self-healing by recreating/repairing schema.
Useful? React with 👍 / 👎.
1 participant
Grace currently persists local working-directory state (the “status” snapshot) and the local object cache index using MessagePack files (gracestatus.msgpack and graceObjectCache.msgpack). This approach becomes impractical at very large scale (for example, 1,000,000+ files) because full-file serialization and deserialization costs dominate normal CLI workflows, especially grace watch, which frequently reads and writes the status snapshot.
This change replaces both MessagePack-backed files with a single SQLite database file stored inside .grace/, prioritizing correctness and safe multi-writer concurrency over raw speed. You have explicitly confirmed:
No backwards compatibility is required.
No migration path is required.
Rename the persisted artifact (new filename).
Multiple commands may write concurrently.
Use a single DB for both concerns.
SQLite synchronous = NORMAL.
Goals
Eliminate full snapshot serialization/deserialization overhead by replacing MessagePack status/object-cache persistence with SQLite.
Support safe concurrent writers from multiple Grace commands/processes without corruption or frequent “database is locked” failures.
Keep CLI behavior functionally equivalent (same semantics of status and object cache), within the existing “local cache” expectations.
Enable incremental updates so that common operations do not rewrite the entire dataset.
Non-goals
Migrating existing .msgpack data into the new database.
Guaranteeing “semantic merge correctness” between competing writers (for example, two simultaneous updates computed from different snapshots). The correctness target is “no corruption, atomic writes, consistent schema”; last-commit-wins semantics are acceptable for a local cache.