feat: token handling

Author: evansims

.serena/memories/development_guidelines.md

@@ -54,13 +54,20 @@ Target: 90%+ coverage
 2. **Green**: Minimal code to make tests pass
 3. **Refactor**: Clean up while keeping tests green
 
-Run coverage: `cargo tarpaulin` or `cargo llvm-cov`
-
 ## Tooling
 
-### Formatting
-Use nightly toolchain for formatting:
+### Build & Test
+Use `just` for common tasks (see `just` or `Justfile` for full list):
+```bash
+just check     # pre-commit: fmt + clippy + test
+just ci        # CI validation: fmt + clippy + doc-check + test
+```
+
+Or cargo with pinned toolchain:
 ```bash
+cargo +1.92 build --workspace
+cargo +1.92 test --workspace --lib
+cargo +1.92 clippy --workspace --all-targets -- -D warnings
 cargo +nightly fmt
 ```
 

.serena/memories/project_overview.md

@@ -6,23 +6,33 @@ Ledger is InferaDB's storage layer — a blockchain database for cryptographical
 ## Tech Stack
 - **Language**: Rust 1.92 (2024 edition)
 - **Storage**: inferadb-ledger-store (custom embedded ACID B+ tree engine)
-- **Consensus**: openraft (Raft implementation)
+- **Consensus**: openraft 0.9 (Raft implementation)
 - **Networking**: gRPC via tonic/prost (HTTP/2 over TCP)
-- **Crypto**: SHA-256, seahash, rs_merkle
-- **Error Handling**: snafu with backtraces
+- **Crypto**: SHA-256, seahash, rs_merkle, Ed25519 (JWT signing), AES-256-GCM (envelope encryption)
+- **Error Handling**: snafu with implicit location tracking (server crates), thiserror (SDK crate)
+- **Builders**: bon crate for type-safe builders
 
-## Crate Structure
-- `inferadb-ledger-types`: Core types, errors, crypto primitives
-- `inferadb-ledger-store`: Embedded B+ tree database engine
-- `inferadb-ledger-state`: Domain state management, indexes, snapshots
-- `inferadb-ledger-raft`: Raft consensus, gRPC services
-- `inferadb-ledger-server`: Server binary entry point
-- `inferadb-ledger-sdk`: Production-grade Rust SDK
-- `inferadb-ledger-test-utils`: Shared test utilities
+## Crate Structure (9 crates)
+- `inferadb-ledger-types`: Core types, errors, crypto primitives, config, token claims, newtype IDs
+- `inferadb-ledger-store`: Embedded B+ tree database engine, crypto key management
+- `inferadb-ledger-proto`: Protobuf code generation and From/TryFrom conversions
+- `inferadb-ledger-state`: Domain state, entity/relationship CRUD, system services (users, signing keys, tokens)
+- `inferadb-ledger-raft`: Raft consensus, transaction batching, rate limiting, saga orchestrator, background jobs
+- `inferadb-ledger-services`: gRPC service implementations (12 services), JwtEngine, LedgerServer assembly
+- `inferadb-ledger-server`: Server binary entry point, bootstrap, CLI configuration
+- `inferadb-ledger-sdk`: Production-grade Rust SDK, retry/circuit-breaker, cancellation, metrics
+- `inferadb-ledger-test-utils`: Shared test utilities, crash injection, proptest strategies
 
 ## Key Concepts
-- **Organization**: Top-level tenant isolation boundary
-- **Vault**: Relationship store with own cryptographic chain
+- **Organization**: Top-level tenant isolation boundary (dual-ID: OrganizationId/OrganizationSlug)
+- **Vault**: Relationship store with own cryptographic chain (dual-ID: VaultId/VaultSlug)
 - **Entity**: Key-value data with TTL/versioning
 - **Relationship**: Authorization tuple (resource, relation, subject)
+- **User**: Identity with email, role, status, token version (dual-ID: UserId/UserSlug)
+- **App**: Organization-scoped client application (dual-ID: AppId/AppSlug)
+- **SigningKey**: Ed25519 JWT signing key with scope and lifecycle
+- **RefreshToken**: Session token family with rotate-on-use and poison detection
 - **Shard**: Multiple organizations sharing a Raft group
+
+## gRPC Services (12)
+Read, Write, Admin, Organization, Vault, User, App, Token, Events, Health, Discovery, Raft

.serena/memories/style_conventions.md

@@ -13,24 +13,32 @@
 ## Linting Rules
 **Denied:**
 - `unsafe_code` - No unsafe code allowed
-- `unwrap_used` - Use snafu error handling
+- `unwrap_used` - Use snafu error handling (server) or thiserror (SDK)
 - `panic` - No panics in production code
+- `todo` - No `todo!()` or `unimplemented!()` allowed
 
 **Warned:**
 - `missing_docs` - Document public items
 - `expect_used` - Prefer proper error handling
-- `todo` - Allowed but flagged
 
 ## Error Handling
-- Use `snafu` with backtraces
-- No `.unwrap()` - use `.context()` or `.ok_or()`
-- Use `?` operator for propagation
+- **Server crates**: snafu with implicit location tracking and `.context()` propagation
+- **SDK crate**: thiserror for consumer-facing error types
+- No `.unwrap()` — use `.context()`, `.ok_or()`, or `?`
+
+## Builders (bon)
+- `#[derive(bon::Builder)]` for simple structs
+- `#[bon]` impl block with `#[builder]` for fallible constructors
+- `#[builder(into)]` for String fields
+- Match `#[builder(default)]` with `#[serde(default)]` for config
 
 ## Documentation
 - Crate-level docs with `//!` comments
 - Document all public types and functions
+- Code examples use ` ```no_run ` (never `ignore` or `text`)
 
 ## Naming
 - Types: PascalCase (OrganizationId, VaultBlock)
 - Functions: snake_case (bucket_id, sha256_concat)
 - Constants: SCREAMING_SNAKE_CASE (EMPTY_HASH)
+- Newtype IDs: `define_id!` / `define_slug!` macros in types/src/types.rs

.serena/memories/suggested_commands.md

@@ -1,39 +1,30 @@
 # Development Commands
 
-## Build
+## Using `just` (preferred)
 ```bash
-cargo build                              # Build all crates
-cargo build --release                    # Release build
-cargo build -p inferadb-ledger-types     # Build specific crate
+just check        # pre-commit: fmt + clippy + test
+just check-quick  # fast pre-commit: fmt + clippy only
+just ci           # CI validation: fmt + clippy + doc-check + test
+just ready        # pre-PR: proto + fmt + clippy + test
+just test         # unit tests only (--workspace --lib)
+just test-ff      # unit tests, stop on first failure
+just test-integration     # integration tests (spawns clusters)
+just test-integration-ff  # integration tests, stop on first failure
+just fmt          # format code (nightly)
+just clippy       # run linter
+just doc-check    # build rustdoc with -D warnings
+just proto        # regenerate protobuf code
+just run          # run server (dev mode)
 ```
 
-## Testing
+## Using cargo directly
 ```bash
-cargo test                               # Run all tests
-cargo test -p inferadb-ledger-state      # Test specific crate
-cargo test -- --nocapture                # With output
-```
-
-## Linting & Formatting
-```bash
-cargo +nightly fmt             # Format code (nightly required)
-cargo +nightly fmt --check     # Check formatting
-cargo clippy --all-targets     # Run clippy
-cargo clippy -- -D warnings    # Clippy with warnings as errors
-```
-
-## Documentation
-```bash
-cargo doc --open               # Generate and open docs
-```
-
-## Running
-```bash
-cargo run -p inferadb-ledger-server           # Run server (dev)
-cargo run -p inferadb-ledger-server --release # Run server (release)
-```
-
-## Full Check Before Commit
-```bash
-cargo +nightly fmt --check && cargo clippy --all-targets -- -D warnings && cargo test
+cargo +1.92 build --workspace              # Build all crates
+cargo +1.92 build -p inferadb-ledger-types # Build specific crate
+cargo +1.92 test --workspace --lib         # Unit tests
+cargo +1.92 test -p inferadb-ledger-state  # Test specific crate
+cargo +1.92 test <name> -- --nocapture     # Single test with output
+cargo +nightly fmt                         # Format (nightly required)
+cargo +1.92 clippy --workspace --all-targets -- -D warnings
+cargo +1.92 doc --workspace --no-deps      # Generate docs
 ```

.serena/memories/task_completion.md

@@ -2,19 +2,25 @@
 
 ## Before Marking Complete
 
-1. **Format**: `cargo +nightly fmt`
-2. **Clippy**: `cargo +1.85 clippy --all-targets -- -D warnings`
-3. **Tests**: `cargo test`
-4. **Build**: `cargo build`
+1. **Format**: `cargo +nightly fmt --check`
+2. **Clippy**: `cargo +1.92 clippy --workspace --all-targets -- -D warnings`
+3. **Tests**: `cargo +1.92 test --workspace --lib`
+4. **Build**: `cargo +1.92 build --workspace`
 
 ## Quick One-Liner
 ```bash
-cargo +nightly fmt && cargo +1.85 clippy --all-targets -- -D warnings && cargo test
+just ci
+```
+
+Or manually:
+```bash
+cargo +nightly fmt --check && cargo +1.92 clippy --workspace --all-targets -- -D warnings && cargo +1.92 test --workspace --lib
 ```
 
 ## Critical Rules
-- Never use `.unwrap()` - always snafu error handling
-- Never use `panic!` in non-test code
+- Never use `.unwrap()` — use snafu `.context()` (server) or thiserror (SDK)
+- Never use `panic!`, `todo!()`, `unimplemented!()`
 - No unsafe code
+- No TODO/FIXME/HACK comments
 - Document all public items
 - Use `?` for error propagation

.serena/project.yml

@@ -87,26 +87,32 @@ initial_prompt: |
   It exposes a gRPC API (HTTP/2 over TCP) and uses TCP for inter-node Raft consensus.
 
   ## Core Terminology
-  - **Namespace**: Storage unit per organization, containing entities and vaults. Isolated with separate Raft consensus per shard.
-  - **Vault**: Relationship store within a namespace. Each vault maintains its own cryptographic chain (state_root, previous_hash, block height).
-  - **Entity**: Key-value data stored in a namespace (users, teams, clients, sessions). Supports TTL, versioning, conditional writes.
-  - **Relationship**: Authorization tuple in a vault: (resource, relation, subject). Used by Engine for permission checks.
-  - **_system namespace**: Global data (user accounts, namespace routing), replicated to all nodes.
-  - **Shard**: Multiple namespaces share a Raft group for efficiency. Vaults within a shard have independent cryptographic chains.
+  - **Organization**: Top-level tenant isolation boundary. Sequential `OrganizationId(i64)` for storage, Snowflake `OrganizationSlug(u64)` for external APIs.
+  - **Vault**: Relationship store within an organization. Each vault maintains its own cryptographic chain (state_root, previous_hash, block height).
+  - **Entity**: Key-value data stored in an organization. Supports TTL, versioning, conditional writes.
+  - **Relationship**: Authorization tuple in a vault: (resource, relation, subject).
+  - **User**: Identity with email, role, status, and token version. Stored in `_system` organization.
+  - **App**: Organization-scoped client application with vault connections.
+  - **SigningKey**: Ed25519 JWT signing key with scope (Global/Organization) and lifecycle (Active/Rotated/Revoked).
+  - **RefreshToken**: Session token family with rotate-on-use and poison detection.
+  - **Team**: Organization-scoped user group.
+  - **Shard**: Multiple organizations sharing a Raft group. Vaults within a shard have independent cryptographic chains.
 
   ## Architecture Highlights
   - Hybrid storage: State commitment (merkleized) is separate from state storage (fast K/V) to avoid write amplification.
   - Bucket-based state roots: 256 buckets with incremental hashing, O(k) complexity where k = dirty keys.
   - Per-vault failure isolation: Divergence in one vault doesn't cascade to other vaults in the same shard.
-  - Leader-assigned sequential IDs: Deterministic for Raft replay (NamespaceId, VaultId, UserId are all int64).
+  - Dual-ID architecture: Internal sequential IDs (`i64`) for storage, Snowflake slugs (`u64`) for APIs. `SlugResolver` translates at gRPC boundaries.
+  - JWT token infrastructure: Ed25519 signing with envelope encryption (AES-256-GCM + AES-KWP), ArcSwap lock-free key cache.
 
   ## Key Design Decisions
   - All operations are idempotent, resolved by Raft total ordering.
   - SHA-256 for all cryptographic commitments; seahash for bucket assignment.
   - Embedded storage backend (single-file, ACID, COW B-trees).
   - Pagination tokens are opaque, HMAC-signed, and include height for consistent reads.
+  - Server crates use snafu for error handling; SDK uses thiserror.
 
-  See DESIGN.md for full specifications.
+  See DESIGN.md for full specifications. See AGENTS.md for development conventions.
 # the name by which the project can be referenced within Serena
 project_name: "InferaDB Ledger"
 
@@ -146,3 +152,8 @@ language_backend:
 # list of regex patterns which, when matched, mark a memory entry as read‑only.
 # Extends the list from the global configuration, merging the two lists.
 read_only_memory_patterns: []
+
+# line ending convention to use when writing source files.
+# Possible values: unset (use global setting), "lf", "crlf", or "native" (platform default)
+# This does not affect Serena's own files (e.g. memories and configuration files), which always use native line endings.
+line_ending: