[pull] main from commontoolsinc:main #139
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* additional cache logging * enabling error logging, some unit test cases needed error logging * adding more debugging for doc ids and queries * scheduler logging --------- Co-authored-by: Claude <[email protected]>
* perf: Cache parsed SourceFiles for type definitions in fixture tests Eliminates redundant parsing of type definition files (es2023.d.ts, dom.d.ts, jsx.d.ts) across 166 fixtures by caching parsed SourceFiles in the CompilerHost. Performance improvements: - Without CHECK_INPUT: 9.7s → 3.4s (65% faster) - With CHECK_INPUT=1: 23.7s → 16.4s (31% faster) - Per-fixture overhead: ~0.09s → ~0.05s (44% reduction) Implementation: - Added sourceFileCache Map at module level - Modified CompilerHost.getSourceFile() to check cache first - Only caches type definition files, not fixture inputs Type definition files are identical across all fixtures and TypeScript SourceFile objects are immutable, making caching safe and effective. * Implement batched type-checking optimization (Chunks 1-2) Add infrastructure for batching TypeScript type-checking across multiple fixtures to improve performance by 4-6x (from ~14s to ~2-3s). Chunk 1: Create batchTypeCheckFixtures() function - Add BatchTypeCheckResult interface with diagnosticsByFile Map and program - Implement batchTypeCheckFixtures() to type-check multiple files in single program - Apply transformCtDirective to all input files - Load environment types using existing envTypesCache pattern - Create single CompilerHost with all files (reuses transformFiles pattern) - Run ts.getPreEmitDiagnostics() once for all files - Filter diagnostics to exclude type definition files - Group diagnostics by file name into Map for efficient lookup Chunk 2: Add precomputedDiagnostics to TransformOptions - Add precomputedDiagnostics field to TransformOptions interface - Modify transformFiles() to use precomputed diagnostics when provided - Use nullish coalescing operator (??) to fall back to computing diagnostics - Maintains backward compatibility - existing behavior unchanged when not provided This enables batching all 166 fixtures into a single program instead of creating separate programs per fixture. * Implement Chunk 3: Integrate batching into fixture test suite Complete the batching optimization by integrating batch type-checking into the fixture test runner, achieving 5.8x performance improvement (from ~14s to ~2.5s with CHECK_INPUT=1). Key changes: - Add loadAllFixturesInDirectory() to collect all fixtures per config - Perform batch type-checking upfront before tests run when CHECK_INPUT=1 - Store diagnostics per config in batchedDiagnosticsByConfig Map - Lookup and pass precomputedDiagnostics to transformFixture in execute() - Fix diagnostic filtering to keep diagnostics without associated files - Initialize diagnostics map with all input files (not just error files) Performance results: - WITH CHECK_INPUT: 14.3s → 2.5s (5.8x faster) - WITHOUT CHECK_INPUT: 1.9s → 1.7s (no regression) The optimization works by: 1. Loading all fixture files in each test configuration directory 2. Type-checking them all together in a single TypeScript program 3. Storing diagnostics keyed by file path in a Map 4. Looking up precomputed diagnostics by path in each test execution 5. Skipping expensive ts.getPreEmitDiagnostics() calls per-fixture * docs: Add comprehensive documentation for batched type-checking optimization Added detailed comment block explaining the batching optimization strategy, performance improvements, and how it works. This completes Chunk 4 of the batched type-checking implementation. Performance summary: - Original: 23.7s with CHECK_INPUT=1 - After SourceFile caching: 16.4s (31% faster) - After batching: 3.8s (84% faster, 6.3x total speedup) The batching optimization achieves 4.3x speedup vs the SourceFile-cached baseline by creating one TypeScript program for all fixtures instead of 166 separate programs. * many fixture updates
…ive (#1966) * feat(transform): optimize && and || to when/unless helpers - Transform `condition && value` to `when(condition, value)` - Transform `condition || fallback` to `unless(condition, fallback)` - Add when() and unless() built-in functions to runner - Add comprehensive test fixtures for logical operator transformations - Add runtime unit tests for when/unless - Add integration pattern tests for when/unless operators * update fixtures to typecheck
* fix(ts-transformers): Widen literal types in value-based type inference Adds consistent literal widening so that `const x = 5; derive(x, fn)` produces `type: "number"` instead of `"enum": [5]` in generated schemas. The fix applies widening at the Type level for all value-based inference: - Direct derive/lift arguments - Closure captures in derive/handler/lift callbacks - cell() initial values Key changes: - Add inferWidenedTypeFromExpression() helper to type-inference.ts - Update expressionToTypeNode() to use widening for closure captures - Apply widening to derive input argument types in schema-injection.ts This preserves literal types where appropriate (explicit annotations, return types, parameter types) while widening inferred value types. * test(ts-transformers): Add comprehensive literal widening test fixture Tests all primitive literal types (number, string, boolean, float) in closure captures to ensure they are properly widened: - `42` → `type: "number"` (not `"enum": [42]`) - `"hello"` → `type: "string"` (not `"enum": ["hello"]`) - `true` → `type: "boolean"` (not `"enum": [true]`) - `3.14` → `type: "number"` (not `"enum": [3.14]`)
…2190) * Remove `schemaifyWish` workaround now `wish` infers type correctly * Use `Default<>` instead of retired second argument
…er reads) (#2174) * feat(memory): add benchmarks for fact operations Add comprehensive benchmarks to measure write and read performance, including isolation benchmarks to identify specific bottlenecks. * perf(memory): add LRU memoization for merkle reference hashing Add bounded LRU cache (1000 entries) to memoize refer() results in reference.ts. refer() is a pure function computing SHA-256 hashes, which was identified as the primary bottleneck via isolation benchmarks. Benchmark results for cache hits: - 3x refer() calls: 44µs vs ~500µs uncached (27x faster) - 10x unclaimed refs: 2.5µs (400k ops/sec) The memoization benefits real-world usage patterns: - Repeated entity access (queries, updates on same docs) - unclaimed({ the, of }) patterns called multiple times - Multi-step transaction flows referencing same content Implementation: - reference.ts: LRU cache using Map with bounded eviction - Updated imports in fact.ts, access.ts, error.ts, entity.ts to use memoized refer() from ./reference.ts instead of merkle-reference The cache uses JSON.stringify as key (~7µs for 16KB) which is ~25x faster than the SHA-256 hash computation (~170µs for 16KB). * perf(memory): cache and reuse SQLite prepared statements Implemented prepared statement caching to eliminate redundant statement preparation overhead on every database operation. Uses a WeakMap-based cache per database connection to ensure proper cleanup and memory safety. Changes: - Added PreparedStatements type and getPreparedStatement() helper - Cached 7 frequently-used SQL statements (EXPORT, CAUSE_CHAIN, GET_FACT, IMPORT_DATUM, IMPORT_FACT, IMPORT_MEMORY, SWAP) - Removed manual finalize() calls as statements are reused - Added finalizePreparedStatements() to close() for cleanup - Updated all database query functions to use cached statements Benchmark results (before → after): - Single GET query: 117.5µs → 53.4µs (54.6% faster / 2.2x speedup) - Single UPDATE: 906.6µs → 705.8µs (22.1% faster) - Batch retract (10): 2.5ms → 1.9ms (24.0% faster) - Query from 1000 docs: 89.6µs → 66.7µs (25.5% faster) - Batch SET (100): 99.4ms → 88.1ms (11.4% faster) - Batch SET (10): 8.6ms → 7.9ms (8.1% faster) - Single SET: 1.2ms → 1.1ms (8.3% faster) Overall, the optimization provides consistent improvements across all operations with particularly strong gains in read-heavy workloads. All 31 existing tests pass without modifications. * perf(memory): reorder datum/fact hashing to leverage merkle sub-object caching The merkle-reference library caches sub-objects by identity during traversal. By computing the datum hash BEFORE the fact hash, the subsequent refer(assertion) call hits the cache when it encounters the payload sub-object, avoiding redundant hashing of the same 16KB payload twice. Before: refer(assertion) then refer(datum) - payload hashed twice After: refer(datum) then refer(assertion) - payload hash reused via WeakMap This ordering matters because: 1. refer(datum) hashes the payload and caches it by object identity 2. refer(assertion) traverses {the, of, is: payload, cause} - when it reaches the 'is' field, the payload object reference hits the WeakMap cache Benchmark results (16KB payload): - set fact (single): 1.1ms → 924.7µs (16% faster) - retract fact (single): 483.8µs → 462.4µs (4% faster) - update fact (single): ~705µs → ~723µs (within noise) * perf(memory): batch label lookups with SELECT...IN via json_each() Previously, getLabels() performed N individual SELECT queries to look up labels for N facts in a transaction. This adds latency proportional to the number of facts being processed. Now uses a single batched query with SQLite's json_each() function to handle an array of 'of' values: SELECT ... WHERE state.the = :the AND state.of IN (SELECT value FROM json_each(:ofs)) This reduces N queries to 1 query regardless of transaction size. Changes: - Added GET_LABELS_BATCH query constant using json_each() for IN clause - Added 'getLabelsBatch' to prepared statement cache - Rewrote getLabels() to collect 'of' values and execute single batch query The optimization benefits workloads with label facts (access control, classification). Benchmarks show ~4% improvement on batch operations, with larger gains expected in label-heavy workloads. * perf(memory): use stored fact hash instead of recomputing with refer() In conflict detection, we were reading a fact from the database and then calling refer(actual) to compute its hash for comparison. But the fact hash is already stored in the database (row.fact) - we were discarding it and recomputing it unnecessarily. Changes: - Added RevisionWithFact<T> type that includes the stored fact hash - Updated recall() to return row.fact in the revision - Use revision.fact directly instead of refer(actual).toString() - Strip 'fact' field from error reporting to maintain API compatibility This eliminates a refer() call (~50-170µs) on the conflict detection path, which is taken for duplicate detection and first insertions. Benchmark results: - set fact (single): ~1.0ms → 846µs (15% faster) - update fact (single): ~740µs → 688µs (7% faster) - retract fact (single): ~428µs → 360µs (16% faster) * fix(memory): correct Reference type annotations and validate benchmarks - Fix Reference type annotations for memoized refer() - Validate Result in benchmarks to catch silent failures - Apply deno fmt * rename benchmark.ts -> memory_bench.ts This makes it work automatically with `deno bench` 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )