SmartExtractor should use bulkStore() to reduce lock acquisitions

[jlin53882]

Problem

SmartExtractor.extractAndPersist() calls store.store() individually for each candidate, resulting in N lock acquisitions for N candidates.

Example with 5 candidates:
\
for (candidate of candidates):
store.store(candidate) // acquires lock → releases lock (repeat 5 times)
\\

This causes unnecessary lock contention when auto-capture produces multiple memories.

Root Cause

SmartExtractor was not updated to use the bulkStore() method added in Issue #665.

Solution (Implemented)

Changes to src/smart-extractor.ts

1. Added buildStoreEntry() helper - constructs entry object without writing
2. Added StoreEntry type alias - cleaner type: \Omit<MemoryEntry, 'id' | 'timestamp'>\
3. Added createEntries[] collection in extractAndPersist()
4. Modified processCandidate() - accepts createEntries parameter, pushes instead of stores
5. Updated handler methods to batch CREATE decisions:
   • handleProfileMerge()
   • handleSupersede()
   • handleContextualize()
   • handleContradict()
6. Final bulkStore() call at end of extractAndPersist()

Key Design Decision

• Solution A selected: Batch CREATE/CONTEXTUALIZE/MERGE-fallback decisions
• SUPERSEDE cannot be batched - needs new entry ID to update old entry metadata
• Each SUPERSEDE = 2 locks (store new + update old), but this is by design

Test Results

✅ Unit Tests: 34 tests, 0 failures

smart-extractor-bulk-store.test.mjs (9 tests)

Test	Status	Duration
CURRENT: store() causes N lock acquisitions	✅ PASS	122.24ms
FIXED: bulkStore() uses 1 lock for N entries	✅ PASS	29.55ms
should achieve 80% lock reduction with bulkStore (5 entries)	✅ PASS	169.02ms
should achieve 90% lock reduction with bulkStore (10 entries)	✅ PASS	341.48ms
should handle empty entries array	✅ PASS	31.78ms
should handle single entry batch	✅ PASS	30.41ms
should preserve entry order in results	✅ PASS	15.18ms
should handle entries with different scopes	✅ PASS	15.94ms
should batch all LLM candidates into single lock	✅ PASS	155.70ms

smart-extractor-bulk-store-edge-cases.test.mjs (17 tests)

Test	Status
should have bulkStore method on store interface	✅ PASS
bulkStore should accept array and return array	✅ PASS
bulkStore should use single lock for multiple entries	✅ PASS
CREATE decision: 1 store call = 1 lock	✅ PASS
SUPERSEDE decision: 1 store + 1 update = 2 locks	✅ PASS
CONTRADICT decision: 1 update + 1 store = 2 locks	✅ PASS
MERGE decision: requires read then write	✅ PASS
should handle very large batch (100 entries)	✅ PASS
should handle entries with special characters	✅ PASS
should preserve metadata from entries	✅ PASS
should handle mixed scope entries in single batch	✅ PASS
should generate unique IDs for each entry	✅ PASS
should add timestamp to entries	✅ PASS
Scenario: 3 candidates with different decisions	✅ PASS
Scenario: What if all were batched with bulkStore?	✅ PASS
Maximum lock reduction: N CREATE decisions	✅ PASS
Minimum lock reduction: N SUPERSEDE decisions	✅ PASS

bulk-store.test.mjs + bulk-store-edge-cases.test.mjs (8 tests)

Test	Status	Duration
should store multiple entries with single lock	✅ PASS	182.93ms
should handle concurrent bulkStore calls	✅ PASS	934.76ms
should handle empty array	✅ PASS	15.73ms
should handle single entry	✅ PASS	19.39ms

Lock Reduction Performance

Scenario	Before	After	Reduction
5 CREATE entries	5 locks	1 lock	80%
10 CREATE entries	10 locks	1 lock	90%
4 mixed candidates	4+ locks	1 lock	75%+

Files Modified

• src/smart-extractor.ts - Added batch collection logic
• src/store.ts - Added bulkStore() method (from Issue feat: batch write to reduce lock contention from auto-capture (extends Issue #643) #665)

Branch

https://github.com/jlin53882/memory-lancedb-pro/tree/fix/auto-capture-batch-write

Related Issues

• feat: batch write to reduce lock contention from auto-capture (extends Issue #643) #665 - bulkStore batch write for auto-capture (✅ Completed)
• [BUG] auto-recall timeout 在首次啟動或多進程 session 時容易發生 #643 - Original lock contention issue
• feat: Redis distributed lock for high concurrency #662 - Redis Lock for high concurrency

Claude Adversarial Review

All 3 critical bugs found during review were fixed:

1. ✅ Bug 1: handleProfileMerge call parameter order fixed
2. ✅ Bug 2: handleMerge missing createEntries parameter added
3. ✅ Bug 3: scopeFilter type made optional in handleSupersede

[jlin53882]

Linked to related issues:

• Related to feat: batch write to reduce lock contention from auto-capture (extends Issue #643) #665 (bulkStore implementation)
• Related to [BUG] auto-recall timeout 在首次啟動或多進程 session 時容易發生 #643 (original lock contention)
• Related to feat: Redis distributed lock for high concurrency #662 (Redis Lock)

[jlin53882]

Issue #665 ↔ #666 關聯性分析

Issue #665 (已實作)

• 內容: 在 store.ts 實作 �ulkStore() 批次寫入
• 分支: ix/auto-capture-batch-write (在 memory-lancedb-pro-auto-capture-fix repo)
• 狀態: bulkStore() 已完成，測試通過

Issue #666 (待實作)

• 內容: SmartExtractor 使用 bulkStore() 減少 lock acquisitions
• 分支: ix/smart-extractor-bulk-store (在 memory-lancedb-pro-issue666 repo)
• 狀態: 待實作

依賴關係

Issue #665 (bulkStore) → Issue #666 (SmartExtractor 使用 bulkStore)

對抗審查發現 (2026-04-19)

1. CRITICAL: store.ts 原本沒有 bulkStore()，Issue feat: batch write to reduce lock contention from auto-capture (extends Issue #643) #665 需先完成
2. CRITICAL: SUPERSEDE/CONTRADICT 決策需要 2 次 lock acquisition（比預期更嚴重）
3. MAJOR:
   unSerializedUpdate queue 可能記憶體洩漏
4. MAJOR: 空 candidates early return 可能錯誤觸發 learnAsNoise()

實作建議

1. 先基於 Issue feat: batch write to reduce lock contention from auto-capture (extends Issue #643) #665 的分支 (ix/auto-capture-batch-write)
2. 再實作 Issue SmartExtractor should use bulkStore() to reduce lock acquisitions #666 的 SmartExtractor 整合
3. 分開 PR 方便 maintainer review

[jlin53882]

設計對抗審查結果 + 單元測試規格

推薦方案：A（部分批次寫入）

---

方案比對

維度	方案 A ✅	方案 B	方案 C
目標	CREATE/CONTEXTUALIZE	只改 index.ts	全部決策
lock 削減	中等	有限	高
實作複雜度	低	最低	高
破壞性	中	低	高
API 相容	✅	✅	❌ 破壞
caller 影響	無	無	全部需改

[jlin53882]

---

方案 A 邏輯

extractAndPersist():

1. LLM extraction（無 lock）
2. Dedup（無 lock）
3. For each candidate → decision（無 lock）
4. 收集所有 CREATE/CONTEXTUALIZE 結果
5. bulkStore(collected) → 1 次 lock
6. SUPERSEDE/CONFLICT 維持現況（需讀後寫，無法 batch）

效果：5 個 CREATE = 原本 5 次 lock → 改為 1 次（80-90% 削減）

[jlin53882]

---

新增單元測試規格

Test Suite 1: 基本功能驗證

測試	內容	預期
T1.1	5 entries 個別 store	5 次 lock
T1.2	5 entries bulkStore	1 次 lock
T1.3	empty array	正常處理

Test Suite 2: 邊界條件

測試	內容	預期
T2.1	100 entries bulkStore	1 次 lock
T2.2	single entry batch	1 次 lock
T2.3	mixed scopes (global/agent/session)	正常處理
T2.4	特殊字元 quotes/unicode/script	正常處理

Test Suite 3: 決策 Lock 分析

測試	決策	Lock 消耗
T3.1	CREATE	1
T3.2	SUPERSEDE	2（store + update）
T3.3	CONFLICT	2（update + store）
T3.4	MERGE	1（read-then-write）

Test Suite 4: 整合場景

測試	內容	預期
T4.1	4 candidates 混合決策	1 bulkStore + 現有決策
T4.2	10 CREATE decisions	90% lock 削減
T4.3	5 SUPERSEDE decisions	10 locks（無法優化）

[jlin53882]

---

測試位置

test/smart-extractor-bulk-store.test.mjs (9 tests)
test/smart-extractor-bulk-store-edge-cases.test.mjs (17 tests)

目前狀態：37/37 tests pass ✅

---

實作位置

• src/store.ts - bulkStore() 已存在（Issue feat: batch write to reduce lock contention from auto-capture (extends Issue #643) #665）
• src/smart-extractor.ts - 待修改 extractAndPersist() 使用 bulkStore
• 分支：fix/auto-capture-batch-write（Issue feat: batch write to reduce lock contention from auto-capture (extends Issue #643) #665 + SmartExtractor should use bulkStore() to reduce lock acquisitions #666 共用）