Skip to content

feat: batch write to reduce lock contention from auto-capture (extends Issue #643) #665

@jlin53882

Description

@jlin53882

問題描述

即使使用 Redis lock(PR #662),在 auto-capture 產生多個記憶時仍然會产生 lock contention。

觀察到的事件(2026-04-19)

\
15:10:46 [RedisLock] Acquired lock memory-write after 1 attempts
15:10:47 [RedisLock] Released lock memory-write
15:10:48 [RedisLock] Acquired lock memory-write after 6 attempts
15:10:50 [RedisLock] Released lock memory-write
15:10:51 [RedisLock] Acquired lock memory-write after 8 attempts
15:11:40 [plugins] memory-lancedb-pro: auto-recall timed out after 60000ms
\\

根本原因

現況

在 index.ts 的 auto-capture 迴圈中,每個 entry 都單獨寫入:

\\ ypescript
// index.ts:2934
for (const entry of entries) {
await store.store({ // <-- 每次都拿 lock
text,
vector,
...
});
}
\\

這導致:

  • N 個記憶 → N 次 lock 取得
  • 即使 Redis lock 很快,仍有重試開銷
  • 在高並發時造成 lock contention

解決方案(已實作)

bulkStore() 方法 (src/store.ts)

\\ ypescript
async bulkStore(
entries: Omit<MemoryEntry, 'id' | 'timestamp'>[],
): Promise<MemoryEntry[]> {
// 單一 lock 取得,寫入全部 entries
return this.runWithFileLock(async () => {
await this.table!.add(fullEntries);
return fullEntries;
});
}
\\

實作內容

  1. 新增 bulkStore() 方法 - 單一 lock 寫入多個 entries
  2. 新增 SmartExtractor 批次收集 - 在 extractAndPersist() 中收集 CREATE 決策
  3. 最終批次寫入 - 一次 bulkStore() 呼叫

測試結果

✅ Unit Tests: 34 tests, 0 failures

測試檔案 測試數 通過 失敗
smart-extractor-bulk-store.test.mjs 9 9 0
smart-extractor-bulk-store-edge-cases.test.mjs 17 17 0
bulk-store.test.mjs 4 4 0
bulk-store-edge-cases.test.mjs 4 4 0
總計 34 34 0

Lock Reduction Performance

情境 改之前 改之後 減少
5 CREATE entries 5 locks 1 lock 80%
10 CREATE entries 10 locks 1 lock 90%
20 concurrent writes ~20 locks ~1-3 locks 85-95%
50 concurrent writes ~50 locks ~3-5 locks 90-94%

Redis Lock 驗證(日誌)

\
[RedisLock] Redis lock manager initialized
[RedisLock] Acquired lock memory-write after 1 attempts
[RedisLock] Released lock memory-write
[bulkStore] 10 entries stored in 166ms
\\

分支

https://github.com/jlin53882/memory-lancedb-pro/tree/fix/auto-capture-batch-write

相關 Issues

狀態

已實作完成,通過所有單元測試

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions