[BUG] ENOENT from proper-lockfile realpath() after proactive stale lock cleanup

[arthurliang]

Bug Description

After the proactive stale lock cleanup (introduced in PR #626 for issues #622/#623) clears a stale .memory-write.lock file, subsequent proper-lockfile calls fail with:

ENOENT: no such file or directory, lstat '/Users/arthurliang/.openclaw/memory/lancedb-pro/.memory-write.lock'

Root Cause

The proactive cleanup (store.ts lines 221-230) unlinks the stale lock file:

if (existsSync(lockPath)) {
  try {
    const stat = statSync(lockPath);
    const ageMs = Date.now() - stat.mtimeMs;
    const staleThresholdMs = 5 * 60 * 1000;
    if (ageMs > staleThresholdMs) {
      try { unlinkSync(lockPath); } catch {}
      console.warn(`[memory-lancedb-pro] cleared stale lock: ${lockPath} ageMs=${ageMs}`);
    }
  } catch {}
}

This runs before lockfile.lock():

const release = await lockfile.lock(lockPath, {
  retries: { retries: 10, factor: 2, minTimeout: 200, maxTimeout: 5000 },
  stale: 10000,
});

In proper-lockfile/lib/lockfile.js, the resolveCanonicalPath function calls options.fs.realpath(file) on the lock target path (line 22). The default for realpath is true in proper-lockfile v4. When the lock file was just deleted by proactive cleanup, realpath() throws ENOENT because the target file does not exist.

Relevant proper-lockfile code (lockfile.js lines 15-22):

function resolveCanonicalPath(file, options, callback) {
    if (!options.realpath) {
        return callback(null, path.resolve(file));
    }
    // Use realpath to resolve symlinks
    // It also resolves relative paths
    options.fs.realpath(file, callback);
}

Observed Timeline

Time	Event
07:35:17	smart-extractor acquired lock, wrote memories, process terminated without releasing
08:35:43.181	Proactive cleanup detected stale lock (ageMs=3635657 ≈ 60 min), deleted .memory-write.lock
08:35:43.184	3ms later: proper-lockfile.realpath() on deleted file → ENOENT → smart-extractor failed

Environment

• memory-lancedb-pro: 1.1.0-beta.10
• OS: macOS 15.6 (arm64)
• OpenClaw gateway (agent main, multi-session)

Proposed Fix

Two complementary fixes:

Fix 1: Pass realpath: false to proper-lockfile

Since the lock path is always an absolute path to a known location, symlink resolution is unnecessary:

const release = await lockfile.lock(lockPath, {
  realpath: false,
  retries: { retries: 10, factor: 2, minTimeout: 200, maxTimeout: 5000 },
  stale: 10000,
});

Fix 2: Recreate lock file after proactive cleanup

After unlinking the stale lock file, recreate it so proper-lockfile has a valid target:

if (ageMs > staleThresholdMs) {
  try { unlinkSync(lockPath); } catch {}
  // Recreate the target file so proper-lockfile can lock it
  try { writeFileSync(lockPath, "", { flag: "wx" }); } catch {}
  console.warn(`[memory-lancedb-pro] cleared stale lock: ${lockPath} ageMs=${ageMs}`);
}

Recommended: Both fixes together

Fix 1 prevents the ENOENT from proper-lockfile. Fix 2 ensures the lock target file exists even if other code paths depend on it. Together they provide defense in depth.

Reproduction Steps

1. Start a session with memory-lancedb-pro
2. Trigger smart-extractor to write memories and acquire the file lock
3. Kill the process (or let session reset) without releasing the lock
4. Wait >5 minutes (stale threshold)
5. Trigger another smart-extractor run
6. Observe: stale lock cleanup runs, deletes lock file, next lock attempt fails with ENOENT

Related Issues

• [BUG] Windows: stale .memory-write.lock not released, blocks smart-extractor writes #622: Stale lock recovery (FIXED in PR fix: resolve stale lock and same-instance write contention (#622, #623) #626)
• [BUG] Same-instance concurrent writes竞争 lock，导致 Lock file is already being held #623: Same-instance write contention (FIXED in PR fix: resolve stale lock and same-instance write contention (#622, #623) #626)
• [BUG] Lock contention between upgrade CLI and plugin causes writes to fail #632: Lock contention between upgrade CLI and plugin (OPEN)
• feat: batch write to reduce lock contention from auto-capture (extends Issue #643) #665: Batch write to reduce lock contention (OPEN)

[AliceLJY]

Thanks for the extremely precise bug report @arthurliang — the 3ms window between proactive unlink and proper-lockfile.realpath() is exactly the race, and the timeline table makes it unambiguous.

Confirming the analysis

Your root-cause is correct. proper-lockfile@4.x calls fs.realpath() on the lock target when options.realpath is left at its default (true). The proactive cleanup in store.ts:221-230 deletes the file right before lockfile.lock() runs → realpath resolves an ENOENT → ENOENT bubbles up.

Preferred fix

Fix 1 (realpath: false) alone is sufficient and preferable.

Reasoning:

• The lock path is always an absolute resolved path built from config (config.dataDir/.memory-write.lock), never a symlink or relative path. realpath adds nothing functional here.
• Fix 2 (recreate empty file with flag: "wx") introduces a second TOCTOU window: between unlinkSync and writeFileSync, another process could already be calling lockfile.lock() on the missing file. Also, proper-lockfile manages lock-file creation itself — creating an empty file first can surface as "lock already acquired" depending on timing.
• Fix 2 (defense-in-depth value) is dominated by Fix 1 in terms of actual race elimination.

Additional defensive change worth considering: in the proactive cleanup block, wrap the unlinkSync in a guard that only runs when the consumer is confident no other writer holds the lock right now. The ageMs > 5min check assumes nothing; if the mtime was bumped by an in-flight writer that just started, you'd unlink a live lock. But that's a separate issue from ENOENT and not blocking this fix.

On #632 / #665 relation

Your note about related issues is on point. #626's proactive cleanup is the direct cause here; a batch-write approach (#665) would reduce the frequency but not eliminate the race. Fix 1 is the right ENOENT-specific patch regardless of whether #665 lands.

Next step

If you want to open a PR with Fix 1 (one-line change + regression test that unlinks the lock then calls acquireLockedWrite()), it would merge quickly. Otherwise @win4r @rwmjhb — this is a genuine regression from #626 with a clean fix, worth prioritizing.

[jlin53882]

Fix Available

PR #674 已修復此問題。

修復內容

1. 新增
   ealpath: false - 跳過 realpath() 調用，避免在 stale lock 被清理後出現 ENOENT
2. 移除 pre-creation - proper-lockfile v4 自己管理 lock artifact，不再需要預先創建檔案
3. 修正 cross-process-lock.test.mjs - 測試實際行為（並發寫入成功）而非實作細節

驗證

• 本地測試：10/11 passed (1 為 Windows 上既有的 skip)
• CI：storage-and-schema ✅ | core-regression ✅

PR: #674