1 of 3: memfix: implement and use safeReadJson to reduce memory usage

[KJ7LNW]

Dependencies

• Closes fix: High memory usage and potential race conditions when reading large JSON files #5331
• Blocks:
  • 2 of 3: corruption fix: Implement atomic read-modify-write for JSON files #5544
  • 3 of 3: scalability fix: refactor task history persistence to use file-based storage #3785
  • feat: Implement task history scanner and recovery tools #5546

Note to Reviewer

The commits tell a clean story, it will be easier to understand what is happening here by looking at each commit individually under "Commits" than by looking at all of the files that were changed.

This PR introduces safeReadJson, a new utility to read JSON files safely and efficiently, reducing memory pressure when loading large JSON files by streaming them instead of double-buffering via JSON.parse.

Key Changes

• safeReadJson Utility:

  • A new function in src/utils/safeReadJson.ts that uses stream-json to parse files without buffering them entirely into memory.
  • Implements file locking via proper-lockfile to prevent race conditions during reads, sharing the lock mechanism with safeWriteJson.
  • Supports extracting specific data using JSON paths.

• Codebase Adoption:

  • Replaced the pattern of JSON.parse(fs.readFile(...)) with safeReadJson across the codebase to improve performance and consistency.

• Refactoring:

  • The file locking logic in safeWriteJson.ts has been extracted into a shared _acquireLock function, now used by both safeReadJson and safeWriteJson.

• Testing:

  • Added comprehensive unit tests for the new safeReadJson utility.
  • Updated existing tests to mock the new utility.

How to Test

1. Run the application and verify that all JSON file reading operations work as expected.
2. Check memory usage when working with large task files (should be reduced).
3. Run the test suite to ensure all functionality remains intact.

Get in Touch

Discord: KJ7LNW

---

Important

Introduces safeReadJson for efficient JSON reading, refactors codebase to use it, and adds comprehensive tests.

• Behavior:
  • Introduces safeReadJson in src/utils/safeReadJson.ts for efficient JSON file reading using streaming and file locking.
  • Replaces JSON.parse(fs.readFile(...)) with safeReadJson in modelCache.ts, modelEndpointCache.ts, and importExport.ts.
• Refactoring:
  • Extracts file locking logic into _acquireLock shared by safeReadJson and safeWriteJson.
• Testing:
  • Adds unit tests for safeReadJson in safeReadJson.spec.ts.
  • Updates existing tests to mock safeReadJson in importExport.spec.ts and cache-manager.spec.ts.
• Misc:
  • Adds rule in .roo/rules/use-safeReadJson.md to enforce safeReadJson usage.

^{This description was created by for bb27ee94901f66f2492dc1327f0ca351c0ca1e5b. You can customize this summary. It will automatically update as commits are pushed.}

[KJ7LNW]

@daniel-lxs please move this into review

[KJ7LNW]

stand by on merging, I have a small change for this, moving back to draft

[KJ7LNW]

rebased on origin/main, ready for review

[daniel-lxs]

Thank you @KJ7LNW!

I left a couple of suggestions, let me know what you think!

[KJ7LNW]

@daniel-lxs I resolved your suggested changes and rebased on v3.23.12

[daniel-lxs]

Thank you @KJ7LNW, looks good to me!

[KJ7LNW]

rebased on v3.23.14

[daniel-lxs]

Thanks for the contribution. At this stage, the impact of these changes isn’t clear and we’re focusing on higher-priority items. Closing for now, but we can reconsider in the future if priorities shift.