the context file size limits and folder support for rule files

[teddyOOXX]

Context

• Added a context file size limit to prevent overflowing the context window.
• folder support for rule files

Implementation

Screenshots

before	after

|

|

|

How to Test

• Create a .clinerules folder and add different files inside it, each containing different rules.
• Create a particularly large file for reading.

Get in Touch

---

Important

Enhance file handling by adding directory support for rule files and enforcing file size limits in custom-instructions.ts and extract-text.ts.

• Behavior:
  • safeReadFile in custom-instructions.ts now supports reading directories and files, handling .clinerules directories recursively.
  • extractTextFromFile in extract-text.ts throws an error if file size exceeds 300KB.
• Tests:
  • Mocks for fs/promises and utils/fs updated in custom-system-prompt.test.ts and custom-instructions.test.ts to support new directory and file size checks.
  • Tests in custom-instructions.test.ts updated to verify directory handling and file size limits.
• Utils:
  • Added getFileSizeInKB and isDirectory functions in fs.ts to support file size checks and directory validation.

^{This description was created by for 84d0e4abfaa0f9a9d630050ef98543b81903ec1d. It will automatically update as commits are pushed.}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 0707abb

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[mrubens]

I think #1926 will end up being a better solution for this (it will respect the maxReadFileLine setting and still let the user read in the beginning of the file + a map of the functions)

[mrubens]

On the .clinerules folder side of things, I would love to discuss a little more before making a change here. A few things going through my mind:

1. I'm not sure that global .clinerules files are the right path forward. Cursor has been going in a different direction with their contextual .cursor/rules files for instance.
2. In a tool like Roo with first-class support for different modes, it seems like a lot of the instructions you would want to give would be mode-specific. I think we should consider how any improvements here would work on specific models.
3. This is kind of a silly one, but I really want to get away from the .clinerules name and switch to .roorules. As part of that, just want to make sure I've thought deeply about the right direction before undergoing the migration.

If you're interested I would love to discuss this more in Discord - I'm MrUbens in there. Thank you for this and all of your other PRs!

[mrubens]

Going to mark this as changes requested until we have a chance to discuss, thank you!

[teddyOOXX]

@mrubens I believe we can add some optimizations for the effectiveness of file reading and retrieval in the future, such as using embeddings for recall. However, this would depend on embedding a local model or an online one. Nonetheless, it could significantly improve the relevance of code queries and reduce the context window for LLMs.
I think you are right about this in the PR, so I have closed it.