feat: add safeguard for large files in readFileTool when maxReadFileLine is -1

[roomote]

Fixes #6155
This PR implements a safeguard to prevent consuming the entire context window when reading large files with maxReadFileLine set to -1.

Problem

When maxReadFileLine is set to -1, the readFileTool reads the full contents of a file. This can be problematic if the file is huge since it could consume the entire context window.

Solution

Added a basic safeguard that:

1. Checks if a file has more than 1000 lines when maxReadFileLine is -1
2. Uses tiktoken to count tokens in the file content
3. If token count exceeds 50,000 (approximately 50% of a typical 100k context window), automatically switches to partial read mode
4. Reads only the first 2000 lines with an informative notice explaining why
5. Includes a fallback for very large files (>5000 lines) when token counting fails

Changes

• Modified src/core/tools/readFileTool.ts to add the safeguard logic
• Added comprehensive tests in src/core/tools/__tests__/readFileTool.spec.ts

Testing

• All existing tests pass
• Added new test cases covering:
  • Files with high token count that trigger the safeguard
  • Files with low token count that do not trigger the safeguard
  • Files under the line threshold that skip token counting entirely
  • Very large files when token counting fails
  • Safeguard not applying when maxReadFileLine is not -1
  • Line ranges bypassing the safeguard
  • Boundary conditions for thresholds

Performance Considerations

• Token counting is only performed for files over 1000 lines to minimize overhead
• The safeguard preserves the existing behavior for smaller files
• The thresholds are configurable constants that can be adjusted if needed

---

Important

Adds a safeguard in readFileTool to handle large files by limiting lines read when maxReadFileLine is -1, with new utility and tests.

• Behavior:
  • Adds safeguard in readFileTool.ts to limit lines read for large files when maxReadFileLine is -1.
  • Uses tiktoken to count tokens and applies safeguard if token count exceeds 50,000.
  • Reads only the first 2000 lines if safeguard is triggered, with a notice.
  • Fallback for very large files (>5000 lines) when token counting fails.
• Utilities:
  • Introduces readLinesWithCharLimit in read-lines-char-limit.ts to read lines up to a character limit.
• Testing:
  • Adds tests in readFileTool.spec.ts for various scenarios including large files, token count checks, and line range handling.
  • Adds tests in read-lines-char-limit.spec.ts for the new utility function.
• Localization:
  • Updates localization files to include safeguard notice in multiple languages.

^{This description was created by for f52f374. You can customize this summary. It will automatically update as commits are pushed.}

[cte]

@roomote-agent This looks promising, but can we get access to the actual context window size of the current model via the reference to the Task object and use that instead of assuming a 100K window size? Additionally, 1,000 lines is a bit low; let's bump that up to 10K lines.

[roomote]

Hi @cte! I see your feedback. Ill work on:

1. Getting the actual context window size from the Task object instead of assuming 100K
2. Increasing the line threshold from 1,000 to 10,000 lines

Let me check out the branch and make these changes.

[ellipsis-dev]

Consider moving the safeguard threshold constants (LARGE_FILE_LINE_THRESHOLD, MAX_TOKEN_THRESHOLD, FALLBACK_MAX_LINES) to a shared configuration or constants module to avoid magic numbers and improve maintainability.

^{This comment was generated because it violated a code review rule: irule_tTqpIuNs8DV0QFGj.}

[ellipsis-dev]

The safeguard notice message is hardcoded in English. For internationalization, consider using the i18n function (e.g., t()) so that the message can be translated for all supported languages.

^{This comment was generated because it violated a code review rule: irule_C0ez7Rji6ANcGkkX.}

[cte]

@roomote-agent Instead of line count let's use file size to detect files that potentially contain a lot of tokens; this should handle the case where there's a file with a huge amount of content on a single line.

[roomote]

Thanks for the feedback @cte! I understand - switching from line count to file size makes sense to better handle files with large amounts of content on single lines. I'll update the implementation to use file size as the primary detection method for potentially token-heavy files. Working on this now.

[roomote]

Hi @cte! Ive successfully implemented both requested changes:

✅ Changes made:

1. Dynamic context window sizing: Now accessing the actual context window size from cline.api.getModel().info.contextWindow instead of assuming 100K
2. Increased line threshold: Bumped LARGE_FILE_LINE_THRESHOLD from 1,000 to 10,000 lines as requested

📝 Implementation details:

• The MAX_TOKEN_THRESHOLD is now calculated as 50% of the actual models context window
• Added fallback to 100K if context window info is not available
• Updated all related tests to mock the api.getModel() method properly
• All tests are passing locally

🔍 CI Status:

• Most checks have passed ✅
• The Windows unit test failure appears to be a pre-existing issue (Ubuntu tests passed fine)

The changes ensure that the safeguard adapts to different model context windows while being less aggressive about triggering for moderately-sized files.

[roomote]

Hi @cte! I've successfully implemented the requested change to use file size instead of line count for detecting large files.

Summary of changes:

1. Replaced line count threshold with file size threshold: Files larger than 100KB now trigger the token count check (instead of files with more than 10,000 lines)

2. Added file size-based safeguard: Files larger than 1MB automatically apply the safeguard if token counting fails

3. Updated notice messages: The safeguard notices now show file size in KB instead of line count

4. Updated all tests: Modified test suite to use file size mocking and updated test descriptions accordingly

This approach better handles edge cases where files might have a huge amount of content on single lines, as you mentioned. The implementation still respects the dynamic context window calculation that was added in the previous commit.

All tests are passing locally, and CI checks are currently running. The changes have been pushed to this PR branch.

[ellipsis-dev]

Consider using the i18n translation function for the safeguard notice message (e.g. on line 550) instead of hardcoded English text, to ensure consistency with other user‐facing messages.

^{This comment was generated because it violated a code review rule: irule_C0ez7Rji6ANcGkkX.}

[daniel-lxs]

Closing in favor of #6319