Remaining Challenges with Line Numbers in File Operations

[KJ7LNW]

Overview

Despite significant improvements to how line numbers are handled in file operations, several fundamental challenges remain that affect model performance and user experience.

Current Challenges

1. Context Token Consumption

Line numbers consume valuable context tokens that could otherwise be used for actual file content. This is especially problematic for:

• Large files where context limits are already a concern
• Complex operations that require understanding multiple files
• Models with smaller context windows

2. Special Character Confusion

Content that naturally contains pipe characters (|) can still be confused with line number delimiters. This particularly affects:

• Markdown tables
• ASCII diagrams
• Code with bitwise OR operations
• Template languages with pipe filters

3. Perception Gap

There remains a fundamental perception gap between what the model sees (content with line numbers) and the actual file content. This requires the model to:

• Mentally filter out line numbers when analyzing content
• Remember to strip line numbers when generating search patterns
• Maintain awareness of the difference between displayed and actual content

4. Indentation and Formatting Challenges

Line numbers can interfere with the model's understanding of code indentation and formatting, especially when:

• Working with whitespace-sensitive languages like Python
• Analyzing complex nested structures
• Determining the exact level of indentation for code generation

Experimental Work in Progress

PR #1889 is an experimental work in progress that aims to ultimately solve these issues by providing a configuration option to enable or disable line number prefixing. This approach would allow users to choose between:

• The current behavior with line numbers (useful for reference and discussion)
• A clean representation without line numbers (better for model understanding and token efficiency)

This experimental work demonstrates the potential benefits of making line numbers optional, particularly for complex file operations where the model struggles with the current approach.

Potential Solutions to Explore

1. Optional Line Numbers: Continue the work started in PR WIP: optionally omit line number from reads and apply_diff requirements #1889 to provide a configuration option for disabling line numbers
2. Alternative Visualization: Explore different ways to indicate line positions without modifying the content
3. Contextual Line Numbers: Only show line numbers in specific contexts where they add value
4. Improved Stripping Heuristics: Continue refining the line number stripping logic for edge cases
5. Model-Specific Formatting: Tailor the presentation of line numbers based on the capabilities of different models

Impact

These challenges lead to:

• Increased error rates in file modifications
• Unnecessary retries and model confusion
• Reduced context efficiency
• Difficulties with special formats and languages

Addressing these remaining challenges would significantly improve the robustness and efficiency of file operations across all models.

[daniel-lxs]

This probably requires a lot of testing with the custom evals to determine if it's something we want to do, @KJ7LNW already mentioned on #1889 that it works well:

I have been using this pull request for months, and it works great

We just need to decide if we want to invest time into creating custom evals and then testing this properly.

Note: This is not an approval, please wait for a dev team member to approve this feature request.

[hannesrudolph]

Roo is working well regrading this IMO. Closing this. Thank you