Codebase Indexing error: Invalid regular expression

[dsent]

App Version

3.25.10

API Provider

Not Applicable / Other

Model Used

N/A

Roo Code Task Links (Optional)

No response

🔁 Steps to Reproduce

1. Enable Codebase Indexing
2. Select any provider and any embeddings model.
3. Click Save
4. Click Start Indexing

💥 Outcome Summary

The process fails with the following error:
Error - Failed during initial scan: Invalid regular expression: /^pqh[A-\/](?=$|\/$)/i: Range out of order in character class

Settings:

📄 Relevant Logs or Errors (Optional)

[roomote]

Hi! I'm here to help out the maintainers and am going to see if I can fix this regex issue. The error indicates an invalid character range in a regular expression pattern used during codebase indexing. I'll investigate and work on a fix!

[roomote]

I've successfully fixed the issue! 🎉

The problem was that the codebase indexing feature was failing when encountering invalid regex patterns in .gitignore files. The specific pattern pqh[A-/] has an invalid character range since / comes before A in ASCII order.

Solution implemented:

• Added robust error handling to gracefully handle invalid gitignore patterns
• When bulk parsing fails, the code now falls back to line-by-line parsing
• Invalid patterns are skipped with warnings while valid patterns are still processed
• The indexing process continues even when some patterns are invalid

Pull Request: #6882

The fix includes comprehensive test coverage for various scenarios including invalid patterns, valid patterns, missing .gitignore files, and mixed valid/invalid patterns. All tests are passing.

This ensures that codebase indexing will no longer fail due to malformed gitignore patterns, making the feature more robust and user-friendly.

[dsent]

Haha, nice. I have a few deliberately incorrect patterns in .gitignore (I've tested corner cases of gitignore processing logic).

[dsent]

@roomote-agent (and humans reading this), by the way, the actual .gitignore processing logic is markedly different from regex. For example, the invalid (by regex standards) range patterns in character classes, like pqh[A-/], are actually parsed and considered correct by git. They are treated as a single character, the starting character of the range, e.g. the mentioned pattern is interpreted as pgh[A] in regex terms.

There's no simple algorithm to recreate the .gitignore logic exactly with regex. I've come up with a translation algorithm (gitignore glob -> regex) that is about 500+ lines long (and I'm still not sure it fully mimics the .gitignore glob logic). No existing library I've found passes all my tests for corner cases (i.e. their logic does not match .gitignore logic 100%).

[roomote]

Thank you @dsent for this insightful explanation! You're absolutely right that .gitignore processing logic is fundamentally different from regex patterns.

Your point about pqh[A-/] being valid in .gitignore and interpreted as pqh[A] is particularly interesting - git treats the invalid range as just the starting character. This is a great example of how gitignore glob patterns have their own unique parsing rules that don't map directly to regex.

I appreciate you sharing your experience with the 500+ line translation algorithm. The complexity you describe highlights why many libraries fail to fully replicate .gitignore behavior. I'll investigate our current implementation to see how we're handling these edge cases and whether we should consider a more robust approach that better aligns with git's actual glob parsing logic rather than trying to force everything through regex patterns.

Let me explore the codebase to understand our current approach and see what improvements we can make based on your insights.

[dsent]

@roomote-agent maybe you shouldn't do anything else, actually. Recreating the .gitignore logic is probably beyond the scope of anything actually useful for the project. I think it's okay to just not fail if one of the patterns seems incorrect. This won't cause problems in most real-world scenarios.

[roomote]

Thank you @dsent for the clear guidance! You're absolutely right - trying to fully recreate .gitignore logic would be overly complex and not particularly beneficial for the project.

I understand and agree with your approach: the implementation should simply handle invalid patterns gracefully without failing, which will work fine for most real-world scenarios. The existing PR #6882 already implements this approach by catching and skipping invalid patterns while continuing to process valid ones.

Since this solution aligns with your recommendation, no further changes are needed. The current implementation will prevent the indexing process from crashing when it encounters patterns like pqh[A-/] while still processing all valid patterns correctly.