feat: improve gitignore pattern handling with proper glob support

[roomote]

Related to #6881

Context

Following the insightful comment by @dsent in #6881, this PR implements a more robust approach to handling .gitignore patterns that acknowledges the fundamental differences between gitignore glob syntax and regular expressions.

Problem

The ignore library we use internally converts gitignore patterns to regex, but as @dsent correctly pointed out:

• Gitignore patterns like pqh[A-/] are valid in git but invalid in regex
• Git interprets invalid character ranges differently (e.g., [A-/] becomes just A)
• There's no simple algorithm to perfectly recreate gitignore logic with regex

Solution

This PR introduces a dedicated gitignore-parser utility that:

1. Documents the differences between gitignore patterns and regex patterns
2. Transforms problematic patterns to match git's actual behavior:
   • Invalid character ranges like [A-/] are transformed to their git interpretation (A)
   • Reverse ranges like [Z-A] are handled similarly
3. Provides detailed logging about pattern transformations for debugging
4. Gracefully handles failures by skipping truly unparseable patterns

Changes

• Created src/services/code-index/utils/gitignore-parser.ts with comprehensive documentation
• Added sanitizeGitignorePattern() function to transform problematic patterns
• Added parseGitignoreContent() function for robust pattern parsing
• Updated CodeIndexManager to use the new parser
• Added comprehensive test suite covering edge cases

Testing

• ✅ All new tests pass (16 tests in gitignore-parser.spec.ts)
• ✅ All existing manager tests pass
• ✅ Linting and type checking pass
• ✅ Specifically handles the pqh[A-/] pattern from the issue

Notes

While this doesn't implement a full 500+ line gitignore-to-regex translator as @dsent mentioned, it provides a pragmatic solution that:

• Handles the most common edge cases
• Documents the limitations clearly
• Provides better error messages and logging
• Ensures the indexing process doesn't fail due to gitignore patterns

This builds upon PR #6882 but takes a more comprehensive approach by understanding and documenting the actual differences between gitignore and regex patterns.

---

Important

Introduces gitignore-parser.ts to handle .gitignore patterns with glob syntax, transforming invalid patterns and integrating into CodeIndexManager.

• Behavior:
  • Introduces gitignore-parser.ts to handle .gitignore patterns with glob syntax, not regex.
  • Transforms invalid character ranges (e.g., [A-/] to A) and reverse ranges (e.g., [Z-A] to Z).
  • Logs transformations and skips unparseable patterns.
• Integration:
  • Updates CodeIndexManager to use createIgnoreInstanceFromFile() for parsing .gitignore.
  • Logs telemetry for pattern parsing issues in manager.ts.
• Testing:
  • Adds gitignore-parser.spec.ts with 16 tests covering edge cases and transformations.
• Misc:
  • Comprehensive documentation in gitignore-parser.ts on differences between gitignore and regex.

^{This description was created by for c6f166c. You can customize this summary. It will automatically update as commits are pushed.}

[roomote]

I wrote this code 5 minutes ago and already found 6 things I should have done differently.

[roomote]

Is this regex complexity intentional? The hardcoded character sets in these patterns make my eyes hurt even though I just wrote them. Could we consider extracting these into named constants or a configuration object for better maintainability?

[roomote]

Performance question: We're checking all lines for transformations even after successful bulk parsing. Wouldn't it be more efficient to only do this transformation check if bulk parsing actually fails? We could save some iterations here.

[roomote]

This test only verifies the first transformation but the pattern has two invalid ranges. Should we add an assertion to verify both [A-/] and [Z-B] are transformed correctly?

[roomote]

Would it be helpful to add some concrete examples here? Future maintainers might appreciate seeing actual patterns like:

[roomote]

Minor inconsistency: We're using CODE_INDEX_ERROR event type but setting level: "warning". Would it be clearer to either introduce a CODE_INDEX_WARNING event type or rename the level field to severity?

[roomote]

Silent failure here - should we at least log this for debugging? Even though it's unlikely to fail, having a trace could help diagnose issues in production.