[Refactor] Refactor the grammar compiler to improve maintainability.

[Seven-Streams]

This PR refactors the structure of the grammar compiler to improve maintainability, and it doesn't change the logic of the grammar compiler.

In this PR, the workflow of MultiThreadCompileGrammar can be represented as:

• Grammar Optimization
• TagDispatch Optimization
• Generate Token Mask Caches
  • Other optimizations
  • Check all possible tokens
    • Check possible tokens in each interval with the optimization technique.

[Copilot]

Pull Request Overview

This PR refactors the grammar compiler code to improve maintainability by extracting large monolithic functions into smaller, more focused helper functions. The main changes involve consolidating tokenizer-related parameters into a TokenizerInfo object and breaking down complex token mask calculation logic into separate methods.

Key Changes

• Method signatures refactored to use TokenizerInfo parameter instead of multiple individual parameters
• Large function bodies extracted into smaller, focused helper methods (GetFirstCharacterMask, CheckAllPossibleTokens, CheckTokensInInterval, etc.)
• MultiThreadCompileGrammar split into TagDispatchOptimization and GenerateTokenMaskCacheForScannableStates

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.