XML Matching Improvements and SpecialTokens

Reliance on single/guessed special tokens only goes so far when we
use constrained outputs because masking-out a potential EOS token
results in infinite generation: we have to account for all possible
candidates in the mask which could normally end generation.

Add SpecialTokens idiomatic extractor as a starting point for this
work and utilize it to feed all EOS tokens to the grammar-building
routines. Add the binary for this library element to examples/ for
@guoqingbao and other developers to have rapid access to what the
SpecialTokens struct actually extracts from any tokenizer.json
provided in ARGV0 or from ./tokenizer.json if none are provided.

Improve XML tool-sled generation. Remaining issue is potential of
XML content within the XML envelope and no ability to mask possibly
infinite strings as anything but infinite due to look-ahead and lazy
regex tricks from interpreted languages not actually compiling to a
finite mask. Use a simple matcher for now, enable env-override by
the user while this gets sorted out (if possible) and critically
enable the grammar generator to honor tool parser override at the
CLI such that `--enforce-parser qwen` produces JSON-constrained
schemas which the parser can then consume.

XML finite masking tracked under:
- guidance-ai/llguidance#306

Multiple EOS token concerns (handled in grammar) under:
- guidance-ai/llguidance#304
- guidance-ai/llguidance#305

Author: RageLtMan

src/core/engine.rs

@@ -20,6 +20,7 @@ use crate::transfer::PdRole;
 use crate::transfer::Transfer;
 use crate::utils::chat_template::Message;
 use crate::utils::config::{EngineConfig, EosTokenId, ModelType, SamplingParams};
+use crate::utils::special_tokens::SpecialTokens;
 use crate::utils::guidance::{build_llg_factory, load_toktrie_from_path};
 use crate::utils::heartbeat::heartbeat_worker;
 use crate::utils::image::{get_image_config, ImageData, ImageProcessConfig};
@@ -100,6 +101,9 @@ pub struct LLMEngine {
     pub model_type: ModelType,
     pub tool_config: ToolConfig,
     pub img_cfg: Option<ImageProcessConfig>,
+    /// SpecialTokens parsed once at engine initialization
+    /// Contains EOS, BOS, and other special token IDs and their string representations
+    pub special_tokens: Arc<SpecialTokens>,
 }
 
 impl LLMEngine {
@@ -466,6 +470,9 @@ impl LLMEngine {
             "default".to_string()
         };
 
+        // Initialize SpecialTokens once at engine startup
+        let special_tokens = Arc::new(SpecialTokens::new(&tokenizer));
+
         let engine = Arc::new(RwLock::new(Self {
             runners,
             scheduler,
@@ -488,6 +495,7 @@ impl LLMEngine {
             tool_config,
             img_cfg,
             model_name,
+            special_tokens,
         }));
         Self::start_engine(engine.clone());
         Ok(engine)

src/server/server.rs

@@ -525,17 +525,17 @@ pub async fn chat_completion(
     if constraint_grammars.is_empty() && !engine_config.enable_tool_grammar {
         crate::log_debug!("[llg] No constraint or tool grammar - not setting guidance");
     } else {
-        // Get EOS token IDs from engine scheduler for building TEXT pattern with EOS bounding
+        // Get SpecialTokens from engine for building TEXT pattern with EOS bounding
         let engine = data.engine.read();
-        let eos_token_ids = engine.scheduler.eos_token_ids();
+        let special_tokens = &engine.special_tokens;
         let llg_grammar = compose_grammars(
             constraint_grammars,
             tool_gram,
             has_tools,
             tool_choice_required,
             forced_tool_name.clone(),
             Some(max_tokens.clone()),
-            eos_token_ids,
+            special_tokens,
         );
         drop(engine); // Explicitly drop the lock guard
         let lark_string = get_lark_from_top_level_grammar(&llg_grammar);

src/tools/mod.rs

@@ -23,7 +23,7 @@ pub struct ToolBuilder {
 }
 
 impl ToolBuilder {
-    fn new(name: String, description: String) -> Self {
+    pub fn new(name: String, description: String) -> Self {
         Self {
             name,
             description,