Add backwards-compatible support for multiple EOS tokens

[hudson-ai]

Motivation

Models like Qwen 3/3.5 define multiple EOS token IDs in their GenerationConfig (e.g. [151645, 151643]), but llguidance only supported a single EOS token. Qwen 3 uses <|im_end|> (151645) to end turns in chat mode and <|endoftext|> (151643) as a general end-of-text marker. When llguidance is configured with only one of these (e.g. the tokenizer's default 151645), the other gets masked out. If the model tries to emit the masked EOS, it's forced to pick garbage tokens and enters an infinite repetition loop, never terminating.

Closes #253
Related: #304

Changes

• Add eos_tokens: Vec<TokenId> to TokTrie with accessors, with_eos_tokens() builder, and validation (asserts IDs are within vocab range)
• Update TokenParser to check the full EOS set for mask computation, token consumption, rollback, and stop detection
• C API: LlgTokenizerInit is unchanged. New LlgTokenizerInitV2 struct (flat, with struct_size for forward compatibility) + llg_new_tokenizer_v2() function for multi-EOS support
  • struct_size enables forward compatibility: the FFI function takes a raw pointer, reads only struct_size bytes, and zero-fills any new fields the caller's header doesn't know about
  • llg_new_tokenizer_v2() validates EOS token IDs against vocab size and returns an error (not a panic) for out-of-range IDs
• Python eos_token parameter now accepts int | list[int] across all entry points
• Add eos_tokens getter property to Python LLTokenizer
• Update type stubs and all Python helper modules (hf, tiktoken, llamacpp)
• C sample tests both v1 and v2 APIs end-to-end

Usage

Python:

from llguidance.hf import from_tokenizer
# Pass multiple EOS tokens
tok = from_tokenizer(hf_tokenizer, eos_token=[151645, 151643])
# Single int still works as before
tok = from_tokenizer(hf_tokenizer, eos_token=151645)

C (v2 API):

LlgTokenizerInitV2 init = {};
init.struct_size = sizeof(init);
init.vocab_size = vocab_size;
init.tok_eos = 151645;
init.tokenize_fn = my_tokenize_fn;
// ...set other fields...
LlgToken extra_eos[] = {151643};
init.tok_eos_extra = extra_eos;
init.tok_eos_extra_count = 1;
LlgTokenizer *tok = llg_new_tokenizer_v2(&init, err, sizeof(err));

Rust:

let mut byte_tok = ByteTokenizer::from_json_str(&tokenizer_json)?;
byte_tok.set_eos_tokens(&[151645, 151643]);
let tok_env = byte_tok.into_tok_env(None)?;

API compatibility

Python: Fully backwards compatible. eos_token still accepts a single int everywhere. The only additions are that it also accepts list[int], and there's a new eos_tokens property.

C API: Fully backwards compatible. LlgTokenizerInit is identical to its pre-PR layout — zero fields added or removed. llg_new_tokenizer() is unchanged. Multi-EOS requires the new LlgTokenizerInitV2 + llg_new_tokenizer_v2(), which are purely additive.

Rust (published crates — toktrie, toktrie_hf_tokenizers, toktrie_tiktoken): Only additive changes — new methods like with_eos_tokens(), eos_tokens(), set_eos_tokens(). No existing signatures changed.

Rust (python_ext — not published): tokenv_from_llamacpp changed from eos_token: u32 to eos_tokens: &[u32]. This is technically a breaking signature change, but python_ext is only consumed internally to build the Python wheel, so no external Rust consumers are affected.

Known limitations

• TokRxInfo.tok_eos still holds only the first (primary) EOS token. Code that reads tok_eos directly rather than going through TokTrie::eos_tokens() will only see the primary one.
• with_info() resets eos_tokens back to vec![info.tok_eos], silently dropping extra EOS tokens. Callers that replace TokRxInfo after setting multi-EOS must re-apply with_eos_tokens().

[Copilot]

Pull request overview

Adds multi-EOS-token support across the tokenizer/trie layers and the parser so models that define multiple EOS IDs (e.g., Qwen/GLM) can terminate correctly without masking alternative EOS tokens.

Changes:

• Extend TokTrie/tokenizer wrappers to carry a list of EOS token IDs (while keeping a primary EOS for compatibility).
• Update TokenParser to treat any configured EOS token as valid for mask computation and stop detection.
• Plumb multi-EOS through the C API and Python bindings/helpers (including typing updates).

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
toktrie_tiktoken/src/lib.rs	Adds a setter to apply multiple EOS tokens to the internal TokTrie.
toktrie_hf_tokenizers/src/lib.rs	Tracks extra EOS tokens and applies them when constructing the TokTrie.
toktrie/src/toktree.rs	Adds eos_tokens storage + builder/accessor; adds unit tests for new behavior.
parser/src/tokenparser.rs	Switches EOS handling from single token to a token set for masking/stop logic.
parser/src/ffi.rs	Extends C tokenizer initialization to accept additional EOS token IDs.
parser/llguidance.h	Exposes the new C init fields for extra EOS token IDs.
c_sample/c_sample.cpp	Documents how to pass multiple EOS tokens via the C API.
python_ext/src/py.rs	Accepts eos_token as `int
python_ext/src/llamatokenizer.rs	Updates llama.cpp bridge to accept multiple EOS token IDs and apply them to TokTrie.
python/llguidance/hf.py	Updates helper typing/docs to allow eos_token as `int
python/llguidance/tiktoken.py	Updates helper typing/docs to allow eos_token as `int
python/llguidance/llamacpp.py	Updates helper typing/docs to allow eos_token as `int
python/llguidance/_lib.pyi	Updates stubs for new eos_tokens property and widened eos_token parameter type.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[sempervictus]

@hudson-ai what would the Rust mechanism be for those when creating an llg factory from a tokenzier?

EDIT: also does this mean i should hold off on hacking-together that text_or_eos bit and just pull the next microversion or are release cycles a bit longer around here? vllm.rs is moving quick so i can always undo a hack later

[hudson-ai]

@hudson-ai what would the Rust mechanism be for those when creating an llg factory from a tokenzier?

EDIT: also does this mean i should hold off on hacking-together that text_or_eos bit and just pull the next microversion or are release cycles a bit longer around here? vllm.rs is moving quick so i can always undo a hack later

If you're working from a tokenizer JSON, you'll want to do something like this:

use toktrie_hf_tokenizers::ByteTokenizer;
use llguidance::{ParserFactory, api::TopLevelGrammar};
use llguidance::toktrie::InferenceCapabilities;
use llguidance::earley::SlicedBiasComputer;

let mut byte_tok = ByteTokenizer::from_json_str(&tokenizer_json)?;
// Probably get this from the generation_config.json?
byte_tok.set_eos_tokens(&[151645, 151643]);
let tok_env = byte_tok.into_tok_env(None)?;

let factory = ParserFactory::new(
    &tok_env,
    InferenceCapabilities::default(),
    &SlicedBiasComputer::general_slices(),
)?;

// factory now produces parsers that allow both EOS tokens in masks
let grammar = TopLevelGrammar::from_lark(r#"start: /[a-z]+/"#.to_string());
let parser = factory.create_parser(grammar)?;

If you already have a TokTrie, you can call trie.with_eos_tokens(&[151645, 151643]) instead and create a TokEnv from there, but the ByteTokenizer path above is probably closest to what you're doing.

RE: holding off on the "hack" -- I think that I can reasonably get a release out in the next week or so, but no strong guarantee. Need to await some review on this PR too. But if you don't mind doing and undoing a hack, go for it 😉

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 7 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 17 out of 17 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

python_ext/src/py.rs:75

• In the LLTokenizer("byte") path, any provided eos_token override is currently ignored (it always returns ApproximateTokEnv::single_byte_env()). Since the Python API now accepts int | list[int] for eos_token, this should either be applied here (e.g., by cloning the underlying trie and calling with_eos_tokens() before wrapping it in an ApproximateTokEnv) or explicitly rejected with a clear error to avoid silent misconfiguration.

        let tok_env: TokEnv = if let Ok(tokenizer_str) = tokenizer.extract::<String>() {
            if tokenizer_str == "byte" {
                ApproximateTokEnv::single_byte_env()
            } else {

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hudson-ai]

Ok after that tremendously long back and forth... @riedgar-ms would you mind reading the C part of this and giving me a sanity check?

[sempervictus]

That multitoken ID grammar is looking pretty elegant right about now 😉

[hudson-ai]

That multitoken ID grammar is looking pretty elegant right about now 😉

Ugh I know. The actual core changes are pretty small -- just lots of API entry-points to be careful of (including "nice" error handling). I've actually considered making this PR more than once over the last few months and have balked every time because I didn't want to make breaking FFI changes that necessitated a major version bump. I think that I got a handle on that now though...

[riedgar-ms]

Why not make this a wrapper for create_tokenizer_v2()? Isn't the code largely the same?

[hudson-ai]

Under the hood, each function is using a different init struct, and consolidating would effectively remove the "living documentation" (slash test) of the old API. I think that removing the commented-out block is a good idea though.

Thoughts?