Add backwards-compatible support for multiple EOS tokens#305

Open

hudson-ai wants to merge 29 commits intoguidance-ai:mainfrom

hudson-ai:multi_eos

Commits on Mar 6, 2026

Commits on Mar 10, 2026

Commits on Mar 11, 2026

Use std::vector instead of new[]/delete[] in c_sample

hudson-ai
and
Copilot
committed
Take a single eos_tokens vector in create_tokenizer_v2

hudson-ai
and
Copilot
committed
Remove stale commented-out v2 snippet from create_tokenizer

hudson-ai
and
Copilot
committed