Improve HF tokenization hack to cover multiple special tokens#649

shawnz wants to merge 3 commits intoguidance-ai:mainguidance-ai/guidance:mainfrom

shawnz:shawn/improve-hf-tokenization-hackshawnz/guidance:shawn/improve-hf-tokenization-hackCopy head branch name to clipboard