Fix tokenizer.encode() to respect add_special_tokens=False parameter#821

Open

ved1beta wants to merge 5 commits intoallenai:mainfrom

ved1beta:Fix_add_special_tokens=False

Commits on Apr 7, 2025

Fix tokenizer.encode() to respect add_special_tokens=False parameter
ved1beta
committed

Commits on May 15, 2025

Commits on May 16, 2025

Merge branch 'main' into Fix_add_special_tokens=False
dirkgr
authored