Skip to content

Add Japanese tokenizer (fugashi) and minimal unit test#223

Open
supernaiter wants to merge 1 commit intomicrosoft:mainfrom
supernaiter:pr-japanese-tokenizer
Open

Add Japanese tokenizer (fugashi) and minimal unit test#223
supernaiter wants to merge 1 commit intomicrosoft:mainfrom
supernaiter:pr-japanese-tokenizer

Commits

Commits on Jul 12, 2025