support generate Qwen model tokenizer? support load tikToken?

[gabohouhou]

now i want to calculate Qwen model input/output tokens length by djl, but Qwen model use qwen.tiktoken not tokenizer.json, is there a way to load qwen.tiktoken or can translate ?

[frankfliu]

Currently DJL use fasttokenizer from huggingface tokenizer. It's possible to implement TikTokenizer in java, would like to contribute?

[gabohouhou]

thank you for your reply and your work for DJL. i am glad to do that，Qwen-72B-Chat model use TikTokenizer，and i find Qwen1.5-72B-Chat use huggingface tokenizer，finally we choose the latter。