Technical Help: Which Tokenizer for Perplexity LLaMa Models?

[Vidyuthr]

Hi everyone!

Does anyone know which tokenizer (and library) we should use to count tokens the exact way the Perplexity models do it?

I'm integrating Perplexity's LLaMa 3.1 models (small/large/huge online and small/large chat) into our product. We need to exactly count input tokens for user quota management, but couldn't find documentation about which exact tokenizer these models use. For example, OpenAI models use Tiktoken.

Thanks!