For the default model, how can you determine how many tokens are in a text?

[jronallo]

I want to test whether I need to split my text before it gets to fastembed, but I'm uncertain how I can use the tokenizer for this model to count the number of tokens in my documents.

Once I get the model with embedding_model = TextEmbedding() how can I get the tokens which would be produced with this model?

[joein]

Hi @jronallo , sorry for the late response

You can access tokenizer with embedding_model.model.tokenizer

[jronallo]

Thank you. I thought I had updated this question by answering myself. Yes, embedding_model.model.tokenizer.encode(text) does what I need.