Support token-based chunking in chunk_by_title using tiktoken

[hamedyosefian]

Currently, chunk_by_title in unstructured.chunking.title uses max_characters for chunking, but I need it to support token-based chunking (e.g., 512 tokens per chunk) using tiktoken.

The current implementation isn't ideal for token-based models like OpenAI GPT, which rely on token limits rather than character limits. How can we modify this to chunk by tokens instead of characters?

Any guidance or suggestions to solve this would be appreciated!

[HarshiniKarumuri]

The easiest way would be to have count_tokens sort of method in your tokenizer and use it to calculate tokens wherever length / number of characters are getting checked in chunk_by_title (will_fit / add_element / remaining_space).

For example, while checking if text fits the current chunk we can have something like self._remaining_space >= self._opts.tokenizer.count_tokens(element.text)

count tokens function could be as something like this

def count_tokens(text: str, model: str = "gpt-4o-mini") -> int:
    import tiktoken
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))