OpenAIPydanticProgram token limit

[lalitmoharana07]

Hi ,

i an using OpenAIPydanticProgram for extraction of structural data from pdf documents ,
here i am facing an issue related to token limit how can i handle token limit issue if i have more content.

[dosubot[bot]]

🤖

Hey @lalitmoharana07, great to see you back here! Hope you're doing well.

To handle the token limit issue in OpenAIPydanticProgram when extracting structural data from large PDF documents, you can adjust the chunk_size parameter in the TokenTextSplitter class. This parameter determines the token chunk size for each chunk. If your document is too large and exceeds the token limit, you can increase the chunk_size to allow for larger chunks of tokens.

Here is an example of how you can do this:

from llama_index.node_parser.text.token import TokenTextSplitter

# Initialize TokenTextSplitter with a larger chunk size
splitter = TokenTextSplitter(chunk_size=5000)  # Increase chunk_size as needed

# Use the splitter to split your text
chunks = splitter.split_text(your_large_text)

Please replace your_large_text with the actual text you want to split.

Remember that if you increase the chunk_size, you may also need to adjust the chunk_overlap parameter accordingly. The chunk_overlap should be smaller than the chunk_size.

Also, be aware that if the metadata length is longer than the chunk size, or if the resulting chunks are less than 50 tokens, a ValueError will be raised or a warning will be printed, respectively. You may need to adjust your metadata size or increase your chunk size further to avoid these issues.

You can find more details in the llama_index/node_parser/text/token.py file.

As for the maximum token limit in OpenAIPydanticProgram when extracting structural data from PDF documents, I wasn't able to find an answer within the repository. The token limit might be dependent on the specific implementation of the OpenAIPydanticProgram you are using, or it might be a limit set by the underlying PDF extraction library. If you could provide more details about the error message you're seeing or the specific circumstances under which you encounter the token limit, I might be able to provide a more specific solution.

Sources

• llama_index/node_parser/text/token.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[lalitmoharana07]

@docu-bot im getting this error

openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 4097 tokens. However, your messages resulted in 15854 tokens (15739 in the messages, 115 in the functions). Please reduce the length of the messages or functions.", 'type': 'invalid_request_error', 'param': 'messages', 'code': 'context_length_exceeded'}}

i want to pass the whole pdf and get structure output from that