Contect size limitation

[touseefurrehmanofficial]

Shubham, I hope you are doing well. First, thanks for resolving recent issues in llminference.cpp. I'm opening this discussion on the topic of context size limitation in this app. Can you please tell me that is this issue is because of llama.cpp or in smollm module. So that maybe I can research on it and solve it if I can.

[touseefurrehmanofficial]

As in pocketpal readme they said they used llama.cpp also for running gguf models, so maybe their repo can help in solving this problem

[shubham0204]

@touseefurrehmanofficial What exactly is the issue with the context size? The context size comes from the model, and the app 'could' only truncate or summarize the chat to ensure that the chat fits in the model's context window.

[touseefurrehmanofficial]

I know context size comes from model but you can take look on IRIS offline GPT and Pocketpal-AI. They use the same models, but for handling larger context sizes, they implement a batching system. In IRIS, for example, they use llama.cpp, yet they still manage to implement batching or chunking of queries—though I'm not sure exactly how.

[touseefurrehmanofficial]

ALso please take look on this conversation maybe it will solve this problem huggingface