Token Sequence Length Exceeds Limit Despite `model_tokens` Parameter in Ollama Model

[JSchmie]

Describe the bug
The model_tokens parameter in the graph_config dictionary is not being applied to the Ollama model within the SmartScraperGraph setup. Despite setting model_tokens to 128000, the output still shows an error indicating that the token sequence length exceeds the model's limit (2231 > 1024), causing indexing errors.

To Reproduce
Steps to reproduce the behavior:

1. Set up a SmartScraperGraph using the code below.
2. Configure the graph_config dictionary, specifying model_tokens: 128000 under the "llm" section.
3. Run the scraper with smart_scraper_graph.run().
4. Observe the error regarding token sequence length.

Expected behavior
The model_tokens parameter should be applied to Ollama's model, ensuring that the model respects the 128000-token length specified without raising indexing errors.

Code

from scrapegraphai.graphs import SmartScraperGraph

ollama_base_url = 'http://localhost:11434'
graph_config = {
    "llm": {
        "model": "ollama/mistral",
        "temperature": 1,
        "format": "json",
        'model_tokens': 128000,
        "base_url": ollama_base_url
    },
    "embeddings": {
        "model": "ollama/nomic-embed-text",
        "base_url": ollama_base_url
    },
}

smart_scraper_graph = SmartScraperGraph(
   prompt='What is this website about?',
   source="my.example.com",
   config=graph_config
)
result = smart_scraper_graph.run()
print(result)

Error Message

Token indices sequence length is longer than the specified maximum sequence length for this model (2231 > 1024). Running this sequence through the model will result in indexing errors.

Desktop:

• OS: Ubuntu 22.04.5 LTS
• Browser: Chromium
• Version:
  • Python 3.12.7
  • scrapegraphai 1.26.7
  • Torch 2.5 (Torch should not be necessary as Ollama is being used)

Additional context: Ollama typically uses the num_ctx parameter to set context length. It seems that model_tokens does not directly influence the model's context length, suggesting a possible oversight or misconfiguration in how the SmartScraperGraph handles token length parameters with Ollama models.

Thank you for taking the time to look into this issue! I appreciate any guidance or suggestions you can provide to help resolve this problem. Your assistance means a lot, and I'm looking forward to any insights you might have on how to apply the model_tokens parameter correctly with Ollama. Thanks again for your help!

[VinciGit00]

please update to the new examples

[VinciGit00]

they are legacy

[JSchmie]

Can you point me towards the latest examples? I followed this one.

[VinciGit00]

look at this link https://github.com/ScrapeGraphAI/Scrapegraph-ai/tree/main/examples

[JSchmie]

look at this link https://github.com/ScrapeGraphAI/Scrapegraph-ai/tree/main/examples

I've looked into the examples, and I noticed that in this example and other examples related to Ollama, the context window is set using model_tokens. However, in the example for simple web scraping, the context window isn’t modified at all.

I really like your project, but without being able to increase the context window to make full use of the model, I won’t be able to use this framework effectively. Could you please provide a short code snippet or guidance on changing the context length in the latest version?

[VinciGit00]

ok, can you specify the context_window inside? like this graph_config = {
"llm": {
"model": "ollama/mistral",
"temperature": 1,
"format": "json",
'model_tokens': 128000,
"base_url": ollama_base_url
}

[VinciGit00]

Btw which model of mistral are you using?
These are the available models https://ollama.com/library/mistral

[JSchmie]

ok, can you specify the context_window inside? like this graph_config = { "llm": { "model": "ollama/mistral", "temperature": 1, "format": "json", 'model_tokens': 128000, "base_url": ollama_base_url }

As you can see from my example, I followed this procedure. I attempted to execute it without embedding for debugging purposes, however, the identical error persists. I am using ollama version 0.3.14.

Btw which model of mistral are you using? These are the available models https://ollama.com/library/mistral

I just use the latest model of mistral, but I also tried llama3.1:8b and 70b which has a context length of 128k and also the gemma2:9b.

[f-aguzzi]

@JSchmie I fixed this in #773. The model_tokens dictionary key was only available with model instances before this, but now it's accessible for all models.

The pull request will be merged into the development branch (pre/beta) first, so a few days will pass before the fix will be available in a stable release.

[JSchmie]

@JSchmie I fixed this in #773. The model_tokens dictionary key was only available with model instances before this, but now it's accessible for all models.

The pull request will be merged into the development branch (pre/beta) first, so a few days will pass before the fix will be available in a stable release.

I have installed your branch using.

pip install --force-reinstall git+https://github.com/ScrapeGraphAI/Scrapegraph-ai.git@768-fix-model-tokens

But unfortunately, I cannot confirm that it works. I still get the error:

Token indices sequence length is longer than the specified maximum sequence length for this model (11148 > 1024). Running this sequence through the model will result in indexing errors

I can confirm that self.model_token is set to 128000. Furthermore, I also tried to set llm_params["num_ctx"] = self.model_token here since ChatOllama also uses num_ctx to set the context window (see documentation here) but it still does not work.

[VinciGit00]

hi please update to the new beta