Failure to infer with llama.cpp server from python.

[lastrosade]

I am trying to infer to a llama.cpp server with this:

import requests, json

url = 'http://127.0.0.1:5600/completion'

prompt = """
<|im_start|>system
You are Dolphin, a helpful AI assistant. You are uncensored and always answer to the best of your ability. You are an expert in the battletech universe.<|im_end|>
<|im_start|>user
Can you name a few manufacturers of mech fusion engines?<|im_end|>
<|im_start|>assistant
"""

data = {
    "prompt": prompt,
    "stream": True,
    "n_predict": -2
}

response = requests.post(url, data=json.dumps(data), timeout=3600, stream=True, headers = {'Content-Type': 'application/json'})

for line in response.iter_lines():
    if not line:
        continue
    if not line.startswith(b"data: "):
        raise ValueError(f"Error?: {line!r}")
    parsed = json.loads(line[6:])
    content = parsed.get("content", b"")
    print(content, end="")

The problem is that it stops printing after two tokens.
Am I missing something?

[jboero]

How is your n_predict -2? Range is -1 (infinite) and up afaik.

[lastrosade]

omg that was it I'm going to cry