🐛 500 error on ontology generation with thinking/reasoning models (Qwen3, DeepSeek-R1, etc.)

[SiNaPsEr0x]

What's happening

If you're running MiroFish with a local reasoning model (Qwen3, DeepSeek-R1, QwQ...) through llama.cpp, every simulation attempt silently dies with a 500 right after the LLM call. No traceback in the terminal, nothing useful in the logs — just 响应: 500 and that's it.

Two bugs, both need fixing

Bug 1 — response_format={"type": "json_object"} in llm_client.py

chat_json() passes this to llama-server, which enforces JSON grammar from token 0. Thinking models generate <think>...</think> before the actual JSON — the grammar constraint fires immediately and the output comes out garbled. The <think> stripping regex that's already in chat() never gets a chance to save the day.

Fix: remove response_format={"type": "json_object"} from the self.chat() call inside chat_json(). The system prompt already asks for JSON-only output, and the existing <think> cleanup handles the rest.

Bug 2 — max_tokens=4096 hardcoded in ontology_generator.py

Even after fixing Bug 1, the model still returns 500. Reasoning models spend thousands of tokens on the <think> block before generating the actual JSON. With max_tokens=4096 the context gets exhausted mid-JSON and json.loads() throws.

Confirmed via llama-server logs — every call terminates at exactly 4096 decoded tokens:

eval time = 244884.20 ms / 4096 tokens

Fix: bump max_tokens=4096 to max_tokens=16384 in the chat_json() call inside ontology_generator.py.

Both fixes together make it work end to end ✅

---

Fixes I applied locally (verified working)

Fix 1 — remove response_format from backend/app/utils/llm_client.py:

# chat_json() before:
response = self.chat(
    messages=messages,
    temperature=temperature,
    max_tokens=max_tokens,
    response_format={"type": "json_object"}  # ← remove this line
)

# chat_json() after:
response = self.chat(
    messages=messages,
    temperature=temperature,
    max_tokens=max_tokens,
)

Fix 2 — increase max_tokens in backend/app/services/ontology_generator.py:

# before:
result = self.llm_client.chat_json(
    messages=messages,
    temperature=0.3,
    max_tokens=4096   # ← change this
)

# after:
result = self.llm_client.chat_json(
    messages=messages,
    temperature=0.3,
    max_tokens=16384  # enough headroom for <think> block + full JSON
)

---

My setup

• llama.cpp llama-server with Qwen3.6-35B-A3B Q4_K (thinking mode on, confirmed thinking=1 in server logs)
• Windows 11, RTX 5070 12GB VRAM
• MiroFish main @ fa0f651

---

Happy to test a fix if you want to push one. 🙌