本地Ollama不同模型部署总结和心得 Tipps for deploying OpenManus with Ollama local LLMs

[Lu233]

首先你必须要使用Ollama中带有tools标签的模型，

其次，就算是带有Tools标签的本地模型，大多数效果并不好，我用了以下模型，一个简单的总结:
llama3-groq-tool-use 8b: 有思考过程，但是保存文件大多数情况失败，经常会输出重复内容，无法正常结束(调用Terminate)
qwen2.5 7b: 无思考过程，保存文件失败率较高，经常会输出重复内容
qwen2.5-coder 14b: 无思考过程，保存文件成功率较高，有概率正常结束(调用Terminate)
incept5/llama3.1-claude: 无思考过程，保存文件成功率较高，但工具调用出错率大
hengwen/watt-tool-8B: 据说在Tooluse Benchmark上排名靠前，然而却不支持OpenManus的工具使用

整体来说，没有哪个本地模型是运行的好的，几乎没有多步思考过程，相对来说qwen2.5-coder 14b 稍好。 OpenManus的API是参考OpenAI格式的，用API的话理论上ChatGPT的效果应该最好。

下面的视频有对OpenManus深入的原理解说和一步步部署教程，有需要的可以看一下
https://youtu.be/oKrpd8d5wFA?si=qB7aUvxPixai9aOd

First, you must use a model in Ollama that has the "tools" tag.

Second, even with models with the "tools" tag, Almost all of them do not perform well. Here is a brief summary of the models I have tested:

llama3-groq-tool-use 8b: Has a reasoning process, but often fails to save files, frequently outputs repetitive content, and cannot properly terminate (calling Terminate).
qwen2.5 7b: No reasoning process, high failure rate when saving files, and frequently outputs repetitive content.
qwen2.5-coder 14b: No reasoning process, higher success rate for saving files, and sometimes terminates properly (calling Terminate).
incept5/llama3.1-claude: No reasoning process, higher success rate for saving files, but has a high error rate in tool calls.
hengwen/watt-tool-8B: Reportedly ranked high in the Tooluse Benchmark, but does not support OpenManus tool usage.
Overall, no local model performs well, and almost none exhibit multi-step reasoning. Relatively speaking, qwen2.5-coder 14b is slightly better.

OpenManus API follows the OpenAI format, so in theory, ChatGPT should provide the best results when using the API.

For an in-depth explanation of OpenManus and a step-by-step deployment tutorial, check out the video below:
https://youtu.be/oKrpd8d5wFA?si=qB7aUvxPixai9aOd

[vincent7f]

I recommend utilizing models of 32B or larger on the Ollama platform. The Qwen2.5 models, ranging from 7B to 32B, have been tested on Ollama by me, and only the 32B model can effectively function as a real AI, generating suitable instructions for AI agents to complete tasks. Additionally, it is worth noting that the models available for download on Ollama are quantized, which means their capabilities are somewhat diminished compared to the original models. From this perspective, the Qwen 2.5:7B model on Ollama will likely perform less effectively than its original version.

[Lu233]

Hi @vincent7f thanks for the hint! may I ask have you used the un-quantized Qwen 32B? or also quantized one. And which GPU do you use. Thanks

[vincent7f]

Hi @vincent7f thanks for the hint! may I ask have you used the un-quantized Qwen 32B? or also quantized one. And which GPU do you use. Thanks

I used the quantized 32b qwen 2.5 on Ollama with 4090。Here is my test and article for your reference.
https://mp.weixin.qq.com/s/WBbA3EYtHSj8CuHp2tO39A

[chiangandy]

Thanks for your clear comments; yes, during my testing, I got same situation and same feeling. I suggest OpenManus can document more clear in user guide to notice which model can support tools and can work on OpenManus especially focus on model name and scale (larger than 32B will be better). It can save user a lot of time trying on this.