Osaurus is slow and responds very late.

[secfb]

Checks

• I have searched existing issues and discussions
• I can reproduce this with the latest main or release

Describe the bug

Osaurus's slowness and extremely slow response times are a real problem. I tested it on the qwen 3.5 9b 4bit MLX version as an LLM user. Using the XLSX plugin, I asked it to write the top 10 OWASP vulnerabilities to Excel. It took 30 minutes, but it still didn't respond. However, when I tried the same prompt and the same LLM model, LM Studio responded in less than a minute. Osaurus's interface is generally good, but it's really very slow. If this problem isn't fixed, I'll switch back to Ollama or LM Studio. This isn't just the case with this prompt; it's very slow with other prompts as well. My computer specifications are an M4 Air with 24GB of RAM. I don't think the RAM and CPU amount are the problem, because LM Studio works flawlessly.

Steps to reproduce

No response

Osaurus version / commit

Osaurus 0.14.16

macOS version

macOS Sequoia 15.7.1

Apple Silicon chip

m4

Xcode version

No response

Logs

Screenshots

No response

[tpae]

Hey @secfb , sorry about this issue. Can you tell me how many tools you have enabled? Can you share some screenshots or video?

[technerd-official]

Same here compared to LM Studio, Osaurus is way to slow (mine has only 17 capabilities enabled) and in the end it does not produce a result in work mode. Tried to fetch a website from github and get some results, it starts to browse, fetch, it thinks but in the end no answer.

[secfb]

As this friend mentioned, there are performance issues, the displayed features don't work, and it responds very slowly. I'm now using Ollama or LM Studio. You can try the prompt I mentioned to identify the problem. You will encounter this situation as well.

[vernonstinebaker]

incredibly, painfully, slow. Using the exact same model, same prompt. Submitted at nearly the same time (Osaurus first, then LM Studio, to give Osaurus every advantage possible). Here's the prompt: As quickly as possible, tell me which LLM you're using.
LM Studio: 00:00:54:60 -- just under a minute. No speed demon, but acceptable considering a large, dense model. (BTY, the model wasn't pre-loaded, I clicked the send button on Osaurus, then clicked the load model button on LM Studio.)
Osaurs: 00:16:XX:XX -- sixteen minutes later and I'm still waiting for a response. Something is certainly wrong. I'm trying to do some debugging. Hopefully we can improve things (significantly).

Tools are disabled.

[vernonstinebaker]

So I have some good news, and some bad(ish) news.

@secfb I'm not sure if you actually ever saw any output after 30 minutes? I'm doubtful. I suspect the actual 'processing' ended well before that.

The Good News:
I've opened a couple of PRs to make some improvements to the processing. The first are related to how models are handled behind the scenes; this should improve the processing speed -- we're still not as quick as LM Studio -- I'm still seeing what else can be done here.

Another PR is related to how Qwen tool calling is managed. Most(?) LLMs seem to output JSON for tool calling, but Qwen3.5 outputs XML, which wasn't being processed, so the new PR adds processing for Qwen XML tool calling. With this PR in place, I'm able to see results (I don't have an Excel related tool installed, so it's just outputting to chat, but presumably it could write to Excel if the tool is available in the local environment. Below is a screenshot that shows results and the stopwatch. 2 minutes, 16 seconds.

The Bad(ish) News:
Not fast, but much better than 30 minutes. Once (if) the PRs get merged, you can test and see if this at least allows you to use the tool, even if it's not (yet) that fast.

[technerd-official]

@vernonstinebaker I don't know why osaurus is slow in the first place. If using llama.cpp then it should be on par. Also MLX is not that good for agentic work I saw. Maybe this guy @jjang-ai is the solution. He doing some incredible work for Mac with adaptive mixed-precision quantization and MLX Studio.
https://github.com/jjang-ai/jangq
https://github.com/jjang-ai/mlxstudio

[vernonstinebaker]

Osaurus isn't using llama.cpp. A closer comparison is LM Studio using MLX, although it's still not purely apples to apples, since the LM Studio MLX Engine is written in Python, not using swift-mlx. Other than things being slow (though faster than before), the tool calling is working. That's not a MLX problem, but a gateway/agent problem, and the issue isn't isolated to MLX vs. c++ and Python just have more people focused and working on implementing and optimizing than Swift has. Standards and implementations for tool calling are still fluid, so regardless of implementation language when new LLMs are released tools (often/usually) need to be updated to work effectively with what's being emitted.

[secfb]

@vernonstinebaker It's really good that it's so fast, because the tool's interface and features are good. However, performance is very important in local LLM. When I tried it with LM Studio using gffuf and mlx, it was still slower than Osarus, and I couldn't figure out why Osarus was slow. I hope it gets much better and I'd like to use it.

[vernonstinebaker]

I'm not declaring victory yet -- there might still be some speed we can squeeze out; I'll keep looking -- however, it's much faster than it was before and it is generating output consistently in my testing. Several related PRs have already been merged, so it would be great if you could test with the latest release @secfb and @technerd-official.

[secfb]

I tried again and it's still very slow. If you don't mind me saying, I uninstalled the program. I won't be checking it anymore. Thank you for your efforts. I switched to AtomicBot-ai, which uses Google's newly developed technology. It's really fast.

https://github.com/AtomicBot-ai/

[vernonstinebaker]

@secfb Are you using Ollama to serve a local model or did you configure to use a cloud provider? If you've switched to using a cloud provider, and you configured Osaurus the same way, you'd probably see similar performance.. I'm curious if you tried that. No worries either way. No tool is a perfect fit for everyone. One thing to keep in mind; if you are using a cloud provider (Google or any other cloud provider) data is leaving your device.

[secfb]

@vernonstinebaker I don't use it in the cloud. I generally use Ollama and LM Studio, and the QWEN 3.5 model for local operations. For Osarus, I'm still experiencing the same performance degradation with the new version.

[vernonstinebaker]

@secfb Thanks for the additional confirmation. I could be misunderstanding the situation, but I think both Ollama and LM Studio are just passing direct LLM responses back to AtomicBot-ai, and it's doing the agent work, whereas Osaurus is undertaking agent work on it's own, so that agent work and tool calling is the bottleneck. I'm hoping someone else can jump in and look at the tool selection and tool calling. Right now I'm just focusing on the LLM part; making sure things don't get stuck and that processing is as fast as possible. There's some progress there, things don't fail completely now, but the (tool selection/tool processing) is still quite slow.