[Question]: Can Oh My OpenCode use local small language models (<32B)?

[TomLucidor]

Prerequisites

• I have searched existing issues and discussions
• I have read the documentation
• This is a question (not a bug report or feature request)

Question

Most of the LLM services mentioned in the Readme are often too expensive. Since OpenCode is FOSS, maybe smaller models (<36B total parameters) can be used through Ollama or Jan or MLX?

The only problems I can see with this issue is:

• Using Hybrid Attention models to 4x the speed but possibly losing performance
• MoE models being reported to be street-smart but not "book smart", needing better harness
• Using multiple models, some for planning/reasoning and others for coding/tools

Context

No response

Doctor Output (Optional)

Question Category

Configuration

Additional Information

No response

[Tetrahedrite]

To discuss this issue properly, it helps to understand the purpose of oh-my-opencode in more detail. This is essentially a harness that divides tasks in parallel through an agent called Sisyphus, performs background operations, and instructs it to keep working without stopping as much as possible.

First, since this project is designed as an OpenCode-based plugin, you can freely use any model from any LLM provider. However, smaller models, especially those under 36B parameters, have significantly poor instruction-following capabilities and easily fall into loops (repeatedly executing the same commands or messages until they hang). Even Google's gemini-3-flash occasionally falls into loops.

The main reason the maintainer adopted Claude Opus 4.5 as the primary orchestrator is likely because this model excels at task decomposition and multi-turn promptsnot simply because it's good at coding.

Below is an explanation of my use cases and experiences.

I've used numerous open-source agents: Claude Code, of course, and I've also tried other models by running a proxy layer with a project called claude-code-router that makes Claude work with third-party models instead. I've also experimented with various combinations using different models through OpenRouter, including OpenHands, CRUSH, and Qwen Code.

My conclusion was: "We're still far from being satisfied with fully open-source models alone."
Open-source models frequently mixed Chinese into prompts and code, fell into loops by themselves, showed inconsistent working styles, and easily forgot what they were doing. Models like GLM-4.6 (now 4.7) and MiniMax M2 (now M2.1) showed decent performance, but they're still limited to "coding" and aren't at the desired size level (< 36B).

If you're genuinely interested in these open-source models, I recommend frequently visiting the LocalLLaMA subreddit, trying out the models there yourself, and carefully considering their limitations to see if they truly integrate well with oh-my-opencode. If you discover a good combination, feel free to share it with the community.

[TomLucidor]

@Tetrahedrite here is the key question: can small models be made to be good IF and orchestrators when there are better scaffolds/harness?

[Tetrahedrite]

@Tetrahedrite here is the key question: can small models be made to be good IF and orchestrators when there are better scaffolds/harness?

It could be, but I think that would be a very tough challenge. Since an LLM is a non-deterministic token generator, we can’t force it to adhere strictly to human requirements for every single element. (It’s just that Opus 4.5 is a model that, in my experience, has a very high probability of following instructions.)

This harness, and indeed every harness, was essentially created in the hopes that the LLM would better understand instructions. You never truly know if an LLM will execute a task well until you actually try it out; this result is simply the outcome of getting a feel for the model through hands-on experience, refining the prompts based on that, and applying them as system prompts.

If you want to improve performance on smaller models, you could enhance the harness with more validation and make it more fault-tolerant (such as enabling quick recovery from loops). However, even with those improvements, I doubt it would work very well.

[Mikec78660]

I have tried several frameworks like opencode and oh-my-opencode. I too want to host the models locally as I am just doing personal projects and don't want to spend money just testing concepts.

The issues I have had relate to VRAM and the quality of the models you can run. I have 2 3090s and 2 P40's. If you run a local model as your sisyphus model you will want a context of at least 256k. This avoids the issues you start to see when context is close to being full and while compacting works great, meaning it actually runs, unlike is vs code for example where I can't get compacting to auto run with local models at all, you tend to loose a lot when it has to run several times. And to get a model that can actually perform in this role reliably you are looking at a model closer to 100B and not quantized to death.

What I have found works pretty well is using smaller (32b) models for the other agents and even the coding agent by enabling the Builder agent. Then using any of the models currently freely available (Minimax M2.1, GLM 4.7, etc) as the sisyphus agent. If the smaller local agents get stuck the sisyphus agent does a great job recognizing that and just does the task itself and cancels the background task. With the Ralph loop and ulw this gives really good results for me so far. Depending on your amount of VRAM you will want to set modelConcurrency to 1 or 2 for the local models.

My goal is to eventually find a local model that can handle the orchestration and use a free tier closed source model (Gemini 3 or something) for the oracle model so it would be used sparingly and wouldn't hit rate limits. Just my experience so far.

[TomLucidor]

@Mikec78660 thanks for the report, assuming we use the local small models with integrated memory, and the "sisyphus agent" with API models as the agent of last resort (OpenRouter API cost saving through routing and selective providers), how can we further optimize the whole system to be as cheap as possible? https://github.com/code-yeongyu/oh-my-opencode?tab=readme-ov-file#agents-your-teammates

P.S. heard of Claude Flow? Could OMOC get to that level of having a whole bag-o-tricks?

[Mikec78660]

I mean you can define any agent you want. So for example if you wanted to breakdown coding into different categories. By default it only separates UI/UX coding from other coding. If you wanted to look for models that specialized in a particular language or environment, or specifically for testing or something you could. Just running locally you will always bump up against memory issue. Running 10 tasks asynchronous even on the same model just isn't possible unless you have pretty extreme hardware just due to the context. A 32gig model with 32k context is doable but a 32gig model with the equivalent of 640k context, even at kv cache q8_0 is huge. So for me I am still limited to 1 or 2 concurrent tasks at a time. So while possible to getting the kind of performance they show with Claude Flow it just doesn't seem practical without enterprise hardware with opencode/OMOC using local models right now. It seems like you would need some kind of memory watcher. So when issuing tasks the sisyphus agent would have to check the VRAM available to determine if a sub-agent could be called or if it would have to be queued and how many instances could be called in parallel.

[TomLucidor]

@Mikec78660 some further questions:

• What about 30B-A3B models? I tried OpenCode with it and it seemed fine?
• How do I set Oh My OpenCode to use mlx-community/NVIDIA-Nemotron-3-Nano-30B-A3B-4bit for example?
  Cross-reference to checking OpenCode out Guides on using this with Claude Code cubist38/mlx-openai-server#122
• How can bad prompt formatting issues be avoided for SLMs?