Small LLM quality on routing and wiki

[SomeOddCodeGuy]

Over the past half year or so I've been looking for ways to improve small model quality in Wilmer, and I think the current state is in a much better place than it was back in September when I last talked about it.

Recently I've made 2 major changes that should help if you ever tried with a small model and feel like the quality wasn't quite there:

1. I updated the categorization workflow to use a style that seems to work far better on small models. I've been testing it using Nemo 12b, and it's gotten the route correctly 9 out of 10 times, with the failure points being Open WebUI autocomplete, tags or title. But it appears to almost always get the main domains like coding, factual, and other correct. There is still a little more screw to tighten if small models still struggle; I can add a third node in there before the current two, re-implementing the "What is the context of what they are asking for" before having it do the validation questions. But right now the validation question style with bullet points is working great for me

2. A couple months back I updated the offline wikipedia api to do a second layer of scoring on top of what txtai does, and this improved the article selection a pretty decent bit. I'm quite happy with it so far, but we'll see how that goes.

Anyhow, wanted to call that out if anyone was curious if any work had been done to help on smaller models. I'm definitely testing them more regularly now.

[amacsmith]

How about now, lol? (It lives while we sleep 🤣)

(off the cuff), I have been possibly, wondering if there would ever exist a need for project based micro-agentic/model system.

• Now that I have some more insight into of the world of distils etc.
  • to get .5 to 1.5 B parameter models that are specifically baked for tools
    • git, windows, react v18, vite, chrome, node, docker, SOLID principles, TDD runner, TDD validator, mkdocs/docusaurus documenter, etc...
  • However, small enough and known dataset to achieve the "aha moment".
  • Then throw a Wilmer or something of the like in front of it to prune/deactivate modes/agents as context narrows.
• All in a possible attempt to get a, Yes very specific, but potentially useful Frankenstein-esk light weight engine to run in front of it.