Structured Generation: Implement LLGuidance from Upstream

[iwr-redmond]

Support for LLGuidance, which uses constrained sampling to facilitate valid JSON output, was added to llama.cpp and then enhanced earlier this year. It's the difference between asking "pretty please", validating the output post-generation, and guaranteeing valid output by supervising each token as it is generated, and it makes working with Small Language Models much more reliable.

Enabling this feature during compilation requires some fiddling with Rust, but is probably the most effective implementation possible given the move away from the llama-cpp-python backend (see #370 for history).

[zhiyuan8]

For structure decoding feature, it is in our nexa server functionality, please try:

nexa serve

then in another terminal

curl -X POST http://127.0.0.1:18181/v1/completions \
-H "Content-Type: application/json" \
-d '{
  "model": "Qwen/Qwen3-0.6B-GGUF",
  "prompt": "Write a hello world program in Python",
  "max_tokens": 150
}'

[iwr-redmond]

I appear to have loaded the wrong wagon. Sorry about that.

That is not structured generation. Structured generation is a feature that can be used to require models to conform their outputs to specific formats, such as choices from a predefined list or particular output fields.

Superficially similar functionality exists in Ollama. However, it works by proposing the structure to the LLM and validating the response upon completion. In essence, it's hoping for the best rather than actually intervening in the generation process.

By comparison, LllamaCPP's structured guidance feature offers the ability to do token-level supervision during generation, filtering the logits to ensure that each generated token matches the required format. This guarantees that the response meets the specified criteria regardless of model hallucinations, refusals, or other runtime errors. Structured generation even works without the model being provided with the desired structure in the user prompt, although generation is more reliable when it knows the expected format.

Practically, what this means is that SLMs can do more with less, and can be relied on for use cases like function calling where even huge models like GPT4 sometimes fail. For a toolkit like Nexa, it would provide, for example, a way for an LLM to trigger a Stable Diffusion image generation with a positive prompt and negative prompt that omits banned words, filtering for these using regex matching, as well as a model filename derived from a list of checkpoints physically present on the user system at runtime.

[iwr-redmond]

The CMakeLists option that needs to be switched on is here.

You'll also need rustc and cargo installed in the runners. According to the documentation, "GitHub-hosted runners include a recent version of the Rust toolchain".