[BUG]: IndexOutOfRangeException with structured GBNF grammar output

[SerialKicked]

Description

Hi,

I think i've tried about everything to get grammar to work properly to no avail. I have read the docs and looked at the issues, but for some reason I can't click the checkboxes. docs/Examples/GrammarJsonResponse.md is outdated btw.

I'm using 0.25 on a Stateless executor: Here's a link to the adapter i wrote

The same grammar that works on KoboldCpp always gives me an error when running LLamaSharp instead. A System.IndexOutOfRangeException: 'Index was outside the bounds of the array.' to be precise.

I'm probably doing something wrong, but I really don't see what. I tried with different Grammar Optimization values out of curiosity, but nope. Oh, and yeah, normal generation works perfectly fine.

Any help would be super welcome because I managed to get structured output to work for both KoboldAPI, and OpenAI-compatible backends. So, that'd very neat to get feature parity for LLamaSharp.

Reproduction Steps

So, example:

I have a BasicBitch class

public class BasicBitch
{
    public string Response { get; set; } = string.Empty;
    public int Confidence { get; set; } = 0;
}

The simplest possible JSON Schema -> GBNF conversion gives:

root ::= "{" space Response-kv "," space Confidence-kv "}" space
Response ::= string | null
Response-kv ::= "\"Response\"" space ":" space Response
Confidence-kv ::= "\"Confidence\"" space ":" space integer

char ::= [^"\\\x7F\x00-\x1F] | [\\] (["\\bfnrt] | "u" [0-9a-fA-F]{4})
integer ::= ("-"? integral-part) space
integral-part ::= [0] | [1-9] [0-9]{0,15}
null ::= "null" space
number ::= ("-"? integral-part) ("." [0-9]+)? ([eE] [-+]? [0-9]+)? space
space ::= | " " | "\n"{1,2} [ \t]{0,20}
string ::= "\"" char* "\"" space

Finally (ignore the implied data):

var pipeline = new DefaultSamplingPipeline()
{
    Temperature = (float)input.Temperature,
    TopP = (float)input.Top_p,
    TopK = input.Top_k,
    TypicalP = (float)input.Typical,
    MinP = (float)input.Min_p,
    RepeatPenalty = (float)input.Rep_pen,
    PenaltyCount = input.Rep_pen_range,
    // input.Grammar has the grammar string stated above. I also tried the one you provide in your json.gbnf file and got same error.
    Grammar = string.IsNullOrWhiteSpace(input.Grammar) ? null : new Grammar(input.Grammar, "root"),
    PenalizeNewline = false,
    PreventEOS = input.Bypass_eos,
    Seed = input.Sampler_seed <= 0 ? 0 : (uint)input.Sampler_seed
};
var sett = new InferenceParams()
{
    MaxTokens = input.Max_length,
    AntiPrompts = input.Stop_sequence?.ToArray() ?? [],
    SamplingPipeline = pipeline
};
string response = string.Empty;
await foreach (var text in Executor.InferAsync(input.Prompt, sett, token)) // <<<---- CRASHES BEFORE FIRST LOOP
{
    if (token.IsCancellationRequested)
        break;
    response += text;
}

I've tried on about 10 different json schemas of various level of complexity, and none work (while they all do in KoboldCpp, so the GBNF conversion should be solid)

Environment & Configuration

• Operating system: WIn11
• .NET runtime version: .NET 8
• LLamaSharp version: 0.25
• CUDA version (if you are using cuda backend): 12
• CPU & GPU device: RTX 3090 24GB

Known Workarounds

No response

[martindevans]

Could you provide the full stacktrace from the IndexOutOfRangeException exception?

[SerialKicked]

Sure, my bad I should have included that from the get go:

Index was outside the bounds of the array.

Stacktrace:
at Internal.Runtime.CompilerHelpers.ThrowHelpers.ThrowIndexOutOfRangeException()
at LLama.Sampling.DefaultSamplingPipeline.Sample(SafeLLamaContextHandle ctx, Int32 index)
at LLama.StatelessExecutor.d__24.MoveNext()
at LLama.StatelessExecutor.d__24.System.Threading.Tasks.Sources.IValueTaskSource<System.Boolean>.GetResult(Int16 token)
at LetheAISharp.API.LlamaSharpAdapter.d__35.MoveNext() in H:\SourceCode\LetheAISharp\Adapters\LlamaSharpAdapter.cs:line 159
at LetheAISharp.API.LlamaSharpAdapter.d__35.MoveNext() in H:\SourceCode\LetheAISharp\Adapters\LlamaSharpAdapter.cs:line 159
at LetheAISharp.LLM.LLMEngine.d__120.MoveNext() in H:\SourceCode\LetheAISharp\LLM\LLMEngine.cs:line 490
at WaifuAI.MainForm.<button1_Click>d__95.MoveNext() in H:\SourceCode\WaifuAI\MainForm.cs:line 1667

(LetheAISharp is the middleware library i'm working on and which is calling LLamaSharp. WaifuAI is just the UI frontend)

[aropb]

I use it and everything works:
https://www.nuget.org/packages/RedPajama

Or:
https://www.nuget.org/packages/Llama.Grammar

More info:
https://thirty25.blog/blog/2025/04/generating-json-with-lms

[SerialKicked]

I use it and everything works: https://www.nuget.org/packages/RedPajama

Or: https://www.nuget.org/packages/Llama.Grammar

More info: https://thirty25.blog/blog/2025/04/generating-json-with-lms

If you have nothing useful to say, don't contribute.

---

Edit:

So, I've been at it for a while. Well, all night..

If the grammar is "obviously" incorrect (garbage grammar string), I get ExecutionEngineException, which is expected. Correct grammar as the one from above, the one from your Example folder, and even like extremely simple stuff like "root ::= ([0-9][0-9]?[0-9]?)" all cause the IndexOutOfRangeException to fire instead.

Am I missing some weird parameter to set in the ModelParams? I tried a few things, to no avail. There's only Encoding I'd see as relevant, but I don't touch that one. I'll try making a custom pipeline so I can debug trace into it.

Edit 2:

So I rolled my own copy of the pipeline so I can debug into it. The line that crashes (in the default pipeline Sample() function is:

var candidateTokenTopK = nativeTopK.Data[checked((int)nativeTopK.Selected)];

Because nativeTopK.Data has no elements. Seems like GrammarOptimizationMode.Extended is the default, I wasn't expecting that.

OKAY, I found the issue.

GrammarOptimizationMode.Extended causes the crash. It works just fine with GrammarOptimizationMode.None and Basic. lol I assumed the default was None, so I thought I checked against both None and Extended.

I looked into the code. And I might be wrong, because I can't trace into the inner calls without using a full copy of the library instead of the nuget, but I think the issue comes with TopK.

All backends I'm aware of assume "TopK = 0" means that the sampler is disabled (you seem to do as well). But here, for grammar optimization, it's just considered like as a normal value. And with TopK set to zero, your "Extended" code crashes. It seems to work okay when TopK is set to something like 60, though, which confirms my hypothesis.

I'd replace that:

                    if (GrammarOptimization == GrammarOptimizationMode.Extended)

By a

                    if (GrammarOptimization == GrammarOptimizationMode.Extended && TopK > 0)

If I were you :)

[martindevans]

Thanks for digging into this. Would you like to open a PR for that fix?

[SerialKicked]

You're welcome! And, sure, now that I found both the bug and the fix, I might as well write the PR too. 😁 I'll get that done Saturday, I have a busy schedule today.

btw, you might like to give my new lib a look, it's pretty much a full LLM C# stack on top of llamasharp (can also plug into kobold and OpenAI APIs). It might be relevant to your interests.

Cheers.