How to create grammars?

[amusingimpala75]

I've looked at the basic BNF syntax and examples in the repo, but when trying to create my own it just crashes with LLAMA_ASSERT(false) on line 2203.

# command responses are in the following format:
# command [arg1] [arg2]
root ::= command argument*
argument ::= "[" ([a-zA-Z_/,.]* ws)* "]"
command ::= [a-zA-Z_] [a-zA-Z_0-9]*
ws ::= [ \t]*

It's got something to do with the 'argument' part, it isn't a problem when I comment out the declaration and usage.

[amusingimpala75]

Huh, well I finally got it work with

# command responses are in the following format (excluding the quotes):
# command [arg1] [arg2]
root       ::= command?
command    ::= identifier (ws argument)*
identifier ::= [a-zA-Z_] [a-zA-Z_0-9]*
argument   ::= "[" [a-zA-Z_ ,.?] "]"
ws         ::= [ \t]*

I'm not sure which part was the issue, but it would be helpful if it were more clear.

[amusingimpala75]

I would like to know, when provided with the grammar is it a strong recommendation for the LLM, or does it have to follow it? Just trying to figure out if my grammar is wrong or if I just need to prompt it better.
The current grammar:

root       ::= command
command    ::= identifier (ws argument)*
identifier ::= [a-zA-Z_] [a-zA-Z_0-9]+
argument   ::= "\"" [a-zA-Z_ .?/]* "\""
ws         ::= " "*

should evaluate to something like

command "arg1"

but it will sometimes output

"command arg1"

instead, which I think is wrong, at least with my current understanding of BNF. It's certainly not what I want.

[KerfuffleV2]

but it will sometimes output "command arg1"

What's before that? Is it something that could match identifier? "command arg1" can match argument.

[amusingimpala75]

No it will literally reply just that, nothing before it. It shouldn’t start with a quote, but it is.

[KerfuffleV2]

I'm not an expert on BNF but for what it's worth I don't see a reason for the behavior you're describing. It also looks like the sampler assigns -INF to the tokens it rejects: https://github.com/ggerganov/llama.cpp/blob/25d43e0eb578b6e73046d9d6644a3a14d460600d/llama.cpp#L2671-L2675

It also doesn't seem like it ever boosts the probability for tokens, only penalizes the ones it "rejects". Quoting "rejects" here because I don't know definitively what that means. One would presume everything that doesn't match a token the grammar can accept should be rejected but using a list for that seems a little weird to me. A lot of the time, it's going to be a list of nearly every token the model supports.