Gbnf checker + validator

[supersteves]

Hi there,

Currently I'm using a gbnf file I've generated myself and the only way I currently have to test it is to ask the LLM to mirror what I say and then see if it emits the same content. Fairly flaky at best.

I'd like to (a) validate my gbnf by running some command (rather than wait for and try to spot a stderr message on the console)
and (b) test whether my input test directly fits the grammar, without invoking the LLM.

I've been through https://github.com/ggerganov/llama.cpp/blob/master/grammars/README.md which isn't entirely clear on some aspects of character escaping (I found the answer in the parser code),
and I've seen https://github.com/ggerganov/llama.cpp/tree/master/examples/gbnf-validator but have no idea how to build/run this.

Any help or pointers would be much appreciated.

[edwin0cheng]

It is called llama-gbnf-validator, and you don't need to build it separately.

[supersteves]

Duh, my bad. Wasn't using my own build. make llama-gbnf-validator at the top level, and I'm good. Thanks.

[Jofresh77]

Did anyone else also have this issue ?

Administrator: PowerShell 7 (x64)

(base) PS C:\Users\...\AppData\Local\llama.cpp\build> cmake --build . --target llama-gbnf-validator
ninja: error: unknown target 'llama-gbnf-validator'

I followed:

• this guide
• Windows 11
• Used MSYS then PowerShell (following the guide)

I can use llama.cpp with no issues and use it actively but I don't know why I can't build the gbnf-validator, actually need it a lot. If somebody has any idea 🙏

[slaren]

It was renamed to test-gbnf-validator.

[Jofresh77]

Still get an unknown target error.. Thanks for the reply though

[Jofresh77]

Actually noticed in the CMakeLists for the tests, that Windows is not supported and will automatically not build the test-gbnf-validator.cpp

.\llama.cpp\tests\CMakeLists.txt

if (NOT WIN32)
# these tests are disabled on Windows because they use internal functions not exported with LLAMA_API
...
llama_build(test-gbnf-validator.cpp)

So there no way to use this tool on Windows ?

[slaren]

With #13980 you should be able to build it by configuring cmake with BUILD_SHARED_LIBS=OFF.

[Jofresh77]

Ok perfect it works, thanks a lot!

[ReinderVosDeWael]

I've been working on a VSCode LSP plugin (https://marketplace.visualstudio.com/items?itemName=ReinderVosdeWael.gbnf-engine) for gbnf. It's currently still early stages, but functional and it catches several errors on top of giving you syntax highlighting. My goal is that it should catch all errors in your editor, though at this stage I'm not willing to guarantee this to bethe case.

That being said, I'm not aware of any invalid GBNF that isn't caught by the LSP. Feedback would be helpful if you run into issues (https://github.com/ReinderVosDeWael/gbnf-lsp)!