Provide more visibility into lower-level schema representations from python

[HansBrende]

There are two things that I would love to be able to do from the python bindings:

1. Given a jsonschema, normalize it (e.g., remove the allOf etc.) and give me back the normalized Schema (even a simple debug-stringified version of the final internal Schema instance would be better than nothing, but preferably, allow me to access the actual Schema object itself so I can recurse over it in any way I want.)

Why: Given that OpenAI is also using this library for structured outputs, this would allow me to quickly sanity-check all my relevant schemas before I send them off to OpenAI to make sure they are how I want them to look before they get turned into a grammar.

2. Given a jsonschema (or Schema, if (1) allows me to access it), give me back the final low-level grammar representation of that schema (again, even a simple debug output would be totally great)

Why: this would allow me even more visibility into the schemas I write to make sure the grammar makes sense, and if not, to tweak the schemas as necessary.

I spent some time digging into the existing python bindings and I can't figure out any way to do either of these things (I thought for sure LLMatcher.grammar_from_json_schema() would work but that just gives me back the same schema I put in, except wrapped in a "grammars" array). But if there is a way that I've missed, please let me know.

[hudson-ai]

Hi @HansBrende , I think you're right that exposing some form of this through the python bindings would be useful, in particular the schema normalization that we do. It would definitely be a good debugging tool.

We'd welcome a PR for this, but otherwise, I'll add it to the backlog :)

[HansBrende]

@hudson-ai that is great to hear, especially for the normalized schema aspect; I see 3 big advantages there: (1) allows for dynamic pre-normalization of schemas, so any schema errors (e.g. unsatisfiability, non-termination, unsupported features) can be caught at project build-time rather than waiting for an OpenAI error in many cases, (2) allows schema "touch-up/sanity-checking" logic to become a lot simpler due to the simplicity of the underlying Schema class, and (3) allows much more expressive power in initially defining the schemas (since OpenAI will flatly reject any schema attributes that aren't at 100% support by llguidance). (Of course, dynamically normalized Schemas would ultimately have to be converted back to valid jsonschema, with descriptions/titles re-inserted, but I can very easily implement those minor details in user-land.)

The low-level grammar is more from a curiosity standpoint, so any kind of debug-stringified representation (or simply the corresponding lark grammar) would be awesome to see. One use-case in that area that I've thought of might be to ditch OpenAI's structured outputs altogether in favor of a CFG constraint that is originally based on the low-level grammar of the initial json schema, but which is more powerful due to, e.g., the insertion of context free constraints on certain strings instead of being limited to regular expressions.

I would love to attempt a PR for this and exercise my time-weakened Rust muscles, however my time commitment to work matters is very high for the foreseeable future so it's doubtful I will be able to squeeze it in any time soon 😞