How to approach refined grammar rules

[alexanderchen750]

I'm trying to define a cfg and add some refinement rule on top of that. For example using the cfg expression defined in the github readme:

import guidance
from guidance import one_or_more, select, zero_or_more
# stateless=True indicates this function does not depend on LLM generations
@guidance(stateless=True)
def number(lm):
    n = one_or_more(select(['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']))
    # Allow for negative or positive numbers
    return lm + select(['-' + n, n])

@guidance(stateless=True)
def operator(lm):
    return lm + select(['+' , '*', '**', '/', '-'], name="op")

@guidance(stateless=True)
def expression(lm):
    # Either
    # 1. A number (terminal)
    # 2. two expressions with an operator and optional whitespace
    # 3. An expression with parentheses around it
    return lm + select([
        number(),
        expression() + zero_or_more(' ') +  operator() + zero_or_more(' ') +  expression(),
        '(' + expression() + ')'
    ])

I'd like to add a rule to stop a zero from being chosen if the previous operator was a '/'. I though about accessing the state of the lm like lm["op"], but stateless functions can't access state. If I set it to stateful, I run into error that I can't select between stateful functions. I thought about using a table to store state, like table["op"] = select(['+' , '*', '**', '/', '-'], name="op"), but it stores a intermediary representation like {{G|137482486511456|G}}.

I'm still new to guidance, so I would appreciate some pointers in the right direction.

[demirciberk]

Short version: what you're hitting isn't a bug, it's a property of how guidance builds grammars. A stateless=True function is called once, at grammar-construction time, to build a static context-free grammar (CFG) tree. There is no "previous token" to inspect at that point — nothing has been generated yet, so lm["op"] can't exist, and mixing in a stateful function breaks the "build once, reuse everywhere" assumption that makes select() work (this is basically the same root cause as the error in #814, "Confusing error message when using capture on stateful function").

So "the previous operator was /" cannot be answered by reading runtime state inside the grammar — it has to be encoded structurally in the CFG itself. That's actually fine here because your rule only depends on the immediate parent production (the operator that dominates the right-hand operand), not on unbounded history, so it's still expressible as a context-free grammar — you just need two flavors of "expression"/"number": a normal one and a "no-leading-zero" one, and only use the restricted one on the right side of /.

Example:

python
import guidance
from guidance import one_or_more, select, zero_or_more

@guidance(stateless=True)
def digit(lm):
return lm + select(['0','1','2','3','4','5','6','7','8','9'])

@guidance(stateless=True)
def number(lm):
n = one_or_more(digit())
return lm + select(['-' + n, n])

@guidance(stateless=True)
def nonzero_number(lm):
# first digit can't be 0, rest can be anything
nz = select(['1','2','3','4','5','6','7','8','9'])
rest = zero_or_more(digit())
n = nz + rest
return lm + select(['-' + n, n])

@guidance(stateless=True)
def operator_no_div(lm):
return lm + select(['+', '', '*', '-'], name="op")

@guidance(stateless=True)
def expression(lm, allow_zero=True):
num = number() if allow_zero else nonzero_number()
return lm + select([
num,
expression() + zero_or_more(' ') + operator_no_div() + zero_or_more(' ') + expression(),
# division: right-hand side restricted to a nonzero expression
expression() + zero_or_more(' ') + '/' + zero_or_more(' ') + expression(allow_zero=False),
'(' + expression(allow_zero) + ')'
])

Notes on this approach:

• allow_zero=False only forces the leading number of the right operand to be nonzero — parenthesized sub-expressions still recurse with allow_zero=False too, so (0+1) after / is also rejected, matching "no zero chosen right after /" more strictly. Loosen/tighten depending on what you actually want ("no literal 0" vs "no expression that could evaluate to 0" — the latter is not just context-free, it's undecidable in general for arbitrary expressions, so scope your rule to what the grammar can actually check syntactically).
• This is the general pattern for "refined" rules in guidance: instead of trying to read state mid-grammar, split the production so the constraint is a different non-terminal chosen by the caller. Anything that depends only on local grammar context (parent node, sibling choice) can be done this way; anything that depends on unbounded prior generation history (truly context-sensitive) is out of scope for guidance's CFG engine (see the CFG description in the guidance README: https://github.com/guidance-ai/guidance#readme, and the 0.2.0 notes on the Rust llguidance grammar engine: 0.2.0 Release! #1093).
• The {{G||G}} string you saw when storing a grammar node in a plain dict is just the internal placeholder for an unresolved grammar node (its repr before it's actually spliced into an lm += and generated) — it confirms you were storing/inspecting a grammar object, not a generated value, which is the same "not generated yet" issue described above.

If your real constraint is more global (e.g. "no division by an expression that evaluates to zero" rather than "no literal 0 right after /"), that's a semantic/runtime check, not a grammar one — you'd need to generate candidate output and validate/retry afterward, since guidance's constrained decoding operates on syntax (CFG), not on evaluated semantics.

References:

• guidance README, "Create your own Guidance functions" / stateless composition: https://github.com/guidance-ai/guidance#create-your-own-guidance-functions
• Confusing error message when using capture on stateful function #814 (stateful function inside select/capture — same class of error): Confusing error message when using capture on stateful function #814
• 0.2.0 Release! #1093 (0.2.0 release notes, Rust llguidance grammar engine details): 0.2.0 Release! #1093

I haven't executed this example against a live model — I only verified it against the documented API (select, one_or_more, zero_or_more, @guidance(stateless=True)) shown in the README. Please test it locally before relying on it, and if you need the "no zero-valued division" (semantic) version rather than "no literal-zero right after /" (syntactic), say so — that needs a different (retry-based) approach.