Support custom output validation failure handler as alternative to sending error to model

[davidbernat]

Question

The use case I will present is a smallest-possible specific example of a broad category of general use cases.

As such, my broader question asks which PydanticAI design features of the Agent and BaseModel classes were written with its solution in mind, so as to most conform to the design patterns of the PydanticAI developers; i.e., "the most PydanticAI way to solve this".

Here is a small specific example: Suppose I want to create a simple agent that responds with a list of strings. For the sake of this discussion the list of strings are generated from the following prompt: "Create several short greetings which I can use when answering a phone from an unknown number. Format your response as a JSON of List of Strings. Provide no other information." Now, for the purpose of this question please do not bog down in any optimizing of prompt engineer, etc. We are using this to discuss a broad category of LLM annoying failure modes in large scale streamlines of enterprise application systems.

To begin, we would presumably use BaseModel to construct an output_type to pass as a parameter to Agent:

class ListOfGreetings(BaseModel):
    greetings: list[str]

agent = Agent(llm, output_type=ListOfComments)

To first order, a very common failure pattern is that the LLM does not response with a strictly formatted JSON, or cannot be parsed, and the Validator of ListOfGreetings will fail. A typical example of a response which answers the semantic request but fails the ListOfGreetings validation would be a text string:

Okay, here are a few options for a short, warm, sophisticated, and subtly engaging greeting: 
* Pleasure to answer your call. To whom am I speaking?
* Is there a way we can answer this call within under five minutes? 
* Hello. This is [insert name]. To whom do I have the pleasure of speaking?
These greetings could be improved were you to provide me more information about your personality.

This gets to the core of my question. IF the ListOfGreetings validation fails, we COULD pass through an additional LLM layer to parse the string, an additional custom AI-enabled Validator/Extractor; for instance, the next prompt returns with very high probability the same LLM response data reformatted as JSON and passes ListOfGreetings validation, by appending the verbatim string response from above, and using an LLM model which may or may not be the same Agent.

The paragraph of text below contains a bullet point list of sentences. Extract the sentences as a JSON list. Provide no other response.\n

In essence, this is simply a handler which is prompted IF ListOfGreetings validation fails, runs and modifies the Agent data, and runs the ListOfGreetings validation again. Strictly speaking this is not a Before/After Validator.

It is, I presume, a very simple and straightforward solution; but one PydanticAI has a specific design pattern to do.

What is that?
And, if applicable, what open source projects are leading the way on this endeavor? I would like to participate.

Additional Context

No response

[DouweM]

@davidbernat

To first order, a very common failure pattern is that the LLM does not response with a strictly formatted JSON, or cannot be parsed, and the Validator of ListOfGreetings will fail.

I don't think this is actually that common; the big model providers all support strict JSON output, where the model is forced to output JSON matching a particular schema. That's the case for 2 out of 3 output modes Pydantic AI supports: https://ai.pydantic.dev/output/#output-modes

Of course your point stands when we're talking about weaker, open source, and/or local models.

IF the ListOfGreetings validation fails, we COULD pass through an additional LLM layer to parse the string, an additional custom AI-enabled Validator/Extractor; for instance, the next prompt returns with very high probability the same LLM response data reformatted as JSON and passes ListOfGreetings validation, by appending the verbatim string response from above, and using an LLM model which may or may not be the same Agent.

If the model's response does not validate, Pydantic AI will already return the validation error to the model and tell it to try again: https://ai.pydantic.dev/tools-advanced/#tool-retries (This is written for tools but applies to output as well).

In essence, this is simply a handler which is prompted IF ListOfGreetings validation fails, runs and modifies the Agent data, and runs the ListOfGreetings validation again. Strictly speaking this is not a Before/After Validator.

If you want to handle asking a (different) model to fix the output yourself, you could use an output function (or text output function) that checks the model's output (function input) and itself calls another agent to produce the right result.

But I think it'd be pretty rare to need that or for it to perform better than the automatic retry we already do.

[davidbernat]

@DouweM

@davidbernat

To first order, a very common failure pattern is that the LLM does not response with a strictly formatted JSON, or cannot be parsed, and the Validator of ListOfGreetings will fail.

I don't think this is actually that common; the big model providers all support strict JSON output, where the model is forced to output JSON matching a particular schema. That's the case for 2 out of 3 output modes Pydantic AI supports: https://ai.pydantic.dev/output/#output-modes

Of course your point stands when we're talking about weaker, open source, and/or local models.

Luckily for us I work in open source models. And, and if always the case, good examples stand out in their applicability in far further design patterns, which I am sure you know. Perhaps you could clarify the setting of those specified output modes.

The example is designed to create an AI-enabled post-processing need from an LLM text output. Could you clarify for us your statement that 2 out of 3 output modes restrict JSON output? Here are the three output restrictions, each using Fruit as classes inheriting from BaseModel, which is a simplified example in the URL you provided. The original example appears to use a fourth structure; which, I presume is equivalent to one of the marker class methods, but I am not seeing that specified in the documents.

ToolOutput requires the LLM to use a Tool.
NativeOutput allows the LLM to reply with its text engines, and sends the expected JSON schema, and uses the configuration of the LLM to offload the 'forcing' requirement of the output to match the JSON schema to the LLM itself, not on the PydanticAI side. This structured output is then further validated by PydanticAI.
PromptedOutput asks the LLM to reply with its text engine, politely requesting the text output match the JSON schema.

When no marker class is used the Agent is presumably using PromptedOutput?

# Tool Output mode
agent = Agent(
    model_name,
    output_type=[ 
        ToolOutput(Fruit, name='return_fruit'),
    ],
)

# Native Output mode
agent = Agent(
    model_name, 
    output_type=NativeOutput(
        [Fruit], 
        name='Fruit',
        description='Return a fruit.'
    ),
)

# Prompted Output mode
agent = Agent(
    model_name,
    output_type=PromptedOutput(
        [Fruit], 
        name='Fruit',
        description='Return a fruit.'
    ),
)

# equivalent provided in the original example
# note: the original example is valid and does not use
# one of the coercion methods
agent = Agent(
    model_name,
    output_type=Vehicle
)

IF the ListOfGreetings validation fails, we COULD pass through an additional LLM layer to parse the string, an additional custom AI-enabled Validator/Extractor; for instance, the next prompt returns with very high probability the same LLM response data reformatted as JSON and passes ListOfGreetings validation, by appending the verbatim string response from above, and using an LLM model which may or may not be the same Agent.

If the model's response does not validate, Pydantic AI will already return the validation error to the model and tell it to try again: https://ai.pydantic.dev/tools-advanced/#tool-retries (This is written for tools but applies to output as well).

Yes, but my understanding is that there is a crucial different between my example and your discussion of tools, as you know.

The baseline PydanticAI behavior is to send a new (fresh) request to the LLM if the result return does not conform to the JSON schema and its requirements (i.e., validation fails). But in this text spool configuration of the example, your mention of the internal retries mechanism would simply send the same request back to the LLM to (presumably) fail again in more or less the same way.

It is for this reason the notion of the "post-processing" step is being asked. Let us move below by contrasting with OutputFunction.

In essence, this is simply a handler which is prompted IF ListOfGreetings validation fails, runs and modifies the Agent data, and runs the ListOfGreetings validation again. Strictly speaking this is not a Before/After Validator.

If you want to handle asking a (different) model to fix the output yourself, you could use an output function (or text output function) that checks the model's output (function input) and itself calls another agent to produce the right result.

But I think it'd be pretty rare to need that or for it to perform better than the automatic retry we already do.

I think we are speaking cross purposes. First, inside PydanticAI is "magic" that coerces one failed LLM output into a retry prompt. In essence, what I am describing is in some ways modifying that "magic" with an additional handle.

If the text output conforms to the BaseModel validator, there is no reason to process its text with an OutputFunction.
This is the encapsulation problem: if an OutputFunction always runs, the manual implementation of the BaseModel validator has to be written, run and checked on the data, and the those above "massage" tools run, then passed as output of the system, which is finally run through the PydanticAI validator. In essence, there should be an OutputFunction setting that only runs the function is the Agent Validator has already failed. Otherwise the design patterns inherent in the other parts of the system fails.

I hope this makes my example more clear. Shrug.
Seems pretty basic to me. Our fundamental disconnect is that you presume models can be configured in NativeOutput -- gosh my phone is getting a spam hold on, these things are so annoying -- to mandate the LLM provide a schema matching result. My advantage is that no such requirement exists for other types of cognitive output structures, and lower quality or inferior models.

1. NativeOutput, if available, i.e,. in open source Ollama, makes the concern somewhat redundant in explicit terms.
2. OutputFunction is, in some ways, the correct solution; but raises the design pattern issue which originated this whole request.

Thanks. Going to go back to watching Deep Impact now. Fish just commanded the installation of the nuclear into the asteroid, but the drilling mechanism got jammed at 75 feet. I think they are about to be blown out into cold deep space. No spoilers.

edit: llama3.2 etc., does not support NativeOutput

[DouweM]

When no marker class is used the Agent is presumably using PromptedOutput?

The default is "tool output", as mentioned in the second paragraph at https://ai.pydantic.dev/output/#structured-output and first at https://ai.pydantic.dev/output/#tool-output :)

The baseline PydanticAI behavior is to send a new (fresh) request to the LLM if the result return does not conform to the JSON schema and its requirements (i.e., validation fails). But in this text spool configuration of the example, your mention of the internal retries mechanism would simply send the same request back to the LLM to (presumably) fail again in more or less the same way.

The validation error will be sent back to the model, so it wouldn't be the same request or necessarily produce the same behavior.

But I understand that a new request to a (potentially different) model with nothing but "turn this text into a JSON object" could perform better with less context to be distracted by.

I think we are speaking cross purposes. First, inside PydanticAI is "magic" that coerces one failed LLM output into a retry prompt. In essence, what I am describing is in some ways modifying that "magic" with an additional handle.

Fair enough, I didn't read the original message as essentially requesting a new "output validation fallback" hook, but that's clear now. That's similar to the request at #3008 for a "tool arg validation fallback"

In essence, there should be an OutputFunction setting that only runs the function is the Agent Validator has already failed.

Output functions, or output validators: https://ai.pydantic.dev/output/#output-validator-functions.

We could extend the @agent.output_validator decorator to take a str -> T callable instead of only T -> T, with a flag to indicate this will run on the raw input string.

Would that work for you?

[davidbernat]

When no marker class is used the Agent is presumably using PromptedOutput?

The default is "tool output", as mentioned in the second paragraph at https://ai.pydantic.dev/output/#structured-output and first at https://ai.pydantic.dev/output/#tool-output :)

You are incorrect and should know better as a language scientist :) You are correct that the description of Tool Output mode begins with "In the default Tool Output mode" but that sentence is both ambiguous and potentially invalid in its interpretation in which you describe its implicature. In the example I provide I am not using Tools, and therefore one would be able to presume ToolOutput is not able to be activated; and ToolOutput can be configured upon itself implying that "default Tool Output" refers to the default settings of ToolOutput, but not that ToolOutput is the default setting for a output_type not specified with a marker class. In reading the class code itself this was not apparent as the configuration of the LLM into its tool mechanics happens at a deeper level. Now that we have that excruciatingly important discussion out of the way I do agree that the language here implies that PydanticAI constructs a tool passed into the LLM which is explicitly expected to be activated by the LLM to create the output: "By default, Pydantic AI leverages the model's tool calling capability to make it return structured data."

This brings us to a perfect Segue into the broader acrobatics: what the heck is Pydantic AI doing under the hood?

1. Where do I print out those modified LLM requests?
2. Even something as simple as an agent.instruction decorator does not seem to contain its previous instructions?
3. There are entire long well-documented pages about streaming events; but those, pardon my French, do not seem user friendly. There are "Parts" that are returned; but even a unified prompt is chunked, and I do not see expository lists of what tools are sent.

For instance, how do I print out every prompt + toolset which is sent to the LLM on each multiple rounds, including as PydanticAI modifies the prompt and toolset to exercise its core utilities of type casting? I have yet to see any discussion at all.

And the quality of PydanticAI effectiveness to be as stochastic as traditional prompt engineering, so the debugging is presumably a prevalent part of the developer workflow. For instance, the CityLocation example works very well, every time. But seemingly small changes, such as asking a robotic assistant to choose a direction from an Enum leads to complete failure until maximum retries are reached. It is no doubt that PydanticAI is a tremendous accomplishment and I have recommended its use for the stack. But mother of pearl what is it doing? What is the standard print-statement level of debugging to see everything.

Please :)
Show Me The Prompt(s), ToolSets, Modifications, and Requests Structures.

The baseline PydanticAI behavior is to send a new (fresh) request to the LLM if the result return does not conform to the JSON schema and its requirements (i.e., validation fails). But in this text spool configuration of the example, your mention of the internal retries mechanism would simply send the same request back to the LLM to (presumably) fail again in more or less the same way.

The validation error will be sent back to the model, so it wouldn't be the same request or necessarily produce the same behavior.

Exactly. We are in total agreement. The quality of PydanticAI as a recommended package depends directly, and some would say only, on the "next level prompt engineering" you are describing implicitly under the hood. So clearly I am behind on that.

But I understand that a new request to a (potentially different) model with nothing but "turn this text into a JSON object" could perform better with less context to be distracted by.

I think we are speaking cross purposes. First, inside PydanticAI is "magic" that coerces one failed LLM output into a retry prompt. In essence, what I am describing is in some ways modifying that "magic" with an additional handle.

Fair enough, I didn't read the original message as essentially requesting a new "output validation fallback" hook, but that's clear now. That's similar to the request at #3008 for a "tool arg validation fallback"

In essence, there should be an OutputFunction setting that only runs the function is the Agent Validator has already failed.

Output functions, or output validators: https://ai.pydantic.dev/output/#output-validator-functions.

We could extend the @agent.output_validator decorator to take a str -> T callable instead of only T -> T, with a flag to indicate this will run on the raw input string.

Would that work for you?

No, I do not think so. I would not feel comfortable prescribing bounds on the issue ticket for any developer yet. We do not have a clear introspection into where the chain of prompt sequences is failing, so we do not have a scope on the "AI-enabled follow-up" for post-Validation failure passthrough. It is important that I specify the output only needs this layer of post-processing if the boilerplate Validator fails, otherwise the Agent ensemble is making additional calls to additional LLMs unnecessarily. Unless you are stating that the extended @agent.output_validator decorator would only take action on str output from the LLM and be bypassed because of its call signature. In that case I am not sure that is the correct solution: if the response is valid JSON with incorrect fields we would have an object, potentially even of type T, which would still be modifiable with the second pass. I would not want to specify the scope on the developer ticket yet.

If you point me to the appropriate training wheels for print-statement management of the LLM-Pydantic back and forth, I will go ahead and implement a sufficient enough example to describe the limitations of what could be extended and why. It is a very valuable bedrock system Pydantic could be building here agnostic to company providers and I am very scared that the only reasonable dumb-engineer naive-level documentation requires my introspection to exist on a proprietary cloud servers set. Besides, there are core user interfaces in open building of AI that would rapidly alter your presumptions about model quality.

Thanks for taking you time :)
PS definitely did get blown out into space.