How does Maybe work with Retries?

[samching]

I'm curious what happens in the following example:

# Apply the patch to the OpenAI client
client = instructor.patch(OpenAI())

def validate_name(cls, v):
    if v.upper() != v:
        raise ValueError("Name must be in uppercase.")
    return v

class UserDetails(BaseModel):
    name: Annotated[str, AfterValidator(validate_name)]
    age: int

MayberUserDetails = Maybe(UserDetails)

try:
    model = client.chat.completions.create(
        model="gpt-3.5-turbo",
        response_model=MaybeUserDetails,
        max_retries=10,
        messages=[
            {"role": "user", "content": "Extract jason is 25 years old"},
        ],
    )
except ValidationError as e:
    print(e)

assert model.name == "JASON"

Let's say there's an ValidationError that occurs, does it bubble up to Retries or does it just return a Maybe object with the Error attribute populated?

[jxnl]

from openai import OpenAI
import instructor
from pydantic import BaseModel, Field, field_validator, AfterValidator
from typing import List, Optional
from typing_extensions import Annotated

# Apply the patch to the OpenAI client
client = instructor.patch(OpenAI())

def validate_name(v):
    if v.upper() != v:
        print("validation error was found")
        raise ValueError("Name must be in uppercase.")
    return v

class UserDetails(BaseModel):
    name: Annotated[str, AfterValidator(validate_name)]
    age: int

MaybeUserDetails = instructor.Maybe(UserDetails)

model = client.chat.completions.create(
    model="gpt-3.5-turbo",
    response_model=MaybeUserDetails,
    max_retries=10,
    messages=[
        {"role": "user", "content": "Extract jason is 25 years old"},
    ],
)

assert model.result.name == "JASON"

Try running this, you'll find that the validation error was found prints once and there are no issues.

1. model.name won't work since MaybeUserDetails does not have name only result

[samching]

Thanks! This is helpful.

And just to be completely clear, if the retries are exhausted and there is still an error, then model.result will be None and model.error will contain the error?

I.e. https://github.com/jxnl/instructor/blob/df8cdeadd1c0fa34f65242fc62d98c106cc3a183/instructor/patch.py#L149

If in the 10th loop the retry still fails, will the error that is raised here be sent to a model call to be formatted accordingly into MaybeUserDetails?

I don't have a strong grasp on how the Maybe class works -- maybe another way of asking this is whether non-LLM errors (i.e. non semantic errors such as ValidationError and JsonDecodeError ) ever get caught as Errors in the Maybe model?

[jxnl]

this is a great question.

if the retires are done, you'll get a validation error,

the maybe.error is just a semantic attribute that the llm sets, where as ValidationError and JsonDecodeError are error messages that the computer system sets.

maybe.error could have easily been maybe.is_missing