how to make the instructor validation more relaxed?

[esteininger]

i want it to not throw an error if just one field is missing. how do i do this?

[jxnl]

make the field optional

[esteininger]

i also tried optional literal per the docs

round_name: Optional[Literal["Series A", "Seed", "Series B", "OTHER"]]

[jxnl]

note the error:

Input should be 'LLC', 'C-Corp' or 'OTHER' [type=enum, input_value='CORPORATION',

it looks like the llm is trying to use CORPORATION which is not an option.
I would put it in the description or use a literal, but still prompt it more aggressively that only those options are allowed.

[jxnl]

same with

Input should be 'Software' or 'OTHER' [type=enum, input_value='Retail'

looks like the LLM is trying to really use other options.
you might want to add more detailed descriptions.

[esteininger]

@jxnl thanks for the quick response.

here's my base prompt:

        I will provide a document.
        Your job is to extract the relevant information from the document and respond in the provided JSON schema.
        Do not make up information. If you don't know the answer, do not make it up.
        Make sure to adhere to the JSON Schema.
        For enum fields, make sure to select from the available options.
        If it doesn't exist in the available enum options, select "OTHER".
        
        Here is the document:
        {corpus}

i also modified each field's description:

    class Company(BaseModel):
        company_name: Optional[str] = Field(
            None, description="The name of the company."
        )
        company_type: Optional[CompanyType] = Field(
            None, description="The type of the company. Only use the provided options."
        )
        company_logo: Optional[HttpUrl] = Field(
            None, description="The URL of the company's logo."
        )
        company_website: Optional[HttpUrl] = Field(
            None, description="The URL of the company's website."
        )
        industry: Optional[IndustryType] = Field(
            None,
            description="The industry the company operates in. Only use the provided options.",
        )
        round_name: Optional[RoundName] = Field(
            None,
            description="The name of the funding round. Only use the provided options.",
        )
        stakeholder_type: Optional[StakeholderType] = Field(
            None, description="The type of stakeholder. Only use the provided options."
        )
        investment_type: Optional[InvestmentType] = Field(
            None, description="The type of investment. Only use the provided options."
        )
        valuation_cap: Optional[Decimal] = Field(
            None, description="The valuation cap of the company."
        )
        capital_contribution: Optional[Decimal] = Field(
            None, description="The capital contribution of the company."
        )

any suggestions? instructor is a great library by the way.

[Mr-Ruben]

Based on my experience, if a regular prompt doesn't get the response you want, using a function call probably won't be better.

Why?
(Just my view)

• When you directly call the model, there's a lot to think about:

understanding the prompt + doing the task + giving a response.

• But with a function call, it's more complicated:

understanding the prompt + reading the function instructions (JSON schema) + doing the task + giving a response (using the schema).

The second way needs more work. If the work needed is more than what is going to be attended/done for each call, then it's better to break it down into simpler tasks or calls.

Follow the rule of divide and conquer.

[esteininger]

i just want some sort of conditional where if the value returned from the LLM != any of the enums, it returns "None".

this would be valuable for a bunch of different use cases.

[jxnl]

you could add a validator that says try to cast it, if it fails, return None.
but having sentiile values is not the point of having a type system,

if you try to int("abc") it should be an error, not None

Extraction GPT is able to correctly return data and the relevant information from the document
Do not make up information. If you don't know the answer, do not make it up. 
For enum fields, make sure to select from the available options if an option is not available, correctly return None, rather than selection an invalid option.

i might just suggest using literals for everything unless you need to use enums later in the pipeline

[esteininger]

when converting the data model to json using data model generator there is a diff causing instructor to not accept the literals.

here’s a gist: https://gist.github.com/esteininger/9b056989de9a0502c978d4bc7bbdfb54

[jxnl]

this seems like not an instructor issue, and i dont hve time to investigate the codegen, best to make an issue with the datamodel gen library.

cause instructor just takes a pydantic model that has a model_dump_schema()