Skip to content

Multimodal inputs in the DraftValidation class #394

@howardbaik

Description

@howardbaik

Prework

Proposal

Continuing on the theme of multimodal inputs, I thought it'd be really cool if the DraftValidation class could support multimodal inputs (images and PDF), just as Validate.prompt() now does. A rough sketch of how that would work, drawing inspiration from the How DraftValidation Works steps:

  1. A statistical summary of your data is generated using the DataScan class.
  2. This summary is converted to JSON format and sent to your selected LLM provider.
  3. NEW: You provide the DraftValidation class with an image or PDF file describing what the data should look like. For example, the image could be a schema diagram, while the PDF could outline the project configuration elements that the data should conform to.
  4. The LLM uses the summary and multimodal input along with knowledge about Pointblank’s validation capabilities to generate a validation plan.
  5. The result is returned as executable Python code that you can use directly or modify as needed

Happy to get feedback from you @rich-iannone, and iterate. I'd love to implement this.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions