Skip to content

[INFERENCE API] Validate the max chunk size against the inference service and model. #133724

@davidkyle

Description

@davidkyle

Description

The max chunk size is limited by the model's context window and therefore model dependant. Rather than having a single max value for all it should be specific to the model or the service provider similar to the logic in #132169

Related to #133718 which removes the limitation

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions