Skip to content

Responses max_logprobs #3552

@iamemilio

Description

@iamemilio

🚀 Describe the new functionality needed

Logprobs are an important tool to help users understand token prediction outcomes. They are currently missing from llama stack responses.

Requirements

  • The Responses Create requests has a top_logprobs field, which gets translated or dropped based on the the inference provider.
  • The response object contains top_lobprobs which is equal to the output value from the provider, or None.

💡 Why is this needed? What if we don't build it?

Users that have max_logprobs set will get bad request errors.

Other thoughts

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions