[Feature] Add Support on Log Probability Value from Returned Response of Gemini Models.

[jacklanda]

Description of the feature request:

How about providing new support for retrieving the log probability of each predicted token by Google models like Gemini?

Something just like the same function illustrated in this OpenAI post: User Logprobs.

What problem are you trying to solve with this feature?

Getting the returned log prob of each generated token does help me (and other users maybe) to confirm the confidence of the model's prediction. Further, this feature can help users compute the perplexity of generated sentences to better understand the textual continuation quality.

Any other information you'd like to share?

No response

[jacklanda]

Any thoughts about it?

[Brian-Ckwu]

I also think it would be extremely helpful if the API could provide the top-k log probabilities of each predicted token.

[krenova]

Really need this as I found Gemini to be hallucinating quite abit on a RAG application.

Btw your title refers to claude and not gemini.
[Feature] Add Support on Log Probability Value from Returned Response of Claude Models

[jacklanda]

Really need this as I found Gemini to be hallucinating quite abit on a RAG application.

Btw your title refers to claude and not gemini. [Feature] Add Support on Log Probability Value from Returned Response of Claude Models

Thanks for reminding 🤗

[simpleusername96]

Many hallucination detection approaches rely on log probability as a key feature. It's one of the most essential elements when building a serious product with a LLM.

[anwang427]

This would be extremely helpful!

[Said-Apollo]

As all others previously mentioned, it would be great to somehow get access to the logsprobs in a similar fashion as OpenAI does so with the models. Based on that we could then e.g. calculate the perplexity score and various other evaluation metrics.

[haugmarkus]

Yep, this would make it possible to use gemini in production

[waveworks-ai]

+1 - useful also for classification tasks.

[lavanyanemani96]

+1