Skip to content

Commit 62f9de4

Browse files
authored
Update README to show to use logprobs parameter (#293)
1 parent c3d2a35 commit 62f9de4

File tree

1 file changed

+24
-0
lines changed

1 file changed

+24
-0
lines changed

README.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -186,6 +186,30 @@ async def async_chat_completion(messages):
186186
asyncio.run(async_chat_completion(messages))
187187
```
188188

189+
#### Fetching logprobs
190+
191+
Logprobs are logarithms of token-level generation probabilities that indicate the likelihood of the generated token based on the previous tokens in the context. Logprobs allow us to estimate the model's confidence in its outputs, which can be used to decide how to optimally consume the model's output (e.g. rejecting low confidence outputs, retrying or ensembling model outputs etc).
192+
193+
```python
194+
from together import Together
195+
196+
client = Together()
197+
198+
response = client.chat.completions.create(
199+
model="mistralai/Mixtral-8x7B-Instruct-v0.1",
200+
messages=[{"role": "user", "content": "tell me about new york"}],
201+
logprobs=1
202+
)
203+
204+
response_lobprobs = response.choices[0].logprobs
205+
206+
print(dict(zip(response_lobprobs.tokens, response_lobprobs.token_logprobs)))
207+
# {'New': -2.384e-07, ' York': 0.0, ',': 0.0, ' also': -0.20703125, ' known': -0.20214844, ' as': -8.34465e-07, ... }
208+
```
209+
210+
More details about using logprobs in Together's API can be found [here](https://docs.together.ai/docs/logprobs).
211+
212+
189213
### Completions
190214

191215
Completions are for code and language models shown [here](https://docs.together.ai/docs/inference-models). Below, a code model example is shown.

0 commit comments

Comments
 (0)