Skip to content

Commit e12c995

Browse files
Ki-SekiRobinPicard
authored andcommitted
fix: Add SamplingParams to model response examples
Updated usage examples to include SamplingParams for response generation and streaming.
1 parent 9f909bd commit e12c995

File tree

1 file changed

+4
-4
lines changed

1 file changed

+4
-4
lines changed

docs/features/models/vllm_offline.md

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -64,7 +64,7 @@ For instance:
6464

6565
```python
6666
import outlines
67-
from vllm import LLM
67+
from vllm import LLM, SamplingParams
6868
from outlines.inputs import Chat
6969

7070
# Create the model
@@ -79,7 +79,7 @@ prompt = Chat([
7979
])
8080

8181
# Call the model to generate a response
82-
response = model(prompt, max_tokens=50)
82+
response = model(prompt, sampling_params=SamplingParams(max_tokens=50))
8383
print(response) # 'Riga'
8484
```
8585

@@ -91,15 +91,15 @@ For instance:
9191

9292
```python
9393
import outlines
94-
from vllm import LLM
94+
from vllm import LLM, SamplingParams
9595

9696
# Create the model
9797
model = outlines.from_vllm_offline(
9898
LLM("microsoft/Phi-3-mini-4k-instruct")
9999
)
100100

101101
# Stream the response
102-
for chunk in model.stream("Tell me a short story about a cat.", max_tokens=50):
102+
for chunk in model.stream("Tell me a short story about a cat.", sampling_params=SamplingParams(max_tokens=50)):
103103
print(chunk) # 'Once...'
104104
```
105105

0 commit comments

Comments
 (0)