Skip to content

Commit f38222a

Browse files
committed
Add demo script to README
1 parent b03974b commit f38222a

File tree

1 file changed

+35
-0
lines changed

1 file changed

+35
-0
lines changed

README.md

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,41 @@ The package installs the command line entry point `llamacpp-cli` that points to
4040

4141
See `llamacpp/cli.py` for a detailed example. The simplest demo would be something like the following:
4242

43+
```python
44+
45+
params = llamacpp.gpt_params(
46+
'./models/7B/ggml_model_q4_0.bin', # model,
47+
"A llama is a ", # prompt
48+
"", # reverse_prompt
49+
512, # ctx_size
50+
100, # n_predict
51+
40, # top_k
52+
0.95, # top_p
53+
0.85, # temp
54+
1.30, # repeat_penalty
55+
-1, # seed
56+
8, # threads
57+
64, # repeat_last_n
58+
8, # batch_size
59+
False, # color
60+
False, # interactive or args.interactive_start
61+
False, # interactive_start
62+
)
63+
model = llamacpp.PyLLAMA(params)
64+
model.add_bos() # Adds "beginning of string" token
65+
model.update_input(params.prompt)
66+
model.print_startup_stats()
67+
model.prepare_context()
68+
69+
model.ingest_all_pending_input(True)
70+
while not model.is_finished():
71+
model.ingest_all_pending_input(not input_noecho)
72+
text, is_finished = model.infer_text()
73+
print(text, end="")
74+
if is_finished:
75+
break
76+
```
77+
4378
## ToDo
4479

4580
- [x] Use poetry to build package

0 commit comments

Comments
 (0)