File tree Expand file tree Collapse file tree 1 file changed +35
-0
lines changed Expand file tree Collapse file tree 1 file changed +35
-0
lines changed Original file line number Diff line number Diff line change @@ -40,6 +40,41 @@ The package installs the command line entry point `llamacpp-cli` that points to
40
40
41
41
See ` llamacpp/cli.py ` for a detailed example. The simplest demo would be something like the following:
42
42
43
+ ``` python
44
+
45
+ params = llamacpp.gpt_params(
46
+ ' ./models/7B/ggml_model_q4_0.bin' , # model,
47
+ " A llama is a " , # prompt
48
+ " " , # reverse_prompt
49
+ 512 , # ctx_size
50
+ 100 , # n_predict
51
+ 40 , # top_k
52
+ 0.95 , # top_p
53
+ 0.85 , # temp
54
+ 1.30 , # repeat_penalty
55
+ - 1 , # seed
56
+ 8 , # threads
57
+ 64 , # repeat_last_n
58
+ 8 , # batch_size
59
+ False , # color
60
+ False , # interactive or args.interactive_start
61
+ False , # interactive_start
62
+ )
63
+ model = llamacpp.PyLLAMA(params)
64
+ model.add_bos() # Adds "beginning of string" token
65
+ model.update_input(params.prompt)
66
+ model.print_startup_stats()
67
+ model.prepare_context()
68
+
69
+ model.ingest_all_pending_input(True )
70
+ while not model.is_finished():
71
+ model.ingest_all_pending_input(not input_noecho)
72
+ text, is_finished = model.infer_text()
73
+ print (text, end = " " )
74
+ if is_finished:
75
+ break
76
+ ```
77
+
43
78
## ToDo
44
79
45
80
- [x] Use poetry to build package
You can’t perform that action at this time.
0 commit comments