Update llm_inference.md

alabulei1 · juntao · commit 4903dc372a7b · 2023-11-06T20:44:05.000Z
Signed-off-by: alabulei1 &lt;vivian.xiage@gmail.com&gt;
Signed-off-by: Michael Yuan &lt;michael@secondstate.io&gt;
diff --git a/docs/develop/rust/wasinn/llm_inference.md b/docs/develop/rust/wasinn/llm_inference.md
@@ -29,19 +29,33 @@ git clone curl -LO https://huggingface.co/wasmedge/llama2/blob/main/llama-2-7b-c
 Run the inference application in WasmEdge.
 
 ```bash
-wasmedge --dir .:. \
-  --nn-preload default:GGML:CPU:llama-2-7b.Q5_K_M.gguf llama-chat.wasm default \
-  --prompt 'Robert Oppenheimer most important achievement is ' \
-  --ctx-size 4096
+wasmedge --dir .:. --nn-preload default:GGML:AUTO:llama-2-7b-chat-q5_k_m.gguf \
+  llama-chat.wasm --prompt-template llama-2-chat
 ```
 
-After executing the command, you may need to wait a moment for the input prompt to appear. Once the execution is complete, the following output will be generated.
+After executing the command, you may need to wait a moment for the input prompt to appear. You can enter your question once you see the `[USER]:` prompt:
 
 ```bash
-Robert Oppenheimer most important achievement is
-1945 Manhattan Project.
-Robert Oppenheimer was born in New York City on April 22, 1904. He was the son of Julius Oppenheimer, a wealthy German-Jewish textile merchant, and Ella Friedman Oppenheimer.
-Robert Oppenheimer was a brilliant student. He attended the Ethical Culture School in New York City and graduated from the Ethical Culture Fieldston School in 1921. He then attended Harvard University, where he received his bachelor's degree.
+[USER]:
+I have two apples, each costing 5 dollars. What is the total cost of these apple
+*** [prompt begin] ***
+<s>[INST] <<SYS>>
+You are a helpful, respectful and honest assistant. Always answer as short as possible, while being safe. <</SYS>>
+
+I have two apples, each costing 5 dollars. What is the total cost of these apple [/INST]
+*** [prompt end] ***
+[ASSISTANT]:
+The total cost of the two apples is 10 dollars.
+[USER]:
+How about four apples?
+*** [prompt begin] ***
+<s>[INST] <<SYS>>
+You are a helpful, respectful and honest assistant. Always answer as short as possible, while being safe. <</SYS>>
+
+I have two apples, each costing 5 dollars. What is the total cost of these apple [/INST] The total cost of the two apples is 10 dollars. </s><s>[INST] How about four apples? [/INST]
+*** [prompt end] ***
+[ASSISTANT]:
+The total cost of four apples is 20 dollars.
 ```
 
 ## Build and run
@@ -70,57 +84,78 @@ curl -LO https://huggingface.co/wasmedge/llama2/blob/main/llama-2-13b-q5_k_m.ggu
 Next, use WasmEdge to load the llama-2-13b model and then ask the model to questions.
 
 ```bash
-wasmedge --dir .:. \
-  --nn-preload default:GGML:CPU:llama-2-13b.Q5_K_M.gguf llama-chat.wasm default \
-  --prompt 'Robert Oppenheimer most important achievement is ' \
-  --ctx-size 4096
+wasmedge --dir .:. --nn-preload default:GGML:AUTO:llama-2-13b-chat-q5_k_m.gguf \
+  llama-chat.wasm --prompt-template llama-2-chat
 ```
 
 After executing the command, you may need to wait a moment for the input prompt to appear. You can enter your question once you see the `[USER]:` prompt:
 
 ```bash
-Robert Oppenheimer most important achievement is
-1945 Manhattan Project.
-Robert Oppenheimer was born in New York City on April 22, 1904. He was the son of Julius Oppenheimer, a wealthy German-Jewish textile merchant, and Ella Friedman Oppenheimer.
-Robert Oppenheimer was a brilliant student. He attended the Ethical Culture School in New York City and graduated from the Ethical Culture Fieldston School in 1921. He then attended Harvard University, where he received his bachelor's degree
+[USER]:
+Who is Robert Oppenheimer?
+*** [prompt begin] ***
+<s>[INST] <<SYS>>
+You are a helpful, respectful and honest assistant. Always answer as short as possible, while being safe. <</SYS>>
+
+Who is Robert Oppenheimer? [/INST]
+*** [prompt end] ***
+[ASSISTANT]:
+Robert Oppenheimer was an American theoretical physicist and director of the Manhattan Project, which developed the atomic bomb during World War II. He is widely regarded as one of the most important physicists of the 20th century and is known for his contributions to the development of quantum mechanics and the theory of the atomic nucleus. Oppenheimer was also a prominent figure in the post-war nuclear weapons debate and was a strong advocate for international cooperation on nuclear weapons control.
 ```
 
-## Optional: Configure the model
-
-You can use environment variables to configure the model execution.
+## Optional: run the model with different CLI
 
-| Option |Default |Function |
-| -------|-----------|----- |
-| LLAMA_LOG |	0	|The backend will print diagnostic information when this value is set to 1|
-|LLAMA_N_CTX |512|	The context length is the max number of tokens in the entire conversation|
-|LLAMA_N_PREDICT |512|The number of tokens to generate in each response from the model|
-
-For example, the following command specifies a context length of 4k tokens, which is standard for llama2, and the max number of tokens in each response to be 1k. It also tells WasmEdge to print out logs and statistics of the model at runtime.
+We also have CLI options for more information.
 
+```bash
+  -m, --model-alias <ALIAS>
+          Model alias [default: default]
+  -c, --ctx-size <CTX_SIZE>
+          Size of the prompt context [default: 4096]
+  -n, --n-predict <N_PRDICT>
+          Number of tokens to predict [default: 1024]
+  -g, --n-gpu-layers <N_GPU_LAYERS>
+          Number of layers to run on the GPU [default: 100]
+  -b, --batch-size <BATCH_SIZE>
+          Batch size for prompt processing [default: 4096]
+  -r, --reverse-prompt <REVERSE_PROMPT>
+          Halt generation at PROMPT, return control.
+  -s, --system-prompt <SYSTEM_PROMPT>
+          System prompt message string [default: "[Default system message for the prompt template]"]
+  -p, --prompt-template <TEMPLATE>
+          Prompt template. [default: llama-2-chat] [possible values: llama-2-chat, codellama-instruct, mistral-instruct-v0.1, mistrallite, openchat, belle-llama-2-chat, vicuna-chat, chatml]
+      --log-prompts
+          Print prompt strings to stdout
+      --log-stat
+          Print statistics to stdout
+      --log-enable
+          Print all log information to stdout
+      --stream-stdout
+          Print the output to stdout in the streaming way
+  -h, --help
+          Print help
 ```
-LLAMA_LOG=1 LLAMA_N_CTX=4096 LLAMA_N_PREDICT=128 wasmedge --dir .:. \
-  --nn-preload default:GGML:CPU:llama-2-7b.Q5_K_M.gguf llama-simple.wasm default \
-  --prompt 'Robert Oppenheimer most important achievement is ' \
-  --ctx-size 4096
 
-...................................................................................................
-[2023-10-08 23:13:10.272] [info] [WASI-NN] GGML backend: set n_ctx to 4096
-llama_new_context_with_model: kv self size  = 2048.00 MB
-llama_new_context_with_model: compute buffer total size =  297.47 MB
-llama_new_context_with_model: max tensor size =   102.54 MB
-[2023-10-08 23:13:10.472] [info] [WASI-NN] GGML backend: llama_system_info: AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 |
-[2023-10-08 23:13:10.472] [info] [WASI-NN] GGML backend: set n_predict to 128
-[2023-10-08 23:13:16.014] [info] [WASI-NN] GGML backend: llama_get_kv_cache_token_count 128
+For example, the following command tells WasmEdge to print out logs and statistics of the model at runtime.
 
-llama_print_timings:        load time =  1431.58 ms
-llama_print_timings:      sample time =     3.53 ms /   118 runs   (    0.03 ms per token, 33446.71 tokens per second)
-llama_print_timings: prompt eval time =  1230.69 ms /    11 tokens (  111.88 ms per token,     8.94 tokens per second)
-llama_print_timings:        eval time =  4295.81 ms /   117 runs   (   36.72 ms per token,    27.24 tokens per second)
-llama_print_timings:       total time =  5742.71 ms
-Robert Oppenheimer most important achievement is
-1945 Manhattan Project.
-Robert Oppenheimer was born in New York City on April 22, 1904. He was the son of Julius Oppenheimer, a wealthy German-Jewish textile merchant, and Ella Friedman Oppenheimer.
-Robert Oppenheimer was a brilliant student. He attended the Ethical Culture School in New York City and graduated from the Ethical Culture Fieldston School in 1921. He then attended Harvard University, where he received his bachelor's degree.
+```
+wasmedge --dir .:. --nn-preload default:GGML:AUTO:llama-2-7b-chat-q5_k_m.gguf \
+  llama-chat.wasm --prompt-template llama-2-chat --log-enable
+..................................................................................................
+llama_new_context_with_model: n_ctx      = 512
+llama_new_context_with_model: freq_base  = 10000.0
+llama_new_context_with_model: freq_scale = 1
+llama_new_context_with_model: kv self size  =  256.00 MB
+llama_new_context_with_model: compute buffer total size = 76.63 MB
+[2023-11-07 02:07:44.019] [info] [WASI-NN] GGML backend: llama_system_info: AVX = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 0 | NEON = 1 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | 
+
+llama_print_timings:        load time =   11523.19 ms
+llama_print_timings:      sample time =       2.62 ms /   102 runs   (    0.03 ms per token, 38961.04 tokens per second)
+llama_print_timings: prompt eval time =   11479.27 ms /    49 tokens (  234.27 ms per token,     4.27 tokens per second)
+llama_print_timings:        eval time =   13571.37 ms /   101 runs   (  134.37 ms per token,     7.44 tokens per second)
+llama_print_timings:       total time =   25104.57 ms
+[ASSISTANT]:
+Ah, a fellow Peanuts enthusiast! Snoopy is Charlie Brown's lovable and imaginative beagle, known for his wild and wacky adventures in the comic strip and television specials. He's a loyal companion to Charlie Brown and the rest of the Peanuts gang, and his antics often provide comic relief in the series. Is there anything else you'd like to know about Snoopy? 🐶
 ```
 
 ## Improve performance
@@ -216,7 +251,7 @@ Next, The prompt is converted into bytes and set as the input tensor for the mod
         .expect("Failed to set prompt as the input tensor");
 ```
 
-Next, excute the model inference.
+Next, execute the model inference.
 
 ```rust
   // execute the inference
@@ -241,7 +276,7 @@ println!("\nprompt: {}", &prompt);
 println!("\noutput: {}", output);
 ```
 
-The code explanation above is simple one time chat with llama 2 model. But we have more!
+The code explanation above is simple [one time chat with llama 2 model](https://github.com/second-state/llama-utils/tree/main/simple). But we have more!
 
 * If you're looking for continuous conversations with llama 2 models, please check out the source code [here](https://github.com/second-state/llama-utils/tree/main/chat).
 * If you want to construct OpenAI-compatible APIs specifically for your llama2 model, or the Llama2 model itself, please check out the surce code [here](https://github.com/second-state/llama-utils/tree/main/api-server).
