test

Author: Kyriection

recipes/experimental/long-context/H2O/README.md

@@ -2,9 +2,9 @@
 
 ### Overview:
 
-Heavy-Hitter Oracle (H2O) is an efficient inference framework of LLMs. During the generative inference of transfomers, the size of KV cache grows linearly with the sequence length (prompt length + generation length) during long context generation. And the size KV cache is usually significantly larger than the model parameters, contrains the inference throughput. H2O identifies the critical KV pairs and evicts other unnecessary ones, maintaining a small cache size thus improving the throughput. 
+Heavy-Hitter Oracle (H2O) is an efficient inference framework of LLMs. During the generative inference of transfomers, the size of KV cache grows linearly with the sequence length (prompt length + generation length) during long context generation. And the size KV cache is usually significantly larger than the model parameters, contrains the inference throughput. H2O identifies the critical KV pairs and evicts other unnecessary ones, maintaining a small cache size thus improving the throughput.
 
-Besides, LLMs usually have poor generation to long sequence during inference. H2O handles this issue by maintaining only heavy-hitter tokens and the most recent tokens. Incorporated with the positional rolling strategy (reassigning the position of each kv with the position in the kv cache instead of the original sequence), H2O can process sequence length much longer than the pretrained context window. 
+Besides, LLMs usually have poor generation to long sequence during inference. H2O handles this issue by maintaining only heavy-hitter tokens and the most recent tokens. Incorporated with the positional rolling strategy (reassigning the position of each kv with the position in the kv cache instead of the original sequence), H2O can process sequence length much longer than the pretrained context window.
 
 Current implementation supports llama-1/2/3, from 7B to 70B. Since H2O only maintains the most important KV pairs, it might missing some important information in the middle content for some knowlege-intensive tasks.
 
@@ -43,6 +43,7 @@ The following example runs inference of Llama-3-8b-instruct on "Needle in a hays
 ```
 # step 1: obtain prompts for evaluation
 python utils/needle_test/prompt.py --model_name meta-llama/Meta-Llama-3-8B-Instruct
+# modify utils/needle_test/config-prompt.yaml to adjust the min/max sequence length for the test
 
 
 # step 2: generation predictions of each prompt

recipes/experimental/long-context/H2O/utils/needle_test/config-prompt.yaml

@@ -5,7 +5,7 @@ prompt:
 
 context:
   min_len: 1000
-  max_len: 8000
+  max_len: 16000
   interval: 10
   manually_select_list: null  # null or a list of context lengths to manually select