Question in reproducing experimental results. #2
Description
HI😊, we are reproducing your experimental results as the baseline of our method.
We are confused by the following questions. 🤔
First, why running the following code needs more than 1 minutes? Since the prompt is short, it is expected to finish very quickly.
In [8]: prompt = "TESLA company is found by"
In [9]: output = model(prompt=prompt)
...(a long time)...Second, after waiting for over 1 min, we finally got the results looks like this:
In [10]: output
Out[10]: {'text': ['Nik']}We think this output is not reasonable and want to know if there are some improper configurations in the following scripts?
@dataclass
class MagicpigConfig:
server_type: str = 'hf'
server_host: str = '127.0.0.1'
server_port: str = '5000'
ssh_server: Optional[str] = None
ssh_key_path: Optional[str] = None
model_name_or_path: str = 'meta-llama/Llama-2-7b-chat-hf'
temperature: float = 0.0
top_k: int = 32
top_p: float = 1.0
random_seed: int = 0
stop_words: list = field(default_factory=list)
sliding_window_size: int = None
threads: int = 1
K: int = 10
L: int = 150
S: float = 4.0
W: int = 64
Q: int = 0
QR: float = 0.0
max_seq_length: int = 4096
max_new_tokens: int = 128If there are improper configurations for short prompt generation, we want to further know what is the most suitable configuration under different prompt length? e.g. 1K, 2K, 4K, 8K.