Skip to content

Commit b3aa5db

Browse files
infwinstoninfwinston
committed
merge
2 parents ea0097e + f2e6ca9 commit b3aa5db

22 files changed

+2009
-520
lines changed

‎README.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,15 +2,16 @@
22
| [**Demo**](https://chat.lmsys.org/) | [**Discord**](https://discord.gg/HSWAKCrnFx) | [**X**](https://x.com/lmsysorg) |
33

44
FastChat is an open platform for training, serving, and evaluating large language model based chatbots.
5-
- FastChat powers Chatbot Arena (https://chat.lmsys.org/), serving over 6 million chat requests for 50+ LLMs.
6-
- Chatbot Arena has collected over 200K human votes from side-by-side LLM battles to compile an online [LLM Elo leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard).
5+
- FastChat powers Chatbot Arena (https://chat.lmsys.org/), serving over 10 million chat requests for 70+ LLMs.
6+
- Chatbot Arena has collected over 500K human votes from side-by-side LLM battles to compile an online [LLM Elo leaderboard](https://leaderboard.lmsys.org).
77

88
FastChat's core features include:
99
- The training and evaluation code for state-of-the-art models (e.g., Vicuna, MT-Bench).
1010
- A distributed multi-model serving system with web UI and OpenAI-compatible RESTful APIs.
1111

1212
## News
13-
- [2023/09] 🔥 We released **LMSYS-Chat-1M**, a large-scale real-world LLM conversation dataset. Read the [report](https://arxiv.org/abs/2309.11998).
13+
- [2024/03] 🔥 We released Chatbot Arena technical [report](https://arxiv.org/abs/2403.04132).
14+
- [2023/09] We released **LMSYS-Chat-1M**, a large-scale real-world LLM conversation dataset. Read the [report](https://arxiv.org/abs/2309.11998).
1415
- [2023/08] We released **Vicuna v1.5** based on Llama 2 with 4K and 16K context lengths. Download [weights](#vicuna-weights).
1516
- [2023/07] We released **Chatbot Arena Conversations**, a dataset containing 33k conversations with human preferences. Download it [here](https://huggingface.co/datasets/lmsys/chatbot_arena_conversations).
1617

‎docs/model_support.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -105,7 +105,7 @@ After these steps, the new model should be compatible with most FastChat feature
105105
## API-Based Models
106106
To support an API-based model, consider learning from the existing OpenAI example.
107107
If the model is compatible with OpenAI APIs, then a configuration file is all that's needed without any additional code.
108-
For custom protocols, implementation of a streaming generator in [fastchat/serve/api_provider.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/api_provider.py) is required, following the provided examples. Currently, FastChat is compatible with OpenAI, Anthropic, Google Vertex AI, Mistral, and Nvidia NGC.
108+
For custom protocols, implementation of a streaming generator in [fastchat/serve/api_provider.py](https://github.com/lm-sys/FastChat/blob/main/fastchat/serve/api_provider.py) is required, following the provided examples. Currently, FastChat is compatible with OpenAI, Anthropic, Google Vertex AI, Mistral, Nvidia NGC, YandexGPT and Reka.
109109

110110
### Steps to Launch a WebUI with an API Model
111111
1. Specify the endpoint information in a JSON configuration file. For instance, create a file named `api_endpoints.json`:
@@ -120,7 +120,7 @@ For custom protocols, implementation of a streaming generator in [fastchat/serve
120120
}
121121
}
122122
```
123-
- "api_type" can be one of the following: openai, anthropic, gemini, or mistral. For custom APIs, add a new type and implement it accordingly.
123+
- "api_type" can be one of the following: openai, anthropic, gemini, mistral, yandexgpt or reka. For custom APIs, add a new type and implement it accordingly.
124124
- "anony_only" indicates whether to display this model in anonymous mode only.
125125

126126
2. Launch the Gradio web server with the argument `--register api_endpoints.json`:

‎fastchat/constants.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,9 +15,12 @@
1515
CONVERSATION_LIMIT_MSG = "YOU HAVE REACHED THE CONVERSATION LENGTH LIMIT. PLEASE CLEAR HISTORY AND START A NEW CONVERSATION."
1616
INACTIVE_MSG = "THIS SESSION HAS BEEN INACTIVE FOR TOO LONG. PLEASE REFRESH THIS PAGE."
1717
SLOW_MODEL_MSG = "⚠️ Both models will show the responses all at once. Please stay patient as it may take over 30 seconds."
18-
RATE_LIMIT_MSG = "**RATE LIMIT OF THIS MODEL IS REACHED. PLEASE COME BACK LATER OR TRY OTHER MODELS.**"
18+
RATE_LIMIT_MSG = "**RATE LIMIT OF THIS MODEL IS REACHED. PLEASE COME BACK LATER OR USE BATTLE MODE (the 1st tab).**"
1919
# Maximum input length
2020
INPUT_CHAR_LEN_LIMIT = int(os.getenv("FASTCHAT_INPUT_CHAR_LEN_LIMIT", 12000))
21+
BLIND_MODE_INPUT_CHAR_LEN_LIMIT = int(
22+
os.getenv("FASTCHAT_BLIND_MODE_INPUT_CHAR_LEN_LIMIT", 24000)
23+
)
2124
# Maximum conversation turns
2225
CONVERSATION_TURN_LIMIT = 50
2326
# Session expiration time

‎fastchat/conversation.py‎

Lines changed: 190 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ class SeparatorStyle(IntEnum):
2222
NO_COLON_TWO = auto()
2323
ADD_NEW_LINE_SINGLE = auto()
2424
LLAMA2 = auto()
25+
LLAMA3 = auto()
2526
CHATGLM = auto()
2627
CHATML = auto()
2728
CHATINTERN = auto()
@@ -34,7 +35,9 @@ class SeparatorStyle(IntEnum):
3435
DEEPSEEK_CHAT = auto()
3536
METAMATH = auto()
3637
YUAN2 = auto()
38+
GEMMA = auto()
3739
CLLM = auto()
40+
DEFAULT = auto()
3841

3942

4043
IMAGE_PLACEHOLDER_STR = "$$<image>$$"
@@ -151,6 +154,19 @@ def get_prompt(self) -> str:
151154
else:
152155
ret += tag
153156
return ret
157+
elif self.sep_style == SeparatorStyle.LLAMA3:
158+
ret = "<|begin_of_text|>"
159+
if self.system_message:
160+
ret += system_prompt
161+
else:
162+
ret += ""
163+
for i, (role, message) in enumerate(self.messages):
164+
if message:
165+
ret += f"<|start_header_id|>{role}<|end_header_id|>\n\n"
166+
ret += f"{message.strip()}<|eot_id|>"
167+
else:
168+
ret += f"<|start_header_id|>{role}<|end_header_id|>\n\n"
169+
return ret
154170
elif self.sep_style == SeparatorStyle.CHATGLM:
155171
# source: https://huggingface.co/THUDM/chatglm-6b/blob/1d240ba371910e9282298d4592532d7f0f3e9f3e/modeling_chatglm.py#L1302-L1308
156172
# source2: https://huggingface.co/THUDM/chatglm2-6b/blob/e186c891cf64310ac66ef10a87e6635fa6c2a579/modeling_chatglm.py#L926
@@ -271,6 +287,14 @@ def get_prompt(self) -> str:
271287
ret += ""
272288
ret = ret.rstrip("<n>") + seps[0]
273289
return ret
290+
elif self.sep_style == SeparatorStyle.GEMMA:
291+
ret = "<bos>"
292+
for role, message in self.messages:
293+
if message:
294+
ret += "<start_of_turn>" + role + "\n" + message + self.sep
295+
else:
296+
ret += "<start_of_turn>" + role + "\n"
297+
return ret
274298
elif self.sep_style == SeparatorStyle.CLLM:
275299
seps = [self.sep, self.sep2]
276300
ret = system_prompt + seps[0]
@@ -283,6 +307,14 @@ def get_prompt(self) -> str:
283307
else:
284308
ret += role + ":"
285309
return ret
310+
elif self.sep_style == SeparatorStyle.DEFAULT:
311+
ret = system_prompt + "\n"
312+
for role, message in self.messages:
313+
if message:
314+
ret += role + ": " + message + "\n"
315+
else:
316+
ret += role + ":"
317+
return ret
286318
else:
287319
raise ValueError(f"Invalid style: {self.sep_style}")
288320

@@ -300,6 +332,10 @@ def set_system_message(self, system_message: str):
300332
"""Set the system message."""
301333
self.system_message = system_message
302334

335+
def get_system_message(self):
336+
"""return the system message."""
337+
return self.system_message
338+
303339
def append_message(self, role: str, message: str):
304340
"""Append a new message."""
305341
self.messages.append([role, message])
@@ -498,6 +534,17 @@ def get_conv_template(name: str) -> Conversation:
498534
)
499535
)
500536

537+
# api-based default template
538+
register_conv_template(
539+
Conversation(
540+
name="api_based_default",
541+
system_message="",
542+
roles=("user", "assistant"),
543+
sep_style=SeparatorStyle.DEFAULT,
544+
sep=None,
545+
)
546+
)
547+
501548
register_conv_template(
502549
Conversation(
503550
name="airoboros_v1",
@@ -782,7 +829,23 @@ def get_conv_template(name: str) -> Conversation:
782829
name="chatgpt",
783830
system_message="You are a helpful assistant.",
784831
roles=("user", "assistant"),
785-
sep_style=None,
832+
sep_style=SeparatorStyle.DEFAULT,
833+
sep=None,
834+
)
835+
)
836+
837+
register_conv_template(
838+
Conversation(
839+
name="gpt-4-turbo-2024-04-09",
840+
system_message=(
841+
"You are ChatGPT, a large language model trained by OpenAI, based on the GPT-4 architecture.\n"
842+
"Knowledge cutoff: 2023-11\n"
843+
"Current date: {{currentDateTime}}\n\n"
844+
"Image input capabilities: Enabled\n"
845+
"Personality: v2"
846+
),
847+
roles=("user", "assistant"),
848+
sep_style=SeparatorStyle.DEFAULT,
786849
sep=None,
787850
)
788851
)
@@ -793,7 +856,7 @@ def get_conv_template(name: str) -> Conversation:
793856
name="pplxai",
794857
system_message="Be precise and concise.",
795858
roles=("user", "assistant"),
796-
sep_style=None,
859+
sep_style=SeparatorStyle.DEFAULT,
797860
sep=None,
798861
)
799862
)
@@ -808,6 +871,84 @@ def get_conv_template(name: str) -> Conversation:
808871
)
809872
)
810873

874+
register_conv_template(
875+
Conversation(
876+
name="claude-3-haiku-20240307",
877+
system_message=(
878+
"The assistant is Claude, created by Anthropic. The current date is "
879+
"{{currentDateTime}}. Claude's knowledge base was last updated in "
880+
"August 2023 and it answers user questions about events before "
881+
"August 2023 and after August 2023 the same way a highly informed "
882+
"individual from August 2023 would if they were talking to someone "
883+
"from {{currentDateTime}}. It should give concise responses to very "
884+
"simple questions, but provide thorough responses to more complex "
885+
"and open-ended questions. It is happy to help with writing, "
886+
"analysis, question answering, math, coding, and all sorts of other "
887+
"tasks. It uses markdown for coding. It does not mention this "
888+
"information about itself unless the information is directly "
889+
"pertinent to the human's query."
890+
),
891+
roles=("user", "assistant"),
892+
sep_style=SeparatorStyle.DEFAULT,
893+
sep=None,
894+
)
895+
)
896+
897+
register_conv_template(
898+
Conversation(
899+
name="claude-3-sonnet-20240229",
900+
system_message=(
901+
"The assistant is Claude, created by Anthropic. The current date is "
902+
"{{currentDateTime}}. Claude's knowledge base was last updated in "
903+
"August 2023 and it answers user questions about events before "
904+
"August 2023 and after August 2023 the same way a highly informed "
905+
"individual from August 2023 would if they were talking to someone "
906+
"from {{currentDateTime}}. It should give concise responses to very "
907+
"simple questions, but provide thorough responses to more complex "
908+
"and open-ended questions. It is happy to help with writing, "
909+
"analysis, question answering, math, coding, and all sorts of other "
910+
"tasks. It uses markdown for coding. It does not mention this "
911+
"information about itself unless the information is directly "
912+
"pertinent to the human's query."
913+
),
914+
roles=("user", "assistant"),
915+
sep_style=SeparatorStyle.DEFAULT,
916+
sep=None,
917+
)
918+
)
919+
920+
register_conv_template(
921+
Conversation(
922+
name="claude-3-opus-20240229",
923+
system_message=(
924+
"The assistant is Claude, created by Anthropic. The current date is "
925+
"{{currentDateTime}}. Claude's knowledge base was last updated on "
926+
"August 2023. It answers questions about events prior to and after "
927+
"August 2023 the way a highly informed individual in August 2023 "
928+
"would if they were talking to someone from the above date, and can "
929+
"let the human know this when relevant. It should give concise "
930+
"responses to very simple questions, but provide thorough responses "
931+
"to more complex and open-ended questions. If it is asked to assist "
932+
"with tasks involving the expression of views held by a significant "
933+
"number of people, Claude provides assistance with the task even if "
934+
"it personally disagrees with the views being expressed, but follows "
935+
"this with a discussion of broader perspectives. Claude doesn't "
936+
"engage in stereotyping, including the negative stereotyping of "
937+
"majority groups. If asked about controversial topics, Claude tries "
938+
"to provide careful thoughts and objective information without "
939+
"downplaying its harmful content or implying that there are reasonable "
940+
"perspectives on both sides. It is happy to help with writing, "
941+
"analysis, question answering, math, coding, and all sorts of other "
942+
"tasks. It uses markdown for coding. It does not mention this "
943+
"information about itself unless the information is directly pertinent "
944+
"to the human's query."
945+
),
946+
roles=("user", "assistant"),
947+
sep_style=SeparatorStyle.DEFAULT,
948+
sep=None,
949+
)
950+
)
951+
811952
# MetaMath default template
812953
# reference: https://github.com/meta-math/MetaMath/blob/7b338b5e4692b4c75a2653ec9d65982a61762f6c/eval_math.py#L58
813954
register_conv_template(
@@ -889,7 +1030,7 @@ def get_conv_template(name: str) -> Conversation:
8891030
Conversation(
8901031
name="bard",
8911032
roles=("0", "1"),
892-
sep_style=None,
1033+
sep_style=SeparatorStyle.DEFAULT,
8931034
sep=None,
8941035
)
8951036
)
@@ -898,8 +1039,25 @@ def get_conv_template(name: str) -> Conversation:
8981039
Conversation(
8991040
name="gemini",
9001041
roles=("user", "model"),
901-
sep_style=None,
1042+
sep_style=SeparatorStyle.DEFAULT,
1043+
sep=None,
1044+
)
1045+
)
1046+
1047+
register_conv_template(
1048+
Conversation(
1049+
name="gemini-dev",
1050+
roles=("user", "model"),
1051+
sep_style=SeparatorStyle.DEFAULT,
9021052
sep=None,
1053+
system_message=(
1054+
"You are a friendly and helpful assistant.\n"
1055+
"Ensure your answers are complete, unless the user requests a more concise approach.\n"
1056+
"When generating code, offer explanations for code segments as necessary and maintain good coding practices.\n"
1057+
"When presented with inquiries seeking information, provide answers that reflect a deep understanding of the field, guaranteeing their correctness.\n"
1058+
"For any non-english queries, respond in the same language as the prompt unless otherwise specified by the user.\n"
1059+
"For prompts involving reasoning, provide a clear explanation of each step in the reasoning process before presenting the final answer."
1060+
),
9031061
)
9041062
)
9051063

@@ -1118,6 +1276,21 @@ def get_conv_template(name: str) -> Conversation:
11181276
)
11191277
)
11201278

1279+
# llama3 template
1280+
# reference: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/blob/main/tokenizer_config.json
1281+
# reference: https://github.com/meta-llama/llama3/blob/0cee08ec68f4cfc0c89fe4a9366d82679aaa2a66/llama/tokenizer.py#L222
1282+
register_conv_template(
1283+
Conversation(
1284+
name="llama-3",
1285+
system_template="<|start_header_id|>system<|end_header_id|>\n\n{system_message}<|eot_id|>",
1286+
roles=("user", "assistant"),
1287+
sep_style=SeparatorStyle.LLAMA3,
1288+
sep="",
1289+
stop_str="<|eot_id|>",
1290+
stop_token_ids=[128001, 128009],
1291+
)
1292+
)
1293+
11211294
register_conv_template(
11221295
Conversation(
11231296
name="chinese-alpaca2",
@@ -1559,7 +1732,7 @@ def get_conv_template(name: str) -> Conversation:
15591732
name="steerlm",
15601733
system_message="",
15611734
roles=("user", "assistant"),
1562-
sep_style=None,
1735+
sep_style=SeparatorStyle.DEFAULT,
15631736
sep=None,
15641737
)
15651738
)
@@ -1612,14 +1785,23 @@ def get_conv_template(name: str) -> Conversation:
16121785
register_conv_template(
16131786
Conversation(
16141787
name="gemma",
1615-
system_message="<bos>",
1616-
roles=("<start_of_turn>user\n", "<start_of_turn>model\n"),
1617-
sep_style=SeparatorStyle.NO_COLON_SINGLE,
1788+
roles=("user", "model"),
1789+
sep_style=SeparatorStyle.GEMMA,
16181790
sep="<end_of_turn>\n",
16191791
stop_str="<end_of_turn>",
16201792
)
16211793
)
16221794

1795+
register_conv_template(
1796+
Conversation(
1797+
name="yandexgpt",
1798+
system_message="",
1799+
roles=("user", "assistant"),
1800+
sep_style=None,
1801+
sep=None,
1802+
)
1803+
)
1804+
16231805

16241806
if __name__ == "__main__":
16251807
from fastchat.conversation import get_conv_template

0 commit comments

Comments
 (0)