original_content.txt

Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Wei-Lin Chiang* 1Lianmin Zheng* 1Ying Sheng2Anastasios N. Angelopoulos1Tianle Li1Dacheng Li1
Banghua Zhu1Hao Zhang3Michael I. Jordan1Joseph E. Gonzalez1Ion Stoica1
Abstract
Large Language Models (LLMs) have unlocked
new capabilities and applications; however, evalu-
ating the alignment with human preferences still
poses significant challenges. To address this is-
sue, we introduce Chatbot Arena, an open plat-
form for evaluating LLMs based on human pref-
erences. Our methodology employs a pairwise
comparison approach and leverages input from
a diverse user base through crowdsourcing. The
platform has been operational for several months,
amassing over 240K votes. This paper describes
the platform, analyzes the data we have collected
so far, and explains the tried-and-true statistical
methods we are using for efficient and accurate
evaluation and ranking of models. We confirm
that the crowdsourced questions are sufficiently
diverse and discriminating and that the crowd-
sourced human votes are in good agreement with
those of expert raters. These analyses collectively
establish a robust foundation for the credibility
of Chatbot Arena. Because of its unique value
and openness, Chatbot Arena has emerged as
one of the most referenced LLM leaderboards,
widely cited by leading LLM developers and
companies. Our demo is publicly available at
https://chat.lmsys.org .
1. Introduction
Recent advancements in large language models (LLMs)
have significantly expanded their capabilities beyond tradi-
tional natural language processing boundaries, addressing a
broad array of general tasks (OpenAI, 2023; Gemini et al.,
2023; Touvron et al., 2023). These developments underscore
the potential of LLMs but also have raised concerns with re-
spect to performance evaluation. Current benchmarks often
fail to capture the nuanced and diverse aspects of these mod-
els, particularly in assessing their alignment with human
*Equal contribution1UC Berkeley2Stanford3UCSD. Corre-
spondence to: Wei-Lin Chiang <weichiang@berkeley.edu>.
LiveStaticCodeforces Weekly ContestsMMLU, HellaSwag, GSM-8KGround TruthChatbot ArenaMT-Bench, AlpacaEvalHuman PreferenceQuestion SourceEvaluationMetricFigure 1. Classification of LLM benchmarks: We categorize
along two dimensions: whether the questions are from a static
dataset or a live, fresh source, and whether the evaluation met-
ric relies on ground truth or (approximated) human preferences.
MMLU (Hendrycks et al., 2020), HellaSwag (Zellers et al., 2019),
GSM-8K (Cobbe et al., 2021), MT-Bench (Zheng et al., 2023b),
and AlpacaEval (Li et al., 2023) are common examples of static
benchmarks. Chatbot Arena is the platform introduced in this
paper.
preferences in real-world, open-ended tasks.
To assess the performance of LLMs, the research community
has introduced a variety of benchmarks. These benchmarks
can be categorized based on two factors: the source of ques-
tions (either static or live) and the evaluation metric (either
ground truth or human preference). According to these fac-
tors, benchmarks can be classified into four categories, as
shown in Figure 1. While a range of benchmarks is benefi-
cial, the most prevalent current method for evaluating LLMs
remains a static, ground-truth-based evaluation, partly be-
cause such evaluations are inexpensive and reproducible.
However, these static, ground-truth-based benchmarks ex-
hibit several limitations. Firstly, the questions within these
benchmarks are not open-ended, hindering the ability to
capture the flexible and interactive use found in real-world
settings (Zheng et al., 2023b). Secondly, the test sets in
these benchmarks are static, meaning they can become con-
taminated over time, which undermines the reliability of
the evaluation results (Yang et al., 2023). Furthermore, for
many complex tasks, establishing a definitive ground truth
is not only challenging but sometimes unattainable. Conse-
quently, current benchmarks fail to adequately address the
needs of state-of-the-art LLMs, particularly in evaluating
user preferences. Thus, there is an urgent necessity for an
open, live evaluation platform based on human preference
that can more accurately mirror real-world usage.
Creating such a benchmark platform entails significant chal-
lenges. It requires the collection of live, fresh, and diverse
user questions to accurately represent real-world scenarios.
1arXiv:2403.04132v1  [cs.AI]  7 Mar 2024
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Additionally, developing scalable, incremental, and efficient
ranking systems is essential for evaluating a large number
of models. Moreover, ensuring the quality of human evalua-
tions is crucial given the noisy nature of human preferences.
To this end, we introduce Chatbot Arena, a benchmarking
platform for LLMs that features anonymous, randomized
battles in a crowdsourced setting. Chatbot Arena is a free
website open to all users.1On this website, a user can ask a
question and get answers from two anonymous LLMs. Af-
terward, the user casts a vote for the model that delivers the
preferred response, with the models’ identities revealed only
after voting. This crowdsourced method effectively gathers
a diverse array of fresh user prompts, accurately reflecting
real-world LLM applications. Armed with this data, we
employ a suite of powerful statistical techniques, ranging
from the statistical model of Bradley & Terry (1952) to the
E-values of V ovk & Wang (2021), to estimate the ranking
over models as reliably and sample-efficiently as possible.
With these tools in hand, we have designed efficient sam-
pling algorithms specifically to select model pairs in a way
that accelerates the convergence of rankings while retaining
statistical validity.
We conduct a thorough analysis of the collected data to en-
sure the credibility of our platform. We demonstrate that
the user-generated questions are sufficiently diverse to en-
compass a wide range of LLM use cases and are sufficiently
challenging to differentiate between models. Furthermore,
we confirm that the crowd-sourced votes are highly consis-
tent with expert evaluations.
We have been running our system since Apr 2023 and have
received over 240K votes from about 90K users in over
100 different languages as of Jan 2024. To encourage user
engagement, we have made over 50 state-of-the-art models
available for free. We also collaborate with leading model
developers such as OpenAI, Google, Anthropic, Mistral,
Hugging Face, and various universities, incorporating their
latest models into our platform. We keep the community
engaged by routinely updating the leaderboard, publishing
analytical blogs, releasing datasets, and sharing information
via tweets. Because of its unique and significant value, our
leaderboard has emerged as one of the most referenced in
the LLM field and has become a benchmark for the industry.
We commit to making our data and code available, ensuring
that this platform is open-source and open-accessible.
We make the following contributions:
•We build the first large-scale crowd-sourced live LLM
evaluation platform with over 1M users visit.2
1https://chat.lmsys.org
2The number was estimated by Google Analytics as of March
2024. Note that user visit may not convert to votes as our website
also offers “direct chat” mode.•We conduct an in-depth analysis of the collected data,
including prompt diversity, quality, vote quality, and
insights on human feedback.
•We will publicly release a human preference dataset with
over 100K pairwise votes collected from Chatbot Arena.
•We design an efficient sampling algorithm that actively
chooses which model pairs to show, such that our sample
efficiency improves, sometimes to a large degree.
2. Related Work
LLM Benchmarks. We briefly review the common LLM
benchmarks, following the classification presented in Fig-
ure 1. The most prevalent benchmarks are static, ground-
truth-based ones, typically in the form of multiple-choice
questions or question-answering tasks with predefined an-
swers and test cases. These benchmarks encompass a range
of topics including language understanding, mathematics,
coding, and logical reasoning. Prominent examples in
this category are MMLU (Hendrycks et al., 2020), Hel-
laSwag (Zellers et al., 2019), GSM-8K (Cobbe et al., 2021),
BigBench (Srivastava et al., 2023), AGIEval (Zhong et al.,
2023), and HumanEval (Chen et al., 2021). Benchmarks
focusing on safety, such as ToxicChat (Lin et al., 2023),
and comprehensive suites like HELM (Liang et al., 2022),
also exist. In addition to closed-ended questions, bench-
marks can include open-ended questions that are evaluated
by human judgment, which can be rated by experts or crowd
workers such as Amazon Mechanical Turk (Karpinska et al.,
2021; Geng et al., 2023; Wang et al., 2023). The recent trend
includes utilizing GPT-4 for approximating human judg-
ment (Chiang & Lee, 2023), with notable instances being
MT-Bench (Zheng et al., 2023b) and AlpacaEval (Li et al.,
2023). In addition to static benchmarks, live benchmarks
that include fresh questions are also available. These ques-
tions can be obtained from annual exams or weekly online
contests such as Codeforces (Li et al., 2022; Huang et al.,
2023). They can also be sourced from human interaction.
Some studies have explored using live human interaction for
reinforcement learning from human preference (Bai et al.,
2022; Ouyang et al., 2022; Touvron et al., 2023). However,
these studies are typically limited to specific organizations.
In this paper, we introduce Chatbot Arena, the first open,
large-scale, and crowdsourced benchmark platform that uti-
lizes live human interaction.
Risks of Static Benchmarks. Static benchmarks have cer-
tain issues, including contamination, saturation, overfitting,
and a lack of human alignment (Yang et al., 2023; Oren
et al., 2023). DynaBench (Kiela et al., 2021) identifies these
challenges and recommends the use of a live benchmark
that incorporates a human-in-the-loop approach for classical
NLP benchmarks. Our system adopts a similar spirit. How-
ever, our focus is on chatting with LLMs, and we implement
2
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
this on a significantly larger user scale.
Ranking System. Ranking systems have been a well-
studied topic in statistics. Related topics include probability
models (Hunter, 2004; Rao & Kupper, 1967), rank elicita-
tion (Szörényi et al., 2015; Busa-Fekete et al., 2014a;b),
and online experiment design (Chernoff, 1992; Karimi
et al., 2021). The Elo rating system has also been used for
LLMs (Bai et al., 2022; Boubdir et al., 2023). Contributing
to this literature, we introduce techniques for accelerating
ranking convergence and detecting abnormalities, specifi-
cally applied to large-scale, real-world settings of LLMs.
Human Preference Dataset. Owing to the significance
of human preferences, several datasets and analyses exist
that incorporate human preferences. These include Ope-
nAssistant (Köpf et al., 2023), HH-RLHF (Bai et al., 2022),
LMSYS-Chat-1M (Zheng et al., 2023a), and synthetic ap-
proximations of human preferences like UltraFeedback (Cui
et al., 2023) and Nectar (Zhu et al., 2023). Our prior data
release, LMSYS-Chat-1M (Zheng et al., 2023a), is similarly
collected via crowdsourcing. However, LMSYS-Chat-1M
comprises solely conversations and lacks human preference
data, rendering it unsuitable for direct use in ranking studies.
This paper focuses on the analysis of preference data for
ranking purposes.
3. Human Preference Data Collection
In this section, we discuss our interface design to collect
human preferences and present summary statistics.
3.1. Interface
Chatbot Arena crowd-sources feedback from users for
model evaluation. Our goal is to design an ease-of-use in-
terface to reduce friction for users to contribute data. Since
we collect feedback from many users, it is difficult to set
a consistent grading rubric across different people. Hence,
we adopt a pairwise comparison mechanism where users
only need to compare two model responses and vote for the
better one, instead of requiring users to provide an absolute
score.
In each battle, two anonymous models are sampled. To
encourage data diversity, we do not preset any input prompt
on the website. Users are free to input any prompt to the
two models. We believe this creates incentives for user en-
gagement, particularly given that we offer a free service.
It also helps us collect a diverse set of inputs representing
real-world usage. After models provide their answers, user
compare them side-by-side and vote for the preferred an-
swer. If a user cannot choose in the first turn, the user can
continue chatting until identifying a winner. For those who
are unsure, we also present two buttons, “tie” or “both are
bad.” Figure 8 shows a screenshot of our interface. Beforeusing our service, users are required to accept terms of use,
which gives us their consent to release the data publicly.
3.2. Data Statistics
We began collecting data in April 2023. As of Jan 2024,
we have received around 240K votes from over 90K users.
Our data involves more than 50 models, including both pro-
prietary models like GPT-4, Claude, and Gemini, as well
as open models such as LLaMA and Mistral. These con-
versations cover more than 100 languages, with 77% being
in English, 5% in Chinese, and the remaining languages,
such as Russian, German, Spanish, French, and Japanese,
each representing less than 2% of the total. Each data point
includes multi-turn conversations between the user and two
LLMs, and a vote to indicate which model the user prefers.
We summarize statistics in Table 1 along with other existing
human preference datasets.
Figure 10 in the Appendix shows the vote count per model.
On average, 8K votes are collected for each model. In Fig-
ure 2, we select a set of representative models and present
their win rate and the number of battles. Note that we em-
ploy non-uniform sampling to concentrate votes on model
pairs that have similar performance due to higher uncer-
tainty. This helps us reduce the number of votes required to
reach stable results. We later develop an adaptive sampling
method and demonstrate its effectiveness against random
sampling. See Section 5 for further analysis.
To ensure anonymity, we use keywords to filter out con-
versations containing model identity such as model name
(e.g., GPT, Claude) or companies (e.g., OpenAI, Anthropic).
To avoid misuse, we adopt OpenAI moderation API to flag
conversations that contain unsafe content. The flagged user
requests account for 3% of the total requests. Figure 9 in
the Appendix shows the number of valid user votes over
time, where we get 1-2K votes per day in recent months and
spikes as we introduce new models or leaderboard updates.
4. From Pairwise Comparisons to Rankings
Our data consists of pairwise comparisons—but how can we
use these comparisons to recover a ranking over all Mmod-
els? This is a well-studied topic in the literature on learning
to rank (Liu et al., 2009), and we present our perspective
here. We let A={(m, m′) :m < m′andm, m′∈[M]}
denote our comparative data set.
We consider a sequential setting, where at time t∈N, we
serve the human a pair of models At∈ A (which we pick),
and in turn we observe the human’s response Ht∈[0,1].
As an example, we might have that At= (1,2)andHt= 1,
indicating that the human prefers model 2 over model 1. In
the ensuing text, we will primarily focus on the binary case—
where Ht∈ {0,1}—but our approach will generalize to
3
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Table 1. Statistics of human preference datasets, including Anthropic HH (Bai et al., 2022), OpenAssistant Conversations (Köpf et al.,
2023), and Chatbot Arena (as of 2024/1/21). The tokens are counted by Llama2’s tokenizer. “Conv” = Conversation. “Lang” = Language.
Dataset # Convs # Models # Users # LangsAvg. # Turns Avg. # Tokens Avg. # Tokens
per Sample per Prompt per Response
Anthropic HH 338,704 - 143 1 2.3 18.9 78.9
OpenAssistant 66,497 - 13,500 35 - 36.9 214.2
Chatbot Arena (20240121) 243,329 50 90,051 149 1.3 94.9 269.0
0.00 0.68 0.69 0.75 0.71 0.76 0.77 0.75 0.76 0.79 0.86 0.90
0.32 0.00 0.50 0.59 0.61 0.59 0.59 0.59 0.62 0.72 0.70 0.80
0.31 0.50 0.00 0.54 0.56 0.50 0.57 0.52 0.69 0.73 0.60 0.87
0.25 0.41 0.46 0.00 0.54 0.48 0.51 0.56 0.58 0.53 0.73 0.84
0.29 0.39 0.44 0.46 0.00 0.42 0.54 0.55 0.58 0.63 0.67 0.76
0.24 0.41 0.50 0.52 0.58 0.00 0.49 0.55 0.58 0.61 0.64 0.73
0.23 0.41 0.43 0.49 0.46 0.51 0.00 0.56 0.58 0.63 0.72 0.71
0.25 0.41 0.48 0.44 0.45 0.45 0.44 0.00 0.54 0.68 0.65 0.62
0.24 0.38 0.31 0.42 0.42 0.42 0.42 0.46 0.00 0.61 0.58 0.61
0.21 0.28 0.27 0.47 0.37 0.39 0.37 0.32 0.39 0.00 0.53 0.57
0.14 0.30 0.40 0.27 0.33 0.36 0.28 0.35 0.42 0.47 0.00 0.52
0.10 0.20 0.13 0.16 0.24 0.27 0.29 0.38 0.39 0.43 0.48 0.00
gpt-4-turbogpt-4-0613mistral-mediummixtral-8x7b-instruct-v0.1gemini-pro-dev-apiclaude-2.1gpt-3.5-turbo-0613claude-instant-1llama-2-70b-chatllama-2-13b-chatllama-2-7b-chatmistral-7b-instruct
mistral-7b-instructllama-2-7b-chatllama-2-13b-chatllama-2-70b-chatclaude-instant-1gpt-3.5-turbo-0613claude-2.1gemini-pro-dev-apimixtral-8x7b-instruct-v0.1mistral-mediumgpt-4-0613gpt-4-turbo
0.10.20.30.40.50.60.70.80.9Model BModel A
02564 1189 1192 858 3053 1991 270 141 157 106 144
2564 0 566 263 756 2227 1025 355 414 409 264 197
1189 566 0 775 371 382 773 103 45 51 40 52
1192 263 775 0 71 744 869 136 862 66 45 61
858 756 371 71 0 74 564 53 31 30 30 37
3053 2227 382 744 74 0 351 650 113 117 75 114
1991 1025 773 869 564 351 0 842 572 388 283 155
270 355 103 136 53 650 842 0 459 241 202 101
141 414 45 862 31 113 572 459 0 383 134 369
157 409 51 66 30 117 388 241 383 0 251 621
106 264 40 45 30 75 283 202 134 251 0 521
144 197 52 61 37 114 155 101 369 621 521 0
gpt-4-turbogpt-4-0613mistral-mediummixtral-8x7b-instruct-v0.1gemini-pro-dev-apiclaude-2.1gpt-3.5-turbo-0613claude-instant-1llama-2-70b-chatllama-2-13b-chatllama-2-7b-chatmistral-7b-instruct
mistral-7b-instructllama-2-7b-chatllama-2-13b-chatllama-2-70b-chatclaude-instant-1gpt-3.5-turbo-0613claude-2.1gemini-pro-dev-apimixtral-8x7b-instruct-v0.1mistral-mediumgpt-4-0613gpt-4-turbo
050010001500200025003000Model BModel A
Figure 2. Win-rate (left) and battle count (right) between a subset of models in Chatbot Arena.
any form of feedback, including the possibility of allowing
the human to express different degrees of preference or to
say the models are tied.
One critical goal is to estimate the win matrix :θ∗(a) =
E[Ht|At=a], for all a∈ A; see the left panel of Figure 2
for an illustration of the (empirical) win matrix. In the
binary case, the aentry in the win matrix corresponds to
the probability the human prefers model a2toa1when
shown the pair a. Finding the win matrix is a relatively
straightforward mean-estimation problem; we will provide
details in Section 5.
Formally, consider a score s(P)∈RM, where Pis a joint
distribution over AandH(by default, we will target a uni-
form distribution over A). Each model has a true score
s(P)m, and better models will have higher scores. In partic-
ular, we have the rank of model m:
rank(P)m= 1 +X
m′∈[M]1{s(P)m′> s(P)m}.(1)
The best model has rank 1. If there is another model tied for
best, they will both get assigned rank 1.
Picking a score. A standard score function in this setting
is the vector of Bradley-Terry (BT) coefficients (Bradley &
Terry, 1952). In the Bradley-Terry model, Ht∈ {0,1}, and
the probability model mbeats model m′is modeled via alogistic relationship:
P(Ht= 1) =1
1 +eξm′−ξm, (2)
where ξis anM-length vector of so-called BT coefficients.
Without loss of generality, we take ξ1= 0(since the model
is invariant to addition in ξ). Our goal is to estimate the pop-
ulation Bradley-Terry coefficients, i.e., those that minimize
the binary cross-entropy:
s(P) = argmin
ξE(A,H)∼P
ℓ
H,1
1 +eξA2−ξA1
,(3)
where ℓis the binary cross-entropy loss, ℓ(h, p) =
−(hlog(p) + (1 −h) log(1 −p)).
Although the BT model technically assumes a parametric
form for the model win rates, the seminal results of Huber
et al. (1967); White (1982) show that maximum likelihood
estimators are still asymptotically normal even when these
assumptions do not hold, so long as the so-called “sandwich”
covariance matrix is used; see Section 5 for details, and see
Appendix B for a nonparametric extension of the Bradley-
Terry model. Finally, we remark that previous evolutions of
our online interface have reported different ranking scores,
such as the Elo score (Elo, 1967) instead of the BT coeffi-
cients. We made this change because the BT coefficients
are better for the purpose of statistical estimation.
4
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
5. Efficient Approximate Ranking
In Section 4 we described how to calculate the win matrix,
score, and rank. Now we describe our estimation proce-
dures.
Win matrix estimation. Estimation of the win ma-
trix is relatively straightforward. Define Xt(a) =
1
Pt(a)Ht1{At=a}, where Pt(a)is the probability of sam-
pling pair aat time t, andXtas the according vector. Then
the estimator is
ˆθT=1
TTX
t=1Xt. (4)
Note that E[Xt(a)] = θ∗(a)for all t, and thus ˆθTis an
unbiased estimator of θ∗. We will furthermore estimate the
covariance matrix as
bΣT=1
TTX
t=1(Xt−ˆθT)(Xt−ˆθT)⊤. (5)
Under the appropriate regularity conditions, we have that
√
TbΣ−1/2(ˆθ−θ∗)→ N (0, Id), (6)
and we construct confidence intervals accordingly. For
an understanding of the appropriate regularity conditions,
see Durrett (2019), Theorem 8.2.8, where condition (ii) is
trivially satisfied so long as Pt(a)> ϵ > 0, and condition
(i) is implied by the almost-sure convergence of Pt(a)to a
limiting distribution P(a).
Estimating the BT scores. To estimate the BT coefficients,
mirroring (3), we perform (reweighted) maximum likeli-
hood estimation on our data points:
s(ˆP) = argmin
ξTX
t=11
P(At)ℓ
Ht,1
1 +eξAt,2−ξAt,1
,
(7)
where At∼P. We perform the inverse weighting by P(At)
because this allows us to target a score with a uniform
distribution over A.
To compute confidence intervals on the BT coefficients, we
employ two strategies: (1) the pivot bootstrap (DiCiccio &
Efron, 1996), and (2) the “sandwich” robust standard errors
outlined in Huber et al. (1967) (see also Freedman (2006)
for an outline of the necessary technical assumptions). Ulti-
mately, based on the results of a simulation study described
in Appendix A, we choose to deploy the sandwich intervals
due to their smaller size in large samples.
Approximate rankings. Finally, we report an approximate
ranking for each model that accounts for the uncertainty
in the estimation of the score. Given an M-dimensional
confidence set Csatisfying
P(s(P)∈ C)≥1−α, (8)we extract an approximate ranking Rm= 1 +P
m′∈[M]1{infCm′>supCm}. The uniform validity of
Cdirectly implies that P(∃m:Rm>rank(P)m)≤α—
i.e., with high probability, no model’s performance is un-
derstated. A guarantee on the other side—that no model’s
performance is overstated—is possible by interchanging the
infandsup. To get the uniform confidence set, we construct
the chi-squared interval implied by the central limit theo-
rem using the sandwich estimate of the variance. In other
words, we construct the interval {ξ:T


ˆV−1/2(ˆξ−ξ)


≤
χ2
1−α,M−1, where ˆξis our MLE of the BT coefficients and
ˆVξis the sandwich variance of the logistic regression.
Active sampling rule. Our sampling rule was to choose
the model pair a∈ A proportionally to the reduction in
confidence interval size by sampling that pair:
Pt(a)∝s
ˆΣt,a,a
|{t:At=a}|−s
ˆΣt,a,a
|{t:At=a}|+ 1.(9)
5.1. Detecting Anomalous Users
On a different note, we take a first step towards identify-
ing anomalous IP addresses in our dataset. In a dataset
ofUunique IPs, we let IP={1, . . . , U }be the set
of all IP addresses. Consider a “test” user, outside this
database, who gives ratings H′
1, . . . , H′
nwhen presented
actions A′
1, . . . , A′
n. The idea of our procedure is to com-
pare the distribution of ratings for the new user to the his-
torical distribution of ratings for a given action. We let
Ha={Ht:At=a}and every time a user submits a vote,
we calculate the following number:
pi=1
|HA′
i|+ 1
1 +X
h∈HA′
i1{h≥H′
i}
. (10)
Under the null hypothesis that HA′
iis exchangeable with
H′
i,piis a valid p-value (see Appendix C for a proof). Fur-
thermore, the dependence of these p-values asymptotically
is negligible.
With this p-value in hand, we can test against this null
hypothesis sequentially by using Fisher’s combination
test (Fisher, 1928) along with a variant of the Bonferroni
correction. In particular, for each user, after their jth vote,
we compute Mj=−2jP
i=1log(pi). At 5 randomly cho-
sen values of jbetween 1 and 100, we identify a user as
anomalous if Mj≥χ2
2j,1−α/5. (The times are randomly
chosen, as to avoid anomalous users strategizing to hack this
p-value.) Despite the heuristic application of this procedure,
it seems to work well in our small-scale tests reported in
Table 5.
5
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Medical Queries and Infor mation ( 0.4% )Role-Playing Games ( 0.4% )Movie R eviews and Discussions ( 0.5% )SQL Database T able Queries ( 0.5% )Web Development Essentials ( 0.7% )Animal Behavior and P et Car e Queries ( 0.4% )Cooking and R ecipes ( 0.7% )Email and L etter W riting Assistance ( 0.8% )Operations & Fleet Management ( 0.8% )Sports and A thletics Queries ( 0.8% )Advanced Mathematical Concepts ( 0.4% )Philosophical T exts & Concepts ( 0.4% )AI Impact and Applications ( 0.9% )Original Jok e Requests ( 0.5% )Poetry W riting & Styles ( 0.8% )Word Play and Phonetics ( 1.0% )
0.30.40.50.60.70.80.91Similarity
Figure 3. Similarity matrix of top-16 topic clusters. The number
followed by the topic label represents the cluster size in percentage.
Note that similarity is computed by cluster’s centroid embeddings,
hence diagonals are always one.
6. Data Analysis
To examine whether Arena’s crowdsourced data reflects
real-world use cases, we conduct topic modeling on the
user prompts. We show how effective are these prompts in
distinguishing models. Lastly, we validate the vote quality
by relabeling data with experts.
6.1. Topic Modeling on User Prompts
To study the prompt diversity, we build a topic modeling
pipeline with BERTopic3(Grootendorst, 2022). We start
with transforming user prompts into representation vectors
using OpenAI’s text embedding model (text-embedding-3-
small). To mitigate the curse of dimensionality for data
clustering, we employ UMAP (Uniform Manifold Approx-
imation and Projection) (McInnes et al., 2020) to reduce
the embedding dimension from 1,536 to 5. We then use
the hierarchical density-based clustering algorithm, HDB-
SCAN, to identify topic clusters with minimum cluster size
32. Finally, to obtain topic labels, we sample 10 prompts
from each topic cluster and feed into GPT-4-Turbo for topic
summarization.
The pipeline identifies 600 clusters covering a wide range of
topics including poetry writing, coding, math, and medical
queries. We present the top-16 topic clusters in Figure 3.
We observe that the largest cluster only accounts for 1%
of the entire set and the rest quickly drop to <0.5%, and
the similarity between clusters is small, showing a long-tail
and diverse distribution. Due to space limit, we present the
similarity matrix and cluster hierarchy of top-64 clusters in
Figure 11 and 12 in Appendix.
6.2. Can Arena Prompts Distinguish Models?
Next, we study how effective are these topic clusters in
distinguishing models strengths. Constructing challenging
prompts has become increasingly difficult due to LLMs’
3https://github.com/MaartenGr/BERTopicTable 2. GPT-4-0613’s win-rate against Llama-2-70b-chat on 30
sample prompts from various topic clusters. We use GPT-4-turbo
as judge to evaluate model responses in pairwise comparison.
Topic Cluster Win-rate Size
Python Game Programming Challenge 96.7% 0.2%
C/C++ Process Multi-Threading 86.7% 0.3%
SQL Query Database Assistance 73.3% 0.2%
Poetry Writing Prompts 66.7% 1.1%
Python Coding Basics 65.0% 0.2%
Linguistic Analysis & Wordplay 58.3% 0.7%
Travel Itinerary Planning 58.3% 0.4%
Movie Recommendations & Ratings 53.3% 0.2%
fast growing capabilities. For example, open models such as
Llama-2-70b-chat can likely answer inquiries about movie
or travel recommendation as good as GPT-4, but not in
other domains such as reasoning or coding. To demon-
strate, we sample 30 prompts from seven topic clusters and
compare the performance of Llama-2-70b-chat and GPT-4.
To control variables, we factor out user votes and consider
LLM-as-judge (Zheng et al., 2023b) to evaluate model re-
sponse. Results are shown in Table 2, where we see GPT-4
has significantly higher win-rate (up to 97%) in clusters that
require coding and reasoning skills. On the other hand, for
clusters with less problem-solving tasks, GPT-4 win-rate
drops to below 60%. We show examples in Appendix D.1.
This result shows models may exhibit varying strengths in
different areas, but also highlights some of the topic clusters
in Chatbot Arena are effective in differentiate models.
Building Challenging Benchmark. To further demonstrate
the prompt quality, we show it is possible to construct a chal-
lenging benchmark with crowd-sourced user prompts. To
ensure both topic coverage and quality, we first run the topic
modeling pipeline and follow a similar procedure in Zheng
et al. (2023a) to select challenging questions sampled from
each topic cluster. Examples prompts and evaluation proce-
dures can be found in the Appendix D.2 and Appendix D.3,
respectively. We observe the selected prompts are highly
effective in differentiating models. In Figure 4, we compare
Arena bench against a widely used LLM benchmark, MT-
Bench (Zheng et al., 2023b). We can see that Arena Bench
effectively reveals a significant gap in performance between
proprietary and the strongest open models.
6.3. Validating Vote Quality
To assess the quality of crowdsourced votes, we randomly
selected 160 battles between GPT-4-Turbo and Llama-2-
13B, as well as GPT-4-Turbo and GPT-3.5-Turbo-0613. We
then asked experts4to label their preference per comparison.
The experts were given the prompts and answers blindly,
and asked to carefully fact-check model’s answer with ex-
ternal resources like search engine. Manually labeling each
4The laborers are graduate students at UC Berkeley.
6
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
02468Llama-2-7B-ChatLlama-2-70B-ChatVicuna-33b-v1.3OpenChat-3.5Starling-LM-7B-alphaMixtral-8x7B-InstructClaude-2.1GPT-3.5-Turbo-0613GPT-4-0613GPT-4-0314GPT-4-Turbo Arena Bench
MT Bench
ScoreModel
Figure 4. Model’s performance between Arena Bench and MT-
Bench, showing an increased gap between open and proprietary
models. Both uses GPT-4 as judge.
Table 3. Pairwise agreement rate between crowd-user, gpt-4 judge,
and experts on pairwise battles. The top part of the table is between
GPT-4-Turbo and Llama-2-13b-chat. The bottom is between GPT-
4-Turbo and GPT-3.5-Turbo-0613.
Llama-2-13b Expert 1 Expert 2 GPT-4
Crowd 72.8% 77.8% 75.6%
Expert 1 - 89.8% 81.0%
Expert 2 - - 78.5%
GPT-3.5-Turbo Expert 1 Expert 2 GPT-4
Crowd 73.8% 83.1% 75.6%
Expert 1 - 79.4% 76.3%
Expert 2 - - 79.3%
data point took on average 3-5 minutes. For reference, we
also use GPT-4 as a judge for pairwise comparisons. The
agreement rate between crowd-users, experts, and GPT-4-
judge are presented in Table 3. The corresponsing win-rate
are shown in Table 4.
To summarize, we observe high agreement rates (72% to
83%) between Arena crowd-user and experts in both setup.
Note that agreement rates between two experts are around
similar levels (79.4% and 89.8%). As for the 10%-20%
disagreement between experts, it is mostly due to some user
prompts don’t have a ground truth answer. Depending on
the preference of the evaluator, sometimes both answers
can be argued as being better than the other one, such as
the examples in Appendix D.4. The gap between crowd-
vs-expert agreement rate and expert-vs-expert agreement
rate (5%-10%) is mostly attributed to crowd user making
mistakes or overlooking factual errors in model’s response.
Overall, the agreement rates presented in Table 3 validate
the decent quality of crowd-sourced votes in Chatbot Arena.Table 4. GPT-4-Turbo’s win-rate across crowd-user, gpt-4 judge,
and experts on pairwise battles against Llama-2-13b and GPT-3.5-
Turbo-0613.
Baseline Arena User Expert 1 Expert 2 GPT-4
Llama-2-13b 81.2% 89.4% 86.9% 78.8%
GPT-3.5-Turbo 76.3% 82.5% 89.4% 79.4%
0.0 0.5 1.0 1.5 2.0 2.5
zephyr-7b-beta (#19-37)wizardlm-13b (#19-37)llama2-70b-steerlm-chat (#17-37)solar-10.7b-instruct-v1.0 (#18-36)dolphin-2.2.1-mistral-7b (#17-37)pplx-70b-online (#17-31)gpt-3.5-turbo-1106 (#15-29)openchat-3.5 (#14-29)llama-2-70b-chat (#14-28)openhermes-2.5-mistral-7b (#12-28)vicuna-33b (#7-25)starling-lm-7b-alpha (#7-26)gpt-3.5-turbo-0314 (#7-22)wizardlm-70b (#7-22)tulu-2-dpo-70b (#6-21)yi-34b-chat (#6-19)claude-instant-1 (#6-19)gemini-pro (#6-18)gpt-3.5-turbo-0613 (#6-18)claude-2.1 (#6-18)mixtral-8x7b-instruct-v0.1 (#4-18)gemini-pro-dev-api (#3-18)claude-2.0 (#3-14)claude-1 (#3-8)mistral-medium (#3-8)gpt-4-0613 (#3-7)gpt-4-0314 (#2)gpt-4-turbo (#1)
corrected
uncorrected
Figure 5. Intervals for the BT coefficients with and without mul-
tiplicity correction. The multiplicity correction, in this case a
chi-square CLT interval, is technically required for the purpose of
calculating the ranking, because it ensures allscores are simulta-
neously contained in their intervals (and the ranking is a function
of all the scores). However, it induces extra conservatism, so we
report both intervals.
7. Experiments
7.1. Ranking system
Computing the rank on real data. In this section, we
report results from our experiments on approximate ranking.
For this experiment, we ran a replay of T= 213 ,576his-
torical votes from our online platform and calculate the BT
coefficients using our earlier-described estimation algorithm
with confidence intervals; see Figure 5 for these intervals
(with and without multiplicity correction; the formal notion
of approximate ranking technically requires multiplicity
correction, but it makes the intervals looser).
Evaluating the coverage of the intervals. A natural follow-
up question is whether or not the intervals are doing their job
correctly: whether they cover the true BT coefficients with
probability at least (and almost exactly) 1−α. Of course,
7
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
0 20000 40000 60000 80000 100000
n0.80.9CoverageCoverage
0 20000 40000 60000 80000 100000
n0.00.51.01.5Average Interval WidthAverage Interval Width
M
4
7
10
15
20
Figure 6. Intervals for the BT coefficients as a function of the
number of samples and the number of models M.
this cannot be evaluated on real data, so we run a simulation.
A vector of BT coefficients is drawn, with each coordinate
sampled i.i.d. from a distribution beta(1/γ,1/γ); we take
γ= 2in Figure 6 (and we vary γin Appendix A). Given
these coefficients, a dataset is synthesized, and the coverage
and average width are computed for each of 20 trials. The
results can be seen in Figure 6 for the uncorrected intervals
The coverage of the intervals behaves as expected, centering
around 1−α, regardless of the number of models. Mean-
while, the more models are included, the larger the intervals
become.
Evaluating the active sampling rule. Next, we discuss the
evaluation of our active sampling rule as Equation (9)for
win matrix estimation. We evaluate this sampling rule by
taking the best fit BT coefficients to our 213,576 point sized
holdout set, and then sampling from that distribution using
our active sampling algorithm. The results are displayed
in Figure 7. It is hard to tell by looking at plots, but the
improvement is substantial: To estimate θ∗to a precision of
0.2, random needs 6,800 samples and adaptive needs 4,400
samples; meanwhile to estimate the score to a precision
of 0.3, random needs 17,200 samples and adaptive needs
16,400 samples. Thus, the random baseline requires 54%
and 5% more data to achieve the same level of precision,
respectively. One can see from the plots in Figure 7 that
these results are not cherry-picked: the sample-efficiency of
our method is better at all values on the horizontal axis.
7.2. Anomalous Users Detection
We evaluate the outlier detection method in Section 5.1.
We construct the evaluation set by manually identifying
25 anomalous users whose inputs are highly repetitive or
meaningless (e.g., asking “hi” for 100 times or inputting
garbled texts). We randomly sample 25 normal users with
0.10 0.12 0.14 0.16 0.18 0.20 0.22 0.24
average width ()
0100002000030000nrandom
pairwise adaptive
0.25 0.30 0.35 0.40 0.45 0.50
average width (s)0100002000030000nFigure 7. Interval widths on the win matrix (upper figure) and on
the BT coefficients (lower figure) as a function of the number of
samples, for random sampling and also adaptive sampling. Im-
provements from adaptive sampling can be seen in both cases,
although they are more subtle on the scale of the score.
Table 5. Confusion matrix of different α. “Pred.” means predicted.
Positive means anomalous and negative means normal.
α= 0.1 Pred. Positive Pred. Negative
Actual Positive 13/14 12/36
Actual Negative 1/14 24/36
α= 0.3 Pred. Positive Pred. Negative
Actual Positive 21/29 4/21
Actual Negative 8/29 17/21
at least 50 votes, and inspect their input prompts to ensure
no abnormal behaviors. As mentioned in Section 5.1, per
user we compute five Mjand identify the user as anomalous
ifMj≥χ2
2j,1−α/5. We present results of two different α
(i.e., the significance leval) in Table 5. We find the detec-
tion method effective (e.g., reaching 90% true positive and
60-70% true negative rate). We inspect the false negative
errors and find those are from users do not always behave
abnormally, making them harder to detect.
8. Discussion
Limitations. Although our user base is extensive, we an-
ticipate that it will primarily consist of LLM hobbyists and
researchers who are eager to experiment with and evaluate
the latest LLMs. This inclination may result in a biased
distribution of users. Additionally, despite the wide array of
topics encompassed by the prompts discussed in previous
sections, the data predominantly comes from our online
chat interface. This source might not accurately reflect the
real-world usage of LLMs in production environments or
specialized domains, potentially leading to a skewed prompt
distribution. Moreover, our study concentrates on assessing
the helpfulness of LLMs but overlooks their safety aspects.
We recognize the possibility and necessity of a parallel
8
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
mechanism to evaluate the safety of these models.
Future Directions. In our future work, we plan to develop
comprehensive topic leaderboards and establish a dedicated
section for multimodal and agent-based LLMs in more dy-
namic, gamified settings, catering to more complex tasks.
We also believe our approach to detecting harmful users
could be improved and made more formally rigorous by
using the theory of nonnegative supermartingales and E-
values (Howard et al., 2020; Waudby-Smith & Ramdas,
2020; V ovk & Wang, 2021; Ramdas et al., 2023); this would
deal with the dependence, but the variants we tried did not
perform well in terms of power.
9. Conclusion
In this paper, we present Chatbot Arena, an open platform
for evaluating LLMs through crowdsourced, pairwise hu-
man preferences. We conduct an in-depth analysis of the
crowdsourced user prompts and preference votes to validate
the diversity and quality. We develop an efficient model
sampling and ranking algorithm. Our dataset including
100K pairwise preference votes will be released for future
research.
Acknowledgments
This project is supported by sponsorship from Kaggle,
MBZUAI, a16z, Together AI, Anyscale, and HuggingFace.
This project is also partly supported by Accenture, AMD,
Google, IBM, Intel, Microsoft, Samsung SDS, SAP, Uber,
and VMware. The authors would like to thank Siyuan
Zhuang for insightful discussion and Tijana Zrni ´c for helpful
feedback on the manuscript.
References
Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-
Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T.,
et al. Training a helpful and harmless assistant with rein-
forcement learning from human feedback. arXiv preprint
arXiv:2204.05862 , 2022.
Boubdir, M., Kim, E., Ermis, B., Hooker, S., and Fadaee, M.
Elo uncovered: Robustness and best practices in language
model evaluation, 2023.
Bradley, R. A. and Terry, M. E. Rank analysis of incom-
plete block designs: I. the method of paired comparisons.
Biometrika , 39(3/4):324–345, 1952.
Busa-Fekete, R., Huellermeier, E., and Szörényi, B.
Preference-based rank elicitation using statistical models:
The case of mallows. In Xing, E. P. and Jebara, T. (eds.),
Proceedings of the 31st International Conference on Ma-
chine Learning , volume 32 of Proceedings of MachineLearning Research , pp. 1071–1079, Bejing, China, 22–
24 Jun 2014a. PMLR. URL https://proceedings.
mlr.press/v32/busa-fekete14.html .
Busa-Fekete, R., Huellermeier, E., and Szörényi, B.
Preference-based rank elicitation using statistical models:
The case of mallows. In Xing, E. P. and Jebara, T. (eds.),
Proceedings of the 31st International Conference on Ma-
chine Learning , volume 32 of Proceedings of Machine
Learning Research , pp. 1071–1079, Bejing, China, 22–24
Jun 2014b. PMLR. URL https://proceedings.
mlr.press/v32/busa-fekete14.html .
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O.,
Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman,
G., et al. Evaluating large language models trained on
code. arXiv preprint arXiv:2107.03374 , 2021.
Chernoff, H. Sequential Design of Experiments , pp.
345–360. Springer New York, New York, NY ,
1992. ISBN 978-1-4612-4380-9. doi: 10.1007/
978-1-4612-4380-9_27. URL https://doi.org/
10.1007/978-1-4612-4380-9_27 .
Chiang, C.-H. and Lee, H.-y. Can large language mod-
els be an alternative to human evaluations? In Rogers,
A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceed-
ings of the 61st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers) , pp.
15607–15631, Toronto, Canada, July 2023. Association
for Computational Linguistics. doi: 10.18653/v1/2023.
acl-long.870. URL https://aclanthology.org/
2023.acl-long.870 .
Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano,
R., et al. Training verifiers to solve math word problems.
arXiv preprint arXiv:2110.14168 , 2021.
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y ., Xie, G.,
Liu, Z., and Sun, M. Ultrafeedback: Boosting language
models with high-quality feedback, 2023.
DiCiccio, T. J. and Efron, B. Bootstrap confidence intervals.
Statistical science , 11(3):189–228, 1996.
Durrett, R. Probability: theory and examples , volume 49.
Cambridge university press, 2019.
Elo, A. E. The proposed uscf rating system, its develop-
ment, theory, and applications. Chess Life , 22(8):242–
247, 1967.
Fisher, R. A. Statistical methods for research workers . Num-
ber 5. Oliver and Boyd, 1928.
Freedman, D. A. On the so-called “huber sandwich esti-
mator”’ and “robust standard errors”’. The American
Statistician , 60(4):299–302, 2006.
9
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Gemini, T., Anil, R., Borgeaud, S., Wu, Y ., Alayrac, J.-B.,
Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A.,
et al. Gemini: a family of highly capable multimodal
models. arXiv preprint arXiv:2312.11805 , 2023.
Geng, X., Gudibande, A., Liu, H., Wallace, E., Abbeel,
P., Levine, S., and Song, D. Koala: A dia-
logue model for academic research. Blog post,
April 2023. URL https://bair.berkeley.edu/
blog/2023/04/03/koala/ .
Grootendorst, M. Bertopic: Neural topic modeling
with a class-based tf-idf procedure. arXiv preprint
arXiv:2203.05794 , 2022.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M.,
Song, D., and Steinhardt, J. Measuring massive multitask
language understanding. In International Conference on
Learning Representations , 2020.
Howard, S. R., Ramdas, A., McAuliffe, J., and Sekhon,
J. Time-uniform chernoff bounds via nonnegative super-
martingales. 2020.
Huang, Y ., Lin, Z., Liu, X., Gong, Y ., Lu, S., Lei, F., Liang,
Y ., Shen, Y ., Lin, C., Duan, N., et al. Competition-level
problems are effective llm evaluators. arXiv preprint
arXiv:2312.02143 , 2023.
Huber, P. J. et al. The behavior of maximum likelihood
estimates under nonstandard conditions. In Proceedings
of the fifth Berkeley symposium on mathematical statistics
and probability , volume 1, pp. 221–233. Berkeley, CA:
University of California Press, 1967.
Hunter, D. R. MM algorithms for generalized Bradley-
Terry models. The Annals of Statistics , 32(1):384 – 406,
2004. doi: 10.1214/aos/1079120141. URL https:
//doi.org/10.1214/aos/1079120141 .
Karimi, M. R., Gürel, N. M., Karlaš, B., Rausch, J., Zhang,
C., and Krause, A. Online active model selection for
pre-trained classifiers. In International Conference on
Artificial Intelligence and Statistics , pp. 307–315. PMLR,
2021.
Karpinska, M., Akoury, N., and Iyyer, M. The perils of
using Mechanical Turk to evaluate open-ended text gen-
eration. In Moens, M.-F., Huang, X., Specia, L., and
Yih, S. W.-t. (eds.), Proceedings of the 2021 Confer-
ence on Empirical Methods in Natural Language Pro-
cessing , pp. 1265–1285, Online and Punta Cana, Domini-
can Republic, November 2021. Association for Computa-
tional Linguistics. doi: 10.18653/v1/2021.emnlp-main.
97. URL https://aclanthology.org/2021.
emnlp-main.97 .Kiela, D., Bartolo, M., Nie, Y ., Kaushik, D., Geiger, A., Wu,
Z., Vidgen, B., Prasad, G., Singh, A., Ringshia, P., et al.
Dynabench: Rethinking benchmarking in nlp. In Pro-
ceedings of the 2021 Conference of the North American
Chapter of the Association for Computational Linguistics:
Human Language Technologies , pp. 4110–4124, 2021.
Köpf, A., Kilcher, Y ., von Rütte, D., Anagnostidis, S.,
Tam, Z.-R., Stevens, K., Barhoum, A., Duc, N. M., Stan-
ley, O., Nagyfi, R., et al. Openassistant conversations–
democratizing large language model alignment. arXiv
preprint arXiv:2304.07327 , 2023.
Langley, P. Crafting papers on machine learning. In Langley,
P. (ed.), Proceedings of the 17th International Conference
on Machine Learning (ICML 2000) , pp. 1207–1216, Stan-
ford, CA, 2000. Morgan Kaufmann.
Li, X., Zhang, T., Dubois, Y ., Taori, R., Gulrajani, I.,
Guestrin, C., Liang, P., and Hashimoto, T. B. Alpacae-
val: An automatic evaluator of instruction-following
models. https://github.com/tatsu-lab/
alpaca_eval , 2023.
Li, Y ., Choi, D., Chung, J., Kushman, N., Schrittwieser, J.,
Leblond, R., Eccles, T., Keeling, J., Gimeno, F., Dal Lago,
A., et al. Competition-level code generation with alpha-
code. Science , 378(6624):1092–1097, 2022.
Liang, P., Bommasani, R., Lee, T., Tsipras, D., Soylu, D.,
Yasunaga, M., Zhang, Y ., Narayanan, D., Wu, Y ., Kumar,
A., et al. Holistic evaluation of language models. arXiv
preprint arXiv:2211.09110 , 2022.
Lin, Z., Wang, Z., Tong, Y ., Wang, Y ., Guo, Y ., Wang,
Y ., and Shang, J. ToxicChat: Unveiling hidden chal-
lenges of toxicity detection in real-world user-AI con-
versation. In Bouamor, H., Pino, J., and Bali, K.
(eds.), Findings of the Association for Computational
Linguistics: EMNLP 2023 , pp. 4694–4702, Singa-
pore, December 2023. Association for Computational
Linguistics. doi: 10.18653/v1/2023.findings-emnlp.
311. URL https://aclanthology.org/2023.
findings-emnlp.311 .
Liu, T.-Y . et al. Learning to rank for information retrieval.
Foundations and Trends ®in Information Retrieval , 3(3):
225–331, 2009.
McInnes, L., Healy, J., and Melville, J. Umap: Uniform
manifold approximation and projection for dimension
reduction, 2020.
OpenAI. Gpt-4 technical report. arXiv preprint
arXiv:2303.08774 , 2023.
10
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Oren, Y ., Meister, N., Chatterji, N., Ladhak, F., and
Hashimoto, T. B. Proving test set contamination in black
box language models. arXiv preprint arXiv:2310.17623 ,
2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright,
C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K.,
Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L.,
Simens, M., Askell, A., Welinder, P., Christiano, P., Leike,
J., and Lowe, R. Training language models to follow
instructions with human feedback, 2022.
Ramdas, A., Grünwald, P., V ovk, V ., and Shafer, G. Game-
theoretic statistics and safe anytime-valid inference. Sta-
tistical Science , 38(4):576–601, 2023.
Rao, P. V . and Kupper, L. L. Ties in paired-comparison
experiments: A generalization of the bradley-terry model.
Journal of the American Statistical Association , 62(317):
194–204, 1967. doi: 10.1080/01621459.1967.10482901.
Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid,
A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A.,
Garriga-Alonso, A., et al. Beyond the imitation game:
Quantifying and extrapolating the capabilities of language
models. Transactions on Machine Learning Research ,
2023.
Szörényi, B., Busa-Fekete, R., Paul, A., and Hüller-
meier, E. Online rank elicitation for plackett-
luce: A dueling bandits approach. In Cortes, C.,
Lawrence, N., Lee, D., Sugiyama, M., and Garnett,
R. (eds.), Advances in Neural Information Process-
ing Systems , volume 28. Curran Associates, Inc.,
2015. URL https://proceedings.neurips.
cc/paper_files/paper/2015/file/
7eacb532570ff6858afd2723755ff790-Paper.
pdf.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P.,
Bhosale, S., et al. Llama 2: Open foundation and fine-
tuned chat models. arXiv preprint arXiv:2307.09288 ,
2023.
V ovk, V . and Wang, R. E-values: Calibration, combination
and applications. The Annals of Statistics , 49(3):1736–
1754, 2021.
Wang, Y ., Kordi, Y ., Mishra, S., Liu, A., Smith, N. A.,
Khashabi, D., and Hajishirzi, H. Self-instruct: Align-
ing language models with self-generated instructions. In
Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Pro-
ceedings of the 61st Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers) ,
pp. 13484–13508, Toronto, Canada, July 2023. Associ-
ation for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.754. URL https://aclanthology.
org/2023.acl-long.754 .
Waudby-Smith, I. and Ramdas, A. Estimating means of
bounded random variables by betting. arXiv preprint
arXiv:2010.09686 , 2020.
White, H. Maximum likelihood estimation of misspeci-
fied models. Econometrica: Journal of the econometric
society , pp. 1–25, 1982.
Yang, S., Chiang, W.-L., Zheng, L., Gonzalez, J. E., and
Stoica, I. Rethinking benchmark and contamination for
language models with rephrased samples. arXiv preprint
arXiv:2311.04850 , 2023.
Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi,
Y . Hellaswag: Can a machine really finish your sentence?
InProceedings of the 57th Annual Meeting of the Asso-
ciation for Computational Linguistics , pp. 4791–4800,
2019.
Zheng, L., Chiang, W.-L., Sheng, Y ., Li, T., Zhuang, S., Wu,
Z., Zhuang, Y ., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E.,
Stoica, I., and Zhang, H. Lmsys-chat-1m: A large-scale
real-world llm conversation dataset, 2023a.
Zheng, L., Chiang, W.-L., Sheng, Y ., Zhuang, S., Wu, Z.,
Zhuang, Y ., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H.,
Gonzalez, J. E., and Stoica, I. Judging LLM-as-a-judge
with MT-bench and chatbot arena. In Thirty-seventh
Conference on Neural Information Processing Systems
Datasets and Benchmarks Track , 2023b. URL https:
//openreview.net/forum?id=uccHPGDlao .
Zhong, W., Cui, R., Guo, Y ., Liang, Y ., Lu, S., Wang,
Y ., Saied, A., Chen, W., and Duan, N. Agieval: A
human-centric benchmark for evaluating foundation mod-
els.arXiv preprint arXiv:2304.06364 , 2023.
Zhu, B., Frick, E., Wu, T., Zhu, H., and Jiao, J. Starling-
7b: Improving llm helpfulness & harmlessness with rlaif,
November 2023.
11
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Figure 8. Screenshot of Chatbot Arena.
4708
435043203838
3794
370634123295
32013144
29342882276027012506
2480
2423
23362294
22522245
222321382101
20581974
19201919
1892185518461832
1783
1770
17671758
1729
171916811657
1656165316441632
1631162016121587
15851578
157315731571
1556
15541529
15131491
1435
14351433
1393132513221316
12841268
1265
125612521242
1233
121212041200
11771176
1163
11441143
112811211108
110710921081
1056
1035
1014970
966961951
932874
848825
813805
786786782
779
768
743729724
722
708705
688
684
667662
657
657
651650640
628
627627626625
613598
580566563
549
548548
539537537536535
535531530
529
525518
516
515
514509
506
504501
489483
482481
480
476472
472469
465
444
442
441439
439
439432
431
428425421
418412409
408406
405404
396394
393384
382380377
372
368366364
362
359
357354352
343341
340340
338
337328
327326326
326325
324
323321
312310310
309301
301299
296
293293
290288
285278
266256248
246245239
233
228225
223222222220
216216206
205200
199
194191
186184184
172149144139
138
134134
131125
125124
123120 112 111 106 103 1019897
9692 8965 60 56
May 2023 Jun 2023 Jul 2023 Aug 2023 Sep 2023 Oct 2023 Nov 2023 Dec 2023 Jan 202401000200030004000
Dates
Figure 9. The number of votes over time
12
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
297792918127726228491775617562175131696815684147841389013852118261172011209104719562813579297686742073297309726670187008671162066018598559615 6 8 8
gpt-3.5-turbo
-0613gpt-4-turbo gpt-4-0613 claude-2.1 gpt-4-0314 claude-instant-1 claude-1 vicuna-33b vicuna-13b llama-2-70b-chat mixtral-8x7b-instruct-
v0.1gpt-3.5-turbo
-1106claude-2.0 llama-2-13b-chat zephyr
-7b-betamistral-medium palm-2 wizar
dlm-70bwizar
dlm-13bvicuna-7b k oala-13b openchat-3.5 mistral-7b-instruct llama-2-7b-chat gemini-pr
ocodellama-34b-instruct oasst-pythia-12b alpaca-13b pplx-70b-
onlinegemini-pr
o-dev
-apigpt-3.5-turbo
-0314yi-34b-chat010k20k30k
Figure 10. The number of votes per model.
Inflation & Monetary P olicyDock er Syslog Service ConfigurationSQL Query Database AssistanceIsrael-Hamas Conflict DynamicsAlternate WWI V ictorious Ger manyBiblical Interpr etation and TheologySimplif ying Quantum MechanicsSustainable Mobility and Solar Ener gyMark et Innovation & Platfor msOvercoming Emotional P aralysisTriangle Angle Bisector Ar eaApple Quantity CalculationsChemical R eactions and CalorimetryChess Notation Magnus vs Hik aruVideogame R ecommendations & InsightsPyTorch AutoEncoder T raining IssuesHigh-P erformance Computing Har dwar eLLM P rompt OptimizationComple x AI R oleplay GuidelinesAI Image Generation P romptsJupiter's Moons Astr onomyTravel Itinerary PlanningGame Engines and Player MovementPython Coding BasicsPython Game P rogramming ChallengesC/C++ P rocess Multi- ThreadingPotter & Rings Cr ossoverErotic Short StoriesMagic & F antasy RPG CampaignsLinguistic Analysis & W ordplayRequesting Original Jok esSong L yrics Analysis
0.10.20.30.40.50.60.70.80.91Similarity Scor eSimilarity Matrix
Figure 11. Similarity matrix of top-64 topic clusters.
13
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
0 0.2 0.4 0.6 0.8 1 1.2Unique Social Gaming Branding
YouTube Content Strategy
Simulated Unfilter ed AIs
AGI Resear ch and P rogress P rediction
Trade R eporting and UA T Coor dination
Operations & Fleet Management
Educational Assessment Strategies
Meaning of Life Inquiry
Philosophical T exts & Concepts
Biblical Theology & Doctrines
Medical Case Summaries
Understanding Depr ession V arieties
Greetings and P ersonal Inquiries
Relationship Communication Issues
Cats and Dog Br eeds
Jessica's R esilient R omance
Medieval F antasy Quest Nar rative
Cyberpunk Sci-F i Narratives
Movie R eviews and Discussions
AI Image P rompt Craf ting
Anime Evolution & R elationships
Israel-Hamas Conflict Analysis
Antisemitism and Historical Inaccuracies
US Presidents Inquiry
Global Geography & Demographics
Simplified Quantum Mechanics Explained
Advanced Mathematical Concepts
Chemical R eactions and Calorimetry
Regression Analysis Essentials
Math Or der of Operations
Sports R ankings and Analysis
Fed Inflation R ate Decisions
Automotive Insights and Comparisons
PC Components & Upgrades
Deter mining T oday's Day P uzzles
Trip Itinerary Planning
Weight Comparison Bricks F eathers
Body R ecomposition W eekly W orkout
Cooking and R ecipes
Horticultur e & Plant Car e
Societal Ideologies & W elfare
Apple Count A fter Eating
Music Discovery & R ecommendations
Guitar Chor ds and Music Theory
Chess Gameplay and Strategies
Riddles Cr eation and Solutions
Poetry W riting & Styles
Original Jok e Requests
Japanese Language T ranslation
Word Play and Phonetics
German Language P ractice
Hexadecimal Message Decryption
ASCII Art Cr eations
Linux vs W indows Comparison
Linux T erminal Command Simulation
Rust P rogramming Essentials
SQL Database T able Queries
Web Development Essentials
AWS Terraform Deployment & Services
Pandas DataF rame Operations
PyTorch Neural Networks
Transfor mer A ttention Mechanisms
Large Language Models Analysis
Understanding LLMs and T rainingHierarchical Clustering
Figure 12. Top-64 clusters visualized in hierarchy. x-axis represents the cosine similarity distance. y-axis shows the topic title per cluster
summarized by gpt-4-turbo.
14
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
llama-2-7b-chat (#7-7)llama-2-13b-chat (#6-7) llama-2-70b-chat (#3-4)gpt-3.5-turbo-0613 (#3-3)claude-2.1 (#3-3)
mixtral-8x7b-instruct-v0.1 (#3-3)gpt-4-0613 (#2-2) gpt-4-turbo (#1-1)0.000.250.500.751.001.25
Intervals (n=23968)
Sandwich
Bootstrap
12000 18000 24000
n0.880.900.920.940.960.981.00CoverageCoverage
12000 18000 24000
n0.180.200.220.240.260.28Average Interval WidthAverage Interval Width
method
sandwich
bootstrap
Figure 13. Replay experiment showing the intervals, coverage, and average interval sizes of the bootstrap and of the sandwich intervals.
The sandwich intervals, though larger in small samples, are more stable, and in large samples, they actually become smaller. We use the
multiplicity corrected version of both intervals, so they both have a coverage of 1. (Coverage here is calculated with respect to the BT
coefficient solution on the full dataset, so it is not as meaningful as in the simulation plot below.)
1 5 9 13 17
M0.00.51.01.52.0
Intervals (n=100000)
Standard
Bootstrap
0 20000 40000 60000 80000 100000
n0.8000.8250.8500.8750.9000.9250.9500.975CoverageCoverage
0 20000 40000 60000 80000 100000
n0.20.40.60.81.01.21.41.6Average Interval WidthAverage Interval Width
method
sandwich
bootstrap
Figure 14. Synthetic experiment. Coefficients are drawn from the BT-coefficient distribution xon the left. Coverage of the uncorrected
intervals is shown in the middle. Line plots of set width are shown on the right, and they almost perfectly match.
A. Confidence Interval Simulation Study
We conduct a simulation study to evaluate the bootstrap confidence intervals versus the sandwich estimator. To a large extent,
both intervals are the same—indeed, their intervals are often identical to the naked eye. Nonetheless, in our experiments,
there are some differences. First, in Figure 13, we conduct a replay study using the same 213576 data points mentioned in
the main text.
We also do a suite of experiments in simulation using the same beta generating process as in the main text, with γ= 2.
The result is shown in Figure 14; results are similar across many choices of the parameter γand the model strength, which
indicates that both intervals will have good coverage and width in the practical conditions we would expose them to.
B. The Nonparametric Bradley-Terry Model
Nonparametric Bradley-Terry. We next consider a nonparametric extension of the Bradley-Terry (BT) model (Bradley &
Terry, 1952) to the case where the ranking is not necessarily transitive. Let G(m)denote the set of all paths to the model m,
i.e.,
G(m) =
g∈ BM−1:gi,1̸=gj,1,∀i̸=j,andgM−1,2=m	
, (11)
where B=A ∪ { (a2, a1) :a∈ A} . Each element of G(m)is a chain of model pairings that leads to m; for example, if
m= 5andM= 6, one element of G(m)is((1,2),(2,4),(4,3),(3,6),(6,5)). Our score function is given by the average
15
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
path-sum of the log odds of the second model winning, over the entirety of G(m):
s(θ)m=1
|G(m)|X
g∈G(m) 
logθ′((1, g1,1))
1−θ′((1, g1,1))+X
a∈glogθ′(a)
1−θ′(a)!
, (12)
where θ′(a) =θ(a)1{a∈ A} + (1−θ((a2, a1)))1{a /∈ A} , with the convention that θ((m, m )) = 1 /2for all m. Note
that for any g∈ G(m)where a∈gandm /∈a, we also have some g′∈ G(m)such that (a2, a1)∈g. Meanwhile, if a∈g
andm∈a, then a= (m′, m)for some m′. Thus, we can compute
s(θ)m=X
a∈A
m/∈a1
2
logθ′(a)
1−θ′(a)+ logθ′((a2, a1))
1−θ′((a2, a1))
+X
m′∈[M]\{m}
logθ′((m′, m))
1−θ′((m′, m))+θ′((1, m′))
1−θ′((1, m′))
(13)
=X
a∈A
m/∈a1
2
logθ(a)
1−θ(a)+ log1−θ(a)
θ(a)
+X
m′∈[M]\{m}
logθ′((m′, m))
1−θ′((m′, m))+θ′((1, m′))
1−θ′((1, m′))
(14)
=X
m′∈[M]\{m}
logθ′((m′, m))
1−θ′((m′, m))+θ′((1, m′))
1−θ′((1, m′))
(15)
=X
m′∈[M]\{m}
(1−21{m′> m}) logθ((m′, m))
1−θ((m′, m))+θ((1, m′))
1−θ((1, m′))
. (16)
This score is always well-defined, and is a simple, smooth function of θ. Its derivative is, for all a∈ A,
∂
∂θ(a)s(θ)m=1{a2=m}(1−21{a1> m})1
θ(a)(1−θ(a))+1{a1= 1, a2̸=m}1
θ(a)(1−θ(a)).(17)
How is the BT score related to the original Bradley-Terry model? In the original Bradley-Terry model, Ht∈ {0,1}, and the
probability of model mbeating model m′is assumed to be given by
θ((m′, m)) =eξm
eξm+eξm′, (18)
for some unknown parameters ξ1, . . . , ξ M—the Bradley-Terry coefficients . The basic goal of the Bradley-Terry model is to
estimate these parameters from the observed outcomes. In our setting, however, we use the outcomes to get a CLT on θ, and
then can immediately recover the coefficients. Taking without loss of generality ξ1= 0, we have that
logθ((1, m′))
1−θ((1, m′))+ logθ((m′, m))
1−θ((m′, m))= logθ((1, m′))
θ((m′,1))+ logθ((m′, m))
θ((m, m′))(19)
= logeξm′(eξm′+ 1)
eξm′+ 1+ logeξm(eξm′+eξm)
eξm′(eξm′+eξm)(20)
=ξm′+ξm−ξm′=ξm (21)
Thus, all the sums over paths in (12) are equal to ξm−ξg1,1.
logθ′((1, g1,1))
1−θ′((1, g1,1))+X
a∈glogθ′(a)
1−θ′(a)(22)
=ξg1,1+ξg1,2−ξg1,1+ξg2,2−ξg2,1+···+ξgM−1,2−ξgM−1,1 (23)
=ξgM−1,2=ξm. (24)
Thus, if the parametric BT model is well-specified, the nonparametric version will exactly recover the Bradley-Terry
coefficients. However, our nonparametric analogue of the BT model has major advantages over the original: it will retain
statistical validity even if Htis not binary, if the win rate is non-transitive, and if the logistic model assumed by the BT
model is misspecified. In practice, the nonparametric BT coefficient can be easily computed by (16).
16
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
C. Valid P-Value
Consider the p-value
pi=1
|HA′
i|+ 1
1 +X
h∈HA′
i1{h≥H′
i}
. (25)
We will prove that this p-value is valid, i.e., that P(pi≤t)≤t, under the null hypothesis that the vector H′= (Ht:At=
A′
i)∥(H′
i)is exchangeable, where ∥denotes the concatenation operator. First, notice that we can equivalently write
pi=1
|H′|X
h∈H′1{h≥H′
i}. (26)
We also have that P(pi≤t)≤P(pi≤k
|H′|), where k=⌊t|H′|⌋/|H′|. Then, since the data points are exchangeable, we
have that H′
iis uniformly distributed among the ranks of H′, soP(pi≤k
|H′|)≤k
|H′|≤t, completing the proof.
17
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
D. Sample Prompts
D.1. Prompt Examples From Different Clusters
Example 1: An example user prompt from the cluster "Python Game Programming Challenge". This prompt distinctively
separates GPT-4-0613 performance from Llama-2-70b-chat. The user request "higher quality code“ from each assistant.
While GPT-4-Turbo provides a correct implementation along with helpful explanation, Llama-2-70b-chat fail to provide any
program and provided a less helpful response overall.
<|User Prompt|>
Develop a snowman game similar to the Hangman game but draws a snowman
figure standing on the snow instead of a hangman. It also downloads (or
uses library) the complete English dictionary to select words for three
levels of difficulty. Think step-by-step, take a deep breath and choose
a higher quality code for each step.
<|GPT-4-0613’s Answer|>
Sure, here’s a step-by-step guide on how to create a Snowman game similar to
Hangman. We’ll use Python for this example.
Step 1: Import necessary libraries
‘‘‘python
import random
import urllib.request
‘‘‘
Step 2: Download the English dictionary
‘‘‘python
url = "https://raw.githubusercontent.com/dwyl/english-words/master/words.txt
"
urllib.request.urlretrieve(url, ’words.txt’)
‘‘‘
Step 3: Load the words from the dictionary and categorize them into three
levels of difficulty
‘‘‘python
with open(’words.txt’) as word_file:
english_words = word_file.read().split()
easy_words = [word for word in english_words if 4 <= len(word) <= 6]
medium_words = [word for word in english_words if 7 <= len(word) <= 9]
hard_words = [word for word in english_words if len(word) >= 10]
‘‘‘
Step 4: Create a function to randomly select a word based on the difficulty
level
‘‘‘python
def get_random_word(difficulty):
if difficulty == ’easy’:
word = random.choice(easy_words)
elif difficulty == ’medium’:
word = random.choice(medium_words)
else:
word = random.choice(hard_words)
return word.upper()
‘‘‘
Step 5: Create a function to print the snowman
18
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
‘‘‘python
def print_snowman(guesses):
snowman = [’ _____ ’, ’ ( ) ’, ’ ( ( ) ) ’, ’(___(___)_)’, ’
( ) ’, ’ ( ) ’, ’ ( ) ’, ’ ( ) ’, ’(
)’, ’ _______ ’]
for i in range(guesses):
print(snowman[i])
‘‘‘
Step 6: Create the main function for the game
‘‘‘python
def play_snowman():
difficulty = input("Choose difficulty (easy, medium, hard): ")
word = get_random_word(difficulty)
guessed_letters = []
guesses = 0
print(’The word contains’, len(word), ’letters.’)
while guesses < 10:
guess = input("Guess a letter: ").upper()
if guess in guessed_letters:
print("You already guessed that letter.")
elif guess in word:
guessed_letters.append(guess)
print("Good job! The word contains that letter.")
else:
guesses += 1
print_snowman(guesses)
print("Sorry, the word does not contain that letter.")
print(’ ’.join(letter if letter in guessed_letters else ’_’ for
letter in word))
if set(guessed_letters) == set(word):
print("Congratulations! You’ve successfully guessed the word.")
break
else:
print("Sorry, you didn’t guess the word. The word was", word)
‘‘‘
Step 7: Run the game
‘‘‘python
play_snowman()
‘‘‘
This is a simple version of the game. You can enhance it by adding more
features like scoring, time limit, etc.
‘‘‘
19
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
<|Llama-2-70b-chat’s Answer|>
Sure! Here’s a step-by-step guide to developing a snowman game similar to
Hangman, using the complete English dictionary for three levels of
difficulty.
Step 1: Setting up the project
*Create a new Python project directory and initialize it with Git.
*Install the necessary libraries:
+ ‘numpy‘ for generating random numbers
+ ‘nltk‘ for accessing the English dictionary
+ ‘pygame‘ for creating the game graphics
+ ‘pydub‘ for playing sound effects (optional)
*Download the English dictionary dataset (e.g., from Kaggle) and store it
in a file named ‘dictionary.txt‘.
Step 2: Loading the dictionary and generating words
*Use the ‘nltk‘ library to read the dictionary file and store the words in
a list.
*Split the list into three parts: easy, medium, and hard words, based on
their length and frequency of use.
*Shuffle the lists to randomize the order of the words.
Step 3: Creating the game graphics
*Use ‘pygame‘ to create a window for the game, with a white background to
represent the snow.
*Draw a snowman figure on the background, using black rectangles for the
body, hat, and scarf.
*Add buttons for the player to submit their guesses.
Step 4: Implementing the game logic
*Create a function to generate a random word from the appropriate
difficulty level and display it on the screen with missing letters (e.g
., _ _ _ _ _ _).
*Create a function to check if a guessed letter appears in the word, and
update the display accordingly (e.g., _ _ _ X _ _).
*Create a function to handle incorrect guesses, such as displaying a
warning message or playing a sound effect.
*Create a function to win the game, such as displaying a congratulations
message or playing a victory sound effect.
Step 5: Adding sound effects (optional)
*Use ‘pydub‘ to play sound effects for various actions in the game, such as
guessing a letter correctly or incorrectly.
*Consider using a library like ‘pygame_mixer‘ to simplify the process of
playing sound effects in pygame.
20
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Step 6: Testing and refining the game
*Test the game thoroughly to ensure that it works as expected, with no bugs
or errors.
*Refine the game mechanics and graphics as needed, based on player feedback
or personal preference.
*Consider adding additional features, such as keeping track of the player’s
score or implementing a time limit for each guess.
Step 7: Packaging and distributing the game
*Package the game as a standalone executable file using a tool like
PyInstaller.
*Distribute the game to friends, family, or online communities, and gather
feedback for future improvements.
This is a high-level overview of the steps involved in creating a snowman
game similar to Hangman. Of course, the details will depend on your
specific implementation and the tools you choose to use. Good luck with
your project!
<|GPT-4-Turbo’s Judgment Verdict|>
GPT-4-0613 Wins
Program Output: Below is the terminal output of the snowman game programmed by GPT-4-0613, demonstrating its
correctness. It is clear that GPT-4-0613’s response is more helpful and relevant to the user’s request.
>>> python snowman.py
Choose difficulty (easy, medium, hard): easy
The word contains 6 letters.
Good job! The word contains that letter.
_ A _ _ _ _
Guess a letter: e
Good job! The word contains that letter.
_ A _ _ _ E
Guess a letter: d
_____
( )
Sorry, the word does not contain that letter.
_ A _ _ _ E
Guess a letter: g
_____
( )
( ( ) )
21
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Example 2: An example user prompt from the cluster "Python Game Programming Challenge". This prompt distinctively
separates GPT-4-0613 performance from Llama-2-70b-chat. Due to limited space, the code from each assistant’s response is
omitted.
<|User Prompt|>
Write a python function that prints a christmas tree in the form of a
triangle of asterisks, where the only input is size.
<|GPT-4-0613’s Code Output|>
>>> print_christmas_tree(5)
∗
∗ ∗ ∗
∗ ∗ ∗ ∗ ∗
∗ ∗ ∗ ∗ ∗ ∗ ∗
∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗ ∗
<|Llama-2-70b-chat’s Code Output|>
>>> print_christmas_tree(5)
* * * * *
* * * *
* * *
* *
*
<|GPT-4-Turbo Verdict|>
GPT-4-0613 Wins
Example 3: An example user prompt from the cluster "Movie Recommendations & Rating". Both assistant retrieve the
correct movie information and gave an almost identical explanation. Therefore, this prompt cannot distinguish GPT-4-0613’s
performance from Llama-2-70b-chat.
<|User Prompt|>
Which film came out first, Blind Shaft or The Mask Of Fu Manchu?
<|GPT-4-0613’s Answer|>
The Mask of Fu Manchu was released in 1932, and Blind Shaft was released in
2003. Therefore, The Mask of Fu Manchu came out first.
<|Llama-2-70b-chat’s Answer|>
The Mask Of Fu Manchu came out first. It was released in 1932, while Blind
Shaft was released in 2003.
<|GPT-4-Turbo Verdict|>
Tie
22
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
Example 4: An example user prompt from the cluster "Travel Itinerary Planning". Both assistant provided an equally
helpful itinerary. Therefore, this prompt cannot distinguish GPT-4-0613’s performance from Llama-2-70b-chat.
<|User Prompt|>
Can you make a list of things I could do with my girlfriend in Lille,
France?
<|GPT-4-0613’s Answer|>
Absolutely, here are some suggestions for activities you and your girlfriend
can enjoy in Lille, France:
1. Visit the Palais des Beaux-Arts: This is one of France’s largest museums
and it’s right in the heart of Lille. It houses collections of European
paintings, a medieval and Renaissance collection, ceramics, French
sculptures and more.
2. Explore the Vieux Lille: Stroll through the charming old town with its
Flemish influenced architecture, chic boutiques, and delightful eateries
.
3. Visit the Lille Cathedral (Basilique-cathedrale Notre-Dame-de-la-Treille)
: This beautiful cathedral is a mix of modern and traditional
architecture.
4. Enjoy a picnic at the Citadel Park: This is a great place to relax, have
a picnic, and enjoy the outdoors.
5. Visit the LaM (Lille Metropole Musee d’art moderne, d’art contemporain et
d’art brut): This museum is home to a unique collection of modern and
contemporary art.
6. Take a boat tour on the Deule River: This is a romantic way to see the
city and its surroundings.
7. Visit the Maison Natale de Charles de Gaulle: This is the birthplace of
Charles de Gaulle and it’s now a museum dedicated to his life and work.
8. Explore the Marche de Wazemmes: This is one of the largest markets in
France and it’s a great place to buy local products.
9. Enjoy a meal at a local restaurant: Lille is known for its delicious
cuisine, so make sure to try some local dishes.
10. Visit the Lille Zoo: This is a small but well-kept zoo that houses about
70 different species.
11. Take a romantic walk through the Jardin des Plantes de Lille: This
botanical garden is a peaceful place to enjoy nature.
12. Visit the Musee de l’Hospice Comtesse: This museum is located in a
former hospital and it’s home to a collection of art and artifacts from
the Middle Ages.
13. Enjoy a performance at the Lille Opera: Check the schedule and book
tickets for a romantic night out.
14. Take a day trip to the nearby city of Bruges: This charming Belgian city
is just a short train ride away.
15. Visit the Le Palais Rihour: This palace houses the Lille Tourist Office
and it’s also home to the Salle du Conclave, which has a beautiful
Gothic fireplace.
Remember to check the opening times and any COVID-19 restrictions before you
visit. Enjoy your trip!
<|Llama-2-70b-chat’s Answer|>
Sure! Lille is a beautiful city in northern France that offers a variety of
activities for couples to enjoy. Here are some ideas for things you could
23
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
do with your girlfriend in Lille:
1. Explore the city’s charming streets and historic landmarks, such as the
Grand Place, the Vieille Bourse, and the Cathedral of Lille.
2. Visit one of the many museums, such as the Palais des Beaux-Arts de Lille
, the Musee d’Art Moderne, d’Art Contemporain et d’Art Brut, or the
Musee de l’Hospice Comtesse.
3. Take a stroll along the canal and enjoy the views of the city from the
waterfront.
4. Visit the famous Lille Christmas market, which takes place every year in
December and features festive stalls, food, and drinks.
5. Go shopping at the city’s many boutiques and galleries, or at the large
shopping centers such as the Euralille or the Lille Europe.
6. Enjoy a romantic dinner at one of Lille’s many excellent restaurants,
such as the Michelin-starred Le Comptoir du Relais or the cozy Le
Bistrot d’Alsace.
7. Take a day trip to the nearby city of Bruges, Belgium, which is known for
its picturesque canals and medieval architecture.
8. Go to a show or concert at the Lille Opera or the Le Zenith concert hall.
9. Visit the nearby town of Roubaix, which is famous for its textile
industry and its beautiful Art Deco architecture.
10. Take a walk in the beautiful Parc de la Citadelle, which offers a
peaceful escape from the city and features a pond, a botanical garden,
and a zoo.
11. Go on a bike ride or a hike in the surrounding countryside, such as the
Parc Naturel Regional Nord-Pas de Calais.
12. Visit the famous Lille flea market, which takes place every Sunday and
offers a wide range of antiques, vintage items, and second-hand goods.
13. Take a cooking class together and learn how to make some of the region’s
delicious specialties, such as carbonnade flamande (beef stew) or tarte
au maroilles (cheese tart).
14. Go on a wine tasting tour and discover the region’s excellent wines,
such as the Chablis or the Champagne.
15. Take a hot air balloon ride over the city and enjoy a bird’s-eye view of
Lille’s beautiful architecture and surroundings.
These are just a few ideas to get you started. Lille is a vibrant and
romantic city that offers many opportunities for couples to have a great
time together. Bon voyage !
<|GPT-4-Turbo Verdict|>
Tie
24
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
D.2. Arena Bench Prompt
Example 1 : An prompt from Arena Bench requiring a very high level of complex problem-solving skill and reasoning while
adhering to real-world application.
Create a flutter app for habit tracking that tracks daily habits for a user.
The user should be able create multiple daily tasks that he wants to track.
And he should be able to group the habits into a set of groups. One the
user completes a task, he is rewarded a number of points per task. The
app should have a page that summarize the total score per group. And the
aggregate score of all the groups of habits. This flutter app needs to be
compilable for both andriod and iOS.
Example 2 : An prompt from Arena Bench requiring a very high level of complex problem-solving skill and reasoning while
adhering to real-world application.
I want to set up a remote raspberry pi zero, powered by a solar panel
with simple wiring. I want to power a small 2W pump, a simple electet
microphone, a custom python script running on the raspberry pi that is used
to classify audio detected by the microphone. What components will I need
to optimise for cost and minimise any electrical work (e.g. soldering)?
What size solar panel will I need to power this whole system?
25
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
D.3. Arena Bench System Prompt
The novel evaluation procedure is as follow: we prompt GPT-4-Turbo with the system prompt displayed below alongside
a user prompt, a reference answer, and 2 assistant’s answers. For reference answer, we present the user prompt with 3
assistants’ answers, GPT-4-Turbo, GPT-4-0314, and Claude-1, to GPT-4-Turbo and ask GPT-4-Turbo to generate an answer
to the prompt. To ensure consistent pairwise judgment, we set up GPT-3.5-Turbo-0301 as the baseline answer for all models
to be compared against. To avoid positional bias, we conduct two judgments per prompt: the first judgment presents the
baseline answer as Assistant A while the second judgment presents the baseline answer as Assistant B. In total, we conduct
700 pairwise comparisons between each model against GPT-3.5-Turbo-0301 across 350 user prompts to calculate a win-rate
against the baseline. Then we project the win-rate on a scale from 0 to 10 by assigning wins with a score of 10, ties with a
score of 5, and losses with a score of 0. Further, we assign a significant win or loss as 3 wins or 3 losses, respectively, and
keeping the other verdicts as a single win, loss, or tie. Finally, we calculate the final score by averaging across the wins,
losses, and ties.
<|System Prompt|>
Please act as an impartial judge and evaluate the quality of the responses
provided by two AI assistants to the user prompt displayed below. Your job
is to evaluate which assistant’s answer is better.
When evaluating the assistants’ answers, compare both assistants’ answers.
You must identify and correct any mistakes or inaccurate information.
Then consider if the assistant’s answers are helpful, relevant, and concise.
Helpful means the answer correctly responds to the prompt or follows the
instructions. Note when user prompt has any ambiguity or more than one
interpretation, it is more helpful and appropriate to ask for clarifications
or more information from the user than providing an answer based on
assumptions. Relevant means all parts of the response closely connect or
are appropriate to what is being asked. Concise means the response is clear
and not verbose or excessive.
Then consider the creativity and novelty of the assistant’s answers when
needed. Finally, identify any missing important information in the
assistants’ answers that would be beneficial to include when responding to
the user prompt.
After providing your explanation, you must output only one of the following
choices as your final verdict with a label:
1. Assistant A is significantly better: [[A>>B]]
2. Assistant A is slightly better: [[A>B]]
3. Tie, relatively the same: [[A=B]]
4. Assistant B is slightly better: [[B>A]]
5. Assistant B is significantly better: [[B>>A]]
Example output: "My final verdict is tie: [[A=B]]".
26
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
D.4. Arena Battles with No Ground Truth
Example 1 : An example of an Arena Battle between GPT-4-Turbo and Llama-2-13B-chat, where depending on different
preferences, either answer can be argued as better than the other. While both answers are correct, GPT-4-Turbo’s answer is
concise and appropriate to the format of multiple choice question. On the other hand, Llama-2-13B-chat provides helpful
explanation for the multiple choice answer.
<|User Prompt|>
Multiple Choice Question
All plants have both a haploid and a diploid form. As the plant changes
from one form to another, this is called:
Multiple choice question.
alternative life cycle
succession
an alternation of generations
<|GPT-4-Turbo’s Answer|>
an alternation of generations
<|Llama-2-13B-chat’s Answer|>
The correct answer is: an alternation of generations.
Plants, like other organisms, have different life cycle stages, and they
can exist in two different forms: haploid and diploid. The haploid form
is a single set of chromosomes, while the diploid form is two sets of
chromosomes. As the plant changes from one form to another, this is called
an alternation of generations. This process involves the alternation of
haploid and diploid phases, and it is a fundamental aspect of the life cycle
of plants.
Alternative life cycle and succession are not correct descriptions of this
process, as they refer to different concepts. Alternative life cycle refers
to the different stages of life an organism may go through, such as larva,
pupa, and adult in insects, while succession refers to the process of one
species replacing another in a particular environment or niche.
Example 2 : An example of an Arena Battle between GPT-4-Turbo and Llama-2-13B-chat, where depending on different
preferences, either answer can be argued as better than the other.
<|User Prompt|>
What are critical elements of a Product Mindset for startup founders?
<|GPT-4-Turbo’s Answer|>
Adopting a Product Mindset is crucial for startup founders as it helps them
focus on creating products that deliver value to customers while achieving
business objectives. Here are some critical elements of a Product Mindset
for startup founders:
1. Customer Centricity:
- Understand and empathize with your customers’ needs, problems, and
motivations.
27
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
- Continuously gather and incorporate customer feedback into the product
development process.
- Prioritize features and improvements that directly address customer pain
points and enhance their experience.
2. Value Proposition:
- Clearly define the unique value your product offers to the market.
- Align your product’s features and capabilities with its value proposition
to ensure it meets the expectations of your target audience.
3. Problem-Solving Focus:
- Identify and focus on solving real problems for users rather than just
creating features for the sake of innovation.
- Validate assumptions and hypotheses through experimentation and user
testing.
4. Iterative Development:
- Embrace an agile development approach, where you build, measure, and learn
in iterative cycles.
- Be open to pivoting or making significant changes based on feedback and
market response.
5. Data-Driven Decision Making:
- Use data to inform decisions about product development, marketing, and
growth strategies.
- Establish key performance indicators (KPIs) to measure the success of the
product and track progress over time.
...
By internalizing these elements, startup founders can develop a strong
Product Mindset that guides their decisions and strategies, ultimately
leading to a product that customers love and that contributes to the
startup’s success.
<|Llama-2-13B-chat’s Answer|>
A product mindset is a way of thinking and approaching product development
that is critical for startup founders. Here are some essential elements of
a product mindset for startup founders:
1. Customer-centricity: A product mindset starts with understanding the
customer’s needs, desires, and pain points. Founders must be obsessed with
their customers and be willing to put in the effort to understand their
needs.
2. Iteration and experimentation: A product mindset is all about
experimentation and iteration. Founders must be willing to try new things,
test hypotheses, and iterate on their product based on customer feedback.
3. Focus on value: A product mindset is focused on delivering value to
the customer. Founders must be able to identify and prioritize the most
important features and functionality that deliver the most value to their
customers.
28
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
4. Collaboration: A product mindset requires collaboration between
cross-functional teams, including engineering, design, and product
management. Founders must be able to work effectively with these teams to
develop a product that meets customer needs.
5. Data-driven decision making: A product mindset is data-driven, and
founders must be comfortable using data to make informed decisions about
their product. This includes analyzing customer feedback, usage data, and
other metrics to inform product development.
...
By embracing these elements of a product mindset, startup founders can
develop a product that meets customer needs, delivers value, and sets their
company up for long-term success.
29