Leaderboard
In short
A leaderboard ranks machine-learning systems by a published score. A competition board scores each team's best submission, and Chatbot Arena ranks language models by human votes between anonymous answers.
- Also called
- model leaderboard, ranking table
- Topic
- Testing and benchmarks
- Level
- Beginner
Chatbot Arena is a public leaderboard for language models. A person asks one question and reads answers from two anonymous models, then votes for the reply they prefer. The site shows the model names only after the vote1. By January 2024 it had collected more than 240,000 votes from about 90,000 users1.
Those pairwise votes become a ranking through a Bradley-Terry model, which estimates a score for each model from how often people prefer it, with confidence intervals on the scores1. Many other language-model benchmarks instead use a fixed test with a predefined answer, such as a multiple-choice exam1.
A competition leaderboard raises a different problem. Blum and Hardt describe contests in which a team can submit repeatedly and see a score computed on a public test set whose labels are withheld2. Teams that adapt to that feedback may overfit the holdout data. The authors report that some public scores are then inflated compared with the final scores2.
Chiang and colleagues also note that a static benchmark's test set can become contaminated over time, which undermines the result1. A rank answers the question that board asked. Arena ranks models by crowd preference on its prompts, not by a ground-truth answer key1.
Example
On Chatbot Arena, someone types one question, reads two anonymous answers, and picks the one they prefer. The leaderboard turns many such votes into model scores1.
Why it matters
A leaderboard makes scores easy to compare. A public competition score can be inflated when teams tune to the test behind it2, and a preference ranking measures something different from a fixed exam1.
Questions people ask
What does the Chatbot Arena leaderboard measure?
It ranks language models from anonymous pairwise votes. A user sees two answers to the same question and picks the one they prefer, and a Bradley-Terry model turns the votes into scores1.
Can a competition leaderboard overstate a result?
Yes. If teams can resubmit and see the public score, they may overfit the holdout data, and some public scores are then inflated compared with the final scores2.