Elo rating
In short
An Elo rating is a score that ranks players by their results against each other, first used in chess. In AI it ranks chatbots by head-to-head votes: beating a higher-rated model gains more points than beating a weaker one.
- Also called
- Elo, Elo score, Elo system
- Topic
- Testing and benchmarks
- Level
- Some background helps
- First described
- Devised by Arpad Elo for chess; adopted by the US Chess Federation in 1960 and by FIDE in 19701.
The system is named after Arpad Elo, a physics professor and chess master. The US Chess Federation adopted it in 1960 and the world chess body FIDE in 19701.
It works on expected results. The bigger the gap between two ratings, the more likely the stronger side is to win. Under FIDE's tables, a player rated about 100 points higher is expected to score 0.64 per game, and about 200 points higher, 0.762. After each game, a player's rating moves by the difference between the actual and expected result, multiplied by a factor K. FIDE uses K of 40, 20 or 10 depending on experience and rating2.
AI leaderboards borrowed the idea for chatbots. On Chatbot Arena, now Arena, people vote between two anonymous answers, and each vote is a "game"3. The site first showed Elo scores, then moved to the closely related Bradley-Terry model, which fits all votes at once rather than updating game by game3.
Most reporting still calls these Elo ratings. Stanford's AI Index said that in March 2026 the top models from Anthropic, xAI, Google and OpenAI were within 25 Elo points of each other on Arena4.
An Elo score only means something relative to the other players in the same pool. A 1,500 on one leaderboard cannot be compared with a 1,500 on another.
Example
If a chatbot rated 1,400 beats one rated 1,500, it gains more points than if it had beaten another 1,400 model, because the win was less expected2.
Why it matters
When news reports say one AI model "leads by 20 points", they are usually quoting Elo-style ratings. Small gaps can be within the margin of error.
Questions people ask
How is Elo calculated?
Each result is compared with what the ratings predicted. The rating change equals the actual score minus the expected score, multiplied by a factor K. FIDE sets K at 40, 20 or 10 depending on the player2.
Do AI leaderboards really use Elo?
Loosely. Arena first reported Elo scores, then switched to the Bradley-Terry model, which is closely related and better suited to statistical estimation3. The results are still widely called Elo ratings4.
What does a 100-point Elo gap mean?
In chess, the higher-rated player is expected to score about 64% against the lower-rated one2. On AI leaderboards, a gap of that size means people preferred one model's answers noticeably more often.