Test-time compute
In short
Test-time compute is the computing power an AI model uses while answering a question, as opposed to while it is being trained. Giving a model more of it, for example to think longer or try several answers, can make its answers more accurate.
- Also called
- Inference-time compute, Test-time scaling, Inference-time scaling
- Topic
- How AI models work
- Level
- Technical
- First described
- Snell, Lee, Xu and Kumar's August 2024 paper studied how test-time compute scales in language models, noting that little earlier research had done so1.
Training compute is spent once, to build a model. Test-time compute, also called inference-time compute, is spent every time someone asks it something. A 2024 paper by Charlie Snell and colleagues put the key question plainly: if a model can use a fixed but non-trivial amount of extra compute on a hard prompt, how much better can it get1?
There are several ways to spend it. A model can write many candidate answers and keep the best, known as best-of-N. It can search through possible steps guided by a separate verifier model that scores each step. Or it can revise its own answer1. Reasoning models spend it by writing long internal chains of thought, and DeepSeek reported that its R1-Zero model learned on its own to use more thinking time during training2.
Snell's team found the best method depends on how hard the question is. Choosing the method per prompt was more than four times as efficient as best-of-N. In a test with equal total compute, a smaller model given extra test-time compute beat a model 14 times larger, on problems the smaller model could already sometimes solve1.
In products this shows up as an effort or budget setting. OpenAI's reasoning effort parameter trades speed and token use against more thorough thinking, and the reasoning tokens are billed as output3. Anthropic says larger thinking budgets can improve quality on complex problems, with diminishing returns and more delay4.
Example
A developer building a chatbot for a UK accountancy firm could set reasoning effort to low for simple questions about filing deadlines and high for tricky tax calculations, paying for extra thinking only where it helps3.
Why it matters
It means AI can get better without building ever-bigger models, by thinking harder on each question. It also means hard questions cost more and take longer to answer.
Questions people ask
Is test-time compute the same as inference?
Closely related. Inference is the act of running a trained model to get an answer. Test-time compute is how much computing that takes, and the term is mostly used when a model deliberately spends extra on a question, for example by thinking longer1.
Why does thinking longer cost more?
Because the thinking is made of tokens like any other text. OpenAI bills reasoning tokens as output tokens even though they are not shown, so a model that thinks more produces a bigger bill3.
Can extra test-time compute replace a bigger model?
Sometimes. In one study, a smaller model given more test-time compute beat a model 14 times larger with the same total compute, but only on problems the smaller model could already partly solve1.
Related terms
Tools that use it
- ChatGPTOpenAI's general chatbot, the most used AI app in the UK and worldwide.
- ClaudeAnthropic's chatbot, strong on writing and on pointing out holes in an argument, with a tight free allowance.
- DeepSeekA free Chinese chatbot with no message cap that stores every conversation on servers in China.
- Gemini 3.1 Deep ThinkGemini's slowest, most thorough reasoning mode for science, maths and engineering problems
Sources
- Snell, Lee, Xu and Kumar (2024), Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters, arXiv:2408.03314
- DeepSeek-AI et al. (2025), DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, arXiv:2501.12948
- OpenAI API documentation, Reasoning models guide
- Anthropic, Claude API documentation, Extended thinking