Compute

In short

Compute is the amount of calculation spent training one model, usually counted as floating-point operations. A bigger model is not automatically a model that used more compute, because the data and the number of training steps also decide the total.

Topic: How models are trained
Level: Some background helps
First described: OpenAI's analysis treated the compute used to train one model, rather than chip speed or data-centre capacity, as the figure tied to the strongest systems. Sevilla and colleagues date that analysis to May 2018; the archived page includes an addendum of 7 November 2019⁵¹.

Compute is the calculation used to train one model⁵. An OpenAI analysis says the relevant figure is not the speed of one GPU and not the capacity of the largest data centre⁵. Its addendum, dated 7 November 2019, says the largest training runs since 2012 doubled about every 3.4 months and grew by more than 300,000 times⁵. Sevilla and colleagues date the analysis to May 2018 and note that it covered 15 systems from 2012 to 2018¹.

Kaplan and colleagues estimate a transformer's training compute from its parameter count². A forward pass is about 2N operations, N excluding embeddings, because a multiply-accumulate counts as two operations². The backward pass is about twice that, so training is about 6N operations per token². Their compute-efficient plan grows model size as about the 0.73 power of compute and tokens as the 0.27 power, and stops well short of convergence².

Hoffmann and colleagues read that result as a 5.5-times larger model and only 1.8-times more tokens when compute rises tenfold³. From more than 400 models, from 70 million to over 16 billion parameters, they find that model size and tokens should scale about equally³. Chinchilla, 70 billion parameters trained on 1.4 trillion tokens, used the same number of operations as Gopher, 280 billion parameters on 300 billion tokens³. The Gopher budget is 5.76 times 10 to the 23 FLOPs, and the smaller Chinchilla has a lower inference cost³.

The abstract says Chinchilla reached 67.5% average accuracy on MMLU, a greater than 7% improvement over Gopher³. The results section says 67.6%, a gain of 7.6%, and a worse score on 4 of 57 tasks³. Sevilla and colleagues reject a single 3.4-month doubling¹. Table 2 gives about 21.3 months from 1952 to 2010, 5.7 months from 2010 to 2022, and 9.9 months for the large-scale trend from September 2015 to 2022¹.

Under EU AI Act Article 51, a general-purpose model is presumed to have high-impact capabilities when the computation used for its training, measured in floating-point operations, is greater than 10 to the 25⁴. That presumption comes into force on 2 August 2025, and the Commission is to amend the threshold by delegated act⁴.

Example

Chinchilla and Gopher used the same training compute, given as 5.76 times 10 to the 23 FLOPs for Gopher, but Chinchilla had 70 billion parameters and 1.4 trillion tokens while Gopher had 280 billion parameters and 300 billion tokens³.

Why it matters

Compute is how labs and regulators compare training runs that used different chips for different lengths of time. Two models can share a training budget and still differ sharply in size and in what they cost to run afterwards.

Questions people ask

Is compute the same as the number of parameters?

No. Parameter count is one input to the total. Kaplan and colleagues estimate about 6N floating-point operations per training token, so the same model trained on more tokens uses more compute². Chinchilla is smaller than Gopher and was trained with the same number of operations³.

How fast has AI training compute grown?

OpenAI reported a doubling about every 3.4 months from 2012, more than 300,000 times over the period they measured⁵. Sevilla and colleagues, on a larger set through 2022, find slower doublings: about 21.3 months before 2010, 5.7 months in the deep-learning era, and 9.9 months for the separate large-scale trend¹.

Does a training-compute figure decide EU AI Act duties by itself?

Article 51 presumes high-impact capabilities when training computation exceeds 10 to the 25 floating-point operations⁴. That rule comes into force on 2 August 2025, and the Commission can amend the threshold⁴. The Act has other classification routes as well⁴.

Related terms

Sources

Back to the glossary