Parameters
In short
Parameters are the numbers inside an AI model that it adjusts while learning and then uses to produce answers. A model's size is usually given as its parameter count, such as 175 billion for GPT-3.
- Also called
- Model parameters, Parameter count, Parameter
- Topic
- The basics
- Level
- Beginner
- First described
- Learning by adjusting the weights of connections in a network was set out by Rumelhart, Hinton and Williams in Nature in October 1986, in the paper that popularised backpropagation.
A neural network is a large web of simple maths operations. Each connection has a number attached that decides how strongly one part influences the next. Those numbers are the parameters. Training is the process of nudging them, again and again, until the model's outputs get closer to the right answers1.
Most parameters are weights. Others are biases, which shift a unit's output up or down; the 1986 paper that popularised training by backpropagation treated the bias as just another weight1. The Open Source Initiative defines parameters as "weights or other configuration settings"2.
Parameter counts have grown fast. OpenAI's GPT-3 had 175 billion parameters in 20203. Meta's first LLaMA models in 2023 ranged from 7 billion to 65 billion4, and its largest Llama 3 model in 2024 had 405 billion5.
More parameters is not the same as better. Meta reported that its 13-billion-parameter LLaMA beat the 175-billion-parameter GPT-3 on most benchmarks4. Some designs, called mixture of experts, only use part of the model for each token: OpenAI's gpt-oss-120b has 117 billion parameters but about 5.1 billion active at once6.
Do not confuse parameters with settings you choose when using a model, such as temperature. Those are sometimes loosely called parameters too, but they are not learned. And many companies no longer publish parameter counts: OpenAI's GPT-4 report gave no model size7.
Example
OpenAI says its 21-billion-parameter gpt-oss-20b can run within 16GB of memory, while the 117-billion-parameter gpt-oss-120b is built to fit on a single 80GB data centre GPU6.
Why it matters
Parameter count is the most quoted measure of a model's size. It hints at cost, memory and the hardware needed to run it, but it is a poor guide to quality on its own.
Questions people ask
What does 7B or 70B mean in a model name?
It is the parameter count in billions. Meta's LLaMA family, for example, came in sizes from 7 billion to 65 billion parameters4.
Are parameters the same as weights?
Nearly. Weights make up most parameters, and biases make up the rest. The backpropagation paper of 1986 treated the bias as one more weight1, so in practice people use the two words almost interchangeably.
Does a model with more parameters always perform better?
No. Training data and method matter a great deal. Meta found its 13-billion-parameter LLaMA beat the 175-billion-parameter GPT-3 on most benchmarks4.
How many parameters does GPT-4 have?
OpenAI has not said. Its GPT-4 technical report states that it contains no details about the architecture, including model size7. Figures quoted online are estimates.
Related terms
Tools that use it
Sources
- Rumelhart, Hinton and Williams (1986), Learning representations by back-propagating errors, Nature 323, 533-536
- Open Source Initiative, The Open Source AI Definition 1.0
- Brown et al. (2020), Language Models are Few-Shot Learners, arXiv:2005.14165
- Touvron et al. (2023), LLaMA: Open and Efficient Foundation Language Models, arXiv:2302.13971
- Grattafiori et al. (2024), The Llama 3 Herd of Models, arXiv:2407.21783
- OpenAI, gpt-oss-120b model card on Hugging Face
- OpenAI (2023), GPT-4 Technical Report, arXiv:2303.08774, section 2