Large language model

Anthropic's documentation describes LLMs as "AI language models with many parameters", trained on "vast amounts of text data", that can generate human-like text, answer questions and summarise¹. In pretraining, the model learns to predict the next word given the text before it¹.

"Large" refers to the number of parameters, the internal numbers set in training, and to the data. OpenAI's GPT-3 had 175 billion parameters and was trained on 300 billion tokens². Meta's Llama 2 came in sizes from 7 billion to 70 billion parameters and was trained on 2 trillion tokens³. Some developers no longer publish sizes: OpenAI's GPT-4 report gave no details of model size⁴.

Prediction alone does not make a helpful assistant. Anthropic notes pretrained models are "not inherently good at answering questions or following instructions", so fine-tuning and reinforcement learning from human feedback are added¹. Llama 2-Chat was fine-tuned for dialogue in the same spirit³.

The GPT-3 paper found that scale let one model do many tasks from a few examples written into the prompt, without retraining². Llama 2 uses the standard transformer architecture³.

An LLM does not look facts up. It produces likely text, which is why it can state false things with confidence, a risk NIST calls confabulation⁵.