Reward model

A reward model reads a prompt and a reply and outputs one number, called a reward. It is trained so the preferred reply scores higher than the rejected one. In the InstructGPT paper, labelers ranked between four and nine replies to the same prompt, and the reward model was trained on every pair inside that ranking¹. The team used reward models with 6 billion parameters, because training a 175 billion parameter reward model was unstable¹.

The chatbot is a different model. InstructGPT then runs reinforcement learning with the PPO algorithm, so the chatbot's replies score higher on the reward model, with a penalty that limits drift from the demonstration-trained version¹. Labelers preferred its 1.3 billion parameter model to the 175 billion parameter GPT-3, which the authors call 100 times fewer parameters¹. They say the result tracks their labelers and researchers, about 40 contractors, not a broader notion of human values¹.

The idea predates chatbots. In 2017, Christiano and colleagues trained a reward predictor from human preferences between short clips of behaviour, then let a reinforcement learning agent maximise the predicted reward. The tasks were Atari games and simulated robot locomotion. People labelled less than 1% of the agent's interactions, and new behaviours took about an hour of human time². In 2019, Ziegler and colleagues applied the same reward-learning idea to text. A model of reward, built by asking people which continuation they preferred, was used for sentiment, descriptive writing and summarisation. Stylistic continuation used 5,000 human comparisons. Summarisation used 60,000³.

OpenAI's November 2022 introduction of ChatGPT says the team ranked two or more replies, trained reward models on those rankings, and fine-tuned with PPO, using the same methods as InstructGPT with slight differences in the data collection. The page now says ChatGPT has evolved substantially since that post, so it does not describe training in 2026⁴. Meta's Llama 3 paper, from July 2024, still trains a reward model on human preference rankings and uses it for rejection sampling, then aligns the chatbot with direct preference optimisation. The authors tried PPO and report that direct preference optimisation used less compute and did better on instruction-following benchmarks⁵.