Training run
In short
A training run is one job of training a model for a chosen number of steps, including restarts after a crash. It is how labs refer to the pre-training, or a later round of tuning, that produced a given model.
- Also called
- Training job, Production training run
- Topic
- How models are trained
- Level
- Beginner
Google's deep-learning tuning guide treats each trial as a training run with a fixed maximum number of steps. It says to keep that number the same across trials in a study, to learn what you can from many short runs, and to reserve the longest "production length" runs for a model you hope to launch. Training for more steps is not automatically better. When you are not limited by compute, extra steps may barely help, or may overfit1.
At the scale of a frontier language model, that production job is a single synchronised piece of work. Meta's Llama 3 paper describes pre-training a model with 405 billion parameters on 15.6 trillion tokens, with a context window of 8,000 tokens, on up to 16K H100 GPUs. A continued stage after that, not the main run, extended the context window to 128,000 tokens using about 800 billion further tokens2.
The job is brittle, and it is not frozen. Because training is synchronous, one GPU failure may require a restart of the entire job. In a 54-day window of Llama 3 405B pre-training, Meta logged 466 interruptions. Forty-seven were planned, including maintenance, configuration changes and dataset updates. The other 419 were unexpected, and about 78% of those were confirmed or suspected hardware faults, with GPU faults the largest share. Automation handled the rest: significant manual intervention was needed only three times, and more than 90% of the elapsed time was useful training2.
A training run is not the whole life of the model. Llama 3's paper treats that pre-training as a run, then describes branching off it to keep training a specialist, for example on a mix that was more than 85% code. Post-training, the later instruction and preference work, sits on top of a pre-trained checkpoint and is described separately2.
Example
Meta's published picture of Llama 3 405B is one pre-training run: 405 billion parameters, 15.6 trillion tokens, up to 16K H100 GPUs, and 466 interruptions inside a 54-day window, 419 of them unplanned2.
Why it matters
The run is what labs schedule, budget and debug. A model's scores belong to the job that produced it, restarts and mid-course data changes included.
Questions people ask
Is a training run the same as pre-training?
No. Pre-training is a stage. A training run is one job. Llama 3's main pre-training was a run on 15.6 trillion tokens, and the paper also describes branching off that run for extra specialist training, then separate post-training on a checkpoint2.
How long does a training run take?
There is no single length. Google's guide says the step budget depends on whether compute is the limit, and that the longest runs should be saved for a model you mean to launch1. Meta published a 54-day window of one Llama 3 pre-training job, not a rule for every run2.
What happens if a chip fails mid-run?
On Llama 3 pre-training, one GPU failure could force a restart of the whole job. In 54 days Meta recorded 466 interruptions, 419 unexpected and mostly hardware. Automation cleared almost all of them, and useful training still took more than 90% of the time2.