Inference

Every AI model has two lives. In training, it is shown huge amounts of data and its internal numbers, called weights, are adjusted over and over. In inference, those numbers are fixed and the model simply applies what it learned to something new. Nvidia describes inference as the point where "capabilities learned during AI training are put to work"¹.

Each time you send a message to a chatbot, the provider runs inference on its servers. For a language model, that means working out the reply one token at a time. Inference skips the hardest part of training: it does not need the backward pass that adjusts the weights².

Inference still needs serious hardware for large models. Meta reports that its Llama 3 model with 405 billion parameters does not fit in the memory of a single machine with eight Nvidia H100 chips at standard precision, so it splits the work across machines or uses lower-precision numbers².

Because training happens once but inference happens every time someone uses a model, chip makers build hardware just for it. Google said in 2017 that its Tensor Processing Unit, in its data centres since 2015, was designed to speed up the inference phase of neural networks³.

Inference is also what you pay for when using an AI API. Anthropic, for example, charges per million tokens in and out, with Claude Sonnet 5.5 at $2 per million input tokens and $10 per million output tokens⁴.