Training and Inference: When AI Learns and When It Answers

Every AI model lives two lives. First it learns: in training, it sees large amounts of data and adjusts its internal numbers, again and again, until its predictions are good. Then it works: in inference, the finished model receives a new input, such as your question, and produces an output without learning anything new. The two phases differ in almost everything: who pays, how long they take, what hardware they need and what can go wrong.

This is part 3 of Level 0 in our learning path AI Fundamentals. You will learn what happens in each phase, why training a frontier model has become an industrial project and why inference, the cheap-looking part, adds up to large costs too.

Two phases of every model

In training, the model makes a prediction for an example, a formula measures the error and an algorithm called backpropagation works out how each weight should change to make the error smaller. Then all weights move a little in that direction. One pass over all the training data is called an epoch, and a model usually needs many of them. The small network in our digit experiment, for example, went through 200 rounds of adjustment.

In inference, the weights stay fixed. The model just calculates: input in, output out. A language model does this once for every token, roughly every word piece, of its answer, which is why long answers take longer and cost more.

How big training runs have become

Training compute, the number of calculations a training run takes, has grown fast. Across the notable models that Epoch AI, a research institute, recorded since 2010, it grew by a factor of about 4.3 per year, our calculation from Epoch AI’s data. The training runs behind today’s frontier models are industrial projects:

The largest run in the data used about 200,000 AI chips at once, and the highest estimated compute cost is about 388 million dollars, counted in 2023 dollars and only for the computing time, not for the researchers or the data. The most power-hungry run drew an estimated 233 megawatts while it was running.

Training after training: fine-tuning and feedback

The big training run is only the first stage. A language model first goes through pretraining: it reads huge amounts of text and learns to predict the next word piece. The result knows a lot, but it does not yet follow instructions well. In fine-tuning, the developer trains it further on a much smaller set of good examples, such as questions with model answers written by people. Then comes reinforcement learning from human feedback (RLHF) or a similar method: people or another model compare pairs of answers, and the model is rewarded for the preferred one. Newer reasoning models also learn from rewards for solving math and programming tasks correctly. These later stages use far less compute than pretraining, but they shape most of what users notice: the tone, when the model refuses, and how it reasons step by step.

Why inference costs add up

A single answer from a language model costs a fraction of a cent. But popular AI services answer billions of requests, every day, and every new user, every longer answer and every model that “thinks” step by step before answering adds computing time. Over the life of a successful model, inference can cost more than training. This is why developers spend so much effort on making inference cheaper: smaller models for simple tasks, compressed weights that need fewer bits per number, and answers that are stored and reused for repeated questions.

Inference on your own device

Inference does not have to happen in a data center. Small models run on laptops and phones, where they answer without sending data anywhere, which helps with privacy and works offline. The limit is memory: every weight has to be loaded, so a model with 8 billion parameters needs several gigabytes even with compressed weights. That is why phones run small models for tasks such as summarizing notifications and hand harder questions to a larger model in the cloud. For companies, running an open model on their own servers is the middle ground: the data stays in-house, but someone has to buy and operate the hardware.

What this means for your projects

If you use an AI model through an API, you pay only for inference, priced per token or per request. Training is something you rarely do yourself; at most you fine-tune an existing model, a short additional training run on your own examples. The practical questions are therefore: How many requests will you send? How long are the inputs and answers? Does a smaller, cheaper model do the job? Our learning path AI Development shows how to answer them in the article on APIs, tokens and context windows.

The next part of this level looks at the three numbers behind the growth of AI models: parameters, data and compute.