Parameters, Data and Compute: Why AI Models Grew So Fast

When a new AI model is announced, three numbers usually come with it: how many parameters it has, how much data it was trained on and how much computing power its training took. These three numbers explain most of the progress of the last fifteen years. The ideas behind today’s models are not much newer than the transformer of 2017; what changed is the scale.

This is part 4 of Level 0 in our learning path AI Fundamentals. You will learn what each of the three numbers means, how fast each has grown according to Epoch AI’s data on notable AI models and why bigger has meant better, so far.

Three numbers behind every model

Parameters are the adjustable numbers inside a model, its weights. They store what the model has learned. Our small digit network had a few thousand; large language models have billions or, by estimates, more than a trillion.

Training data is the amount of examples a model learns from. For language models it is counted in tokens, word pieces of about four characters in English, and today reaches into the trillions.

Training compute is the number of calculations the whole training takes, counted in FLOP (floating-point operations, simple calculations with decimal numbers). It depends on the other two: a common rule of thumb for transformer models is that training compute is about six times the number of parameters times the number of training tokens. A model that is twice as big and trained on twice as much data needs about four times the compute.

How fast they grew

Epoch AI, a research institute that tracks the development of AI, estimates the training compute of 478 of its notable models. We fitted a trend line through the models published since 2010. The result: training compute grew by a factor of about 4.3 per year, which means it doubled roughly every 5.7 months. The parameters grew by a factor of about 2.2 per year, the training data by a factor of about 3.

Compute grows fastest because it combines the other two: bigger models trained on more data for longer. For comparison, the computing power of a single chip grew far more slowly; the rest came from using many more chips at once, as part 3 of this level showed.

Where the training data comes from

For language models, most training data is text from the public web, collected by crawlers and filtered heavily: duplicates, spam and low-quality pages are removed, because quality matters as much as quantity. Books, scientific papers, code from public repositories and transcribed audio add to it. Image and video models learn from pictures and clips paired with descriptions. Two trends shape this data today. Developers increasingly use synthetic data, examples generated by other models, such as math problems with worked solutions. And the use of copyrighted material for training is contested: court cases and new rules deal with what may be used, and the EU AI Act requires providers of general-purpose AI models to publish a summary of the content they trained on.

Why parameter counts can mislead

A bigger parameter count does not always mean a better or more expensive model. Many large models today are mixtures of experts: they contain many specialized sub-networks, and for each word piece only a few of them are active. Such a model may have hundreds of billions of parameters in total but use only a fraction of them for each answer, which makes it much cheaper to run. In the other direction, distillation trains a small model to imitate a large one, so that a model with a few billion parameters can match much larger models from a year or two earlier. When you compare models, look at benchmark results and prices, not at the parameter count alone.

The largest training runs

The table lists the notable models with the most training compute. The numbers are estimates by Epoch AI, because developers rarely publish them. The largest run in the data needed about 10 to the power of 27.0 calculations, a one followed by more than two dozen zeros.

Why bigger has meant better

Since around 2020, researchers have found that the error of language models falls in a surprisingly regular way when compute, parameters and data grow together. These scaling laws let developers predict how good a model will be before they train it, and they explain why companies invest billions in ever larger training runs. A second finding shaped the field: for a given amount of compute, it pays to balance model size and data, rather than to build the biggest possible model on too little data.

Growth has limits. Training runs need more chips, more electricity and more data each year, and high-quality text from the internet is finite. That is why developers now also look for other ways to improve models, such as letting them “think” longer before they answer, which shifts compute from training to inference.

The last part of Level 0 opens the box: what a neural network actually is, with the neurons, layers and weights of our own small network.