{"author":{"@type":"Person","name":"Alexander Bock","url":"https://abxk.ai/en/autor/alexander-bock/"},"categories":["AI Development"],"content":{"markdown":"When a new AI model is announced, three numbers usually come with it: how many parameters it has, how much data it was trained on and how much computing power its training took. These three numbers explain most of the progress of the last fifteen years. The ideas behind today's models are not much newer than the transformer of 2017; what changed is the scale.\n\nThis is part 4 of Level 0 in our learning path AI Fundamentals. You will learn what each of the three numbers means, how fast each has grown according to Epoch AI's data on notable AI models and why bigger has meant better, so far.\n\n## Three numbers behind every model\n\n**Parameters** are the adjustable numbers inside a model, its weights. They store what the model has learned. Our small digit network had a few thousand; large language models have billions or, by estimates, more than a trillion.\n\n**Training data** is the amount of examples a model learns from. For language models it is counted in **tokens**, word pieces of about four characters in English, and today reaches into the trillions.\n\n**Training compute** is the number of calculations the whole training takes, counted in **FLOP** (floating-point operations, simple calculations with decimal numbers). It depends on the other two: a common rule of thumb for transformer models is that training compute is about six times the number of parameters times the number of training tokens. A model that is twice as big and trained on twice as much data needs about four times the compute.\n\n| Number | What it measures | Unit | Grows when |\n|---|---|---|---|\n| Parameters | Adjustable weights of the model | Count | The model gets bigger |\n| Training data | Examples the model learns from | Tokens, images, data points | More data is collected |\n| Training compute | Calculations of the whole training | FLOP | Either of the two grows, or training runs longer |\n\n## How fast they grew\n\nEpoch AI, a research institute that tracks the development of AI, estimates the training compute of {{\u003c num \"ai_models\" \"with_compute\" \u003e}} of its notable models. We fitted a trend line through the models published since 2010. The result: training compute grew by a factor of about {{\u003c num \"ai_models\" \"compute_growth_per_year\" \u003e}} per year, which means it doubled roughly every {{\u003c num \"ai_models\" \"compute_doubling_months\" \u003e}} months. The parameters grew by a factor of about {{\u003c num \"ai_models\" \"params_growth_per_year\" \u003e}} per year, the training data by a factor of about {{\u003c num \"ai_models\" \"data_growth_per_year\" \u003e}}.\n\n![Scatter chart on a logarithmic scale: training compute of notable AI models by publication date since 1950; the points rise slowly until about 2010 and very steeply afterward, with language models at the top](/images/learn/en/parameters-data-compute-compute.webp \"Training compute of notable AI models. Each step on the vertical axis means ten times more.\")\n\nCompute grows fastest because it combines the other two: bigger models trained on more data for longer. For comparison, the computing power of a single chip grew far more slowly; the rest came from using many more chips at once, as part 3 of this level showed.\n\n![Bars: average growth per year since 2010 for training compute, training data and parameters of notable AI models](/images/learn/en/parameters-data-compute-growth.webp \"Average growth per year since 2010, our calculation from Epoch AI's data.\")\n\n## Where the training data comes from\n\nFor language models, most training data is text from the public web, collected by crawlers and filtered heavily: duplicates, spam and low-quality pages are removed, because quality matters as much as quantity. Books, scientific papers, code from public repositories and transcribed audio add to it. Image and video models learn from pictures and clips paired with descriptions. Two trends shape this data today. Developers increasingly use **synthetic data**, examples generated by other models, such as math problems with worked solutions. And the use of copyrighted material for training is contested: court cases and new rules deal with what may be used, and the EU AI Act requires providers of general-purpose AI models to publish a summary of the content they trained on.\n\n## Why parameter counts can mislead\n\nA bigger parameter count does not always mean a better or more expensive model. Many large models today are **mixtures of experts**: they contain many specialized sub-networks, and for each word piece only a few of them are active. Such a model may have hundreds of billions of parameters in total but use only a fraction of them for each answer, which makes it much cheaper to run. In the other direction, **distillation** trains a small model to imitate a large one, so that a model with a few billion parameters can match much larger models from a year or two earlier. When you compare models, look at benchmark results and prices, not at the parameter count alone.\n\n## The largest training runs\n\nThe table lists the notable models with the most training compute. The numbers are estimates by Epoch AI, because developers rarely publish them. The largest run in the data needed about 10 to the power of {{\u003c num \"ai_models\" \"largest_compute_log10\" \u003e}} calculations, a one followed by more than two dozen zeros.\n\n{{\u003c data \"models-largest\" \u003e}}\n\n## Why bigger has meant better\n\nSince around 2020, researchers have found that the error of language models falls in a surprisingly regular way when compute, parameters and data grow together. These **scaling laws** let developers predict how good a model will be before they train it, and they explain why companies invest billions in ever larger training runs. A second finding shaped the field: for a given amount of compute, it pays to balance model size and data, rather than to build the biggest possible model on too little data.\n\nGrowth has limits. Training runs need more chips, more electricity and more data each year, and high-quality text from the internet is finite. That is why developers now also look for other ways to improve models, such as letting them \"think\" longer before they answer, which shifts compute from training to inference.\n\nThe last part of Level 0 opens the box: what a neural network actually is, with the neurons, layers and weights of our own small network.","text":"When a new AI model is announced, three numbers usually come with it: how many parameters it has, how much data it was trained on and how much computing power its training took. These three numbers explain most of the progress of the last fifteen years. The ideas behind today\u0026rsquo;s models are not much newer than the transformer of 2017; what changed is the scale.\nThis is part 4 of Level 0 in our learning path AI Fundamentals. You will learn what each of the three numbers means, how fast each has grown according to Epoch AI\u0026rsquo;s data on notable AI models and why bigger has meant better, so far.\nThree numbers behind every model Parameters are the adjustable numbers inside a model, its weights. They store what the model has learned. Our small digit network had a few thousand; large language models have billions or, by estimates, more than a trillion.\nTraining data is the amount of examples a model learns from. For language models it is counted in tokens, word pieces of about four characters in English, and today reaches into the trillions.\nTraining compute is the number of calculations the whole training takes, counted in FLOP (floating-point operations, simple calculations with decimal numbers). It depends on the other two: a common rule of thumb for transformer models is that training compute is about six times the number of parameters times the number of training tokens. A model that is twice as big and trained on twice as much data needs about four times the compute.\nParameters, Data and Compute: Why AI Models Grew So Fast Number What it measures Unit Grows when Parameters Adjustable weights of the model Count The model gets bigger Training data Examples the model learns from Tokens, images, data points More data is collected Training compute Calculations of the whole training FLOP Either of the two grows, or training runs longer How fast they grew Epoch AI, a research institute that tracks the development of AI, estimates the training compute of 478 of its notable models. We fitted a trend line through the models published since 2010. The result: training compute grew by a factor of about 4.3 per year, which means it doubled roughly every 5.7 months. The parameters grew by a factor of about 2.2 per year, the training data by a factor of about 3.\nTraining compute of notable AI models. Each step on the vertical axis means ten times more. Compute grows fastest because it combines the other two: bigger models trained on more data for longer. For comparison, the computing power of a single chip grew far more slowly; the rest came from using many more chips at once, as part 3 of this level showed.\nAverage growth per year since 2010, our calculation from Epoch AI\u0026#39;s data. Where the training data comes from For language models, most training data is text from the public web, collected by crawlers and filtered heavily: duplicates, spam and low-quality pages are removed, because quality matters as much as quantity. Books, scientific papers, code from public repositories and transcribed audio add to it. Image and video models learn from pictures and clips paired with descriptions. Two trends shape this data today. Developers increasingly use synthetic data, examples generated by other models, such as math problems with worked solutions. And the use of copyrighted material for training is contested: court cases and new rules deal with what may be used, and the EU AI Act requires providers of general-purpose AI models to publish a summary of the content they trained on.\nWhy parameter counts can mislead A bigger parameter count does not always mean a better or more expensive model. Many large models today are mixtures of experts: they contain many specialized sub-networks, and for each word piece only a few of them are active. Such a model may have hundreds of billions of parameters in total but use only a fraction of them for each answer, which makes it much cheaper to run. In the other direction, distillation trains a small model to imitate a large one, so that a model with a few billion parameters can match much larger models from a year or two earlier. When you compare models, look at benchmark results and prices, not at the parameter count alone.\nThe largest training runs The table lists the notable models with the most training compute. The numbers are estimates by Epoch AI, because developers rarely publish them. The largest run in the data needed about 10 to the power of 27.0 calculations, a one followed by more than two dozen zeros.\nThe largest training runs by compute ModelOrganizationPublishedTraining compute (FLOP)Domain GPT-6 AstraOpenAISeptember 3, 20269.3 × 1026Multimodal Grok 4xAIJuly 9, 20255.0 × 1026Language GPT-4.5OpenAIFebruary 27, 20253.8 × 1026Language Grok 3xAIFebruary 17, 20253.5 × 1026Language GPT-5OpenAIAugust 7, 20256.6 × 1025Multimodal Llama 4 Behemoth (preview)Meta AIApril 5, 20255.1 × 1025Multimodal Gemini 1.0 UltraGoogle DeepMindDecember 6, 20235.0 × 1025Multimodal Composer 2.5CursorMay 18, 20263.9 × 1025Language Llama 3.1-405BMeta AIJuly 23, 20243.8 × 1025Language Claude 3.7 SonnetAnthropicFebruary 24, 20253.4 × 1025Language Source as for the first table on this page.\nWhy bigger has meant better Since around 2020, researchers have found that the error of language models falls in a surprisingly regular way when compute, parameters and data grow together. These scaling laws let developers predict how good a model will be before they train it, and they explain why companies invest billions in ever larger training runs. A second finding shaped the field: for a given amount of compute, it pays to balance model size and data, rather than to build the biggest possible model on too little data.\nGrowth has limits. Training runs need more chips, more electricity and more data each year, and high-quality text from the internet is finite. That is why developers now also look for other ways to improve models, such as letting them \u0026ldquo;think\u0026rdquo; longer before they answer, which shifts compute from training to inference.\nThe last part of Level 0 opens the box: what a neural network actually is, with the neurons, layers and weights of our own small network.\n","words":1023},"date":"2026-10-18","description":"What parameters, training data and compute mean, how fast each has grown since 2010 according to Epoch AI's data and why more of all three made AI models better.","image":"https://abxk.ai/images/cover/parameters-data-compute.webp","json":"https://abxk.ai/en/ai-development/fundamentals/parameters-data-compute/index.json","kind":"page","language":"en","learning_path":{"category":"ai-development","level":0,"level_name":"Start here"},"modified":"2026-10-18","related_tools":["Runpod","Pinecone","MindStudio"],"section":"ai-development","seo_title":"Parameters, Data and Compute: How AI Models Grow","tags":["parameters","compute","scaling","ai-basics","beginner"],"title":"Parameters, Data and Compute: Why AI Models Grew So Fast","translations":[{"json":"https://abxk.ai/de/ai-development/fundamentals/parameters-data-compute/index.json","language":"de","url":"https://abxk.ai/de/ai-development/fundamentals/parameters-data-compute/"}],"url":"https://abxk.ai/en/ai-development/fundamentals/parameters-data-compute/"}