Every robot brain, every self-driving car and every chatbot runs on chips. When a new AI chip is announced, one number gets the headlines: how many calculations it can do per second. But that number alone says surprisingly little about how fast a chip runs AI in practice. The real limit is often somewhere else: how fast the chip can get data from its memory.
This is part 5 of Level 1 in our learning path Physical AI. Using data from Epoch AI, a research institute that tracks AI hardware, we compare the top AI chips of each year since 2017. You will learn the three numbers that describe an AI chip, why compute has grown more than twice as fast as memory speed and what this memory wall means for the speed of AI answers.
Three numbers that describe an AI chip
| Number | What it means | Unit | Why it matters |
|---|---|---|---|
| Compute | Calculations per second, here with 16-bit numbers | TFLOP/s (trillions per second) | Sets the limit for large, regular math such as training |
| Memory | How much data fits directly next to the chip | GB | Decides whether a model fits on one chip or must be split |
| Memory bandwidth | How fast data moves between memory and chip | TB/s (terabytes per second) | Sets the limit whenever the chip must read a lot of data for little math |
Today’s AI chips are mostly graphics processors (GPUs) or chips built only for AI, such as Google’s TPUs. Both contain thousands of small arithmetic units that work in parallel, which suits the matrix math of neural networks. In the chip sales that Epoch AI tracks since the start of 2024, NVIDIA accounts for 65.1% of the computing power sold.
Compute grew faster than memory speed
We took the most powerful widely used AI chip of each year, from NVIDIA, AMD or Google, and compared it with the 2017 top chip. Compute grew 20.1-fold and memory capacity 18-fold. Memory bandwidth, however, grew only 8.9-fold, from 0.9 to 8 terabytes per second.
| Year | Chip | TFLOP/s | Memory bandwidth (TB/s) | Memory (GB) | Calculations per byte read |
|---|---|---|---|---|---|
| 2017 | NVIDIA V100 SXM2 | 125 | 0.9 | 16 | 139 |
| 2018 | NVIDIA V100 DGXS 16 GB | 125 | 0.9 | 16 | 139 |
| 2019 | NVIDIA V100S PCIe 32 GB | 130 | 1.1 | 32 | 115 |
| 2020 | NVIDIA A100 SXM4 80 GB | 312 | 2.0 | 80 | 153 |
| 2021 | AMD Radeon Instinct MI250X | 383 | 3.3 | 128 | 117 |
| 2022 | NVIDIA H100 SXM5 80GB | 989 | 3.4 | 80 | 295 |
| 2023 | AMD Instinct MI300X | 1,307 | 5.3 | 192 | 247 |
| 2024 | NVIDIA B200 | 2,250 | 7.7 | 180 | 292 |
| 2025 | AMD Instinct MI355X | 2,517 | 8.0 | 288 | 315 |
Data: Epoch AI, Data on machine learning hardware, retrieved on October 18, 2026, licensed CC BY 4.0. For each year, the most powerful GPU or TPU chip by NVIDIA, AMD or Google with figures for compute (16-bit), bandwidth and memory; calculations per byte: our calculation.
The last column of the table shows the consequence. The 2017 top chip could do 139 calculations in the time it took to read one byte from memory. The fastest chip in the table can do 315. A program that does fewer calculations per byte than that leaves part of the chip idle, waiting for data. Engineers call this the memory wall.
Why answering one user wastes most of the chip
When a language model writes an answer, it produces one word piece (token) at a time. For each token, the chip has to read all the model’s weights from memory once, and it does about two calculations per weight. With 16-bit numbers, each weight is two bytes, so the chip performs about one calculation per byte it reads. That is far below the 315 the chip could handle: when it serves a single user, it uses at most 0.3% of its compute.
The trick is to serve many users at once. If a chip produces the next token for 64 conversations in one go, it reads the weights once but uses them 64 times: about 64 calculations per byte. This is called batching, and it is the main reason why AI providers can answer millions of users with a limited number of chips. It also explains why the speed of a single answer depends so much on memory bandwidth, not only on compute.
A worked example with our own assumption: a language model with 70 billion parameters needs 140 GB for its weights in 16-bit form. In 2017, that would have taken at least 9 top chips just to hold it; today it fits on 1. For a single user, memory bandwidth limits the speed to at most about 57 tokens per second on the fastest chip in the table, compared with about 6 on the 2017 chip, if the model could have fit on it. Real systems reach less, because no chip uses its full bandwidth all the time.
What chip designers do about it
The memory wall shapes nearly every new design:
- Faster memory stacked next to the chip. High-bandwidth memory (HBM) sits in stacks right beside the processor and is one of the most expensive parts of an AI chip.
- Smaller numbers. Storing weights with 8 or 4 bits instead of 16 halves or quarters the bytes per weight, which raises the calculations per byte. The price is a small loss of precision that has to be checked for each model.
- More memory on the chip itself. Some chips designed for answering, not training, keep large parts of a model in fast on-chip memory.
- Splitting the work. Large models are spread across many chips connected by fast links, so that each chip reads only a part of the weights.
For physical AI, the same trade-off appears in a smaller format. A robot or a car has to run its models on board, within a power budget of tens or hundreds of watts instead of the kilowatts of a data center rack. Every byte it moves costs time and energy, which is why models for robots and vehicles are often smaller, use fewer bits and are tuned closely to their chip.
How we analyzed the data
The chip data comes from Epoch AI’s database of machine learning hardware. For each year, we took the most powerful GPU or TPU from NVIDIA, AMD or Google with information on 16-bit compute, memory and bandwidth. Calculations per byte are compute divided by bandwidth. The worked example uses our own simplified assumptions: two calculations per parameter and token, 16-bit weights and no other memory traffic. Real numbers depend on the software, the model and the chip’s actual bandwidth.