“The model learns from data” is easy to say. What does it mean in practice? How many examples does a model need, what happens if nobody tells it the right answers, and how does a machine learn by trial and error? Instead of repeating textbook claims, we ran the experiments ourselves, on a public dataset that anyone can use, with code that gives the same numbers on every run.
This is part 2 of Level 0 in our learning path AI Fundamentals. You will learn the three ways machines learn, supervised, unsupervised and reinforcement learning, and see what each of them achieved in our experiment. You do not need to run any code to follow along.
The experiment
The data is a classic of machine learning: 1,797 images of handwritten digits from 0 to 9, collected by researchers at Boğaziçi University in Istanbul and published in the UCI Machine Learning Repository. Each image has only 8 × 8 = 64 pixels, so the task is small enough to run in seconds on a laptop, and big enough to show the real effects. We held back 540 images that no model saw during training and measured every result on them, because a model that only does well on its training data has not learned anything useful.
| Way of learning | What the model gets | What it learns | Our result |
|---|---|---|---|
| Supervised | Images with the right digit as a label | To name the digit of a new image | 96.7% of the test images correct |
| Unsupervised | The same images without labels | Groups of similar images | 79.4% fall into a group that matches their digit |
| Reinforcement | No examples, only rewards for reaching a goal | A strategy for a task | Steps to the goal fell from 68.3 to 8.1 |
Supervised learning: examples with answers
In supervised learning, every training example comes with the right answer, called a label. The model sees an image of a 7 together with the label “7”, thousands of times for all digits, and adjusts its internal numbers until it predicts the labels well. We used logistic regression, one of the simplest methods, and trained it again and again with more examples.
With only 10 examples, the model recognized 43.1% of the test images: far better than guessing, which would hit one digit in ten, but useless in practice. With 100 examples it reached 88.3%, and with all 1,257 training images 96.7%. The curve shows a rule that holds far beyond digits: the first examples bring the biggest gains, and every further step costs more data for less improvement. This is why collecting the first few hundred good labels is usually the most valuable investment in a machine learning project.
| Training examples | Test images recognized correctly |
|---|---|
| 10 | 43.2% |
| 20 | 76.7% |
| 50 | 83.6% |
| 100 | 88.3% |
| 200 | 91.9% |
| 400 | 94.3% |
| 800 | 95.7% |
| 1,257 | 96.7% |
Our own experiment: dataset Optical Recognition of Handwritten Digits (E. Alpaydin, C. Kaynak, UCI Machine Learning Repository, license CC BY 4.0), 1797 images of 8 × 8 pixels, 540 of them held back for testing; scikit-learn 1.9.1, fixed random seed, so every run gives the same numbers.
Unsupervised learning: finding groups without answers
In unsupervised learning, the model gets no labels at all. It has to find structure on its own. We gave the same images, without the digits, to k-means, a method that sorts data into a fixed number of groups of similar items, and asked for ten groups.
k-means never learned what a “7” is. Still, the groups it found matched the digits surprisingly well: if you label each group with the digit that is most common in it, 79.4% of all images land in the right group, and all 10 digits get a group of their own. The method found that images of the same digit look alike. Unlabeled data, in other words, still contains structure. In business, the same idea finds customer segments or unusual transactions, where nobody has labels to begin with.
Reinforcement learning: trial, error and reward
In reinforcement learning, there are no examples at all. An agent acts in an environment, gets a reward when it reaches a goal and learns from experience which actions pay off. We built a small maze: a 5 × 5 grid with walls, a start in one corner and a goal in the opposite one. The agent, using a method called Q-learning, started without any knowledge and moved at random.
In its first ten attempts, the agent needed 68.3 steps on average to reach the goal. In its last ten, 8.1, close to the shortest possible path of 8 steps. Nobody showed it the way; the reward alone was enough. The same principle trains game-playing AI, walking robots and, in a more complex form called reinforcement learning from human feedback, the behavior of large language models. It works best where an agent can try often and safely, which is why games and simulations are its natural home.
How we ran the experiment
We used the Python library scikit-learn and the copy of the digit dataset that comes with it. For the learning curve, we drew training sets of different sizes from the training images five times each and report the average; all results are measured on the same held-back test images. k-means ran without the labels, which we used only afterward to check the groups. The maze agent used Q-learning with a reward of 1 for reaching the goal and a small penalty for every step, so that shorter paths pay off. All random choices use a fixed seed, so the numbers on this page come out the same on every run; the dataset, the test share and the seed are also listed below each table. The next part of this level explains the two phases of every AI model’s life: training, when it learns, and inference, when it answers. For a broader introduction to classification and model families, see how machine learning works in our learning path AI Development.