Spam or not spam, fraud or normal payment, which department should get this ticket: most decisions that companies hand to machine learning are classification, a choice between a few fixed categories. A classifier is easy to build. Knowing whether it is good is harder, because a single number such as “96% correct” can hide exactly the errors that matter.
This is part 2 of Level 1 in our learning path AI Fundamentals. We use the digit recognizer from Level 0, a logistic regression trained on handwritten digits, and look at its results the way a professional would: with accuracy, a confusion matrix, precision and recall, and the threshold that decides between them.
Accuracy and what it hides
Accuracy is the share of correct answers. Our classifier recognized 96.7% of the 540 test images, which means 18 errors. That sounds good, and for digits it is. But accuracy treats all errors as equal, and it can be misleading when the categories are unbalanced. If only 1 in 100 payments is fraud, a “classifier” that calls every payment normal reaches 99% accuracy and catches no fraud at all. Before you trust an accuracy figure, ask how often each category occurs and which errors are expensive.
The confusion matrix: which errors happen
A confusion matrix counts, for every true category, what the model predicted. Correct answers sit on the diagonal; every cell off the diagonal is a specific kind of mistake.
The matrix shows what accuracy hides: the errors are not spread evenly. The most common mistake was an 8 read as a 1, 5 times. The digit the model found hardest was the 8: it recognized only 88.5% of them. In a real project, this is where you would look next: at the hard examples, at whether more training data for that category helps, or whether the input itself, here a tiny 8 × 8 image, loses the detail that tells the categories apart.
Precision and recall
For a single category, two questions matter, and they pull in opposite directions:
- Precision: of everything the model flagged as “9”, how much really was a 9? Low precision means many false alarms.
- Recall: of all the real 9s, how many did the model find? Low recall means many misses.
| The model says “9” | The model says “not 9” | |
|---|---|---|
| It really is a 9 | Hit (true positive) | Miss (false negative) |
| It is not a 9 | False alarm (false positive) | Correct rejection (true negative) |
Precision counts the hits among all alarms, recall the hits among all real 9s. A spam filter needs high precision, because a real email in the spam folder is costly. A fraud check or a medical screening test needs high recall, because a missed case is costly. Which matters more is not a technical question but a business decision.
The threshold decides
A classifier does not just say “9” or “not 9”. It gives a probability, and you choose the threshold above which it counts as a 9. We varied this threshold for our model and counted the results on the test images.
At a low threshold of 0.1, the model found every 9, a recall of 100%, but its precision was only 79.4%. At a high threshold of 0.9, every flagged image really was a 9, a precision of 100%, but it missed many of them: recall fell to 55.6%. Same model, same data, very different behavior. The threshold is one of the most important settings of any classifier, and the default of 0.5 is rarely the right one for a real business case.
| Threshold | Precision | Recall | False alarms | Missed 9s |
|---|---|---|---|---|
| 0.1 | 79.4% | 100.0% | 14 | 0 |
| 0.3 | 93.0% | 98.2% | 4 | 1 |
| 0.5 | 96.0% | 88.9% | 2 | 6 |
| 0.7 | 97.8% | 83.3% | 1 | 9 |
| 0.9 | 100.0% | 55.6% | 0 | 24 |
Our own experiment: dataset Optical Recognition of Handwritten Digits (E. Alpaydin, C. Kaynak, UCI Machine Learning Repository, license CC BY 4.0), 1797 images of 8 × 8 pixels, 540 of them held back for testing; scikit-learn 1.9.1, fixed random seed, so every run gives the same numbers.
What to ask about any classifier
When someone presents a classifier, these questions get you past the headline number:
- How was it tested? Only results on data the model never saw count.
- How often is each category? With rare categories, accuracy is almost meaningless.
- Which errors happen? Ask for the confusion matrix, not just one number.
- Precision or recall? Which kind of error is more expensive in your process, and is the threshold set accordingly?
- What does a person check? For costly decisions, uncertain cases near the threshold should go to a human.
The next part shows a classifier that scores perfectly on its training data and still fails: the decision tree, and the problem of overfitting.