{"author":{"@type":"Person","name":"Alexander Bock","url":"https://abxk.ai/en/autor/alexander-bock/"},"categories":["AI Development"],"content":{"markdown":"A decision tree works like a game of twenty questions. \"Is the pixel in the middle dark? If yes: is the pixel at the top left dark too?\" After a few such questions, the tree reaches an answer. Trees are popular in business because anyone can follow their logic, and they are the building block of some of the most successful methods for table data. They also show the most important trap in machine learning more clearly than any other model: **overfitting**.\n\nThis is part 3 of Level 1 in our learning path AI Fundamentals. You will learn how a tree finds its questions, what happened when we let our trees grow deeper and deeper on the handwritten digits from Level 0, and how combining many trees into a random forest solves the problem.\n\n## How a tree finds its questions\n\n![A decision tree with three levels: each box shows the 8 × 8 image with one marked pixel and asks whether it has more ink than a threshold; the eight leaves at the bottom show the most common digit and its share, from almost pure zeros and nines to a mixed leaf](/images/learn/en/decision-trees-overfitting-tree.webp \"The first three levels of a real tree from our experiment. Each leaf shows its most common digit.\")\n\nDuring training, the tree looks at all possible questions, for every pixel and every brightness value, and picks the one that best separates the digits: after the split, each group should contain as few different digits as possible. Then it repeats the search inside each group, and so on. Each question is a **node**, each final group a **leaf**, and the number of questions from the top to a leaf is the **depth** of the tree.\n\nThe figure shows the first three levels of a real tree from our experiment. Its very first question checks a pixel in the middle of the image: if it has almost no ink, the digit is most likely a 0, because a 0 is open in the middle. After three questions, this tree recognizes {{\u003c num \"ml_experiments\" \"tree3_test_pct\" \u003e}}% of the test images. Most of its leaves are still mixed; to tell all ten digits apart, it needs many more questions.\n\nThe tree never stops on its own as long as a leaf still contains mixed digits. If you let it, it keeps asking until every training image sits in a leaf of its own kind. That is the moment it stops learning patterns and starts memorizing examples.\n\n## Our experiment: deeper is not better\n\nWe trained trees with a maximum depth from 1 to 20 on the training images and measured the share of correct answers twice: on the training images themselves and on the test images that the tree had never seen.\n\n![Line chart: accuracy on training data and on test data by tree depth from 1 to 20; training accuracy climbs to 100 percent, test accuracy stops improving at around 85 percent](/images/learn/en/decision-trees-overfitting-depth.webp \"Training accuracy keeps rising with depth; test accuracy stops improving. The gap is overfitting.\")\n\nThe result shows overfitting in its purest form. With enough depth, the tree recognized {{\u003c num \"ml_experiments\" \"tree_deep_train_pct\" \u003e}}% of its training images: perfect. On the test images, it reached only {{\u003c num \"ml_experiments\" \"tree_deep_test_pct\" \u003e}}%. The best test result, {{\u003c num \"ml_experiments\" \"tree_best_test_pct\" \u003e}}%, came at a depth of {{\u003c num \"ml_experiments\" \"tree_best_depth\" \u003e}}; every question beyond that only helped the tree remember individual training images, including their quirks and noise.\n\n{{\u003c data \"ml-tree\" \u003e}}\n\nThis is why the gap between training and test results is the first thing to check in any machine learning project. A model that is much better on its training data than on new data has learned its examples, not the task. The opposite problem exists too: our shallowest trees were poor on both kinds of data, because a few questions are simply not enough to tell ten digits apart. That is called **underfitting**.\n\n| Training result | Test result | Diagnosis | What usually helps |\n|---|---|---|---|\n| Low | Low | Underfitting: the model is too simple | A more flexible model, better features, longer training |\n| High | Much lower | Overfitting: the model memorized its examples | Limit complexity, more data, combine models |\n| High | Close to training | Good fit | Confirm on fresh data before relying on it |\n\n## The fix: many trees instead of one\n\nA **random forest** trains many trees, each on a slightly different random sample of the training data and with a random subset of the features to choose from at each question. Every tree overfits in its own way, but their errors differ, so when the trees vote, many errors cancel out. Our forest of 200 trees recognized {{\u003c num \"ml_experiments\" \"forest_test_pct\" \u003e}}% of the test images, compared with {{\u003c num \"ml_experiments\" \"tree_test_pct\" \u003e}}% for a single tree grown without limits.\n\n{{\u003c data \"ml-models\" \u003e}}\n\nThe single tree is the weakest method in this table, and the forest of the same kind of trees is among the strongest. The same idea, combining many weak models, is called an **ensemble**, and a related method called gradient boosting, which trains each new tree to fix the errors of the previous ones, is among the most successful methods for business data stored in tables.\n\n## How to avoid overfitting\n\n- **Always test on held-back data.** Never judge a model by its training results alone.\n- **Limit the complexity.** For trees, a maximum depth or a minimum number of examples per leaf; for other models, comparable settings.\n- **Use more data.** The more examples, the harder it is to memorize them all.\n- **Combine models.** Ensembles like random forests reduce the variance of single models.\n- **Keep a final test set untouched.** If you tune your settings on the test data many times, you overfit to the test data too.\n\nThe next part compares six classic methods on the same data and explains cross-validation, the standard way to compare models fairly.","text":"A decision tree works like a game of twenty questions. \u0026ldquo;Is the pixel in the middle dark? If yes: is the pixel at the top left dark too?\u0026rdquo; After a few such questions, the tree reaches an answer. Trees are popular in business because anyone can follow their logic, and they are the building block of some of the most successful methods for table data. They also show the most important trap in machine learning more clearly than any other model: overfitting.\nThis is part 3 of Level 1 in our learning path AI Fundamentals. You will learn how a tree finds its questions, what happened when we let our trees grow deeper and deeper on the handwritten digits from Level 0, and how combining many trees into a random forest solves the problem.\nHow a tree finds its questions The first three levels of a real tree from our experiment. Each leaf shows its most common digit. During training, the tree looks at all possible questions, for every pixel and every brightness value, and picks the one that best separates the digits: after the split, each group should contain as few different digits as possible. Then it repeats the search inside each group, and so on. Each question is a node, each final group a leaf, and the number of questions from the top to a leaf is the depth of the tree.\nThe figure shows the first three levels of a real tree from our experiment. Its very first question checks a pixel in the middle of the image: if it has almost no ink, the digit is most likely a 0, because a 0 is open in the middle. After three questions, this tree recognizes 48% of the test images. Most of its leaves are still mixed; to tell all ten digits apart, it needs many more questions.\nThe tree never stops on its own as long as a leaf still contains mixed digits. If you let it, it keeps asking until every training image sits in a leaf of its own kind. That is the moment it stops learning patterns and starts memorizing examples.\nOur experiment: deeper is not better We trained trees with a maximum depth from 1 to 20 on the training images and measured the share of correct answers twice: on the training images themselves and on the test images that the tree had never seen.\nTraining accuracy keeps rising with depth; test accuracy stops improving. The gap is overfitting. The result shows overfitting in its purest form. With enough depth, the tree recognized 100% of its training images: perfect. On the test images, it reached only 85.2%. The best test result, 85.7%, came at a depth of 10; every question beyond that only helped the tree remember individual training images, including their quirks and noise.\nDecision tree: accuracy on training and test data by depth Tree depthTraining dataTest data 120.1%19.3% 232.2%31.1% 349.2%48.0% 570.6%67.2% 894.8%83.0% 1099.4%85.7% 15100.0%85.2% 20100.0%85.2% Source as for the first table on this page.\nThis is why the gap between training and test results is the first thing to check in any machine learning project. A model that is much better on its training data than on new data has learned its examples, not the task. The opposite problem exists too: our shallowest trees were poor on both kinds of data, because a few questions are simply not enough to tell ten digits apart. That is called underfitting.\nDecision Trees and Overfitting: When a Model Learns by Heart Training result Test result Diagnosis What usually helps Low Low Underfitting: the model is too simple A more flexible model, better features, longer training High Much lower Overfitting: the model memorized its examples Limit complexity, more data, combine models High Close to training Good fit Confirm on fresh data before relying on it The fix: many trees instead of one A random forest trains many trees, each on a slightly different random sample of the training data and with a random subset of the features to choose from at each question. Every tree overfits in its own way, but their errors differ, so when the trees vote, many errors cancel out. Our forest of 200 trees recognized 96.5% of the test images, compared with 85.2% for a single tree grown without limits.\nSix classic methods on the same data MethodTest images recognized correctly Logistic regression96.7% k-nearest neighbors (k = 3)98.7% Decision tree85.2% Random forest (200 trees)96.5% Support vector machine98.3% Neural network (64 neurons)97.8% Source as for the first table on this page.\nThe single tree is the weakest method in this table, and the forest of the same kind of trees is among the strongest. The same idea, combining many weak models, is called an ensemble, and a related method called gradient boosting, which trains each new tree to fix the errors of the previous ones, is among the most successful methods for business data stored in tables.\nHow to avoid overfitting Always test on held-back data. Never judge a model by its training results alone. Limit the complexity. For trees, a maximum depth or a minimum number of examples per leaf; for other models, comparable settings. Use more data. The more examples, the harder it is to memorize them all. Combine models. Ensembles like random forests reduce the variance of single models. Keep a final test set untouched. If you tune your settings on the test data many times, you overfit to the test data too. The next part compares six classic methods on the same data and explains cross-validation, the standard way to compare models fairly.\n","words":936},"date":"2026-10-18","description":"How a decision tree decides, why a deeper tree scores 100% on its training data and still fails on new data, and how a random forest fixes it, shown with our data.","image":"https://abxk.ai/images/cover/decision-trees-overfitting.webp","json":"https://abxk.ai/en/ai-development/fundamentals/decision-trees-overfitting/index.json","kind":"page","language":"en","learning_path":{"category":"ai-development","level":1,"level_name":"Beginner"},"modified":"2026-10-18","related_tools":["Runpod","Pinecone","MindStudio"],"section":"ai-development","seo_title":"Decision Trees and Overfitting, Shown with Data","tags":["decision-trees","overfitting","random-forest","machine-learning","beginner"],"title":"Decision Trees and Overfitting: When a Model Learns by Heart","translations":[{"json":"https://abxk.ai/de/ai-development/fundamentals/decision-trees-overfitting/index.json","language":"de","url":"https://abxk.ai/de/ai-development/fundamentals/decision-trees-overfitting/"}],"url":"https://abxk.ai/en/ai-development/fundamentals/decision-trees-overfitting/"}