{"author":{"@type":"Person","name":"Alexander Bock","url":"https://abxk.ai/en/autor/alexander-bock/"},"categories":["AI Development"],"content":{"markdown":"Most data in a company has no labels. Nobody has marked which customers belong together, which machine readings are normal or which documents cover the same topic. **Unsupervised learning** works with exactly this kind of data. It does not learn to predict a known answer; it looks for structure: groups of similar items, or a simpler way to describe the data.\n\nThis is part 5 of Level 1 in our learning path AI Fundamentals. We return to the handwritten digits, this time without telling the methods which digit each image shows. You will learn how **k-means clustering** finds groups, why the hardest question is how many groups to look for, and how **principal component analysis (PCA)** compresses each image from 64 numbers to a handful, plus what that does to the groups.\n\n## How k-means finds groups\n\nk-means needs one setting from you: the number of groups, called **k**. Then it repeats two simple steps:\n\n1. **Assign.** Every data point joins the group whose center is closest.\n2. **Update.** Every group center moves to the average of its members.\n\nAt the start, the centers are placed at random. After a few rounds, the assignments stop changing, and the method is done. Because the result depends on the random start, it is usually run several times, and the best run is kept. We used ten runs.\n\n![Three panels: on the left, unlabeled points with three randomly placed centers; in the middle, each point colored by its nearest center; on the right, the centers have moved to the middle of their groups](/images/learn/en/clustering-and-pca-steps.webp \"k-means repeats two steps: assign every point to the nearest center, then move each center to the middle of its group.\")\n\n## Our experiment: ten groups, no labels\n\nWe asked k-means for ten groups of the {{\u003c num \"ml_experiments\" \"images\" \u003e}} digit images, without the labels. Afterward, we used the labels only to check what each group contains.\n\n{{\u003c data \"ml-kmeans\" \u003e}}\n\nSome groups are almost pure. The group that mostly contains zeros is {{\u003c num \"ml_experiments\" \"kmeans_best_purity_pct\" \u003e}}% zeros, because a zero looks like nothing else. Other groups are mixed: in the group where 8 is the most common digit, only {{\u003c num \"ml_experiments\" \"kmeans_worst_purity_pct\" \u003e}}% of the {{\u003c num \"ml_experiments\" \"kmeans_worst_size\" \u003e}} images are actually 8s. Almost all of the others, {{\u003c num \"ml_experiments\" \"kmeans_worst_second_n\" \u003e}} images, are 1s: by the method's measure of distance, many 1s are closer to the average 8 than to the average 1. The 1s end up split across several groups. Overall, {{\u003c num \"ml_experiments\" \"kmeans_majority_pct\" \u003e}}% of the images land in the group of their own digit. That is impressive for a method that never saw a single label, and it is also a warning: groups found by an algorithm are similar by its measure of distance, not necessarily by the meaning you care about.\n\n## How many groups? The methods disagree\n\nWith the digits, we know the right answer: ten. With real business data, nobody does. Two common methods are supposed to help.\n\nThe **elbow method** runs k-means for many values of k and plots how far the points are from their group centers, in total. More groups always reduce this distance, so you look for the \"elbow\", the point where adding groups stops paying off. The **silhouette score** checks for every point whether it is much closer to its own group than to the next one; a higher average score means cleaner groups.\n\n![Two line charts for 2 to 15 groups: on the left, the total distance to the group centers falls steadily without a sharp bend; on the right, the silhouette score rises to a peak at 9 groups and stays almost flat after that](/images/learn/en/clustering-and-pca-elbow.webp \"The elbow method points to 8 groups, the silhouette score to 9. The true number of digits is 10.\")\n\nOn our data, the elbow method pointed to {{\u003c num \"ml_experiments\" \"elbow_k\" \u003e}} groups and the silhouette score to {{\u003c num \"ml_experiments\" \"silhouette_k\" \u003e}}. Neither found the true ten. The curve bends only gently, and the best silhouette score, {{\u003c num \"ml_experiments\" \"silhouette_best\" \"2\" \u003e}} on a scale from −1 to 1, is low, which means that the groups overlap a lot. This is typical for real data. The methods give you a range to try, but the final number of groups is a decision you make based on what the groups are for: five customer segments that a sales team can work with may be more useful than the twelve that score slightly better.\n\n## PCA: the same images in fewer numbers\n\nEach digit image consists of 64 numbers, one per pixel. But the pixels are not independent: if the middle of a 0 is empty, its neighbors are probably empty too. **Principal component analysis (PCA)** uses this. It finds new directions in the data, called **principal components**, ordered by how much of the variation between the images they capture. The first direction captures the most, the second the most of what is left, and so on. Keeping only the first few directions describes each image with fewer numbers while losing as little as possible.\n\n![Line chart: share of the variation between the digit images kept by the first 1 to 64 principal components; the curve rises steeply at first and passes 50 percent at 5 components, 90 percent at 21 and 95 percent at 29](/images/learn/en/clustering-and-pca-variance.webp \"How much of the variation between the images the first principal components keep.\")\n\nThe curve rises steeply at first. Just {{\u003c num \"ml_experiments\" \"pca_for_50\" \u003e}} directions keep {{\u003c num \"ml_experiments\" \"pca_5_pct\" \u003e}}% of the variation, {{\u003c num \"ml_experiments\" \"pca_for_90\" \u003e}} keep more than 90%, and {{\u003c num \"ml_experiments\" \"pca_for_95\" \u003e}} keep more than 95%. The last directions add almost nothing: each of them describes a single pixel at the left or right edge of the image, and those pixels are nearly always empty.\n\nThe first two directions alone keep only {{\u003c num \"ml_experiments\" \"pca_first2_pct\" \u003e}}% of the variation, but they are enough to draw the whole dataset on a flat page. In this map, some digits form clear islands: zeros at the bottom, fours on the right, threes on the left and sevens at the top. Others overlap in the middle, among them the 1s, 8s and 9s, the digits that k-means mixed up most.\n\n![Scatter chart: 600 digit images placed by their first two principal components; zeros, fours, threes and sevens are highlighted and form separate regions at the bottom, right, left and top, while the other digits are gray and overlap in the middle](/images/learn/en/clustering-and-pca-map.webp \"600 digit images on the first two principal components. Four digits are highlighted; the others are gray.\")\n\n## Compress first, then cluster\n\nFewer numbers per image make every later step faster and cheaper. But do the groups suffer? We ran k-means again after compressing the images with PCA.\n\n{{\u003c data \"ml-pca\" \u003e}}\n\nWith only two directions, k-means put just {{\u003c num \"ml_experiments\" \"pk2_pct\" \u003e}}% of the images in the group of their digit, because too much information was lost. With {{\u003c num \"ml_experiments\" \"pca_for_90\" \u003e}} directions, the result was {{\u003c num \"ml_experiments\" \"pk21_pct\" \u003e}}%, as good as with all 64 pixels ({{\u003c num \"ml_experiments\" \"pk64_pct\" \u003e}}%), with a third of the numbers. The directions that compression dropped carried little that mattered for the groups. This is why PCA and related methods are a standard first step before clustering, search and visualization. The embeddings that AI systems use to compare texts and images follow a related idea: they describe each item with a list of numbers in which similar items end up close together.\n\n| Question | Clustering | PCA |\n|---|---|---|\n| What does it do? | Puts similar items into groups | Describes items with fewer numbers |\n| What do you choose? | The number of groups | The number of directions to keep |\n| Typical uses | Customer segments, grouping documents, spotting unusual cases | Charts of complex data, faster training, removing noise |\n| Main risk | Groups that are similar by distance but not by meaning | Throwing away a small detail that matters |\n\n## How we ran the experiment\n\nWe used the Python library scikit-learn and its copy of the digit dataset, with all {{\u003c num \"ml_experiments\" \"images\" \u003e}} images and pixel values scaled from 0 to 1. k-means ran ten times from different random starts for every number of groups, and the best run was kept. To find the elbow automatically, we took the point with the largest distance below a straight line from the first to the last point of the curve. The labels were used only to evaluate the groups, never to form them. All random choices use a fixed seed, so the numbers on this page come out the same on every run. Level 2 of this learning path turns to deep learning: what changes when networks get many layers.","text":"Most data in a company has no labels. Nobody has marked which customers belong together, which machine readings are normal or which documents cover the same topic. Unsupervised learning works with exactly this kind of data. It does not learn to predict a known answer; it looks for structure: groups of similar items, or a simpler way to describe the data.\nThis is part 5 of Level 1 in our learning path AI Fundamentals. We return to the handwritten digits, this time without telling the methods which digit each image shows. You will learn how k-means clustering finds groups, why the hardest question is how many groups to look for, and how principal component analysis (PCA) compresses each image from 64 numbers to a handful, plus what that does to the groups.\nHow k-means finds groups k-means needs one setting from you: the number of groups, called k. Then it repeats two simple steps:\nAssign. Every data point joins the group whose center is closest. Update. Every group center moves to the average of its members. At the start, the centers are placed at random. After a few rounds, the assignments stop changing, and the method is done. Because the result depends on the random start, it is usually run several times, and the best run is kept. We used ten runs.\nk-means repeats two steps: assign every point to the nearest center, then move each center to the middle of its group. Our experiment: ten groups, no labels We asked k-means for ten groups of the 1,797 digit images, without the labels. Afterward, we used the labels only to check what each group contains.\nk-means with ten groups, without labels: size and purity of each group Most common digit in the groupImages in the groupShare of that digitNext most common digits (images) 017998.9%2 (1), 6 (1) 416898.8%0 (1), 5 (1) 618297.3%1 (2), 8 (2) 515787.3%8 (6), 9 (6) 719885.9%4 (8), 3 (7) 318185.6%2 (13), 9 (6) 217684.1%1 (24), 8 (3) 18961.8%9 (20), 8 (6) 924157.7%8 (48), 5 (41) 822645.1%1 (99), 2 (8) Source as for the first table on this page.\nSome groups are almost pure. The group that mostly contains zeros is 98.9% zeros, because a zero looks like nothing else. Other groups are mixed: in the group where 8 is the most common digit, only 45.1% of the 226 images are actually 8s. Almost all of the others, 99 images, are 1s: by the method\u0026rsquo;s measure of distance, many 1s are closer to the average 8 than to the average 1. The 1s end up split across several groups. Overall, 79.4% of the images land in the group of their own digit. That is impressive for a method that never saw a single label, and it is also a warning: groups found by an algorithm are similar by its measure of distance, not necessarily by the meaning you care about.\nHow many groups? The methods disagree With the digits, we know the right answer: ten. With real business data, nobody does. Two common methods are supposed to help.\nThe elbow method runs k-means for many values of k and plots how far the points are from their group centers, in total. More groups always reduce this distance, so you look for the \u0026ldquo;elbow\u0026rdquo;, the point where adding groups stops paying off. The silhouette score checks for every point whether it is much closer to its own group than to the next one; a higher average score means cleaner groups.\nThe elbow method points to 8 groups, the silhouette score to 9. The true number of digits is 10. On our data, the elbow method pointed to 8 groups and the silhouette score to 9. Neither found the true ten. The curve bends only gently, and the best silhouette score, 0.19 on a scale from −1 to 1, is low, which means that the groups overlap a lot. This is typical for real data. The methods give you a range to try, but the final number of groups is a decision you make based on what the groups are for: five customer segments that a sales team can work with may be more useful than the twelve that score slightly better.\nPCA: the same images in fewer numbers Each digit image consists of 64 numbers, one per pixel. But the pixels are not independent: if the middle of a 0 is empty, its neighbors are probably empty too. Principal component analysis (PCA) uses this. It finds new directions in the data, called principal components, ordered by how much of the variation between the images they capture. The first direction captures the most, the second the most of what is left, and so on. Keeping only the first few directions describes each image with fewer numbers while losing as little as possible.\nHow much of the variation between the images the first principal components keep. The curve rises steeply at first. Just 5 directions keep 54.5% of the variation, 21 keep more than 90%, and 29 keep more than 95%. The last directions add almost nothing: each of them describes a single pixel at the left or right edge of the image, and those pixels are nearly always empty.\nThe first two directions alone keep only 28.5% of the variation, but they are enough to draw the whole dataset on a flat page. In this map, some digits form clear islands: zeros at the bottom, fours on the right, threes on the left and sevens at the top. Others overlap in the middle, among them the 1s, 8s and 9s, the digits that k-means mixed up most.\n600 digit images on the first two principal components. Four digits are highlighted; the others are gray. Compress first, then cluster Fewer numbers per image make every later step faster and cheaper. But do the groups suffer? We ran k-means again after compressing the images with PCA.\nCompressing with PCA: directions kept, variation kept and k-means afterward Directions kept (of 64)Variation keptk-means: images in the group of their digit 228.5%59.4% 554.5%71.7% 1380.3%78.7% 2190.3%79.6% 64100.0%79.4% Source as for the first table on this page.\nWith only two directions, k-means put just 59.4% of the images in the group of their digit, because too much information was lost. With 21 directions, the result was 79.6%, as good as with all 64 pixels (79.4%), with a third of the numbers. The directions that compression dropped carried little that mattered for the groups. This is why PCA and related methods are a standard first step before clustering, search and visualization. The embeddings that AI systems use to compare texts and images follow a related idea: they describe each item with a list of numbers in which similar items end up close together.\nClustering and PCA: Finding Structure Without Labels Question Clustering PCA What does it do? Puts similar items into groups Describes items with fewer numbers What do you choose? The number of groups The number of directions to keep Typical uses Customer segments, grouping documents, spotting unusual cases Charts of complex data, faster training, removing noise Main risk Groups that are similar by distance but not by meaning Throwing away a small detail that matters How we ran the experiment We used the Python library scikit-learn and its copy of the digit dataset, with all 1,797 images and pixel values scaled from 0 to 1. k-means ran ten times from different random starts for every number of groups, and the best run was kept. To find the elbow automatically, we took the point with the largest distance below a straight line from the first to the last point of the curve. The labels were used only to evaluate the groups, never to form them. All random choices use a fixed seed, so the numbers on this page come out the same on every run. Level 2 of this learning path turns to deep learning: what changes when networks get many layers.\n","words":1326},"date":"2026-10-18","description":"How k-means finds groups in data nobody labeled, why choosing the number of groups is hard, and how PCA shrinks 64 pixels to a few numbers, tested on real data.","image":"https://abxk.ai/images/cover/clustering-and-pca.webp","json":"https://abxk.ai/en/ai-development/fundamentals/clustering-and-pca/index.json","kind":"page","language":"en","learning_path":{"category":"ai-development","level":1,"level_name":"Beginner"},"modified":"2026-10-18","related_tools":["Runpod","Pinecone","MindStudio"],"section":"ai-development","seo_title":"Clustering and PCA Explained with a Real Experiment","tags":["clustering","k-means","pca","unsupervised-learning","beginner"],"title":"Clustering and PCA: Finding Structure Without Labels","translations":[{"json":"https://abxk.ai/de/ai-development/fundamentals/clustering-and-pca/index.json","language":"de","url":"https://abxk.ai/de/ai-development/fundamentals/clustering-and-pca/"}],"url":"https://abxk.ai/en/ai-development/fundamentals/clustering-and-pca/"}