Dimensionality reduction without the hand waving

The defect images in Clustering when nobody labelled anything were clustered on eight principal components rather than on several hundred raw measurements. That step is usually described in a sentence and skipped, which is a mistake, because the eight components decided what the clustering could possibly find. Anything the reduction discarded was not available to any later step.

Principal component analysis is worth doing by hand once, on numbers small enough that the totals close exactly. After that it stops being a black box and becomes a rotation.

Five points, two columns, one calculation

Five observations with two measurements each: (2, 1), (3, 3), (4, 2), (5, 5), (6, 4).

Subtract the means, which are 4 and 3, because PCA is about spread around the centre and nothing else. The centred points are (-2, -2), (-1, 0), (0, -1), (1, 2) and (2, 1).

Compute the three numbers that describe the spread, dividing by four because there are five observations.

The variance of the first column is 4 plus 1 plus 0 plus 1 plus 4, divided by 4, which is 2.5. The variance of the second is 4 plus 0 plus 1 plus 4 plus 1, divided by 4, which is also 2.5. Their covariance is the products added, 4 plus 0 plus 0 plus 2 plus 2, divided by 4, which is 2.0.

The total variance in the data is 2.5 plus 2.5, which is 5.

Now the only step that needs a result rather than arithmetic. For a two by two matrix with equal diagonal entries and off-diagonal entries of 2.0, the two directions of interest are the diagonals, and the variances along them are the diagonal entry plus and minus the off-diagonal entry: 4.5 and 0.5. You can check this without any linear algebra by projecting.

Project each centred point onto the direction that runs up and to the right at 45 degrees, which means adding its two coordinates and dividing by the square root of two. The five projections are minus 2.828, minus 0.707, minus 0.707, plus 2.121, plus 2.121. Square them, add, divide by four: 18 divided by 4, which is 4.5.

Project onto the perpendicular direction, subtracting the coordinates instead. The projections are 0, minus 0.707, plus 0.707, minus 0.707, plus 0.707. Squared and added and divided by four: 2 divided by 4, which is 0.5.

Those two add to 5, the total variance, exactly. That is the entire content of PCA. It finds a new set of perpendicular axes, ordered so the first captures as much of the spread as possible, then the second as much of what is left, and so on, and it accounts for all of the variance without creating or destroying any. Jolliffe and Cadima describe it as creating new uncorrelated variables that successively maximise variance, and the scikit-learn documentation as decomposing a dataset into successive orthogonal components that explain a maximum amount of the variance.

Keeping one component here retains 4.5 of 5, which is 90 percent, and turns two columns into one. That percentage is the only claim the method makes.

What the 90 percent does not mean

Three misreadings of that number cause real damage.

It is not 90 percent of the information. It is 90 percent of the squared spread. If the thing you care about lives in a direction with little spread, PCA throws it away first and reports a high retained percentage while doing it. A defect that shows up as a small consistent shift in one measurement, against columns that vary widely for irrelevant reasons, is exactly this case. PCA never looks at the target. It cannot know which directions matter, because nothing told it what matters.

It is not scale invariant. Variance carries the square of whatever unit the column is in. The scikit-learn documentation is precise about the default: PCA centres but does not scale the input data. So a column of rupees with values in the hundreds of thousands will dominate the first component over a column of ratios between zero and one, and the resulting components describe the choice of units rather than the phenomenon. Standardise every column first unless the columns are already commensurable, and put the standardiser in the same pipeline as the reducer so it is fitted on training rows only, for the reasons in The data comes first.

A component is not a concept. The first component here is a 45 degree direction, which is a weighted mixture of both original columns. Sometimes a mixture is interpretable and someone can name it. Often it is not, and naming it anyway is where PCA acquires its reputation for hand waving. The honest statement is that the component is a direction, its loadings say which columns contribute, and whether it means anything is a subject-matter question.

For the practical side, the documentation notes that when only a subset of components is needed the randomized solver is much more efficient, with a time complexity proportional to the number of components rather than to the smaller dimension of the data, and that IncrementalPCA processes mini-batches for data that does not fit in memory. Reduction is not usually the expensive part of a pipeline.

t-SNE and UMAP answer a different question

PCA preserves variance globally and is a linear map, so it cannot unfold a curved structure. Two methods dominate the case where the goal is a picture rather than a transform, and both are commonly misread.

Van der Maaten and Hinton's t-SNE gives each datapoint a location in a two or three dimensional map, improving on earlier stochastic neighbour embedding by being easier to optimise and by avoiding the crowding of points at the centre of the map. It works by matching a distribution over pairwise neighbour relationships in the original space with one in the map, which is a local objective: it tries hard to keep near things near, and makes no comparable promise about far things.

McInnes, Healy and Melville's UMAP is built on a different foundation, described as Riemannian geometry and algebraic topology, and they report that it rivals t-SNE for visualisation quality while running faster and preserving more global structure. They also note a practical difference: it has no computational restriction on the embedding dimension, so unlike t-SNE it is viable as a general purpose reduction step rather than only as a way to draw a picture.

The most useful writing on how to read the output is Wattenberg, Viegas and Johnson's interactive article, which sets out what a t-SNE plot does not license. Three findings are worth memorising.

Cluster sizes carry no information. Their statement is direct: you cannot see relative sizes of clusters in a t-SNE plot, because the algorithm equalises them by adjusting for density. A blob twice the width of another is not twice as common or twice as varied.

Distances between clusters may mean nothing. They write that distances between well-separated clusters in a t-SNE plot may mean nothing, and that capturing global geometry requires tuning the perplexity parameter, with no single value working across datasets whose clusters differ in size. Two groups drawn at opposite ends of the picture are not thereby more different than two drawn adjacent.

Random data can look structured. At low perplexity, points with no structure at all form clumps, and recognising those clumps as noise is part of reading these plots. The commonly cited range for perplexity, which they quote as typically between 5 and 50, is a starting point rather than a safe default, and anyone presenting one embedding without having looked at several is presenting a parameter choice.

How to use these in a pipeline without lying to yourself

• Fit the reducer on the training rows only. PCA computes its directions from the data, so fitting it on the whole dataset before splitting puts test-set structure into the training representation. It is the same leakage path as scaling before the split, and it inflates the reported score exactly as described in Train, test, and the lie of a single score.
• Choose the number of components against the downstream task, not against a retained-variance target. Ninety-five percent is a habit, not a criterion. Try several counts, score the model that uses them, and pick on that.
• Check whether reduction helps at all. Tree ensembles handle uninformative columns cheaply, as noted in Random forests and why averaging works, so reducing before a forest often costs accuracy and buys nothing. Distance-based methods, which includes k-means and kernel support vector machines, benefit far more.
• Do not feed a t-SNE or UMAP embedding into a classifier without a very specific reason. These embeddings are optimised for viewing, depend on parameters and on random initialisation, and the coordinates are not a stable representation of anything.
• Report what a picture is evidence of. Well-separated groups are evidence that a separation exists in the original space. The number of groups, their sizes, and the distances between them are not conclusions the picture supports.

The eight components that went into the defect clustering retained 91 percent of the variance in the measurement table. The fourth cluster, the one that turned out to be a failing overhead light, was visible because lighting produced a very large, very consistent spread across most of the columns, which is precisely the sort of thing PCA puts in an early component. Had the difference of interest been a subtle one instead, the same eight components would have discarded it and reported the same 91 percent.

That is the shape of every method in this series. Each one optimises a stated quantity exactly, and the work that decides whether it was worth doing sits in the gap between that quantity and the decision it was supposed to inform.

References

1. Decomposing signals in components, Principal component analysis. scikit-learn documentation, version 1.9.0, 2026.
2. Principal component analysis, a review and recent developments. Ian T. Jolliffe and Jorge Cadima, Philosophical Transactions of the Royal Society A, volume 374, issue 2065, 2016.
3. Visualizing Data using t-SNE. Laurens van der Maaten and Geoffrey Hinton, Journal of Machine Learning Research, volume 9, pages 2579 to 2605, 2008.
4. UMAP, Uniform Manifold Approximation and Projection for Dimension Reduction. Leland McInnes, John Healy and James Melville, arXiv, 2018.
5. How to Use t-SNE Effectively. Martin Wattenberg, Fernanda Viegas and Ian Johnson, Distill, 2016.