Support vector machines and the margin

A quality team had 340 labelled images of surface defects and no prospect of more, because labelling one takes a metallurgist twenty minutes. Three hundred and forty rows and several hundred derived measurements per image is the shape of problem where a support vector machine is still the sensible first call, and it is worth understanding what it optimises, because it is not the same thing that logistic regression optimises.

A fitted logistic model puts a boundary somewhere reasonable. It does not ask where the most defensible place for that boundary is. That question has an answer, and the answer is geometric.

The widest gap, computed

Four training cases in two features. Two of one class at coordinates (3, 4) and (4, 3). Two of the other at (1, 2) and (2, 1).

Infinitely many straight lines separate these four points. The support vector machine picks the one that leaves the largest clear corridor on both sides.

By symmetry that line is where the sum of the two coordinates equals 5. The two positive points both sit where the sum is 7, and the two negative points both sit where the sum is 3, so the halfway point is 5 and the corridor runs from 3 to 7.

Turning that into a distance takes one more step. Write the boundary as a weight vector and an offset: weights of 1 and 1, offset minus 5. The value of that expression at (3, 4) is 2. To convert this into a distance in the plane, divide by the length of the weight vector, which is the square root of 2, giving a distance of 1.414. The full corridor is twice that, about 2.83 units wide, and no other separating line does better.

The convention in the literature is to rescale the weights so the nearest points sit at exactly plus and minus one, which here means weights of 0.5 and 0.5 and an offset of minus 2.5. The margin is then one divided by the length of the weight vector, so maximising the margin is the same thing as minimising the length of the weight vector. That is why the optimisation is written as a minimisation, and it is the only piece of translation needed to read the standard formulation.

Now add a fifth training case, of the negative class, at (0.5, 0.5). It changes nothing. It is not near the corridor, it is not consulted, and deleting it leaves the fitted model identical. The scikit-learn documentation names the points that do matter: the samples lying on or within the margin boundaries are the support vectors, and when the problem is not linearly separable the support vectors are the samples within the margin boundaries.

That property is unusual and useful. Most of the training set can be discarded after fitting. The stored model is a handful of rows and their coefficients, which is why a support vector machine on 340 images is a small object.

The soft margin, and the setting that governs it

Real defect images overlap. Some borderline cases sit on the wrong side of any straight boundary, and a method that insists on separating everything will contort itself around them.

The soft margin allows violations and charges for them, and the price is the C parameter. The documentation states the trade directly: C trades off misclassification of training examples against simplicity of the decision surface, a low value makes the decision surface smooth while a high value aims at classifying all training examples correctly, and if there are many noisy observations C should be decreased, since decreasing it corresponds to more regularisation.

Read that alongside the arithmetic above. Raising C narrows the corridor so it can dodge individual awkward points. Lowering it widens the corridor and accepts that some points fall inside. Which is right is the bias and variance question from Bias and variance in plain terms, arriving with an explicit dial attached.

The margin idea is not confined to this one method. Bartlett, Freund, Lee and Schapire used the same quantity, the gap between the correct answer and the best wrong one, to explain why boosted classifiers keep improving on test data after their training error reaches zero. Wide margins going with good generalisation is a recurring result rather than a property of one algorithm.

The kernel trick is a change of coordinates you never pay for

Some problems have no straight boundary at all. The smallest example is four points: (0, 0) and (1, 1) of one class, (0, 1) and (1, 0) of the other. Draw them. No line separates the two diagonals.

Add one derived coordinate, the squared difference of the two features. For (0, 0) it is 0. For (1, 1) it is 0. For (0, 1) and (1, 0) it is 1. In that single new coordinate the two classes sit at 0 and at 1, and a threshold at 0.5 separates them perfectly. The problem was never impossible. It was stated in the wrong coordinates.

Constructing such coordinates by hand does not scale, because the useful ones are usually products of pairs or triples of features and there are a great many of those. The kernel trick is the observation that the optimisation never needs the coordinates themselves, only the inner products between pairs of points, and those can often be computed without building the coordinates.

Work the degree two polynomial kernel through and it becomes concrete. For two dimensional points, the square of the ordinary inner product expands to the first coordinates squared multiplied together, plus twice the cross terms multiplied together, plus the second coordinates squared multiplied together. That is exactly the inner product you would get by mapping each point to the three coordinates: first squared, root two times the product of both, second squared. One multiplication and one squaring stands in for a trip through a three dimensional space, and for higher degrees it stands in for a trip through a space with hundreds of coordinates.

The documentation lists the kernels available, of which the radial basis function is the usual default, along with the parameter that shapes it: gamma defines how much influence a single training example has, and the larger it is, the closer other examples must be to be affected. A large gamma produces a boundary that wraps tightly around individual training points, which is overfitting expressed as geometry. C and gamma interact, and the documentation's advice is to search them with cross-validation, spaced exponentially far apart.

Two searched parameters on a small dataset is exactly the situation Raschka's review warns about, since the score used to pick them is not an estimate of future performance any more. His recommendations for small data, nested cross-validation and the combined five by two cross-validated F test for comparing algorithms, exist for this case.

What it costs, and where it stops

It does not scale in rows. The documentation gives the complexity of the solver used by the libsvm-based implementation as scaling between the number of features times the number of samples squared, and the number of features times the number of samples cubed. Doubling the training set can multiply the fitting time by eight. For the linear case, LinearSVC uses a different algorithm that can scale almost linearly to millions of samples, so the linear kernel and the non-linear ones are practically different tools.

It is not scale invariant. Distances drive everything, so a feature measured in millimetres and a feature measured in metres do not carry equal weight. The documentation is unambiguous that scaling the data is highly recommended, either to a fixed interval or to zero mean and unit variance, and that the scaler belongs inside a pipeline so it is fitted on training data only. Fitting it on everything first is one of the leakage paths in The data comes first.

It does not produce probabilities. The distance from the boundary is a confidence ordering, not a frequency. The documentation notes that support vector machines do not directly provide probability estimates and that these are calculated using an expensive five-fold cross-validation, and advises that if confidence scores are required but they do not have to be probabilities, it is better to use the decision function directly. Anyone planning to multiply the output by a cost, as in The cost of a wrong answer, needs the calibrated version and needs to check it against the tests in What probability buys you.

A non-linear kernel is opaque. With a linear kernel the model is a weight per feature and reads like the boundary in Logistic regression and decision boundaries. With a radial basis function kernel the boundary is a sum of bumps around stored training points and no coefficient corresponds to anything a reviewer can name.

Where it still earns its place

Fernandez-Delgado and colleagues, testing 179 classifiers over 121 datasets, put Gaussian kernel support vector machines second only to random forests, at 92.3 percent of the maximum accuracy achieved per dataset against 94.1 percent for the best forest. On a broad sweep of ordinary problems it is a strong default that happens not to be the strongest.

The cases where it is genuinely the better answer share a shape: few rows, many features, and a boundary you expect to be smooth. Text classification with sparse high dimensional inputs. Measurement data from instruments where every column is a real quantity on a comparable scale. Small labelled sets where a forest has too little data to average anything, and a booster has too little to correct anything.

Three hundred and forty defect images with several hundred measurements each is that shape. The team scaled the features, searched C and gamma over a coarse exponential grid with nested cross-validation, and shipped a model that stored 61 support vectors out of 340 rows.

What made the project awkward was not the classifier. It was that a second set of 9,000 images existed with no labels at all, and nobody could say whether they contained two kinds of defect or five.

References

1. Support Vector Machines. scikit-learn documentation, version 1.9.0, 2026.
2. Do we Need Hundreds of Classifiers to Solve Real World Classification Problems?. Manuel Fernandez-Delgado, Eva Cernadas, Senen Barro and Dinani Amorim, Journal of Machine Learning Research, volume 15, pages 3133 to 3181, 2014.
3. Boosting the margin, a new explanation for the effectiveness of voting methods. Peter Bartlett, Yoav Freund, Wee Sun Lee and Robert E. Schapire, The Annals of Statistics, volume 26, issue 5, pages 1651 to 1686, 1998.
4. Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning. Sebastian Raschka, arXiv, 2018.