How a dataset becomes features

An energy retailer had a demand forecast that was accurate all day and wrong at midnight. Every night, around the turn of the hour, the predicted load jumped by an amount nobody could explain from anything happening on the grid. The analyst who found it had been looking at the model for a week. The cause was in the second column of the training table.

Hour of day had been stored as an integer, 0 through 23. To the model, 23:00 and 00:00 were the two furthest points in the day, twenty-three units apart, while 00:00 and 06:00 were only six units apart. Midnight is one minute after 23:59. The table said otherwise, and the model believed the table.

The model never sees your data

Whatever your data actually is, orders or patients or sensor traces, the fitting procedure receives a rectangle of numbers. Somebody decided how each real thing in the world became a number in that rectangle. That decision is the feature layer, it is made by a person rather than by the optimiser, and it is not checked by anything downstream.

What a model actually learns makes the point that training only adjusts parameters to make a loss small. The optimiser has no way to notice that your integer encoding of the hour has a cliff in it, because from where it sits there is no cliff, only a column. Whatever geometry you hand it is the geometry it works in.

That is why the feature layer carries so much weight. It is the one place where knowledge about the domain enters a system that otherwise has none.

Working the midnight problem by hand

Take the hour column seriously for a moment and do the arithmetic three ways.

As an integer. Hour 23 and hour 0 sit 23 apart. Hour 0 and hour 6 sit 6 apart. Any method that treats the column as a magnitude, which includes every linear model and every distance-based method, now believes late evening is roughly four times further from midnight than dawn is.

As twenty-four indicator columns. One-hot encoding removes the false ordering: every hour becomes its own column, and no hour is nearer to any other. It also throws away the true ordering. The model cannot learn that 02:00 and 03:00 behave similarly unless it sees enough of both to learn each separately, and any hour that is thin in the training data gets no help from its neighbours.

As a position on a circle. Map the hour to an angle and take its sine and cosine. Hour 0 becomes the point (1.000, 0.000). Hour 23 becomes (0.966, -0.259). Hour 6 becomes (0.000, 1.000). Now measure straight-line distances. Midnight to 23:00 is 0.261. Midnight to 06:00 is 1.414. Late evening is about five and a half times closer to midnight than dawn is, which is what a person who has looked at a load curve would say.

Three encodings of the same fact. One of them contains a lie, one of them discards information, and one of them puts the day's shape into the coordinates so the model does not have to discover it.

The scikit-learn documentation carries a version of this on real data. Its time-related feature engineering example takes an hourly bike-sharing series and compares approaches. A gradient boosting model given the raw time fields reaches a mean absolute error of 0.044 plus or minus 0.003, with a root mean squared error of 0.068 plus or minus 0.005. A linear model given the same fields one-hot encoded reaches 0.142 plus or minus 0.014 and 0.184 plus or minus 0.020 on the same two measures, better than three times worse on the first. The example then rebuilds the linear model's inputs with trigonometric and periodic spline encodings of the same clock, which is the whole point of it: the model family did not change, the columns did.

Scale is not cosmetic

The second thing that happens between a table and a fit is scaling, and its effect is larger than most people expect.

The scikit-learn preprocessing guide states the reason directly: standardisation is "a common requirement for many machine learning estimators", which "might behave badly if the individual features do not more or less look like standard normally distributed data". The mechanism is not mysterious. A regulariser that penalises the sum of squared coefficients penalises a coefficient on a column measured in rupees far more than the same relationship expressed in lakhs. A distance calculation is dominated by whichever column happens to have the widest range.

The documentation's own example puts a number on it. Using the wine recognition dataset, it runs the same pipeline twice, principal components followed by logistic regression, changing nothing but whether the inputs were standardised first. Unscaled, test accuracy is 35.19 percent with a log loss of 1.18. Standardised, test accuracy is 96.30 percent with a log loss of 0.0739. Same data, same model, same hyperparameter search. One line of preprocessing.

A caution that costs nothing to observe: the scaler is part of the model, not part of the data. Its mean and standard deviation are fitted on the training rows only. Computing them across the whole table before splitting puts test information into training, which is the preprocessing leakage described in The data comes first.

Categories, and the cost of each option

Most business data is categorical, and categorical columns are where the representation choices get expensive.

The scikit-learn guide is blunt about the naive option. Assigning integers to categories with an OrdinalEncoder produces a column that estimators "would interpret as being ordered, which is often not desired". Product code 7 is not more than product code 3.

One-hot encoding fixes the ordering and costs width. Twelve product categories cost twelve columns and nobody minds. Forty thousand postcodes cost forty thousand columns, most of which are almost always zero, and now the model has more parameters than it has rows to fit them with. The documentation names this case: target encoding "is useful with categorical features with high cardinality, where one-hot encoding would inflate the feature space", and gives location fields such as postcode or region as the classical example.

Target encoding replaces the category with a statistic of the outcome inside it. The average claim value for this postcode, the churn rate for this plan. One column, whatever the cardinality. In a comparison across four ways of handling the categorical fields of a wine reviews dataset, the scikit-learn example reports target encoding giving the best test error, one-hot next, ordinal after that, and dropping the categories entirely worst.

The catch is exactly the one to expect. If the encoding is computed from the same rows the model is fitted on, the target has been copied into the inputs, and the model can read the answer off a column. The library handles this internally by cross fitting, and the documentation flags the consequence in a form worth memorising: for TargetEncoder, fit(X, y).transform(X) does not equal fit_transform(X, y), and training data should always go through fit_transform. If you build your own target encoding in SQL, that cross fitting is yours to implement, and the score you get if you forget will look excellent.

What the feature layer costs to own

Every feature is a small permanent liability, and it is worth pricing them before adding the fortieth one.

It has to exist at prediction time. A feature computed from a field populated overnight cannot serve a decision made at nine in the morning.
It has to be computed the same way twice. The training version comes from an analyst's notebook against a warehouse; the serving version comes from application code against a live database. When those two definitions drift apart, nothing errors. The score just decays.
It has to survive its own source. A column derived from a vendor's category taxonomy changes when the vendor reorganises the taxonomy, and nobody will tell you.
It has to earn its width. Cardinality is not free, and a high-cardinality field encoded carelessly is the fastest way to memorise the training set.

The cheapest feature set that answers the question is the correct one. This is not minimalism for its own sake. Each column is a thing that can silently change.

Why this outranks the algorithm choice

Grinsztajn, Oyallon and Varoquaux benchmarked tree-based models against neural networks across 45 tabular datasets, with a hyperparameter search they report as totalling around 20,000 compute hours per learner. Their conclusion is that tree-based models remain the state of the art on medium-sized tabular data of roughly ten thousand samples. The interesting part is their explanation of why. The advantages they identify are that trees tolerate uninformative features, that they are unaffected by rotations of the input space, and that they can fit irregular functions easily.

Read that as a statement about representation rather than about trees. Two of those three properties describe how a method copes with the coordinate system it was handed. Neural networks are more sensitive to that coordinate system, which is another way of saying that on tabular data the geometry of your columns is doing much of the work that people attribute to the model.

The practical consequence is an ordering. Before comparing algorithms, get the representation right: no false orderings, no scale mismatch, no categorical field that inflates the matrix beyond the row count, and no column that will not exist when the prediction is made. After that, the model comparison is a real comparison. Before it, you are mostly measuring which algorithm best tolerates your encoding.

What to check before the first fit

A short pass over the feature table catches most of this, and it takes an hour.

For every numeric column, ask whether the distance between two values means anything. If it does not, it should not be numeric.
For every categorical column, count the distinct values and decide the encoding from that count rather than from habit.
For every cyclical quantity, hour, weekday, month, compass bearing, check that the wrap-around is represented.
For every column, name the system it comes from and the moment it is populated.
Fit the scaler inside the pipeline, never before the split.

The energy retailer's fix took an afternoon. Hour of day became a sine and a cosine, weekday stayed one-hot because seven categories cost nothing, and the temperature columns were standardised inside the pipeline rather than in the extract. The midnight jump disappeared, and the overall error improved as well, because the model had stopped spending capacity on repairing a coordinate system nobody had meant to give it.

None of that told anyone whether the forecast was actually good enough to schedule against. That question needs a different tool, and it is the one most teams get wrong first: how the data is split, and what a single score does and does not tell you.

References

Preprocessing data. scikit-learn documentation, version 1.9.0, 2026.
Importance of Feature Scaling. scikit-learn documentation, version 1.9.0, 2026.
Time-related feature engineering. scikit-learn documentation, version 1.9.0, 2026.
Comparing Target Encoder with Other Encoders. scikit-learn documentation, version 1.9.0, 2026.
Why do tree-based models still outperform deep learning on tabular data?. Leo Grinsztajn, Edouard Oyallon and Gael Varoquaux, arXiv, 2022.