Data pipelines that do not rot

The nightly job had run for eleven months without a failure. On a Tuesday it ran again, finished in the usual nineteen minutes, wrote the usual number of rows, and reported success. Nothing paged anyone.

That afternoon an upstream team had shipped a change: the amount column on the orders feed moved from rupees to paise, so every value was now a hundred times larger. It was still an integer. It was still non-null. It was still within any range check that existed, because none did.

The model kept scoring. Its most important feature, average order value over 30 days, drifted by two orders of magnitude across a fortnight as the 30-day window filled with new-unit rows. Nobody noticed for nine days, and the person who noticed was a category manager who thought a dashboard looked wrong.

A pipeline that fails is an inconvenience. A pipeline that keeps running while it stops telling the truth is the expensive kind, and it is the ordinary kind. It is also the thing The first week of an AI project asks about when it asks who is called at 3am when a score is missing, and the answer at this company was that nobody would be, because nothing would appear to be wrong.

The failure that does not raise an exception

Polyzotis, Zinkevich, Roy, Breck and Whang built and deployed a validation system for exactly this class of problem at Google, and their framing is the one to adopt. They observe that while a great deal of machine learning research has focused on improving the accuracy and efficiency of training and inference algorithms, there is less attention in the equally important problem of monitoring the quality of data fed to machine learning, and that the importance of this problem is hard to dispute: errors in the input data can nullify any benefits on speed and accuracy for training and inference. Their conclusion is a data-centric approach to machine learning that treats training and serving data as an important production asset, on par with the algorithm and infrastructure used for learning.

Their system runs as part of TFX and is, in their description, used by hundreds of product teams to continuously monitor and validate several petabytes of production data per day. The challenges they name are the ones every team meets at smaller scale: the ability of ML pipelines to soldier on in the face of unexpected patterns, schema-free data, or training and serving skew.

Soldiering on is the exact problem. An ordinary service that receives malformed input throws an error and somebody fixes it in an hour. A data pipeline receives malformed input, computes an average anyway, and hands a plausible number to a model that has no way to know the number is wrong.

Where a pipeline actually breaks

Each of the five is worth naming precisely, because the fix is different in each case.

Late rows read as missing. The job runs at 02:00 and reads yesterday's partition. A regional system uploads at 02:40 on days when its batch is large. Those rows are absent when the job reads, so a customer with fourteen orders is aggregated as a customer with eleven. The pipeline is not wrong about what it read. It is wrong about when it read.

A schema change absorbed silently. The rupees-to-paise change above, or a category added to an enumeration, or a column renamed with the old name kept as a null-filled alias for compatibility. Where the schema is inferred from the data rather than declared, any change is by definition consistent with the schema.

A join that drops what it cannot match. Discussed below, because it is the one that quietly changes who is in your dataset.

Units that change while types do not. Currency, seconds against milliseconds, percent against proportion, weight in grams against kilograms. Every one of these passes a type check and every one of these multiplies or divides a feature by a constant nobody chose.

Two code paths computing the same feature. The training feature is a SQL window function over a warehouse table; the serving feature is Python over a Redis counter. They agree for a year, then somebody fixes a bug in one of them.

The join that changes who is in the data

The orders pipeline processes about 1.2 million rows a day and enriches them from a customers table snapshotted at 02:00. The enrichment is an inner join on customer_id.

Customers who registered after the 02:00 snapshot are not in it. On a typical day that is about 36,000 orders, roughly 3 percent, and those orders vanish from the output. The job reports 1,164,000 rows written, which nobody compares against 1,200,000, because nobody wrote down that they should be equal.

Three percent sounds ignorable. It is not, because the dropped rows are not a random 3 percent. They are entirely first-day customers, which is the single segment where behaviour differs most from everyone else. The training data now systematically under-represents new customers, and the model trained on it is worst exactly where it is used most: on a customer's first session. That is a distribution shift the team introduced themselves, of the kind The data comes first describes arriving from the outside.

The fix is not a better join. It is an assertion: output_rows == input_rows, failing loudly, with the unmatched keys written somewhere a human can look at them. Then a left join, an explicit unknown_customer category, and a feature that records whether the customer record was missing, because that fact is itself predictive.

What a declared schema buys

TensorFlow Data Validation makes the distinction that matters: the schema codifies properties which the input data is expected to satisfy, such as data types or categorical values, and can be modified or replaced by the user. A developer can rely on automatic schema construction, where an initial schema is built from statistics computed over training data available in the pipeline, but the schema is then a checked-in artefact that a human owns and a code review can change.

That last property is the whole point. An inferred schema recomputed on every run cannot detect a change, because it becomes the change. A declared schema is a statement about what the world is supposed to look like, so the world can contradict it.

For skew between the two sides of the system, the same tooling detects distribution skew between training and serving data, which occurs when the distribution of feature values for training data is significantly different from serving data. For change across time, drift detection is supported between consecutive spans of data, such as between different days of training data, and is expressed in terms of L-infinity distance for categorical features and approximate Jensen-Shannon divergence for numeric features. Their list of common problems is worth pinning above a desk: missing data such as features with empty values; labels treated as features, so that your model gets to peek at the right answer during training; and features with values outside the range you expect.

The second of those is leakage arriving through the pipeline rather than through the split, which is worth knowing about after reading How a dataset becomes features.

The assertions worth writing on day one

Validation is unrewarding to write and it is the highest-return code in the repository. Ten assertions catch most of what actually happens.

• Row count within a band. Yesterday's count, plus or minus 20 percent, with day-of-week accounted for. A pipeline that wrote 4 percent of normal has failed even though it succeeded.
• Null rate per column, against a declared maximum. customer_id at most 0 percent, promo_code at most 60 percent. A column that jumps from 2 percent null to 40 percent is a broken upstream field.
• Range and median per numeric column. The paise change moves the median order value from about Rs 1,240 to about Rs 124,000. A single assertion that the median sits between 800 and 2,500 would have caught it in nineteen minutes rather than nine days.
• Category set membership. The set of payment_method values, compared against the declared set. New values are not errors, but they must be seen by a person before they reach a feature encoder that will treat them as unknown.
• Join cardinality. Input rows against output rows, and the count of unmatched keys.
• Freshness. The maximum event timestamp in the partition, against the wall clock. If the newest row is eleven hours old, the feed is stale regardless of how many rows arrived.
• Uniqueness of the key. Duplicate order_id values are how near-duplicate rows end up on both sides of a split.
• Distribution comparison against the previous span. The drift measure above, on the ten features the model actually uses most.
• Training and serving agreement. Sample 1,000 live scoring requests a day, recompute their features through the training path, and assert the two agree. This one catches the failure nothing else catches.
• The label's own arrival curve. How many labels for last week have arrived by now, against how many arrived by the same point in previous weeks.

Each assertion needs an owner and a documented response. An alert that fires into a channel where three people all assume somebody else is looking is worse than no alert, because the team now believes it has coverage.

Undeclared consumers

Sculley and colleagues named the failure mode that turns a working pipeline into an immovable one. Under the software engineering framework of technical debt, they find it is common to incur massive ongoing maintenance costs in real-world ML systems, and the risk factors they list include boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies and changes in the external world.

Undeclared consumers are what makes a pipeline unfixable. The table your job writes is readable by everyone in the warehouse. Within a year, a finance dashboard, a weekly export to a partner, and somebody's retention analysis all read it. None of them told you. When you correct the currency bug, three things break and none of them are yours, so the correction gets reverted and the wrong column stays wrong with a comment above it.

The countermeasures are unglamorous and they work. Write to a named, versioned output rather than a table anyone may discover. Log reads where the platform allows it. Publish a deprecation window and mean it. Treat a schema change as an interface change with a release note, because that is what it is.

Rebuildability is the real test

Shankar, Garcia, Hellerstein and Parameswaran interviewed eighteen machine learning engineers about how production systems are actually kept alive, and reduced what they heard to three variables that govern success for a production ML deployment: velocity, validation and versioning.

Versioning is the one people underrate in pipeline work. The test of a pipeline is not whether it ran this morning. It is whether you can rebuild last quarter's training table today and get the same rows, byte for byte, after the upstream data has been corrected twice and a column has been renamed once.

That capability costs something specific: immutable partitioned raw storage, no in-place updates, transformation code in version control with the pipeline run recording its own commit hash, and every derived table carrying the version of the code that produced it. Teams that skip it spend a week each time a result is questioned and usually cannot answer.

A pipeline that does not rot is not one that never breaks. It is one that tells you within an hour, in a message a specific person is responsible for reading, and one whose past output can be reconstructed after the fact. Everything else is a job that runs.

Once the data is trustworthy, the next thing that goes missing is the record of what you did with it, which is where the next article starts.

References

1. Data Validation for Machine Learning. Neoklis Polyzotis, Martin Zinkevich, Sudip Roy, Eric Breck and Steven Whang, Proceedings of Machine Learning and Systems, 2019.
2. TensorFlow Data Validation: Checking and analyzing your data. TensorFlow Extended documentation, Google, 2021.
3. Hidden Technical Debt in Machine Learning Systems. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo and Dan Dennison, Advances in Neural Information Processing Systems, 2015.
4. Operationalizing Machine Learning: An Interview Study. Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein and Aditya G. Parameswaran, arXiv, 2022.