Deploying a model is not the finish line

The release goes out at 09:40 on a Thursday. The model artefact is version 7, trained the previous week, evaluated carefully, signed off. The service starts, the health check passes, latency is normal, and the error rate is zero.

The feature transformer in the serving image is still version 6. Between the two versions somebody had added a merchant category to the encoder's vocabulary, which shifted every category after it by one index. So grocery arrived at the model as fuel, fuel as pharmacy, and so on down the list.

No exception was raised, because the indices were all valid. Scores stayed in their usual range. Precision fell by about four points, which is inside the week-to-week noise nobody had characterised. It ran for nineteen days.

The model was fine. The release was not, and no test in the pipeline was looking at the join between the two.

What is actually being deployed

A model version is not a file. It is at least three things that must move together, and treating any of them as separate is where this class of failure comes from.

• The transformation code. Whatever turns a raw request into the vector the model expects, including every vocabulary, bucket boundary and normalisation constant.
• The fitted parameters. The artefact everybody thinks of as the model.
• The contract. The names, types and permitted ranges of the inputs, and the shape and meaning of the output.

Package the three as one immutable unit, tagged with the run id from Experiment tracking for people who forget, and make the serving code refuse to start when the transformer version recorded in the artefact does not match the one it loaded. That single assertion would have turned nineteen days into zero seconds.

Google's own guidance pushes the same idea one level further out. At the more automated level, they write, you deploy a whole training pipeline, which automatically and recurrently runs to serve the trained model as the prediction service. The unit of release becomes the process that produces models rather than any individual model, which is the only way retraining stops being a manual event that somebody has to remember.

Sculley and colleagues had already named why this bites harder in machine learning than in ordinary services: their list of risk factors includes boundary erosion, entanglement, configuration issues and undeclared consumers. A configuration mismatch between two halves of one system is the plainest possible instance of the third.

The rollout, sized honestly

The instinct to ship at 100 percent because the offline numbers were good is the instinct to find out about a problem from a customer.

Three stages, and each answers a different question.

Shadow. The new version scores live traffic and its output goes nowhere except a log. This answers whether the serving path works: latency under real load, features resolving, no missing values in the request shape. It cannot answer whether the model is better, because nothing acted on it, and teams that stop here learn less than they think.

Canary. A small share of real decisions use the new version. This is where quality is measured, so it is worth doing the arithmetic on how long it takes.

Take a service making 40,000 decisions a day with a 4 percent error rate on the outcome you care about. A 5 percent canary sees 2,000 decisions a day. The standard error of an error rate measured on 2,000 outcomes is the square root of 0.04 times 0.96 divided by 2,000, which is 0.0044, or 0.44 percentage points. Because the control arm is nineteen times larger, its own noise is small enough to ignore, so that figure is roughly the standard error of the comparison too.

A regression from 4 percent to 6 percent is 4.5 standard errors. You will see it on day one. A regression from 4 percent to 4.5 percent needs the standard error down near 0.17 points, which needs about 13,000 canary decisions, or roughly a week at 5 percent. That is the whole scheduling decision, done in three lines: a canary catches a catastrophe overnight and a regression in a week. Plan the ramp on the second number, not the first.

Ramp. 5, 25, 50, 100, with a defined wait and a defined metric at each step, and the wait derived from the arithmetic above rather than from how the release feels.

Where the outcome takes weeks to arrive, as with the 120-day chargeback lag in The first week of an AI project, none of these stages can measure the outcome directly. Then the canary compares what it can compare now: the score distribution, the rate of cases crossing the decision threshold, the agreement between old and new versions, and the disagreement cases sampled for human review. Say out loud that you are measuring a proxy, and say when the real number will arrive.

Rollback is a decision you make before you launch

Every deployment plan should be able to answer four questions in one sentence each, and if it cannot, the launch is not ready.

• What is the trigger? A named metric crossing a named value, not a judgement call made by whoever is awake.
• How long does it take? Reverting to the previous version should be a single action and it should be measured in minutes. If it requires a rebuild, it is not a rollback.
• What happens to work in flight? Requests already scored, queues already ranked, and a customer already shown a decision by the version being withdrawn.
• Who can pull it without asking? One named on-call engineer. An escalation path is not a rollback plan.

Shankar and colleagues found teams doing exactly this in practice, keeping a known-good model as a landing place rather than treating rollback as an exception. One of their participants described the arrangement plainly: if the production model drops and the calibration model is still performing within a specified range, we'll fall back to the calibration model until someone will fix the production model. Others simply switched to a less economic model and had to just cut the losses.

The organisational half is in their interviews too. For each model, at any point in time, some ML engineer would be on call, or primarily responsible for it, and teams kept a central queue of production ML bugs that every engineer added tickets to and processed tickets from. A model with no name attached to it is a model nobody will roll back at 02:00.

Testing a thing whose output you cannot specify

Ordinary software testing assumes you can state the correct answer. Breck, Cai, Nielsen, Salib and Sculley started their rubric from the observation that you often cannot: testing and monitoring are key considerations for ensuring the production-readiness of an ML system, and for reducing technical debt of ML systems, but it can be difficult to formulate specific tests, given that the actual prediction behavior of any given model is difficult to specify a priori. Their answer was 28 specific tests and monitoring needs, drawn from experience with a wide range of production ML systems, presented as a road-map to improve production readiness and pay down ML technical debt.

The move worth copying is what they test instead of accuracy. You cannot assert that a score is correct. You can assert a great deal around it:

• The training and serving paths produce identical features for the same input. Take 1,000 logged requests, push them through both, and fail the build on any mismatch.
• The model reproduces a fixed set of golden predictions to the last decimal after a rebuild.
• No feature in the contract is missing, constant, or out of range on live traffic.
• The full pipeline runs end to end on a small fixture in continuous integration, including the transformer and the serving wrapper.
• The service degrades in a defined way when a feature store lookup times out, rather than blocking or returning a silent default.

That last one is where serving meets the user, and the interface consequences are worked through in Putting a model behind an interface. A default value returned quietly on timeout is a wrong answer delivered with the same confidence as a right one.

Retraining is a deployment

The most common way a working system decays is that retraining is treated as maintenance rather than as a release.

A retrained model is a new model. It gets the same evaluation, the same canary, the same rollback trigger and the same record. When retraining is automatic, the gate has to be automatic too: the new candidate is promoted only if it beats the incumbent on the held-out period by more than the run-to-run variation established earlier, and only if its input data passed validation.

Google's guidance is direct about why the cadence exists at all: models can decay in more ways than conventional software systems because of constantly evolving data profiles, and to capture the evolving and emerging patterns, you need to retrain your model with the most recent data. Automating that loop is worth doing. Automating it without a promotion gate means an automated system that quietly ships a worse model every Sunday.

Before you ramp

Seven things, all answerable in a sentence, none of them about model quality.

1. The artefact, transformer and contract ship as one unit, and the service refuses to start on a version mismatch.
2. The rollback trigger is written down with a number in it.
3. Rollback takes one action and under five minutes.
4. One named person is on call for this model this week.
5. Feature parity between training and serving is asserted in continuous integration, not hoped for.
6. The canary duration comes from the detectable-effect arithmetic, not from the calendar.
7. Somebody is watching a dashboard that would have caught a four-point precision drop, which is the subject of the next article, because a nineteen-day failure is a monitoring failure long before it is a deployment failure.

References

1. The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction. Eric Breck, Shanqing Cai, Eric Nielsen, Michael Salib and D. Sculley, Proceedings of IEEE Big Data, 2017.
2. MLOps: Continuous delivery and automation pipelines in machine learning. Google Cloud Architecture Center, 2024.
3. Operationalizing Machine Learning: An Interview Study. Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein and Aditya G. Parameswaran, arXiv, 2022.
4. Hidden Technical Debt in Machine Learning Systems. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo and Dan Dennison, Advances in Neural Information Processing Systems, 2015.