Monitoring for silent failure

A telecom operator scores 8,000 support tickets a day for whether they need a specialist. Above 0.50 the ticket is routed to the specialist team, which can handle about 2,000 tickets a day. At launch, 18 percent of tickets crossed the threshold, so the queue ran at about 1,440 a day with room to spare.

In week 2 the company launched a self-service app. It was a success. Customers began resolving password resets, balance queries and SIM swaps without opening a ticket at all, so the tickets that still arrived were systematically harder than the tickets the model had been trained on.

Nothing failed. The pipeline was green, the service returned 200s, latency was flat, and the model did exactly what it had been fitted to do. It simply started saying "specialist" more often, because more of what it saw genuinely looked like specialist work.

By week 5 the specialist queue was over capacity. By week 12 the backlog was around 41,000 tickets and a director was asking why response times had tripled. The drift dashboard, which existed and was well built, had raised its alert in week 8.

The three things that rot

Lu, Liu, Dong, Gu, Gama and Zhang open their review of the field with the definition worth keeping: concept drift describes unforeseeable changes in the underlying distribution of streaming data over time. That covers three distinguishable failures, and telling them apart decides what you do next.

The inputs change. The mix of tickets shifts because a new app absorbed the easy ones. Nothing about the relationship between a ticket and its correct routing has changed; the model is simply being asked a different question than the one it was tested on. This is the distribution shift set out in The data comes first, arriving through the front door.

The relationship changes. A new product launches and "cannot activate" now means something different from what it meant last quarter. The same input maps to a different correct answer. Retraining on fresh data fixes this one; retraining on stale data does not.

The system changes around the model. A feature stops being computed, an upstream unit changes, a fallback silently substitutes a default. This is not drift at all, and treating it as drift sends the team to retrain a model that was never the problem.

Google's own guidance states the underlying asymmetry plainly: models can decay in more ways than conventional software systems, because of constantly evolving data profiles. A web service that stops working tells you. A model that stops working keeps answering.

Two dashboards of the same system

Both plots come from the telecom system above.

On the left is a drift statistic computed weekly on the input distribution. The team chose its own levels: 0.10 to investigate and 0.25 to alert. The statistic climbs from 0.01 in week 0 to 0.38 in week 12, reaching 0.10 early in week 4 and 0.25 early in week 8. As a measurement of change in the inputs it is doing its job well.

On the right is the share of scored tickets landing above the 0.50 threshold, which is the number that decides how many tickets enter the specialist queue. Capacity is 2,000 tickets against 8,000 scored, so the operational ceiling is 0.25. That share crosses in week 5.

Now put the arithmetic on the gap. From week 5 the daily excess over capacity runs 80, then 320, 480, 640, 880, 1,040, 1,200 and 1,280 tickets as the share climbs from 26 to 41 percent. Held for a week each, that is about 41,000 tickets of backlog by week 12, of which roughly 6,300 accumulated before the drift alert fired.

The alert was not late because the threshold was too high. It was late because it was watching a statistic about the inputs rather than the quantity that governs whether the system works. The right alarm is on the consequence, and the drift statistic is a diagnostic you read after the alarm, not the alarm itself.

The label arrives too late to be your alarm

The obvious objection is that all of this would be caught by measuring accuracy. It usually would not, and the reason is timing.

Shankar, Garcia, Hellerstein and Parameswaran interviewed eighteen machine learning engineers about production practice, and one of them described the situation most teams are actually in: I have no idea how well models actually perform on live data, feedback is always delayed by at least 2 weeks, sometimes we might not have feedback, so when we realize maybe something went wrong, it could have been 2 weeks ago.

Two weeks is a good case. A chargeback can take 120 days. A churn label takes a full billing cycle plus a grace period. A "was this routing correct" label exists only for tickets a human later reclassified, which is a biased sample of the tickets that went to the wrong place.

That timing problem is why monitoring has to be built in layers, ordered by how fast each layer can tell you something.

1. Input signals, available immediately. Row counts, null rates, schema conformance and the distribution of each feature the model actually uses. TensorFlow Data Validation expresses this comparison between consecutive spans of data in terms of L-infinity distance for categorical features and approximate Jensen-Shannon divergence for numeric features, which is a reasonable default when a team has no opinion of its own. It also detects distribution skew between training and serving data, which occurs when the distribution of feature values for training data is significantly different from serving data.
2. Output signals, available immediately. The distribution of scores, the share above each decision threshold, and the rate of each downstream action. This layer is cheap, needs no labels, and is the layer the telecom was missing.
3. Proxy outcomes, available in hours or days. Human override rate, escalation rate, reopen rate, time to resolution. When a specialist reassigns a ticket back to general support within four minutes, that is a label about the model arriving a hundred days before the real one.
4. True outcomes, available eventually. Accuracy, precision at the operating point, calibration. These are the numbers that matter and they cannot be your alerting layer. They are your monthly review.

Calibration belongs in layer four for a specific reason worth remembering from What probability buys you: a model can keep its ranking almost perfectly while its stated probabilities drift, and a threshold set on those probabilities then means something different from what it meant at launch.

Alerting on the right thing

The same interview study found the failure that follows over-alerting, in the words of the engineers living with it. People were getting bombed with these alerts, said one. There was nothing critical in most of the alerts, said another, who added: you typically ignore most alerts, I guess on record I'd say 90 percent of them aren't immediate.

An alert that is ignored 90 percent of the time is not a safety mechanism. It is a small daily tax that produces the feeling of coverage. Four rules keep the count survivable.

• Alert on things a person will act on tonight. Everything else is a dashboard or a weekly report. A feature's distribution moving slightly is not a night-time event; the review queue exceeding capacity is.
• Give every alert a name, an owner and a written first action. If nobody can say what to do at 02:00, the alert is a notification.
• Set thresholds from the operational constraint, not from the statistic. 2,000 tickets a day is a real number that came from headcount. 0.25 on a drift index came from a blog post.
• Alert on rate of change as well as on level. The telecom's share crossed capacity in week 5, but it had been rising for four consecutive weeks before that. A monotone rise over four periods is itself worth a message, because it turns a threshold breach into a forecast.

That last rule is what would have bought the team four weeks. Nothing about it requires labels, a platform, or a drift library.

When the alarm is real

The first instinct is to retrain. Resist it for an hour, because retraining a model that was not the problem costs a week and hides the actual cause.

• Check the pipeline first. The daily assertions from Data pipelines that do not rot answer this in minutes. A feature that has been null for three days looks exactly like drift on every distribution chart and is not drift.
• Then check what changed on your own side. Deployments, feature flags, upstream releases. The version discipline in Deploying a model is not the finish line exists so this question has an answer.
• Then look at the inputs against the outputs. If inputs moved and the score distribution moved with them, the world changed and retraining is the right response. If the score distribution moved while the inputs held still, something in your system changed and retraining will bury the evidence.
• Then decide between retraining and re-thresholding. These are different repairs. Where the mix of inputs changed but the mapping did not, moving the threshold restores the operating point immediately and costs nothing, using the reasoning in The cost of a wrong answer. Where the mapping changed, only new labelled data helps.

The telecom's answer, when they finally worked it through, was the cheap one. The relationship between a ticket and its correct routing had not changed at all. The mix had. Raising the threshold from 0.50 to 0.62 brought the queue back under capacity the same afternoon, and the retraining that followed six weeks later was a planned piece of work rather than an emergency.

Twelve weeks of degradation, one afternoon of repair, and the difference between the two was a chart nobody had drawn. Which raises the question the next article is about: what all of this actually costs, per request, per month, and at the scale somebody has already promised.

References

1. \"We Have No Idea How Models will Behave in Production until Production\": How Engineers Operationalize Machine Learning. Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein and Aditya G. Parameswaran, arXiv, 2024.
2. TensorFlow Data Validation: Checking and analyzing your data. TensorFlow Extended documentation, Google, 2021.
3. Learning under Concept Drift: A Review. Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama and Guangquan Zhang, arXiv, 2020.
4. MLOps: Continuous delivery and automation pipelines in machine learning. Google Cloud Architecture Center, 2024.