Reading a model you did not build

A retention team had a churn model running against every account, driving a discount offer worth several crore a year. It had been in production for two years. The person who built it had left eighteen months earlier. There was a serialised model file, a scheduled job, and a notebook that no longer ran because two of its imports had moved.

The question in the room was whether to keep it. Nobody could answer, because nobody could say what it did.

This is a more common position than the one the rest of this series has assumed. Most people meet models they did not build: bought from a vendor, inherited from a team that reorganised, or fitted by somebody who has moved on. The skill is different from building. It is closer to reading a building's wiring without the drawings.

Three questions, in order

Everything worth knowing sits under three headings, and the order matters because a bad answer to the first makes the others irrelevant.

What was it fitted to. Not the algorithm, the target. Which column was called the outcome, how it was defined, and by what process a row came to carry a one rather than a zero.

What goes in. Every field, where it comes from, and the moment in the day it is populated.

What comes out, and what happens next. A score, a rank, a decision. Then the threshold, the action, and who is accountable for the action.

Start with the target because that is where the most expensive errors live and where nobody looks. Obermeyer, Powers, Vogeli and Mullainathan examined a commercial system used across United States health systems to flag patients needing extra care. The model worked; it predicted its target accurately. Its target was health care spending rather than illness, and because less is spent caring for Black patients at the same level of sickness, the model systematically under-identified them. At a given risk score, Black patients were considerably sicker. Correcting the target would raise the share of Black patients receiving additional help from 17.7 percent to 46.5 percent.

An audit that had checked the algorithm, the features, and the accuracy would have passed that system. The defect was one line above all of it.

What to ask for, before you start reverse-engineering

There is an established answer to what documentation should accompany a released model, and quoting it is more effective than inventing a list of your own.

Mitchell and colleagues proposed model cards, recommending that "released models be accompanied by documentation detailing their performance characteristics", specifically to clarify intended use cases and to "minimize their usage in contexts for which they are not well suited". The part of their proposal that does the most work in a real audit is the requirement for evaluation broken out by group rather than only in aggregate. An overall score conceals a segment that fails, and the segment that fails is the one that will produce the complaint.

For the data underneath, the equivalent is the datasheet described in The data comes first: motivation, composition, collection process, and intended use.

So the request to a vendor or to an internal predecessor is short and specific.

• The target definition in words, with the query or process that generated it.
• The list of input fields with their source systems and their population times.
• The training period, the population it was drawn from, and what was excluded.
• How the evaluation was split, and the score with its spread, broken out by the segments that matter to us.
• The threshold currently in use and who set it.
• The date of the last retraining and the trigger for the next one.

A vendor who cannot supply that list is telling you something. A vendor who supplies an aggregate accuracy figure and nothing else is telling you the same thing more politely.

Testing a model you cannot read

Assume you get nothing. The model is a file with a predict method. There is still a great deal you can establish from the outside, and it is worth doing in this order.

Replay it on recent data. Score the last complete period and compare against what actually happened. This is the single most informative hour available, because it measures the model as it stands today against the world as it is now, with no dependence on any claim about how it was built.

Break the score out by segment. Region, channel, tenure, product, account size. An aggregate that looks acceptable frequently contains a segment where the model is worse than the trivial baseline. That segment is a live liability.

Score the trivial baseline on the same period. Last month's value, the majority class, the rule the operations team used before the model existed. If the model does not clearly beat it, given the error bars discussed in Train, test, and the lie of a single score, the maintenance cost has no return against it.

Check the calibration. Bucket the scores, count outcomes, plot the reliability curve. If the model's numbers feed an expected-value rule anywhere downstream, What probability buys you explains why this is not optional.

Probe the inputs. Take a real row, change one field, and watch the output. Sweep one field across its range with everything else fixed. This tells you what the model is sensitive to and, more usefully, what it ignores. A model that barely responds to the field the business believes is central is not doing what people think it is doing.

Check the input distribution against the training period, if you know it. Fields whose distribution has moved are where the score has quietly stopped meaning what it meant.

Two cautions about the last technique. Single-feature probing describes local behaviour, not a global rule, and correlated inputs make its results easy to over-read: moving one field to a value that never co-occurs with the rest of the row asks the model about a case it has never seen. Treat the output as a lead to investigate, not as an explanation.

That caution generalises. Rudin argues that for high-stakes decisions, trying "to explain black box models, rather than creating models that are interpretable in the first place, is likely to perpetuate bad practices and can potentially cause catastrophic harm". Her position is a strong one and it is directly relevant when the audit's conclusion is being written: a post-hoc explanation of an opaque model is a description of a second model, and reporting it as though it were the first one's reasoning overstates what has been established. If the decision is consequential and the incumbent is opaque, "replace it with something legible" is a legitimate recommendation rather than a purist one.

Turning it into something an organisation can act on

An audit that ends in a document nobody owns changes nothing. Two published frameworks help here, and both are worth knowing by name in a governance conversation.

Raji and colleagues set out an internal auditing framework covering the development lifecycle end to end, in which each stage produces documents that together form an audit report, assessed against the organisation's own stated values or principles. The design point is that the audit happens before deployment and inside the organisation, so problems are traceable to their source rather than discovered afterwards from outside.

The NIST AI Risk Management Framework, released as version 1.0 in January 2023, organises the same territory around four functions: Govern, Map, Measure and Manage. Its practical value in a company that has none of this is that it supplies a vocabulary that a risk function and an engineering team can both use, which is often the actual blocker.

Neither framework will tell you whether your churn model is any good. What they give you is a shape for the answer that survives the meeting.

Deciding whether to keep it running

The audit ends in one of four recommendations, and naming them in advance keeps the conversation from drifting into a defence of the model's honour.

Keep and document. It beats the baseline, the segments are sound, the calibration holds, and the gap is that nothing is written down. Cheapest good outcome.

Keep and constrain. It performs on most of the population and fails on an identifiable part. Narrow its remit to where it works and route the rest elsewhere. This is usually available and usually not considered.

Retrain on a defensible basis. The structure is fine and the model is stale, or the split was random when it should have been by time or by group.

Retire it. It does not beat the baseline, or the target is wrong in the way the health care example was wrong. A model driving crore-scale spend on a defective target is not a technical debt item, it is a live commercial and ethical exposure.

The retention team's model landed on the third and then the second. Replayed on the last two quarters it beat the baseline overall, comfortably on accounts under two years old and not at all on the long-tenured accounts that carried most of the revenue. Probing showed why: the strongest input was a support-contact count that behaved completely differently for accounts on the legacy contract. The model was retrained by time-based split, its remit narrowed to accounts under two years, and the long-tenured book went back to a named-account process while a proper model for it was scoped.

The write-up was four pages: target definition, input inventory with source systems, replay results by segment, the reliability curve, the threshold and its owner, and the date of the next review. It took nine days. The model had been running for two years on a decision worth several crore, and those nine days were the first time anyone had established what it was doing.

That is the whole of the foundations, in the end. A model is parameters fitted to a sample, the sample decides more than the algorithm, the representation decides more than the model family, one score is an estimate with a width, the error has parts you can diagnose, a probability is a claim you can check, and a threshold is a budget decision somebody owns. Everything after this is a particular method. None of it replaces those questions.

References

1. Model Cards for Model Reporting. Margaret Mitchell and colleagues, ACM Conference on Fairness, Accountability, and Transparency, 2018.
2. Closing the AI Accountability Gap, Defining an End-to-End Framework for Internal Algorithmic Auditing. Inioluwa Deborah Raji and colleagues, ACM Conference on Fairness, Accountability, and Transparency, 2020.
3. Stop Explaining Black Box Machine Learning Models for High Stakes Decisions and Use Interpretable Models Instead. Cynthia Rudin, Nature Machine Intelligence, volume 1, 2019.
4. Dissecting racial bias in an algorithm used to manage the health of populations. Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, Science, volume 366, 2019.
5. AI Risk Management Framework. National Institute of Standards and Technology, AI RMF 1.0, 2023.