The cost of a wrong answer

The fraud review team had one request for the next release. Precision was at 67 percent and the analysts were tired of opening cases that turned out to be nothing. Could the model be more precise before it flagged anything.

It could. Raising the threshold pushed precision to 84 percent, the analysts were happier, and the quarter's fraud losses went up by roughly two hundred thousand rupees. Nobody had done anything wrong. The team had optimised the metric that was being complained about, which is not the same as the metric that was costing money.

The confusion matrix, on real volumes

Work it with numbers rather than symbols. Ten thousand transactions in a month, of which one hundred are fraudulent. The model produces a score between zero and one for each, and a threshold turns that score into an action.

At a threshold of 0.50 the model flags 60 transactions. Forty of them are fraud, twenty are not. So it catches 40 frauds, misses 60, and raises 20 false alarms.

• Precision, the share of flags that were right: 40 divided by 60, which is 67 percent.
• Recall, the share of frauds that were caught: 40 divided by 100, which is 40 percent.

At a threshold of 0.20 the model flags 375 transactions. Seventy-five are fraud, three hundred are not.

• Precision: 75 divided by 375, which is 20 percent.
• Recall: 75 divided by 100, which is 75 percent.

Precision fell by more than two thirds. On any dashboard, the second policy looks dramatically worse. It is the one that saves money, and the reason is that neither number knows what a mistake costs.

Putting money on each cell

Two figures turn this from an argument into an arithmetic problem. The business supplies them, not the modelling team.

A missed fraud costs an average of 8,000 rupees: the loss, the chargeback handling, the write-off. A false alarm costs about 200 rupees: a few minutes of an analyst's time and a small amount of customer friction.

At the 0.50 threshold: 60 missed at 8,000 is 480,000, plus 20 false alarms at 200 is 4,000. Total 484,000 rupees.

At the 0.20 threshold: 25 missed at 8,000 is 200,000, plus 300 false alarms at 200 is 60,000. Total 260,000 rupees.

The less precise policy is cheaper by 224,000 rupees a month. And the ratio in the costs tells you where the threshold should sit without any searching. Acting on a case is worth it when the expected cost of not acting exceeds the expected cost of acting: p * 8000 against (1 - p) * 200. Setting those equal gives p = 200 / 8200, which is 0.024. The break-even point is two and a half percent, not fifty.

That is the general form. When a false negative costs E and a false positive costs R, the threshold that minimises expected cost is R / (R + E). A default of 0.5 asserts that the two mistakes cost the same, which is true in almost no application anyone gets paid to build.

The scikit-learn documentation has a worked version on the German credit dataset, where the cost matrix says a bad credit approved is five times worse than a good credit refused. Its business metric at the default threshold is -209. Tuning the threshold against that same metric moves it to 0.03 and the business metric to -143. Same model, same features, same training. The threshold was the whole intervention.

Why the usual summary numbers choose badly

Run the standard metrics over the two fraud policies above and watch them all recommend the expensive one.

Accuracy. At 0.50 the model gets 9,920 of 10,000 right, which is 99.2 percent. At 0.20 it gets 9,675 right, which is 96.75 percent. Accuracy prefers the policy that costs more. It also prefers doing nothing at all: a model that never flags anything is 99 percent accurate, because 99 percent of transactions are legitimate.

F1. The harmonic mean of precision and recall gives 0.50 at the higher threshold and 0.32 at the lower one. It picks the option that costs an extra 224,000 rupees a month, because it treats a missed fraud and a wasted five minutes as equally weighted.

Area under the ROC curve. It is threshold-independent, so it says nothing about the operating point at all, and it is optimistic under imbalance. Saito and Rehmsmeier made this concrete by comparing a balanced set of 1,000 positives against 1,000 negatives with an imbalanced one of 1,000 positives against 10,000 negatives. The same point in ROC space, the same counts of true and false positives, indicates good performance in the balanced case and poor performance in the imbalanced one, and the ROC curve looks the same either way. Their conclusion is that the precision-recall plot is the more informative view when the classes are imbalanced, because precision responds to the thing that changed.

None of these metrics is broken. Each answers a question. The question they answer is not "which policy should we run", and that is the question the meeting is actually about.

The metric encodes what somebody decided to care about

There is a version of this failure that is much more serious than a suboptimal threshold, and it happens one level up, when the target itself is chosen for convenience.

Obermeyer, Powers, Vogeli and Mullainathan examined a commercial algorithm used across United States health systems to identify patients with complex needs, affecting millions of people. The algorithm predicted health care costs, on the reasonable-sounding basis that expensive patients are sick patients. Because less money is spent caring for Black patients than for White patients at the same level of illness, the cost proxy encoded that gap as lower predicted need. At a given risk score, Black patients were considerably sicker. Correcting the target would raise the share of Black patients receiving additional help from 17.7 percent to 46.5 percent.

Their own summary of the lesson is the sentence to remember: the choice of convenient, seemingly effective proxies for ground truth can be an important source of algorithmic bias in many contexts. No threshold tuning reaches this. The cost matrix was written into the label.

So the metric conversation has two levels. Which mistake costs more, given this target, and whether this target is the thing anyone actually cares about. The second is harder to ask and nobody in the modelling review is incentivised to raise it.

Getting the numbers out of the business

The arithmetic above needs two figures that rarely exist in writing. Asking for them directly usually fails, because "what does a false positive cost" is not a question anybody has been asked before. Three approaches work better.

Ask what happens next. Walk the false positive path: who receives the flag, how long they spend, what the customer experiences. Walk the false negative path the same way. The costs fall out of the process description rather than out of an estimate.

Ask for a ratio rather than an amount. People who cannot price a missed fraud will happily say that missing one is worth about forty wasted reviews. That ratio is all the threshold formula needs.

Offer the current position back. The team is running at 0.5 today, which asserts the two mistakes cost the same. Stating that out loud usually produces an immediate correction, and the correction is the number you wanted.

Some costs are not money and should not be converted into money to make the sum tidy. A wrongly refused loan and a wrongly approved one are not exchangeable at any rate. Where that is the case, the honest output is not a single optimal threshold but a table of operating points with their consequences spelled out, handed to whoever is accountable for the decision.

What follows for practice

• Report the confusion matrix at the threshold you propose to run, in counts, on realistic volumes. Rates hide the imbalance that drives the cost.
• State the threshold as a business decision with a named owner, not as a model parameter.
• Compute the break-even threshold from the cost ratio first and use it as the starting point.
• Choose the threshold on validation data, never on the sealed test set, for the reasons set out in Train, test, and the lie of a single score.
• Check the calibration of the scores before trusting the expected-cost arithmetic. Every term in that comparison is multiplied by a probability, and What probability buys you covers what happens when the probability is overstated.
• Revisit the threshold when the base rate moves. A fraud rate that doubles changes the optimal operating point even though the model has not changed at all.

The fraud team went back to 0.20 and then, once the costs were written down properly, to 0.05. Precision settled at 12 percent, which looked terrible and was reported alongside the monthly loss figure so that it stayed in context. The analysts got a queue sorted by expected loss rather than by score, so the expensive cases arrived first, and the cases nobody would ever have time for stopped being opened at all.

The remaining question in that project was not about thresholds. It was that the model itself had been bought from a vendor, and nobody in the room could say what was in it.

References

1. Post-tuning the decision threshold for cost-sensitive learning. scikit-learn documentation, version 1.9.0, 2026.
2. The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. Takaya Saito and Marc Rehmsmeier, PLOS ONE, 2015.
3. Dissecting racial bias in an algorithm used to manage the health of populations. Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan, Science, volume 366, 2019.