The first week of an AI project

The kickoff runs forty minutes. A card issuer wants to use machine learning to reduce fraud losses, the sponsor is the head of risk operations, and by the end there is a shared drive, a channel, and a date six weeks out. Everyone leaves satisfied.

Six weeks later there is a model. It ranks transactions well. It is never switched on, because the fraud analysts can review 200 cases a day and that number was fixed by headcount before the project started. Nobody had asked what the analysts do with a score.

The model was not the problem. The first week was.

The decision, not the prediction

Start by naming the decision that will be made differently, by a named person or a named system, on a named schedule. If no answer survives that sentence, there is no project yet.

At the card issuer the answer is precise once someone asks. Twenty analysts review flagged transactions. The queue is capped at 200 cases a day. Every case is reviewed by a human, decided within eight minutes, and either released or held. The model does not decide anything. It orders a queue whose length is set by payroll.

That reframing changes the whole brief. The question is no longer how accurately fraud can be predicted. It is how much of the fraud can be moved into the top 200 scores.

Zinkevich puts the same instinct first in his rules for engineering teams at Google: don't be afraid to launch a product without machine learning, and before formalizing what your machine learning system will do, track as much as possible in your current system. A team that cannot describe the current decision in numbers cannot recognise an improvement to it.

The baseline, in the units the business already uses

Get the current numbers on a whiteboard before anyone opens a notebook.

• 400,000 card transactions a day.
• About 0.05 percent are fraudulent, so roughly 200 fraudulent transactions a day.
• The existing rules engine flags far more than the team can handle, and the queue is truncated at 200 cases.
• Of those 200 reviewed cases, about 48 turn out to be fraud. That is a precision of 24 percent.
• So 48 frauds are caught and 152 are missed.
• The average loss on a missed fraud is Rs 12,000.

The missed losses are 152 times Rs 12,000, which is Rs 1,824,000 a day, or roughly Rs 55 million a month. That is the number the project is competing against, and it took twenty minutes to produce.

Now translate the ask. The sponsor wants fraud losses down 30 percent. Thirty percent of Rs 1,824,000 is Rs 547,200 a day, which at Rs 12,000 a case is about 46 more frauds caught per day. Since the queue stays at 200 cases, catching 48 plus 46 means 94 of the 200 reviewed cases must be genuinely fraudulent.

The business ask, converted, is: raise precision at rank 200 from 24 percent to about 47 percent. That is a modelling target a team can pursue, argue about, and fail at honestly. "Reduce fraud with AI" is not.

The ceiling is worth computing too. If every one of the 200 daily frauds were ranked in the top 200, the missed losses would go to zero and the project would be worth Rs 1.82 million a day. No model can be worth more than that without more analysts. Knowing the ceiling stops a sponsor from asking for 60 percent in month four.

Choosing where to sit between those points is the argument worked through in The cost of a wrong answer. The first week is where the inputs to that argument get collected, not where the threshold gets chosen.

What the data looked like at the moment of the decision

The third question of week one is which facts existed at the instant the decision has to be made, and it catches more projects than any modelling error.

The analytics warehouse has a fraud table. It is excellent. It contains the chargeback date, the disputed amount, the merchant category as later corrected by the operations team, and a flag for whether the customer was refunded. Every one of those columns is written days or weeks after the transaction. A model trained on them will score beautifully and predict nothing, for the reason set out in The data comes first.

The scoring model gets what the authorisation message carries at 200 milliseconds: amount, merchant identifier, card token, country, terminal type, and whatever the feature store has already computed about this card from earlier transactions. That list is usually much shorter than the warehouse list, and writing it down in week one is what stops a six-week detour.

Ask the same question of the label. When does a transaction become known fraud? At this issuer, a chargeback can arrive up to 120 days after the transaction. That single fact reshapes the project: the freshest three months of data have incomplete labels, evaluation has to account for it, and the retraining cadence cannot be weekly on a 120-day label lag. It also decides what monitoring can possibly detect, which is the subject of Monitoring for silent failure.

The system around the model

Amershi and colleagues watched software teams at Microsoft build applications with machine learning in them and reported on the result. They considered a nine-stage workflow process informed by prior experiences developing AI applications and data science tools, and found that various Microsoft teams have united this workflow into preexisting, well-evolved, Agile-like software engineering processes. Their three differences from ordinary software engineering are worth reading in week one rather than month three: discovering, managing and versioning the data is much more complex and difficult than in other software engineering; model customisation and reuse need skills software teams do not usually have; and AI components are harder to keep as distinct modules because models may be entangled in complex ways and show non-monotonic error behaviour.

Google's own architecture guidance states the proportion bluntly: only a small fraction of a real-world ML system is composed of the ML code, and the required surrounding elements are vast and complex.

Sculley and colleagues gave that surrounding mass its name. Using the software engineering framework of technical debt, they find it is common to incur massive ongoing maintenance costs in real-world ML systems, and list the risk factors to account for in system design: boundary erosion, entanglement, hidden feedback loops, undeclared consumers, data dependencies, configuration issues, changes in the external world, and system-level anti-patterns.

The practical consequence for week one is a short question addressed to whoever owns the surrounding systems. Who runs the job that produces the features. What happens when it is late. Who is called at 3am when a score is missing. If those answers do not exist, the project's real first deliverable is a pipeline, not a model, which is why the next article in this series is about pipelines rather than about algorithms.

What to refuse

Refusing well is the most valuable thing a senior person does in week one, and it is easier when the refusal names a condition rather than a judgement.

• No named decision owner. If nobody's action changes when the score changes, the output is a report. Build the report, cheaply, and revisit.
• No measurable current state. Without the 24 percent, the 200, and the Rs 12,000, there is nothing to beat and no way to be found wrong. Spend week one instrumenting instead. Zinkevich's second rule is exactly this, and it is second for a reason.
• A label that does not exist yet. "Predict which customers will be dissatisfied" with no dissatisfaction ever recorded is a labelling project wearing a modelling project's clothes. Say so, price the labelling, and let the sponsor decide.
• A target defined only as a percentage improvement in a metric nobody currently computes. Ask what number it is today. If the answer is a shrug, that is the first month of work.
• A deadline that assumes the data is ready. It is not. It has never once been.
• Success criteria that will be chosen after the results are in. Write the acceptance threshold down in week one, sign it, and store it where it cannot be edited quietly.

None of these refusals kills a project. Each converts a vague commitment into a specific one, which is the only kind that can be delivered.

The one page

Week one should end with a single page that anybody in the company can read in four minutes.

1. The decision. An analyst reviews a case and holds or releases it, 200 times a day.
2. The current numbers. 400,000 transactions, 200 frauds, 200 reviews, 48 caught, Rs 1.82 million missed daily.
3. The change we are attempting. Precision at rank 200, from 24 percent to 47 percent.
4. What we will use to decide it. The authorisation message plus the card's prior history, and nothing that arrives after the transaction.
5. How it will be judged. A held-out period later than the training period, with the 120-day label lag stated, because a single average over a random split answers a different question, as Train, test, and the lie of a single score works through.
6. What we are not doing. No automatic blocking, no change to the queue length, no new data collection, this quarter.
7. What would make us stop. Precision below 30 percent at rank 200 on the held-out period, or a feature dependency nobody will own.

Zinkevich's fourth rule is the right closing note: keep the first model simple and get the infrastructure right. A simple model on a clean path, running end to end, tells you more in week three than a strong model in a notebook tells you in month three, because only one of them has met the queue.

The next article follows that path one step further back, to the pipeline that has to deliver those features every morning without anyone noticing when it stops telling the truth.

References

1. Hidden Technical Debt in Machine Learning Systems. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo and Dan Dennison, Advances in Neural Information Processing Systems, 2015.
2. Rules of Machine Learning: Best Practices for ML Engineering. Martin Zinkevich, Google Developers, 2025.
3. Software Engineering for Machine Learning: A Case Study. Saleema Amershi, Andrew Begel, Christian Bird, Robert DeLine, Harald Gall, Ece Kamar, Nachiappan Nagappan, Besmira Nushi and Thomas Zimmermann, ICSE-SEIP, 2019.
4. MLOps: Continuous delivery and automation pipelines in machine learning. Google Cloud Architecture Center, 2024.