The cost model nobody writes down

The pilot ran for six weeks with forty internal users. It cost about as much as lunch. The demo was excellent, the feature was approved, and somebody wrote "roll out to all users in Q3" on a slide.

Nobody had multiplied anything.

The rollout is a shopping assistant on product pages. Four million sessions a month, about 12 percent of visitors open the assistant, and a conversation runs about three exchanges. That is roughly 1.44 million generations a month. The pilot had produced about 900.

What follows is the arithmetic that should have been on that slide. The numbers are worked in the open so you can substitute your own, and the point of the exercise is not the total. It is that the largest term in the model turns out to be a promise somebody made about waiting time.

What a generation costs

Two quantities decide everything: how many tokens the accelerator has to move, and how many requests it can work on at once.

Pope, Douglas, Chowdhery, Devlin, Bradbury, Levskaya, Heek, Xiao, Agrawal and Dean published measurements on exactly this problem, studying efficient generative inference for Transformer models in one of its most challenging settings: large deep models, with tight latency targets and long sequence lengths. Their reported result gives a firm anchor for the arithmetic: a low-batch-size latency of 29ms per token during generation, using int8 weight quantization, and a 76 percent model FLOPS utilization during large-batch-size processing of input tokens, on a 540 billion parameter model with a 2,048 token context.

Take those two numbers as the boundary conditions of the tradeoff, and assume, as this team's contract does, an inference slice at 32 dollars an hour and an exchange rate of 84 rupees to the dollar. Assume a request with a 3,200 token prompt and a 250 token answer.

Serving one request at a time. At 29ms per generated token, 250 tokens take 7.25 seconds. One slice therefore completes about 497 requests an hour, and 32 dollars divided by 497 is about 6.4 cents per request.

Serving twenty-four at a time. Batching slows each individual stream. Suppose per-token latency rises from 29ms to 45ms, which is a deliberately pessimistic assumption. Each stream now produces 22 tokens a second, but there are 24 of them, so the slice emits 528 tokens a second. At 250 tokens per answer that is 2.11 requests a second, or 7,603 an hour, and 32 dollars divided by 7,603 is about 0.42 cents per request.

Fifteen times cheaper. The user waits 11.25 seconds instead of 7.25, which is 55 percent longer.

Now multiply by 1.44 million generations a month.

One at a time: about 92,000 dollars a month, roughly Rs 77 lakh.
Twenty-four at a time: about 6,050 dollars a month, roughly Rs 5.1 lakh.

The model is identical in both rows. The prompt is identical. The only difference is a queueing decision, and it is a 15x difference in the bill. Anybody who has costed a machine learning feature by asking what the model costs has already missed the largest term.

The prefill side deserves its own line, because it responds to a different lever. Processing the 3,200 token prompt is compute-bound and parallel, which is why the paper reports high utilisation there. Cutting the prompt to 1,400 tokens by retrieving three passages instead of five reduces that work by about 56 percent, and whether three passages are enough is exactly the question Retrieval-augmented generation, honestly works through. Prompt length is a cost lever most teams never touch because nobody owns it.

The latency budget the arithmetic has to fit inside

Eleven seconds is not a number you can hide. It has to be designed for, and the design constraint is not the model's total time but the time before the interface responds at all.

Google's guidance on user-visible responsiveness sets the levels: pages should have an INP of 200 milliseconds or less, and LCP should occur within 2.5 seconds of when the page first starts loading, both measured at the 75th percentile of page loads, segmented across mobile and desktop devices.

An 11 second generation and a 200 millisecond interaction budget are not in conflict, as long as the 200 milliseconds buys an acknowledgement rather than an answer. That gives a budget with real slots in it:

0 to 200ms. The interface responds to the click. Nothing about the model is involved.
200 to 300ms. Embed the query and search the index.
300 to 600ms. Prefill the 3,200 token prompt.
645ms. First token appears, and the answer streams from there.
11.9s. Last token.

Streaming is what makes the second column affordable, because it decouples the number the user experiences from the number the finance team pays for. That is the same argument Putting a model behind an interface makes from the user's side, arriving here from the ledger's side.

One more piece of arithmetic belongs in the budget, because it is the one that surprises teams at the tail. Suppose retrieval fans out to five index shards and each shard has a 1 in 100 chance of exceeding 400ms. The probability that at least one is slow is 1 minus 0.99 to the fifth, about 5 percent. Grow the fan-out to 100 shards and it becomes 1 minus 0.99 to the hundredth, about 63 percent. A tail that is invisible on one component is the common case once you have enough components, so the p99 you promise has to be budgeted per component, not per request.

Does the feature pay for itself

A cost model is only half an argument. The other half is what the thing is worth, and it should be written in the same units.

Suppose the assistant lifts conversion by 0.4 percentage points on a Rs 2,400 average order. Each conversation is then worth about 0.004 times Rs 2,400, which is Rs 9.60.

A three-exchange conversation costs three generations.

Batched at 24: 3 times Rs 0.35, which is Rs 1.05 per conversation. The feature returns about nine times its serving cost.
Served one at a time: 3 times Rs 5.38, which is Rs 16.14 per conversation. The feature loses about Rs 6.50 every time somebody uses it.

Same model, same lift, same customers. One version is a good business and the other is a subsidy, and the sentence that decides which is "responses must feel instant", said in a meeting by somebody who was not asked to price it.

This is also where the fine-tuning question gets its honest answer. A smaller model that has been trained properly is not only cheaper to train; it is cheaper on every single request forever. Hoffmann and colleagues made the point with a specific comparison: for compute-optimal training, the model size and the number of training tokens should be scaled equally, and their Chinchilla uses the same compute budget as Gopher but with 70B parameters and 4 times more data, while uniformly and significantly outperforming Gopher at 280B, GPT-3 at 175B, Jurassic-1 at 178B and Megatron-Turing NLG at 530B. A model a quarter of the size, serving better answers, on every request of its life. The comparison of approaches in Fine-tuning against prompting, a real comparison is incomplete until this row is added to it, because a one-off training cost amortises and a per-request cost does not.

The lines nobody writes down

The serving bill is the part everyone eventually computes. The rest of the cost is the part Sculley and colleagues warned about when they observed that, under the software engineering framework of technical debt, it is common to incur massive ongoing maintenance costs in real-world ML systems, driven by risk factors including data dependencies, configuration issues, hidden feedback loops and changes in the external world.

Priced out for this one feature, at plausible rates:

Retraining and re-evaluation. Eight releases a year, each needing 500 freshly labelled examples at Rs 40 each: Rs 160,000 a year, plus the engineering days around it.
On-call. One engineer carrying the pager for this system, at roughly 0.2 of a full-time role once you count the interruptions rather than the incidents.
Human review of the output. If 4 percent of 1.44 million monthly generations are corrected by a person taking 90 seconds, that is 57,600 corrections, 1,440 hours, and roughly nine full-time people. This line is frequently larger than the compute line and it almost never appears on the slide.
The evaluation set itself. Maintaining it, growing it, and re-labelling the parts that went stale, which is the standing cost behind Evaluating a system that can say anything.
The monitoring. Cheap to run, and the reason the 4 percent correction rate is a known number rather than a guess, as Monitoring for silent failure sets out.
The idle capacity. Traffic at 20:00 is not traffic at 04:00. A slice provisioned for peak and billed by the hour is paid for at both. Peak-to-average ratios of three are ordinary, so a naive plan pays three times the average cost unless something scales down.

That last line has a straightforward fix and it is worth saying plainly: not every request needs the same treatment. Route the 60 percent of queries that a cached answer or a small model handles correctly to the cheap path, and reserve the expensive path for the rest. A 60 percent deflection at these rates takes the serving bill from Rs 5.1 lakh to about Rs 2 lakh a month, which is a larger saving than any prompt optimisation is going to produce.

The one page

Before a pilot becomes a rollout, one page, seven lines, each with a number on it.

Volume. Requests a month, at launch and at the twelve-month projection, with peak-to-average stated.
Shape. Prompt tokens and output tokens per request, measured on real traffic rather than assumed.
Throughput. Requests per hour per unit of hardware at the batch size you will actually run.
Unit cost. Currency per request, derived from lines 2 and 3, not quoted from a demo.
Latency commitment. The p50 and p99 you are promising, and the batch size that commitment forces.
Value per request. In the same currency, from a measured effect and not a hoped-for one.
The operating lines. Review labour, retraining, on-call, evaluation, idle capacity.

If line 6 is not at least three times line 4 plus the per-request share of line 7, the feature is a subsidy, and it is far better to know that in week one, when The first week of an AI project is asking what should improve, than in the quarter after the rollout.

None of this is difficult arithmetic. It is arithmetic nobody is assigned, which is why the cost model is the document that does not exist on most projects, and why the first person to write it usually changes the plan.

Which leaves one thing standing between a system that works and a system somebody else can run: the record of what it does, who decided it, and what happens when it is wrong.

References

Efficiently Scaling Transformer Inference. Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Anselm Levskaya, Jonathan Heek, Kefan Xiao, Shivani Agrawal and Jeff Dean, arXiv, 2022.
Training Compute-Optimal Large Language Models. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark and colleagues, arXiv, 2022.
Web Vitals. Philip Walton and Barry Pollard, web.dev, Google, 2024.
Hidden Technical Debt in Machine Learning Systems. D. Sculley, Gary Holt, Daniel Golovin, Eugene Davydov, Todd Phillips, Dietmar Ebner, Vinay Chaudhary, Michael Young, Jean-Francois Crespo and Dan Dennison, Advances in Neural Information Processing Systems, 2015.