Experiment tracking for people who forget

The message arrives on a Thursday in March. Compliance wants to know why this particular model was chosen over the alternatives considered in November.

The answer exists. It is in a notebook, in a cell whose output still reads 0.883, sitting under a heading that says "v4 with the new features". The code in that notebook has been edited since. The table it read has been backfilled twice. The engineer who ran it works somewhere else now. Rerunning the cell produces 0.861, and nobody in the room can say whether that is because the data changed, the code changed, or the run was simply lucky the first time.

Four days go into reconstructing an answer that took twenty minutes to produce and was never written down.

The same configuration does not give the same number

Before anything about tooling, the uncomfortable fact underneath it.

Picard scanned random seeds across popular computer vision architectures and reported what the spread actually is. On CIFAR 10 with a short training budget and 10,000 seeds, accuracy averaged 90.02 percent with a standard deviation of 0.23, and the runs ranged from 89.01 percent to 90.83 percent. The gap between the worst seed and the best seed was 1.82 points, from nothing but the initialisation. His comment on that gap is the part worth quoting: such difference is widely considered as significant in the community, to the point of being an argument for publication at very selective venues, whereas we know here that this is just the effect of finding a lucky or cursed seed.

The same scan on ImageNet with a pre-trained ResNet50 gave 75.48 percent with a standard deviation of 0.10 over 50 seeds, and he makes the same observation there: a 0.5 percent difference in accuracy on ImageNet is widely considered significant in the computer vision community, whereas we know here that it is entirely due to the change of seed.

Do the arithmetic on your own team's decisions with those numbers. Suppose a variant beats the incumbent by 0.4 points, each measured once, with a per-run standard deviation around 0.23. The standard error of the difference between two single runs is 0.23 times the square root of two, about 0.33, so 0.4 is 1.2 standard errors. That is not evidence. Run five seeds of each and the standard error of the difference falls to 0.23 times the square root of two-fifths, about 0.15, and the same 0.4 becomes 2.7 standard errors. The decision changed because the measurement changed, not because the model did.

Bouthillier and colleagues went further and modelled the whole benchmarking process, finding that variance due to data sampling, parameter initialization and hyperparameter choice impact markedly the results. Their most useful finding for a team on a budget is counter-intuitive: adding more sources of variation to an imperfect estimator approaches better the ideal estimator at a 51 times reduction in compute cost. Randomising the data split and the initialisation across a handful of runs buys more truth than perfecting one run.

The practical upshot is that a tracked run is not bookkeeping. It is the only way to know whether a difference you are about to act on exists. Train, test, and the lie of a single score makes the same argument about the split; this is the same argument about the seed.

What a run has to record

A record is useful only if a stranger can use it to reproduce the result without asking you anything. That sets the list.

1. The code, as a commit hash, from a clean working tree. Not a branch name. A run launched from uncommitted changes should either refuse to start or write the diff into the record.
2. The data, as an identifier that cannot be edited in place. A partition path with a date, a table snapshot id, a content hash. "The orders table" is not an identifier, because the orders table in March is not the orders table in November.
3. Every parameter, dumped whole. Not the ones you were sweeping. All of them, including the defaults, because next quarter's default will be different.
4. The seed, and the fact that it was set. A run whose seed was not fixed should say so in the record rather than leave a reader assuming it was.
5. The environment. Library versions, driver version, hardware. A result that only appears on one GPU generation is a fact worth having.
6. The metrics, plural, with the evaluation set identified. One number is a summary of a decision, not a record of it.
7. The artefacts. The fitted model, the predictions on the evaluation set, and the confusion counts. Predictions are the cheapest thing to store and the most useful thing to have later, because every metric anybody asks for afterwards can be recomputed from them without touching the model.

That last point deserves its own line. Storing 40,000 predicted scores and their labels costs under a megabyte. It converts every future question about that model from a rerun into a query.

Reproducibility is a property of a system

Pineau and colleagues describe the reproducibility programme run at NeurIPS in 2019 and the reasoning behind it. Their definition is worth borrowing exactly: reproducibility, that is obtaining similar results as presented in a paper or talk, using the same code and data when available, is a necessary step to verify the reliability of research findings. Their programme had three components: a code submission policy, a community-wide reproducibility challenge, and the inclusion of the Machine Learning Reproducibility checklist as part of the paper submission process.

The mechanism there is the transferable part. None of the three depends on individuals being careful. A submission policy, a checklist attached to the artefact, and somebody else attempting the reproduction are all system properties, and they work on tired people at the end of a quarter.

The same three translate directly into a team. Runs are launched by a script that refuses a dirty tree, so the code submission policy is enforced by the runner. A run record has required fields, so the checklist is a schema. And once a quarter somebody other than the author reproduces the current production model from its record, which is the reproduction attempt.

That third practice is the one teams skip and the one that finds the problems. It is also cheap: one afternoon, once a quarter, and it either succeeds or it hands you a list of what your record was missing.

Versioning, velocity, and the thing in between

Shankar, Garcia, Hellerstein and Parameswaran interviewed eighteen machine learning engineers about production practice and found three variables that govern success for a production ML deployment: velocity, validation and versioning. Their engineers ran experiments fast and deliberately, and one of their observations reframes what tracking is for: it's actually okay for experiments and models not to make it to production, what matters is making sure ideas can be prototyped and validated quickly.

Tracking serves velocity rather than opposing it. The team that records everything can throw work away without losing it, which is what makes throwing work away comfortable.

They also name the anti-patterns, and both are recognisable. Keeping GPUs warm is unnecessarily parallelising experiments rather than focusing on high-value work: a queue that is always full looks like productivity and is often the absence of a decision about what to try next. Retrofitting an explanation is constructing the reason a result worked after seeing that it worked, rather than understanding the mechanism beforehand. A tracked hypothesis, written before the run, is the only defence against the second one, because the record shows what you expected.

When the run cannot be reproduced

It will happen. What matters is what the team does in the next hour.

• Check the data first. In practice this is the cause more often than the code. Compare row counts and the ten summary statistics from Data pipelines that do not rot between the recorded snapshot and today's. A backfill that corrected six months of a column will move every number downstream of it.
• Then check the seed and the ordering. Non-determinism from parallel data loading, from unordered aggregation, and from certain GPU kernels is real and is not a bug in your code.
• Then check the environment. A library minor version can change a default.
• Write down which of the three it was. That sentence, stored on the run, is worth more than the rerun. It tells the next person where to look.

If the answer is that nobody can tell, the honest response is to treat the number as unverified and to stop citing it in decisions. A result nobody can reproduce is a rumour with a decimal point.

The habit that costs ten minutes

None of this needs a platform. It needs a discipline that a tired person can follow at 18:40 on a Friday.

Every run writes one row to one table: timestamp, author, commit hash, data snapshot id, a parameter dump, the seed, the metrics, and a path to the predictions. Every run is launched from a script, never from a notebook cell, and the notebook calls the script. Every claim in a document carries the run id that produced it. Every model that goes to production carries the run id of the run that trained it, which is what makes the deployment record in the next article possible at all.

Ten minutes of setup per project, and about fifteen seconds per run. Against four days in March, and against an evaluation number that turns out to have been a lucky seed, that is not a close call.

The record only becomes valuable when somebody asks a question about the past. The next article is about the moment that guarantees somebody will: putting the model in front of real traffic.

References

1. Torch.manual_seed(3407) is all you need: On the influence of random seeds in deep learning architectures for computer vision. David Picard, arXiv, 2021.
2. Accounting for Variance in Machine Learning Benchmarks. Xavier Bouthillier, Pierre Delaunay, Mirko Bronzi, Assya Trofimov, Brennan Nichyporuk, Justin Szeto, Naz Sepah, Edward Raff, Kanika Madan, Vikram Voleti, Samira Ebrahimi Kahou, Vincent Michalski, Dmitriy Serdyuk, Tal Arbel, Chris Pal, Gael Varoquaux and Pascal Vincent, arXiv, 2021.
3. Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program). Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d'Alche-Buc, Emily Fox and Hugo Larochelle, arXiv, 2020.
4. Operationalizing Machine Learning: An Interview Study. Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein and Aditya G. Parameswaran, arXiv, 2022.