Training a network without wasting a month

A training script is started on a Friday. The progress bar settles on an estimate of 26 days, and everybody agrees to let it run over the weekend and see. On Monday the loss is lower than it was, which proves nothing, and nobody can say whether day 26 will be better than day 6.

Do the arithmetic before starting instead. A dataset of 1,280,000 examples at a batch size of 256 gives 5,000 steps per epoch. Ninety epochs is 450,000 steps. If a step takes five seconds, that is 2,250,000 seconds, which is the 26 days on the progress bar. Every one of those four numbers is a decision, and three of them can be changed.

The batch size is a hardware decision with a statistics bill

Increase the batch to 2,048 and the step count falls by a factor of eight, to 56,250. The step gets slower, but not eight times slower, because a small batch leaves most of an accelerator idle. If the larger step takes twelve seconds rather than forty, the run finishes in 675,000 seconds, or 7.8 days. Same epochs, same data seen, one third of the calendar.

The bill arrives as instability. A larger batch gives a less noisy estimate of the gradient, so the sensible step to take with it is larger, and a learning rate tuned for 256 will crawl at 2,048.

Goyal and colleagues addressed this directly, with a hyper-parameter-free linear scaling rule for adjusting learning rates as a function of minibatch size, and a warmup scheme that overcomes optimisation challenges early in training. They report no loss of accuracy up to a minibatch size of 8,192, and train ResNet-50 with that batch on 256 GPUs in one hour while matching small minibatch accuracy. The rule is what it says: multiply the batch by eight and multiply the learning rate by eight. The warmup exists because that scaled rate is destructive in the first few hundred steps, when the weights are still random.

There is a ceiling. McCandlish, Kaplan, Amodei and the OpenAI Dota Team show that a simple and easy-to-measure statistic they call the gradient noise scale predicts the largest useful batch size across supervised learning, reinforcement learning and generative modelling, and that this scale grows during training as the loss falls. Past that size, doubling the batch stops halving the number of steps, and the extra compute buys almost nothing. They frame it as a tradeoff between time efficiency and compute efficiency, with useful batch sizes running from the tens of thousands for ImageNet up to millions for their Dota agents.

Two practical consequences. If your accelerator cannot hold the batch you want, accumulate gradients over several forward passes before stepping: eight passes at 256 is arithmetically a batch of 2,048, at the cost of eight forward and backward passes per step rather than one. And do not raise the batch indefinitely on the assumption that bigger is faster, because there is a measurable point where it stops being true.

The learning rate, found rather than guessed

The learning rate is the single setting most likely to waste the month. Too small and the run does not converge in the budget. Too large and the loss diverges, or units die in the manner shown in Why activation functions matter, and the run converges to something worse while looking fine.

Leslie Smith described a way to find it without a search. Rather than fixing the rate, let it vary cyclically between reasonable boundary values, which he reports gives improved classification accuracy without the need to tune and often in fewer iterations. His method for finding those boundaries is the useful part on its own: linearly increase the learning rate of the network for a few epochs and watch the loss. It falls, keeps falling, then turns sharply upward. The rate at which it turns is the ceiling, and something below it is the working value.

That test costs a few minutes on a small slice of the data. Compare it to the cost of finding out on day nine.

Once the working value is known, the schedule matters. Loshchilov and Hutter's warm restart method repeatedly lowers the rate and then resets it, which they propose as a simple technique to improve the anytime performance of stochastic gradient descent, reporting error rates of 3.14 percent on CIFAR-10 and 16.21 percent on CIFAR-100.

Smith, Kindermans, Ying and Le point out that the schedule and the batch size are two views of one thing. They obtain equivalent training and validation curves by increasing the batch size during training instead of decaying the learning rate, across plain stochastic gradient descent, momentum, Nesterov momentum and Adam, reaching the same test accuracies in the same number of epochs but with fewer parameter updates. Their headline is that this trains ResNet-50 on ImageNet to 76.1 percent validation accuracy in under 30 minutes. The transferable idea is that what falls over a run is the ratio of noise to signal in the gradient, and either lever moves it.

Regularisation, and one specific trap

A network with more parameters than examples will fit the noise, which is the mechanism in Bias and variance in plain terms arriving with more parameters than that article contemplated.

Srivastava, Hinton, Krizhevsky, Sutskever and Salakhutdinov's method is to randomly drop units, along with their connections, from the network during training, which prevents units co-adapting. Their framing is worth keeping: during training the procedure samples from an exponential number of different thinned networks, and at test time a single unthinned network with smaller weights approximates averaging all of them. Dropout is an ensemble made cheap, which is the same argument as Random forests and why averaging works, applied inside one model.

Weight decay is the other standard tool, and it carries a trap worth naming. Loshchilov and Hutter show that L2 regularisation and weight decay are equivalent for standard stochastic gradient descent, when rescaled by the learning rate, but that this is not the case for adaptive gradient algorithms such as Adam. Many implementations label the L2 term "weight decay" anyway. Their decoupled version, which applies the decay outside the adaptive update, is the reason AdamW exists as a separate optimiser, and using plain Adam with a weight decay argument is not the same thing.

Knowing when to stop

A 26 day run is a bet that day 26 is better than day 6. That bet can be settled with evidence rather than patience.

• Evaluate on a validation split, on a fixed schedule. Training loss falls almost monotonically and reports nothing about generalisation. The distinction is the whole subject of Train, test, and the lie of a single score.
• Keep the best checkpoint, not the last one. If validation loss bottoms out at epoch 40 and drifts upward through epoch 90, the model you want is on disk from epoch 40 only if somebody saved it.
• Stop on a patience rule. Halt if the validation metric has not improved in, say, ten evaluations. Set the number in advance, because a rule invented while staring at a curve is not a rule.
• Watch the gap, not the level. A widening gap between training and validation loss is overfitting, and more epochs will widen it. A high validation loss with no gap is underfitting, and more epochs might help.

Before starting the long run

Most of a month is lost in the first hour, to something that a short run would have caught.

• Overfit ten examples deliberately. A correct training loop can drive the loss to nearly zero on a batch of ten. If it cannot, the bug is in the code, not the data, and finding that out costs two minutes.
• Run one epoch on one percent of the data. Confirm the loss falls, the checkpoint writes, the evaluation runs and the metric is computed on the right thing.
• Print the fraction of dead units and the per-layer gradient magnitudes at step 100. The diagnosis in Backpropagation worked by hand works on any network and takes one line.
• Fix and record the seed, the data version and every setting above. A result that cannot be reproduced is not a result, and a run this long will not be repeated casually.
• Write down, before starting, what number would make this model worth deploying. Without it, the run ends when somebody gets bored.

What none of this settles

The settings in this article change how quickly a given architecture reaches a given quality. They do not change what that quality is. If the data has a leak of the kind described in The data comes first, the perfectly tuned run produces a perfectly tuned illusion, faster.

And the arithmetic at the top is the honest form of the question. Steps per epoch times epochs times seconds per step. Anyone can compute it before the weekend rather than after, and the answer usually shows that the expensive setting is not the one being argued about.

This is the last article in the series. Starting from a weighted sum and a switch, everything since has been the same two ideas rearranged: multiply things together and add them up, then put something non-linear in the way so the stack cannot collapse. The architectures differ in what they let each unit see. The training loop, and its arithmetic, does not change at all.

References

1. Accurate, Large Minibatch SGD, Training ImageNet in 1 Hour. Priya Goyal, Piotr Dollar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia and Kaiming He, arXiv, 2017.
2. An Empirical Model of Large-Batch Training. Sam McCandlish, Jared Kaplan, Dario Amodei and the OpenAI Dota Team, arXiv, 2018.
3. Don't Decay the Learning Rate, Increase the Batch Size. Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying and Quoc V. Le, arXiv, 2017.
4. Cyclical Learning Rates for Training Neural Networks. Leslie N. Smith, arXiv, 2015.
5. Decoupled Weight Decay Regularization. Ilya Loshchilov and Frank Hutter, arXiv, 2017.
6. Dropout, A Simple Way to Prevent Neural Networks from Overfitting. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever and Ruslan Salakhutdinov, Journal of Machine Learning Research, volume 15, 2014.