Our techniques enable a linear reduction in training time with ∼90% efficiency as we scale to large minibatch sizes, allowing us to train an accurate 8k minibatch ResNet-50 model in 1 hour on 256 GPUs.įor k workers each with a per-worker minibatch of size n, following ( 4), gradient aggregation must be performed over the entire set of k n examples according to 1 k n ∑ j ∑ x ∈ B j l ( x, w t ). Using this simple approach, accuracy of our models is invariant to minibatch size (up to an 8k minibatch size). All other hyper-parameters are kept fixed. For all minibatch sizes we set the learning rate as a linearįunction of the minibatch size and apply a simple warmup phase for the first few epochs of training. We present a simple and general technique for scaling distributed synchronous SGD to minibatches of up to 8k images while maintaining the top-1 error of small minibatch training. Error range of plus/minus two standard deviations is shown. Figure 1: ImageNet top-1 validation error vs. But as model and data scale grow, so does training time discovering the potential and limits of large-scale deep learning requires developing novel techniques to keep training time manageable. Moreover, this pattern generalizes: larger datasets and neural network architectures consistently yield improved accuracy across all tasks that benefit from pre-training. Take the profound impact in computer vision as an example: visual representations learned by deep convolutional neural networks show excellent performance on previously challenging tasks like ImageNet classification and can be transferred to difficult perception problems such as object detection and segmentation. We are in an unprecedented era in AI research history in which the increasing data and model scale is rapidly improving accuracy in computer vision, speech Internet-scale data with high efficiency. This system enables us to train visual recognition models on In one hour, while matching small minibatch accuracy. With these simple techniques, ourĬaffe2-based system trains ResNet-50 with a minibatch size of 8192 on 256 GPUs Optimization challenges early in training. Result, we adopt a linear scaling rule for adjusting learning rates as aįunction of minibatch size and develop a new warmup scheme that overcomes When training with large minibatch sizes up to 8192 images. Specifically, we show no loss of accuracy This paper, we empirically show that on the ImageNet dataset large minibatchesĬause optimization difficulties, but when these are addressed the trained

Must be large, which implies nontrivial growth in the SGD minibatch size. Yet to make this scheme efficient, the per-worker workload Potential solution to this problem by dividing SGD minibatches over a pool of Larger networks and larger datasets result in longer training times that impede Deep learning thrives with large neural networks and large datasets.

Accurate 5 in 1