This article's translation was prepared in anticipation of the course launch
Distributed training on multiple high-performance computing instances can reduce the training time of modern deep neural networks on large datasets from several weeks to hours or even minutes, making this training technique predominant in practical deep learning applications. Users need to understand how to share and synchronize data across multiple instances, which significantly impacts scalability efficiency. Additionally, users must know how to deploy a training script that operates on a single instance across multiple instances.
In this article, we will discuss a fast and simple way to achieve distributed training using the open-source deep learning library Apache MXNet and the distributed training framework Horovod. We will visually demonstrate the performance advantages of the Horovod framework and show how to write an MXNet training script to work distributedly with Horovod.
What is Apache MXNet
– an open-source deep learning framework used to create, train, and deploy deep neural networks. MXNet abstracts the complexities associated with implementing neural networks, offers high performance and scalability, and provides APIs for popular programming languages such as , , , , , , and others.
Distributed Training in MXNet with Parameter Server
uses a parameter server approach. It employs a set of parameter servers to collect gradients from each worker, perform aggregation, and send updated gradients back to the workers for the next optimization iteration. Determining the right ratio of servers to workers is key to effective scaling. If there is only one parameter server, it may become a bottleneck in conducting computations. Conversely, if too many servers are used, the "many-to-many" connection can congest all network connections.
What is Horovod
– an open framework for distributed deep learning developed at Uber. It uses efficient communication technologies between multiple GPUs and nodes, such as NVIDIA Collective Communications Library (NCCL) and Message Passing Interface (MPI), to distribute and aggregate model parameters across workers. It optimizes network bandwidth usage and scales well when working with deep neural network models. Currently, it supports several popular machine learning frameworks, namely , TensorFlow, Keras, and PyTorch.
Integration of MXNet and Horovod
MXNet integrates with Horovod through the distributed training API defined in Horovod. In Horovod, the communication APIs horovod.broadcast(), horovod.allgather() and horovod.allreduce() are implemented using asynchronous callbacks from the MXNet engine as part of its task graph. This allows data dependencies between communication and computations to be easily managed by the MXNet engine, avoiding performance loss due to synchronization. The distributed optimizer object defined in Horovod horovod.DistributedOptimizer expands Optimizer in MXNet is set up to invoke the corresponding Horovod APIs for distributed parameter updates. All these implementation details are transparent to end-users.
Quick Start
You can quickly start training a small convolutional neural network on the MNIST dataset using MXNet and Horovod on your MacBook.
To get started, install mxnet and horovod from PyPI:
pip install mxnet
pip install horovodNote: If you encounter an error during pip install horovod, you may need to add the variable MACOSX_DEPLOYMENT_TARGET=10.vv, where vv – this is the version of your MacOS version, for example, for MacOSX Sierra you would write MACOSX_DEPLOYMENT_TARGET=10.12 pip install horovod
Then install OpenMPI .
Finally, download the test script mxnet_mnist.py and run the following commands in the MacBook terminal in the working directory:
mpirun -np 2 -H localhost:2 -bind-to none -map-by slot python mxnet_mnist.pyThis will launch training on two cores of your processor. The output will be as follows:
INFO:root:Epoch[0] Batch [0-50] Speed: 2248.71 samples/sec accuracy=0.583640
INFO:root:Epoch[0] Batch [50-100] Speed: 2273.89 samples/sec accuracy=0.882812
INFO:root:Epoch[0] Batch [50-100] Speed: 2273.39 samples/sec accuracy=0.870000Performance Demonstration
When training the ResNet50-v1 model on the ImageNet dataset with 64 GPUs in eight instances p3.16xlarge On AWS cloud, each EC2 instance contains 8 NVIDIA Tesla V100 GPUs, achieving a training throughput of 45,000 images/sec (i.e., the number of samples trained per second). The training completed in 44 minutes after 90 epochs with the best accuracy of 75.7%.
We compared this with distributed training using MXNet with a parameter server approach across 8, 16, 32, and 64 GPUs against a single parameter server with a worker-to-server ratio of 1:1 and 2:1, respectively. The results can be seen in Figure 1 below. The left y-axis shows the number of images trained per second in bars, while the lines on the right y-axis reflect the scaling efficiency (i.e., the ratio of actual throughput to ideal throughput). As you can see, the choice of the number of servers affects scaling efficiency. When using a single parameter server, scaling efficiency drops to 38% with 64 GPUs. To achieve the same level of scaling efficiency as with Horovod, the number of servers must be doubled relative to the number of workers.
Figure 1. Comparison of distributed training using MXNet with Horovod and with a parameter server.
In Table 1 below, we compared the total instance cost when running experiments on 64 GPUs. Using MXNet along with Horovod provides the best throughput at the lowest costs.
Table 1. Cost comparison between Horovod and the parameter server with a 2:1 server-to-worker ratio.
Steps to Reproduce
In the following steps, we will show you how to reproduce the results of distributed training using MXNet and Horovod. To learn more about distributed training with MXNet, read .
Step 1
Create a cluster of homogeneous instances with MXNet version 1.4.0 or higher and Horovod version 0.16.0 or higher to utilize distributed training. You will also need to install libraries for GPU training. For our instances, we chose Ubuntu 16.04 Linux with GPU Driver 396.44, CUDA 9.2, cuDNN library 7.2.1, NCCL communicator 2.2.13, and OpenMPI 3.1.1. You can also use , where these libraries are already pre-installed.
Step 2
Add the ability to work with the Horovod API to your training script in MXNet. The script below, based on the MXNet Gluon API, can be used as a simple template. The lines highlighted in bold are necessary if you already have a corresponding training script. Here are some critical changes that need to be made for training with Horovod:
- Set the context according to the local Horovod rank (line 8) to ensure that the training is performed on the correct GPU core.
- Pass initial parameters from one worker to all (line 18) to ensure that all workers start with the same initial parameters.
- Create Horovod DistributedOptimizer (line 25) to update parameters in a distributed manner.
For the complete script, refer to the Horovod-MXNet examples. and .
1 import mxnet as mx
2 import horovod.mxnet as hvd
3
4 # Horovod: initialize Horovod
5 hvd.init()
6
7 # Horovod: pin a GPU to be used to local rank
8 context = mx.gpu(hvd.local_rank())
9
10 # Build model
11 model = ...
12
13 # Initialize parameters
14 model.initialize(initializer, ctx=context)
15 params = model.collect_params()
16
17 # Horovod: broadcast parameters
18 hvd.broadcast_parameters(params, root_rank=0)
19
20 # Create optimizer
21 optimizer_params = ...
22 opt = mx.optimizer.create('sgd', **optimizer_params)
23
24 # Horovod: wrap optimizer with DistributedOptimizer
25 opt = hvd.DistributedOptimizer(opt)
26
27 # Create trainer and loss function
28 trainer = mx.gluon.Trainer(params, opt, kvstore=None)
29 loss_fn = ...
30
31 # Train model
32 for epoch in range(num_epoch):
33 ...Step 3
Log into one of the workers to run distributed training using the MPI directive. In this example, distributed training is initiated on four instances with 4 GPUs each, resulting in a total of 16 GPUs in the cluster. The stochastic gradient descent (SGD) optimizer will be used with the following hyperparameters:
- mini-batch size: 256
- learning rate: 0.1
- momentum: 0.9
- weight decay: 0.0001
As we scale from one GPU to 64 GPUs, we linearly scaled the learning rate according to the number of GPUs (from 0.1 for 1 GPU to 6.4 for 64 GPUs), while keeping the number of images per GPU at 256 (from a batch of 256 images for 1 GPU to 16,384 for 64 GPUs). The weight decay and momentum parameters were adjusted as the number of GPUs increased. We employed mixed precision training with float16 data type during the forward pass and float32 for gradients, to accelerate float16 computations supported by NVIDIA Tesla GPUs.
$ mpirun -np 16
-H server1:4,server2:4,server3:4,server4:4
-bind-to none -map-by slot
-mca pml ob1 -mca btl ^openib
python mxnet_imagenet_resnet50.pyConclusion
In this article, we explored a scalable approach to distributed model training using Apache MXNet and Horovod. We demonstrated the effectiveness of scaling and cost efficiency compared to a parameter server approach on the ImageNet dataset, on which the ResNet50-v1 model was trained. We also outlined the steps you can take to modify an existing script to initiate training on multiple instances using Horovod.
If you are just starting with MXNet and deep learning, visit the installation page , to set up MXNet first. We also highly recommend reading the article , to get started.
If you have already worked with MXNet and want to try distributed training with Horovod, check out the , set it up with MXNet, and follow the example or .
*cost calculated based on AWS for EC2 instances
Theory and practice of using ClickHouse in real applications. Alexander Zaitsev (2018)
Source: habr.com
