PS+: A Simple Yet Effective Framework for Fast Training on Parameter Server

A. Long Jin, Wenchao Xu, Song Guo, Bing Hu, Kwan Yeung

Research output: Contribution to journalArticlepeer-review

1 Citation (Scopus)

Abstract

In distributed training, workers collaboratively refine the global model parameters by pushing their updates to the Parameter Server and pulling fresher parameters for the next iteration. This introduces high communication costs for training at scale, and incurs unproductive waiting time for workers. To minimize the waiting time, existing approaches <italic>overlap communication and computation</italic> for deep neural networks. Yet, these techniques not only require the layer-by-layer model structures, but also need significant efforts in runtime profiling and hyperparameter tuning. To make the overlapping optimization <italic>simple</italic> and <italic>generic</italic>, in this paper, we propose a new Parameter Server framework. Our solution <italic>decouples</italic> the dependency between push and pull operations, and allows workers to <italic>eagerly</italic> pull the global parameters. This way, both push and pull operations can be easily overlapped with computations. Besides, the overlapping manner offers a different way to address the straggler problem, where the stale updates greatly retard the training process. In the new framework, with adequate information available to workers, they can explicitly modulate the learning rates for their updates. Thus, the global parameters can be less compromised by stale updates. We implement a prototype system in PyTorch and demonstrate its effectiveness on both CPU/GPU clusters. Experimental results show that our prototype saves up to 54&#x0025; less time for each iteration and up to 37&#x0025; fewer iterations for model convergence, achieving up to 2.86&#x00D7; speedup over widely-used synchronization schemes.

Original languageEnglish
Pages (from-to)1-13
Number of pages13
JournalIEEE Transactions on Parallel and Distributed Systems
DOIs
Publication statusAccepted/In press - 2022
Externally publishedYes

Keywords

  • Computational efficiency
  • Computational modeling
  • Data models
  • Distributed training
  • Hardware
  • machine learning
  • parameter server
  • Servers
  • Synchronization
  • Training

Fingerprint

Dive into the research topics of 'PS+: A Simple Yet Effective Framework for Fast Training on Parameter Server'. Together they form a unique fingerprint.

Cite this