3 Hits in 7.9 sec

Densifying Assumed-sparse Tensors: Improving Memory Efficiency and MPI Collective Performance during Tensor Accumulation for Parallelized Training of Neural Machine Translation Models [article]

Derya Cavdar, Valeriu Codreanu, Can Karakus, John A. Lockman III, Damian Podareanu, Vikram Saletore, Alexander Sergeev, Don D. Smith II, Victor Suthichai, Quy Ta, Srinivas Varadharajan, Lucas A. Wilson, Rengan Xu (+1 others)
2019 arXiv   pre-print
This paper describes modifications made to the Horovod MPI-based distributed training framework to reduce memory usage for transformer models by converting assumed-sparse tensors to dense tensors, and  ...  Recent attempts to parallelize the official TensorFlow "Transformer" model across multiple nodes have hit roadblocks due to excessive memory use and resulting out of memory errors when performing MPI collectives  ...  Acknowledgement The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources that have contributed to the research results reported  ... 
arXiv:1905.04035v1 fatcat:2hia5qr35zbw3n3rtnn7c4svsq

PyTorch Distributed: Experiences on Accelerating Data Parallel Training [article]

Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, Soumith Chintala
2020 arXiv   pre-print
Recent advances in deep learning argue for the value of large datasets and large models, which necessitates the ability to scale out model training to more computational resources.  ...  Data parallelism has emerged as a popular solution for distributed training thanks to its straightforward principle and broad applicability.  ...  This design avoids densifying sparse tensors and communicating empty values, which is especially helpful for NLP models.  ... 
arXiv:2006.15704v1 fatcat:qwebz4p6hjfnfbm46bsdpflxhq

Area Chairs

Adrian Clark, Alessio Bue, Ist Genova, Andrea Cavallaro, Queen Mary, Andrea Torsello, Andrew Fitzgibbon, Antonio Robles, Kelly Nicta -Canberra, Barbara Caputo, Bjorn Stenger, Edmond Boyer (+28 others)
Foreword It is our great pleasure to welcome you to York for the 27th British Machine Vision Conference (BMVC).  ...  York is a campus University, sited in parkland and famous for its lakes and waterfowl. The conference is sited on the Heslington West campus on the outskirts of the historic city of York.  ...  We believe that our results demonstrate high level of localization accuracy for our system, which could be very effective in most cases when the GPS signal is lost, for both day and nighttime.  ...