A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Filters
Densifying Assumed-sparse Tensors: Improving Memory Efficiency and MPI Collective Performance during Tensor Accumulation for Parallelized Training of Neural Machine Translation Models
[article]
2019
arXiv
pre-print
Preserved Fulltext
This paper describes modifications made to the Horovod MPI-based distributed training framework to reduce memory usage for transformer models by converting assumed-sparse tensors to dense tensors, and ...
Recent attempts to parallelize the official TensorFlow "Transformer" model across multiple nodes have hit roadblocks due to excessive memory use and resulting out of memory errors when performing MPI collectives ...
Acknowledgement The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources that have contributed to the research results reported ...
arXiv:1905.04035v1
fatcat:2hia5qr35zbw3n3rtnn7c4svsq
PyTorch Distributed: Experiences on Accelerating Data Parallel Training
[article]
2020
arXiv
pre-print
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2020; you can also visit the original URL.
The file type is application/pdf.
Recent advances in deep learning argue for the value of large datasets and large models, which necessitates the ability to scale out model training to more computational resources. ...
Data parallelism has emerged as a popular solution for distributed training thanks to its straightforward principle and broad applicability. ...
This design avoids densifying sparse tensors and communicating empty values, which is especially helpful for NLP models. ...
arXiv:2006.15704v1
fatcat:qwebz4p6hjfnfbm46bsdpflxhq
Area Chairs
unpublished
Preserved Fulltext
A copy of this work was available on the public web and has been preserved in the Wayback Machine. The capture dates from 2018; you can also visit the original URL.
The file type is application/pdf.
Foreword It is our great pleasure to welcome you to York for the 27th British Machine Vision Conference (BMVC). ...
York is a campus University, sited in parkland and famous for its lakes and waterfowl. The conference is sited on the Heslington West campus on the outskirts of the historic city of York. ...
We believe that our results demonstrate high level of localization accuracy for our system, which could be very effective in most cases when the GPS signal is lost, for both day and nighttime. ...
fatcat:qy4idlefnzf6xjwaxjjpd75tzy