3 Hits in 2.9 sec

DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution [article]

Keshav Santhanam, Siddharth Krishna, Ryota Tomioka, Tim Harris, Matei Zaharia
2021 arXiv   pre-print
In this work we propose DistIR, an expressive intermediate representation for distributed DNN computation that is tailored for efficient analyses, such as simulation.  ...  optimization time by an order of magnitude for certain regimes.  ...  DistIR is an intermediate representation (IR) for distributed computation based on the static single assignment (SSA) form.  ... 
arXiv:2111.05426v1 fatcat:5zzv4vmtgvhc3bi2ptp5ntyrsq

Automap: Towards Ergonomic Automated Parallelism for ML Models [article]

Michael Schaarschmidt and Dominik Grewe and Dimitrios Vytiniotis and Adam Paszke and Georg Stefan Schmid and Tamara Norman and James Molloy and Jonathan Godwin and Norman Alexander Rink and Vinod Nair and Dan Belov
2021 arXiv   pre-print
The rapid rise in demand for training large neural network architectures has brought into focus the need for partitioning strategies, for example by using data, model, or pipeline parallelism.  ...  We present the prototype of an automated partitioner that seamlessly integrates into existing compilers and existing user workflows.  ...  Distir: An intermediate representation for optimizing distributed neural networks.  ... 
arXiv:2112.02958v1 fatcat:tlda37oxgjeezggohojvh4sdni

GSPMD: General and Scalable Parallelization for ML Computation Graphs [article]

Yuanzhong Xu, HyoukJoong Lee, Dehao Chen, Blake Hechtman, Yanping Huang, Rahul Joshi, Maxim Krikun, Dmitry Lepikhin, Andy Ly, Marcello Maggioni, Ruoming Pang, Noam Shazeer (+4 others)
2021 arXiv   pre-print
We present GSPMD, an automatic, compiler-based parallelization system for common machine learning computations.  ...  It allows users to write programs in the same way as for a single device, then give hints through a few annotations on how to distribute tensors, based on which GSPMD will parallelize the computation.  ...  We are also grateful to Mike Burrows, David Majnemer, and Naveen Kumar for their valuable and constructive feedback on earlier drafts of the paper.  ... 
arXiv:2105.04663v2 fatcat:ex4uk6t3orewnkgay4p76kdeka