Filters








244 Hits in 1.8 sec

MLPerf Training Benchmark [article]

Peter Mattson, Christine Cheng, Cody Coleman, Greg Diamos, Paulius Micikevicius, David Patterson, Hanlin Tang, Gu-Yeon Wei, Peter Bailis, Victor Bittorf, David Brooks, Dehao Chen (+24 others)
2020 arXiv   pre-print
We therefore present MLPerf, an ML benchmark that overcomes these challenges.  ...  But ML training presents three unique benchmarking challenges absent from other domains: optimizations that improve training throughput can increase the time to solution, training is stochastic and time  ...  ACKNOWLEDGEMENTS In this section, we acknowledge all those who helped produce the first set of results or supported the overall benchmark development.  ... 
arXiv:1910.01500v3 fatcat:ciwfjyu3x5crrm2fy2275g3wiy

Developing a Recommendation Benchmark for MLPerf Training and Inference [article]

Carole-Jean Wu and Robin Burke and Ed H. Chi and Joseph Konstan and Julian McAuley and Yves Raimond and Hao Zhang
2020 arXiv   pre-print
Training andInference Suites.  ...  To advance the state of understanding and enable machine learning system development and optimization for the commerce domain, we aim to define an industry-relevant recommendation benchmark for the MLPerf  ...  THE MLPERF RECOMMENDATION BENCHMARK ADVISORY BOARD The advisory board was formed in October 2019.  ... 
arXiv:2003.07336v2 fatcat:lwsjokiqwvgatjv77e336woaby

Scale MLPerf-0.6 models on Google TPU-v3 Pods [article]

Sameer Kumar, Victor Bitorff, Dehao Chen, Chiachen Chou, Blake Hechtman, HyoukJoong Lee, Naveen Kumar, Peter Mattson, Shibo Wang, Tao Wang, Yuanzhong Xu, Zongwei Zhou
2019 arXiv   pre-print
The recent submission of Google TPU-v3 Pods to the industry wide MLPerf v0.6 training benchmark demonstrates the scalability of a suite of industry relevant ML models.  ...  MLPerf defines a suite of models, datasets and rules to follow when benchmarking to ensure results are comparable across hardware, frameworks and companies.  ...  Methods We present performance optimization techniques to optimize MLPerf 0.6 training time on TPU-v3 pods. We use [6, 7] for all the MLPerf 0.6 benchmarks.  ... 
arXiv:1909.09756v3 fatcat:gwovrhmjizghrnjwcw7vsyb23m

Demystifying the MLPerf Benchmark Suite [article]

Snehil Verma, Qinzhe Wu, Bagus Hanindhito, Gunjan Jha, Eugene B. John, Ramesh Radhakrishnan, Lizy K. John
2019 arXiv   pre-print
We find that application benchmarks such as MLPerf (although rich in kernels) exhibit different features compared to kernel benchmarks such as DeepBench.  ...  We present a study on its characteristics and how the MLPerf benchmarks differ from some of the previous deep learning benchmarks like DAWNBench and DeepBench.  ...  MLPerf initial release v0.5 consists of benchmarks only for training, with inference benchmarks expected shortly.  ... 
arXiv:1908.09207v1 fatcat:xcakw5urojhidpetjwdvwfs7sa

Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark

Cody Coleman, Matei Zaharia, Daniel Kang, Deepak Narayanan, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Chris Ré
2019 ACM SIGOPS Operating Systems Review  
We show similar findings with entries to the MLPERF v0.5 benchmark.  ...  To address this problem, we recently introduced DAWNBENCH, a benchmark competition focused on end-to-end training time to achieve near-state-of-the-art accuracy on an unseen dataset-a combined metric called  ...  v0.5 training benchmark.  ... 
doi:10.1145/3352020.3352024 fatcat:qqiqkgwgunbutj673q4dqxie6u

Exploring the limits of Concurrency in ML Training on Google TPUs [article]

Sameer Kumar and James Bradbury and Cliff Young and Yu Emma Wang and Anselm Levskaya and Blake Hechtman and Dehao Chen and HyoukJoong Lee and Mehmet Deveci and Naveen Kumar and Pankaj Kanwar and Shibo Wang and Skye Wanderman-Milne and Steve Lacy and Tao Wang and Tayo Oguntebi and Yazhou Zu and Yuanzhong Xu and Andy Swing
2021 arXiv   pre-print
We also present performance resultsfrom the recent Google submission to the MLPerf-v0.7 benchmark contest, achieving record training times from16 to 28 seconds in four MLPerf models on the Google TPU-v3  ...  Recent results in language understanding using neural networks have required training hardware of unprecedentedscale, with thousands of chips cooperating on a single training run.  ...  Like systems benchmark suites which have come before it, the MLPerf benchmark suite is pushing performance forward and our MLPerf v0.7 Training submission on Google TPU-v3 and TPU-v4 systems showcase the  ... 
arXiv:2011.03641v3 fatcat:d6kjkdjmrvdw7go2m4vbysde4e

AIBench Training: Balanced Industry-Standard AI Training Benchmarking [article]

Fei Tang, Wanling Gao, Jianfeng Zhan, Chuanxin Lan, Xu Wen, Lei Wang, Chunjie Luo, Jiahui Dai, Zheng Cao, Xingwang Xiong, Zihan Jiang, Tianshu Hao (+21 others)
2021 arXiv   pre-print
We contribute by far the most comprehensive AI training benchmark suite.  ...  The specification, source code, and performance numbers are available from the AIBench homepage https://www.benchcouncil.org/aibench-training/index.html.  ...  Totally, MLPerf Training V0.5 includes seven benchmarks.  ... 
arXiv:2004.14690v4 fatcat:34dn54tmjbhuhfcefsttf62ceu

Analysis of DAWNBench, a Time-to-Accuracy Machine Learning Performance Benchmark [article]

Cody Coleman, Daniel Kang, Deepak Narayanan, Luigi Nardi, Tian Zhao, Jian Zhang, Peter Bailis, Kunle Olukotun, Chris Re, Matei Zaharia
2019 arXiv   pre-print
We show similar findings with entries to the MLPERF v0.5 benchmark.  ...  To address this problem, we recently introduced DAWNBench, a benchmark competition focused on end-to-end training time to achieve near-state-of-the-art accuracy on an unseen dataset---a combined metric  ...  v0.5 training benchmark.  ... 
arXiv:1806.01427v2 fatcat:oaxei3hhufa2dkqgqhs2jftmmu

Performance Characteristics of Virtualized GPUs for Deep Learning

Scott Michael, Scott Teige, Junjie Li, John Michael Lowe, George Turner, Robert Henschel
2020 2020 IEEE/ACM International Workshop on Interoperability of Supercomputing and Cloud Technologies (SuperCompCloud)  
In this paper, we investigate the performance characteristics of vGPUs for both traditional HPC workloads and for deep learning training and inference workloads.  ...  Using NVIDIA's vDWS virtualization software, we perform a series of HPC and deep learning benchmarks on both non-virtualized (bare metal) and vGPUs of various sizes and configurations.  ...  Training v0.5 benchmark.  ... 
doi:10.1109/supercompcloud51944.2020.00008 fatcat:ndq5hcwczfb5rn7hvza27764l4

MLPerf-Bench presentation at HPCA'22: "MLPerf design space exploration and production deployment"

Grigori Fursin
2022 Zenodo  
Invited talk at MLPerf-Bench @ HPCA'22 tutorial. Related resources: Collective Mind framework (CM or CK2) CK-powered MLPerf inference automation MLCommons OctoML  ...  / mlperf-bert / scripts to run MLPerf BERT benchmark / project root / package / mlperf-loadgen / scripts to download and install MLPerf benchmark / dataset-imagenet / scripts to download ImageNet / project  ...  as a database of reusable artifacts and automations / project root / program / mlperf-image-classification / scripts to run MLPerf IC benchmark / mlperf-bert / scripts to run MLPerf BERT benchmark / project  ... 
doi:10.5281/zenodo.6475385 fatcat:d5ocp45qhnc3bfp4wn2j5mkeda

Benchmarking Contemporary Deep Learning Hardware and Frameworks:A Survey of Qualitative Metrics [article]

Wei Dai, Daniel Berleant
2019 arXiv   pre-print
Because MLPerf is a benchmark organization working with industry and academia, and offering deep learning benchmarks that evaluate training and inference on deep learning hardware devices, the survey also  ...  mentions MLPerf benchmark results, benchmark metrics, datasets, deep learning frameworks and algorithms.  ...  Currently MLPerf members already have submitted the MLPerf Training Results v0.5 and MLPerf Training Results v0.6, and the deep learning reference results v0.5 will be released soon.  ... 
arXiv:1907.03626v4 fatcat:cabjqm655bcw5euk2hessp4axy

Benchmarking Contemporary Deep Learning Hardware and Frameworks: A Survey of Qualitative Metrics

Wei Dai, Daniel Berleant
2019 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI)  
Because MLPerf is a benchmark organization working with industry and academia, and offering deep learning benchmarks that evaluate training and inference on deep learning hardware devices, the survey also  ...  mentions MLPerf benchmark results, benchmark metrics, datasets, deep learning frameworks and algorithms.  ...  Currently MLPerf members already have submitted the MLPerf Training Results v0.5 and MLPerf Training Results v0.6, and the deep learning reference results v0.5 will be released soon.  ... 
doi:10.1109/cogmi48466.2019.00029 dblp:conf/cogmi/DaiB19 fatcat:3tziv3stuvddnetxc6ha66bcgq

MLPerf HPC: A Holistic Benchmark Suite for Scientific Machine Learning on HPC Systems [article]

Steven Farrell, Murali Emani, Jacob Balma, Lukas Drescher, Aleksandr Drozd, Andreas Fink, Geoffrey Fox, David Kanter, Thorsten Kurth, Peter Mattson, Dawei Mu, Amit Ruhela (+31 others)
2021 arXiv   pre-print
In this paper, we introduce MLPerf HPC, a benchmark suite of large-scale scientific machine learning training applications driven by the MLCommons Association.  ...  MLPerf is a community-driven standard to benchmark machine learning workloads, focusing on end-to-end performance metrics.  ...  MLPerf Training benchmarks [17] aim to measure the performance of training models while MLPerf Inference benchmarks [18] aim to measure how fast systems can produce results using a trained model.  ... 
arXiv:2110.11466v2 fatcat:qb6qfyklefb4bcufuj3eozjzcm

MLPerf Tiny Benchmark [article]

Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Jeremy Holleman, Nat Jeffries, Csaba Kiraly, Pietro Montino, David Kanter, Sebastian Ahmed, Danilo Pau, Urmish Thakker, Antonio Torrini (+10 others)
2021 arXiv   pre-print
To meet this need, we present MLPerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems.  ...  Additionally, MLPerf Tiny implements a modular design that enables benchmark submitters to show the benefits of their product, regardless of where it falls on the ML deployment stack, in a fair and reproducible  ...  Additionally, each benchmark has a reference implementation that includes training scripts, pre-trained models, and C code implementations.  ... 
arXiv:2106.07597v4 fatcat:ps4y36uq4nevxfbe7p3tne4opu

MLPerf Inference Benchmark [article]

Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman (+35 others)
2020 arXiv   pre-print
There is a clear need for industry-wide standard ML benchmarking and evaluation criteria. MLPerf Inference answers that call.  ...  In this paper, we present our benchmarking method for evaluating ML inference systems.  ...  ACKNOWLEDGEMENTS MLPerf Inference is the work of many individuals from multiple organizations.  ... 
arXiv:1911.02549v2 fatcat:jewandlmivctjb7wywadxuuqju
« Previous Showing results 1 — 15 out of 244 results