Filters








52 Hits in 6.3 sec

MLPerf Mobile Inference Benchmark [article]

Vijay Janapa Reddi, David Kanter, Peter Mattson, Jared Duke, Thai Nguyen, Ramesh Chukka, Ken Shiring, Koan-Sin Tan, Mark Charlebois, William Chou, Mostafa El-Khamy, Jungwook Hong (+15 others)
2022 arXiv   pre-print
This paper presents the first industry-standard open-source machine learning (ML) benchmark to allow perfor mance and accuracy evaluation of mobile devices with different AI chips and software stacks.  ...  MLPerf Mobile will evolve and serve as an open-source community framework to guide research and innovation for mobile AI.  ...  MLPerf Inference is another industry-standard open benchmark (Reddi et al., 2020) .  ... 
arXiv:2012.02328v4 fatcat:uu7p5nk2pzhepnayhwwtspklwa

MLPerf Inference Benchmark [article]

Vijay Janapa Reddi, Christine Cheng, David Kanter, Peter Mattson, Guenther Schmuelling, Carole-Jean Wu, Brian Anderson, Maximilien Breughe, Mark Charlebois, William Chou, Ramesh Chukka, Cody Coleman (+35 others)
2020 arXiv   pre-print
There is a clear need for industry-wide standard ML benchmarking and evaluation criteria. MLPerf Inference answers that call.  ...  In this paper, we present our benchmarking method for evaluating ML inference systems.  ...  ACKNOWLEDGEMENTS MLPerf Inference is the work of many individuals from multiple organizations.  ... 
arXiv:1911.02549v2 fatcat:jewandlmivctjb7wywadxuuqju

MLPerf Tiny Benchmark [article]

Colby Banbury, Vijay Janapa Reddi, Peter Torelli, Jeremy Holleman, Nat Jeffries, Csaba Kiraly, Pietro Montino, David Kanter, Sebastian Ahmed, Danilo Pau, Urmish Thakker, Antonio Torrini (+10 others)
2021 arXiv   pre-print
To meet this need, we present MLPerf Tiny, the first industry-standard benchmark suite for ultra-low-power tiny machine learning systems.  ...  MLPerf Tiny measures the accuracy, latency, and energy of machine learning inference to properly evaluate the tradeoffs between systems.  ...  In this paper, we present MLPerf Tiny, an open-source benchmark suite for TinyML systems.  ... 
arXiv:2106.07597v4 fatcat:ps4y36uq4nevxfbe7p3tne4opu

Benchmarking Contemporary Deep Learning Hardware and Frameworks:A Survey of Qualitative Metrics [article]

Wei Dai, Daniel Berleant
2019 arXiv   pre-print
Because MLPerf is a benchmark organization working with industry and academia, and offering deep learning benchmarks that evaluate training and inference on deep learning hardware devices, the survey also  ...  This paper surveys benchmarking principles, machine learning devices including GPUs, FPGAs, and ASICs, and deep learning software frameworks.  ...  A MACHINE LEARNING BENCHMARK ORGANIZATION MLPerf is a machine learning benchmark organization that offers useful benchmarks that evaluate training and inference on deep learning hardware devices.  ... 
arXiv:1907.03626v4 fatcat:cabjqm655bcw5euk2hessp4axy

Benchmarking Contemporary Deep Learning Hardware and Frameworks: A Survey of Qualitative Metrics

Wei Dai, Daniel Berleant
2019 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI)  
Because MLPerf is a benchmark organization working with industry and academia, and offering deep learning benchmarks that evaluate training and inference on deep learning hardware devices, the survey also  ...  This paper surveys benchmarking principles, machine learning devices including GPUs, FPGAs, and ASICs, and deep learning software frameworks.  ...  A MACHINE LEARNING BENCHMARK ORGANIZATION MLPerf is a machine learning benchmark organization that offers useful benchmarks that evaluate training and inference on deep learning hardware devices.  ... 
doi:10.1109/cogmi48466.2019.00029 dblp:conf/cogmi/DaiB19 fatcat:3tziv3stuvddnetxc6ha66bcgq

Benchmarking TinyML Systems: Challenges and Direction [article]

Colby R. Banbury, Vijay Janapa Reddi, Max Lam, William Fu, Amin Fazel, Jeremy Holleman, Xinyuan Huang, Robert Hurtado, David Kanter, Anton Lokhmotov, David Patterson, Danilo Pau (+5 others)
2021 arXiv   pre-print
Recent advancements in ultra-low-power machine learning (TinyML) hardware promises to unlock an entirely new class of smart applications.  ...  However, continued progress is limited by the lack of a widely accepted benchmark for these systems.  ...  INTRODUCTION Machine learning (ML) inference on the edge is an increasingly attractive prospect due to its potential for increasing energy efficiency (Fedorov et al., 2019) , privacy, responsiveness  ... 
arXiv:2003.04821v4 fatcat:rodnh7fd6fa57low2j4jcuqbc4

Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices [article]

Chunjie Luo, Xiwen He, Jianfeng Zhan, Lei Wang, Wanling Gao, Jiahui Dai
2020 arXiv   pre-print
The application of deep learning on the mobile and embedded devices is taken more and more attentions, benchmarking and ranking the AI abilities of mobile and embedded devices becomes an urgent problem  ...  Considering the model diversity and framework diversity, we propose a benchmark suite, AIoTBench, which focuses on the evaluation of the inference abilities of mobile and embedded devices.  ...  Fig. 3 : 3 The typical modules of different models 46], primarily developed by Facebook's AI Research lab (FAIR), is an open-source machine learning library based on the Torch library.  ... 
arXiv:2005.05085v1 fatcat:72p3whrkfvhvzpl6ih7pfc2bi4

FLBench: A Benchmark Suite for Federated Learning [article]

Yuan Liang, Yange Guo, Yanxia Gong, Chunjie Luo, Jianfeng Zhan, Yunyou Huang
2021 arXiv   pre-print
The goal is to build a machine learning model from the data sets distributed on multiple devices so-called an isolated data island, while keeping their data secure and private.  ...  Hence, it becomes a promising platform for developing novel federated learning algorithms. Currently, FLBench is open sourced and in fast evolution. We package it as an automated deployment tool.  ...  These benchmarks have provided various metrics and results for machine learning training and inference.  ... 
arXiv:2008.07257v3 fatcat:2fxajr5s6rguddldvbgu66jub4

A Survey on Edge Performance Benchmarking [article]

Blesson Varghese and Nan Wang and David Bermbach and Cheol-Ho Hong and Eyal de Lara and Weisong Shi and Christopher Stewart
2020 arXiv   pre-print
In this context, the performance characteristics of such systems will need to be captured rapidly, which is referred to as performance benchmarking, for application deployment, resource orchestration,  ...  It then systematically classifies previous research to identify the system under test, techniques analyzed, and benchmark runtime in edge performance benchmarking.  ...  Devices that run machine learning workloads: The benchmarking of machine-learning-specific workloads for various devices was presented in (65) .  ... 
arXiv:2004.11725v2 fatcat:gyqqgfqf5fe2jk2ntlnd2itmli

A Survey on Edge Performance Benchmarking

Blesson Varghese, Nan Wang, David Bermbach, Cheol-Ho Hong, Eyal De Lara, Weisong Shi, Christopher Stewart
2021 ACM Computing Surveys  
In this context, the performance characteristics of such systems will need to be captured rapidly, which is referred to as performance benchmarking, for application deployment, resource orchestration,  ...  It then systematically classifies previous research to identify the system under test, techniques analyzed, and benchmark runtime in edge performance benchmarking.  ...  Devices that run machine learning workloads: The benchmarking of machine-learning-specific workloads for various devices was presented in (65) .  ... 
doi:10.1145/3444692 fatcat:75kmnweazzeppefekfyrxmemze

FedScale: Benchmarking Model and System Performance of Federated Learning at Scale [article]

Fan Lai, Yinwei Dai, Sanjay S. Singapuram, Jiachen Liu, Xiangfeng Zhu, Harsha V. Madhyastha, Mosharaf Chowdhury
2022 arXiv   pre-print
We present FedScale, a federated learning (FL) benchmarking suite with realistic datasets and a scalable runtime to enable reproducible FL research.  ...  FedScale is open-source and actively maintained by contributors from different institutions at http://fedscale.ai. We welcome feedback and contributions from the community.  ...  We also thank FedScale contributors and users from many different academic institutions and industry for their valuable inputs.  ... 
arXiv:2105.11367v5 fatcat:6xfm4h37znecbkrqijjs3lo4um

QuTiBench: Benchmarking Neural Networks on Heterogeneous Hardware [article]

Michaela Blott, Lisa Halder, Miriam Leeser, Linda Doyle
2019 arXiv   pre-print
Neural Networks have become one of the most successful universal machine learning algorithms. They play a key role in enabling machine vision and speech recognition for example.  ...  None of the existing benchmarks support essential algorithmic optimizations such as quantization, an important technique to stay on chip, or specialized heterogeneous hardware architectures.  ...  Ce Zhang at ETH Zurich, and the Deephi team for insights and support. Miriam Leeser is supported in part by the National Science Foundation under Grant No. 1717213.  ... 
arXiv:1909.05009v2 fatcat:hkd53al5k5ecbinf7pqbw4afcq

MARVEL - D5.2: Technical evaluation and progress against benchmarks – initial version

Toni Heittola, Tuomas Virtanen
2022 Zenodo  
The benchmarking strategy was defined in WP1, and this document describes how the benchmarking is implemented for the components in the Minimum Viable Product (MVP) of the MARVEL project.  ...  The purpose of this deliverable is to describe in detail the technical evaluation and progress against benchmarks.  ...  • Keycloack, an open-source software product that provides single sign-on to applications and services.  ... 
doi:10.5281/zenodo.6322699 fatcat:d5lpwby5szg4fih77sp5rjoyae

Collective Knowledge: organizing research projects as a database of reusable components and portable workflows with common APIs

Grigori Fursin
2020 figshare.com  
and hardware for ML and AI in terms of speed, accuracy, energy, size, and various costs.  ...  of missing packages.This article presents several industrial projects where the modular CK approach was successfully validated to automate benchmarking, auto-tuning, and co-design of efficient software  ...  Machine Learning benchmarking initiatives such as MLPerf [52] , MLModelScope [38] , and Deep500 [36] attempt to standardize machine learning model benchmarking and make it more reproducible.  ... 
doi:10.6084/m9.figshare.12988361.v1 fatcat:njmhurelvjcc7pv44fi2r2bp4y

Collective Knowledge: organizing research projects as a database of reusable components and portable workflows with common APIs

Grigori Fursin
2020 figshare.com  
and hardware for ML and AI in terms of speed, accuracy, energy, size, and various costs.  ...  of missing packages.This article presents several industrial projects where the modular CK approach was successfully validated to automate benchmarking, auto-tuning, and co-design of efficient software  ...  Machine Learning benchmarking initiatives such as MLPerf [52] , MLModelScope [38] , and Deep500 [36] attempt to standardize machine learning model benchmarking and make it more reproducible.  ... 
doi:10.6084/m9.figshare.12988361.v3 fatcat:2rp433zuzvchpcgblbigk4jdvq
« Previous Showing results 1 — 15 out of 52 results