45,788 Hits in 5.5 sec

Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training [article]

Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr, Martin Jaggi
2021 arXiv   pre-print
We study two factors in neural network training: data parallelism and sparsity; here, data parallelism means processing training data in parallel using distributed systems (or equivalently increasing batch  ...  parallelism and sparsity on neural network training.  ...  of the effects of data parallelism on sparse neural network training.  ... 
arXiv:2003.11316v3 fatcat:6sf52pdz5zbj5l7xongmc52lqy

Truly Sparse Neural Networks at Scale [article]

Selima Curci, Decebal Constantin Mocanu, Mykola Pechenizkiyi
2022 arXiv   pre-print
To achieve this goal, we introduce three novel contributions, specially designed for sparse neural networks: (1) a parallel training algorithm and its corresponding sparse implementation from scratch,  ...  Recently, sparse training methods have started to be established as a de facto approach for training and inference efficiency in artificial neural networks. Yet, this efficiency is just in theory.  ...  Acknowledgement We thank the Google Cloud Platform Research Credits program for granting us the necessary resources to run the Extreme large sparse MLPs experiments.  ... 
arXiv:2102.01732v2 fatcat:xw4pnoj5zfafvilmk34odczt5m

Training Large Scale Deep Neural Networks on the Intel Xeon Phi Many-Core Coprocessor

Lei Jin, Zhaokang Wang, Rong Gu, Chunfeng Yuan, Yihua Huang
2014 2014 IEEE International Parallel & Distributed Processing Symposium Workshops  
In this paper, we propose a many-core algorithm which is based on a parallel method and is used in the Intel Xeon Phi many-core systems to speed up the unsupervised training process of Sparse Autoencoder  ...  However, it will face problems when being applied to deal with large scale data due to its intensive computation from many levels of training process against large scale data.  ...  ACKNOWLEDGMENT This work is funded in part by China NSF Grants (No. 61223003), and the USA Intel Labs University Resea rch Program.  ... 
doi:10.1109/ipdpsw.2014.194 dblp:conf/ipps/JinWGYH14 fatcat:tpmcupt4indklfoj3dtwm65rze

TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learning [article]

Riyadh Baghdadi, Abdelkader Nadir Debbagh, Kamel Abdous, Fatima Zohra Benhamida, Alex Renda, Jonathan Elliott Frankle, Michael Carbin, Saman Amarasinghe
2020 arXiv   pre-print
In this paper, we demonstrate a compiler that can optimize sparse and recurrent neural networks, both of which are currently outside of the scope of existing neural network compilers (sparse neural networks  ...  Our demonstration includes a mapping of sparse and recurrent neural networks to the polyhedral model along with an implementation of our approach in TIRAMISU, our state-of-the-art polyhedral compiler.  ...  and sparse neural networks.  ... 
arXiv:2005.04091v1 fatcat:zqeblrvhqjh6xjy6i6nquualza

Sparsely-Connected Neural Networks: Towards Efficient VLSI Implementation of Deep Neural Networks [article]

Arash Ardakani, Carlo Condo, Warren J. Gross
2017 arXiv   pre-print
Recently deep neural networks have received considerable attention due to their ability to extract and represent high-level abstractions in data sets.  ...  In fact, they contain most of the deep neural network parameters.  ...  INTRODUCTION Deep neural networks (DNNs) have shown remarkable performance in extracting and representing high-level abstractions in complex data (Lecun et al. (2015) ).  ... 
arXiv:1611.01427v3 fatcat:upamabr47jgwxapurtpf4xokqa

Parallel Cross-Sparse Filtering Networks and Its Application on Fault Diagnosis of Rotating Machinery

Shan Wang, Baokun Han, Huaiqian Bao, Jinrui Wang, Zongzhen Zhang, Haidong Shao
2022 Journal of Sensors  
this paper proposed a parallel network based on Cr-SF.  ...  In view of the feature extraction advantages of cross-sparse filtering (Cr-SF), which can be regarded as an unsupervised minimum entropy learning method using the maximization of the proxy of sparsity,  ...  As shown in Figure 1 , cross sparse filtering is the variant of SF, Which can be regarded as a twolayer neural network.  ... 
doi:10.1155/2022/9259639 fatcat:zzjni2jd7vc3rl7qstpzv3pg6m

Combining Deep Learning and Adaptive Sparse Modeling for Low-dose CT Reconstruction [article]

Ling Chen, Zhishen Huang, Yong Long, Saiprasad Ravishankar
2022 arXiv   pre-print
Each proposed trained block consists of a deterministic MBIR solver and a neural network.  ...  In our experiments, we study combinations of supervised deep network reconstructors and sparse representations-based (unsupervised) learned or analytical priors.  ...  Cynthia McCollough, the Mayo Clinic, the American Association of Physicists in Medicine, and the National Institute of Biomedical Imaging and Bioengineering for providing the Mayo Clinic data.  ... 
arXiv:2205.09587v1 fatcat:cys3c3rmh5aqdkcgzmgwc7yq64

Certifai: A Toolkit for Building Trust in AI Systems

Jette Henderson, Shubham Sharma, Alan Gee, Valeri Alexiev, Steve Draper, Carlos Marin, Yessel Hinojosa, Christine Draper, Michael Perng, Luis Aguirre, Michael Li, Sara Rouhani (+13 others)
2020 Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence  
At its foundation, Cortex Certifai generates counterfactual explanations, which are synthetic data points close to input data points but differing in terms of model prediction.  ...  Cortex Certifai is a framework that assesses aspects of robustness, fairness, and interpretability of any classification or regression model trained on tabular data, without requiring access to its internal  ...  Besides, due to the lack of hardware and libraries, the training efficiency provided by unstructured sparse neural networks can not be mapped to the parallel processors.  ... 
doi:10.24963/ijcai.2020/735 dblp:conf/ijcai/Liu20a fatcat:p746gszfqbdwhgdhr4b4wye2zu

Guest Editors' Introduction to the Special Issue on Machine Learning Architectures and Accelerators

Xuehai Qian, Yanzhi Wang, Avinash Karanth
2020 IEEE transactions on computers  
.), compiler-assisted optimizations, parallel computing techniques such as data parallelism and model parallelism, distributed training algorithms, federated learning, to name a few.  ...  Various forms of DNNs have been proposed, including Convolutional Neural Networks, Recurrent Neural Networks, Deep Reinforcement Learning, Transformer model, etc.  ...  Further on, it is our pleasure to thank the Editor-in-Chief Ahmed Louri and Associate Editors Tao Li and James Hoe for their continuous help and support with all our organizational questions in connection  ... 
doi:10.1109/tc.2020.2997574 fatcat:vfng262tlvagrmfudtv44x75ly

Scalable and Sustainable Deep Learning via Randomized Hashing [article]

Ryan Spring, Anshumali Shrivastava
2016 arXiv   pre-print
Due to the sparse gradient updates, our algorithm is ideally suited for asynchronous and parallel training leading to near linear speedup with increasing number of cores.  ...  We present a novel hashing based technique to drastically reduce the amount of computation needed to train and test deep networks.  ...  This work has the users train neural networks on their local data, and then periodically transmit their models to a central server.  ... 
arXiv:1602.08194v2 fatcat:fo2pjpzsivgzxnmxurmlalfvda

Editorial introduction: special issue on advances in parallel and distributed computing for neural computing

Jianguo Chen, Ahmad Salah
2020 Neural computing & applications (Print)  
In scientific research and practical applications, clusters of computers and accelerators (e.g., GPUs) are routinely used to train and run various neural network models.  ...  The papers in the special issue represent a broad spectrum of parallel and distributed computing, machine learning models, and neural network models.  ...  The paper by Yuedan Chen et al. focused on the general sparse matrix-sparse matrix (SpGEMM)-the basic kernels in many machine learning and neural computing, and designed a partitioning and parallelization  ... 
doi:10.1007/s00521-020-04887-7 fatcat:aydztes3cjfrffsw7ext7g6c4i

Memory-Efficient Deep Learning on a SpiNNaker 2 Prototype

Chen Liu, Guillaume Bellec, Bernhard Vogginger, David Kappel, Johannes Partzsch, Felix Neumärker, Sebastian Höppner, Wolfgang Maass, Steve B. Furber, Robert Legenstein, Christian G. Mayr
2018 Frontiers in Neuroscience  
Deep Rewiring (DEEP R) is a training algorithm which continuously rewires the network while preserving very sparse connectivity all along the training procedure.  ...  Yet, these techniques are not applicable to the case when neural networks have to be trained directly on hardware due to the hard memory constraints.  ...  This result indicates that training of sparse neural networks with DEEP R can be highly parallelized on the SpiNNaker 2 system, with good speedup and per-core memory reduction.  ... 
doi:10.3389/fnins.2018.00840 pmid:30505263 pmcid:PMC6250847 fatcat:exackg5sbrdstnjqohhnagxmlm

Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks [article]

Soojeong Kim, Gyeong-In Yu, Hojin Park, Sungwoo Cho, Eunji Jeong, Hyeonmin Ha, Sanha Lee, Joo Seong Jeong, Byung-Gon Chun
2019 arXiv   pre-print
In this paper, we propose Parallax, a framework that optimizes data parallel training by utilizing the sparsity of model parameters.  ...  The employment of high-performance servers and GPU accelerators for training deep neural network models have greatly accelerated recent advances in deep learning (DL).  ...  Data Parallel Distributed Training A DL model refers to a neural network architecture, which is trained via gradient descent; the loss value of the model is calculated from forward computations, and the  ... 
arXiv:1808.02621v3 fatcat:flymv2t6lnh23pkzivelnjby44

DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs [article]

Da Zheng, Chao Ma, Minjie Wang, Jinjing Zhou, Qidong Su, Xiang Song, Quan Gan, Zheng Zhang, George Karypis
2021 arXiv   pre-print
Graph neural networks (GNN) have shown great success in learning from graph-structured data. They are widely used in various applications, such as recommendation, fraud detection, and search.  ...  The combination of these design choices allows DistDGL to train high-quality models while achieving high parallel efficiency and memory scalability.  ...  However, GNN mini-batch training is different from other neural networks due to the data dependency between vertices.  ... 
arXiv:2010.05337v3 fatcat:6xvbvgvyczgsdhyfwolyhpmvwi

Design and Implementation of Blind Source Separation Based on BP Neural Network in Space-Based AIS

Chengjie Li, Lidong Zhu, Zhongqiang Luo, Zhen Zhang, Yilun Liu, Ying Yang
2021 Frontiers in Space Technologies  
the problem of blind source separation with BP neural network, using the generated suitable data set to train the neural network, thereby automatically generating a traditional blind signal separation  ...  In this paper, to improve system processing power and security, according to the characteristics of neural network that can efficiently find the optimal solution of a problem, proposes a method that combines  ...  Then the trained BP neural network system can simulate the relationship between the initial data the expected data.  ... 
doi:10.3389/frspt.2021.756478 fatcat:52m25q5inrds7prutanwuneniu
« Previous Showing results 1 — 15 out of 45,788 results