Filters








8,256 Hits in 6.2 sec

Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training [article]

Namhoon Lee, Thalaiyasingam Ajanthan, Philip H. S. Torr, Martin Jaggi
2021 arXiv   pre-print
and generally, establishing a better account of the effects of data parallelism and sparsity on neural network training.  ...  for the effect of data parallelism, and further, difficulty of training under sparsity.  ...  effects of data parallelism and sparsity on neural network training to this end.  ... 
arXiv:2003.11316v3 fatcat:6sf52pdz5zbj5l7xongmc52lqy

Certifai: A Toolkit for Building Trust in AI Systems

Jette Henderson, Shubham Sharma, Alan Gee, Valeri Alexiev, Steve Draper, Carlos Marin, Yessel Hinojosa, Christine Draper, Michael Perng, Luis Aguirre, Michael Li, Sara Rouhani (+13 others)
2020 Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence  
Cortex Certifai is a framework that assesses aspects of robustness, fairness, and interpretability of any classification or regression model trained on tabular data, without requiring access to its internal  ...  Cortex Certifai can be configured and executed using a command-line interface (CLI), within jupyter notebooks, or on the cloud, and the results are recorded in JSON files and can be visualized in an interactive  ...  Robert Peharz for helpful discussions and feedback on drafts of this paper.  ... 
doi:10.24963/ijcai.2020/735 dblp:conf/ijcai/Liu20a fatcat:p746gszfqbdwhgdhr4b4wye2zu

Accelerating Training of Deep Neural Networks via Sparse Edge Processing [chapter]

Sourya Dey, Yinan Shao, Keith M. Chugg, Peter A. Beerel
2017 Lecture Notes in Computer Science  
We propose a reconfigurable hardware architecture for deep neural networks (DNNs) capable of online training and inference, which uses algorithmically pre-determined, structured sparsity to significantly  ...  The overall effect is to reduce network complexity by factors up to 30x and training time by up to 35x relative to GPUs, while maintaining high fidelity of inference results.  ...  Conclusion and Future Work This work presents a flexible architecture that can perform both training and inference of large and deep neural networks on hardware.  ... 
doi:10.1007/978-3-319-68600-4_32 fatcat:skwynctmxveepp3jrhteb2hm5u

The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation [article]

Orevaoghene Ahia, Julia Kreutzer, Sara Hooker
2021 arXiv   pre-print
A "bigger is better" explosion in the number of parameters in deep neural networks has made it increasingly challenging to make state-of-the-art networks accessible in compute-restricted environments.  ...  We introduce the term low-resource double bind to refer to the co-occurrence of data limitations and compute resource constraints.  ...  In particular, we want to control the effects of (1) network sparsity, (2) training data size, (3) target language, and (4) domain shifts.  ... 
arXiv:2110.03036v1 fatcat:z4o4boau25cwrbtxqyhjlevvgy

Algorithm to Compilation Co-design: An Integrated View of Neural Network Sparsity [article]

Fu-Ming Guo, Austin Huang
2021 arXiv   pre-print
However, operationalizing those benefits and understanding the end-to-end effect of algorithm design and regularization on the runtime execution is not often examined in depth.  ...  Reducing computation cost, inference latency, and memory footprint of neural networks are frequently cited as research motivations for pruning and sparsity.  ...  Acknowledgments and Disclosure of Funding Both authors are employeed by Fidelity Investments personal investing. They have no conflicts of interest to disclose.  ... 
arXiv:2106.08846v2 fatcat:bfx3lvvpzvbbffzeqic2g4ffye

Sparse and dense matrix multiplication hardware for heterogeneous multi-precision neural networks

Jose Nunez-Yanez, Mohammad Hosseinabady
2021 Array  
We initially investigate the effects of quantization and sparsity on the accuracy of neural networks with convolution, dense and recurrent layers observing better tolerance to pruning when recurrent layers  ...  The methodology involves quantization-sparsity aware training and it is applied to a case study consisting of human activity classification.  ...  Acknowledgements This research was funded by the Royal Society Industry fellowship, INF\R2\192044 Machine Intelligence at the Network Edge (MINET), EPSRC HOPWARE EP\RV040863\1, EPSRC ENEAC EP\N002539\1  ... 
doi:10.1016/j.array.2021.100101 fatcat:nr32njqu4bawdc2dihiqi6mv2e

Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks

Jack Turner, Jose Cano, Valentin Radu, Elliot J. Crowley, Michael OrBoyle, Amos Storkey
2018 2018 IEEE International Symposium on Workload Characterization (IISWC)  
In this paper we unify the two viewpoints in a Deep Learning Inference Stack and take an across-stack approach by implementing and evaluating the most common neural network compression techniques (weight  ...  Convolutional Neural Networks (CNNs) are extremely computationally demanding, presenting a large barrier to their deployment on resource-constrained devices.  ...  The opinions expressed and arguments employed herein do not necessarily reflect the official views of these funding bodies.  ... 
doi:10.1109/iiswc.2018.8573503 dblp:conf/iiswc/TurnerCRCOS18 fatcat:hxxhuovm6fhyhheg55vtwyvsoi

Characterising Across-Stack Optimisations for Deep Convolutional Neural Networks [article]

Jack Turner, José Cano, Valentin Radu, Elliot J. Crowley, Michael O'Boyle, Amos Storkey
2018 arXiv   pre-print
In this paper we unify the two viewpoints in a Deep Learning Inference Stack and take an across-stack approach by implementing and evaluating the most common neural network compression techniques (weight  ...  Convolutional Neural Networks (CNNs) are extremely computationally demanding, presenting a large barrier to their deployment on resource-constrained devices.  ...  The opinions expressed and arguments employed herein do not necessarily reflect the official views of these funding bodies.  ... 
arXiv:1809.07196v1 fatcat:wxevr5hprveiro5lg2aie5nnem

Exposing Hardware Building Blocks to Machine Learning Frameworks [article]

Yash Akhauri
2020 arXiv   pre-print
Further, we develop a library that supports training a neural network with custom sparsity and quantization.  ...  We focus on how to design topologies that complement such a view of neurons, how to automate such a strategy of neural network design, and inference of such networks on Xilinx FPGAs.  ...  IMPLEMENTING NEURAL NETWORKS ON FPGAS To aid our understanding of how a neural network is optimized for inference on an FPGA, we glance at the HLS-RFNoC workflow.  ... 
arXiv:2004.05898v1 fatcat:g5a5fly4szfkdlw5kppysx2kia

AR-Net: A simple Auto-Regressive Neural Network for time-series [article]

Oskar Triebe, Nikolay Laptev, Ram Rajagopal
2019 arXiv   pre-print
In this paper we present a new framework for time-series modeling that combines the best of traditional statistical models and neural networks.  ...  This eliminates the need to know the exact order of the AR-process and allows to learn sparse weights for a model with long-range dependencies.  ...  The views and opinions of authors expressed herein do not necessarily state or reflect those of the funding source.  ... 
arXiv:1911.12436v1 fatcat:6jaukggnefcnlfcnk44opjdgny

Cross-Channel Intragroup Sparsity Neural Network [article]

Zhilin Yu, Chao Wang, Xin Wang, Qing Wu, Yong Zhao, Xundong Wu
2020 arXiv   pre-print
Modern deep neural networks rely on overparameterization to achieve state-of-the-art generalization. But overparameterized models are computationally expensive.  ...  We then present a novel training algorithm designed to perform well under the constraint imposed by the CCI-Sparsity.  ...  (b) The effect of group size (G = 4, 8, and 16, s = 1, 2, and 4, respectively) on the model accuracy with CCI-Sparsity imposed on a pre-trained sparse MobileNetV2 (75% sparsity ratio).  ... 
arXiv:1910.11971v2 fatcat:shdchslviva7jec46retb2omdy

Divergent representations of ethological visual inputs emerge from supervised, unsupervised, and reinforcement learning [article]

Grace W. Lindsay, Josh Merel, Tom Mrsic-Flogel, Maneesh Sahani
2022 arXiv   pre-print
Here we compare the representations learned by eight different convolutional neural networks, each with identical ResNet architectures and trained on the same family of egocentric images, but embedded  ...  Specifically, the representations are trained to guide action in a compound reinforcement learning task; to predict one or a combination of three task-related targets with supervision; or using one of  ...  ACKNOWLEDGMENTS Many thanks to Yuval Tassa, Arunkumar Byravan, and Diego Aldarondo for their input on the manuscript.  ... 
arXiv:2112.02027v2 fatcat:g6fwjcz5w5e3zjbxgvkmxkffza

Mitigating Asymmetric Nonlinear Weight Update Effects in Hardware Neural Network Based on Analog Resistive Synapse

Chih-Cheng Chang, Pin-Chun Chen, Teyuh Chou, I-Ting Wang, Boris Hudec, Che-Chia Chang, Chia-Ming Tsai, Tian-Sheuan Chang, Tuo-Hung Hou
2018 IEEE Journal on Emerging and Selected Topics in Circuits and Systems  
Asymmetric nonlinear weight update is considered as one of the major obstacles for realizing hardware neural networks based on analog resistive synapses because it significantly compromises the online  ...  New insights on engineering activation functions and a threshold weight update scheme effectively suppress the undesirable training noise induced by inaccurate weight update.  ...  The advantages of sparsity has been widely recognized in the field of deep neural networks [28] .  ... 
doi:10.1109/jetcas.2017.2771529 fatcat:bam2qsepfbc2lkrz3vdxsqjare

Beyond Word-based Language Model in Statistical Machine Translation [article]

Jiajun Zhang, Shujie Liu, Mu Li, Ming Zhou, Chengqing Zong
2015 arXiv   pre-print
phrase boundary in the large-scale monolingual data in order to enlarge the training set; 3, how to alleviate the data sparsity problem due to the huge vocabulary size of phrases.  ...  Language model is one of the most important modules in statistical machine translation and currently the word-based language model dominants this community.  ...  Finally, the deep neural network (DNN) is leveraged to investigate the data sparsity problem of the phrase-based language model.  ... 
arXiv:1502.01446v1 fatcat:ejtqo2fezjdehapy4ikqpuch3e

A Constrained, Weighted-L1 Minimization Approach for Joint Discovery of Heterogeneous Neural Connectivity Graphs [article]

Chandan Singh, Beilun Wang, Yanjun Qi
2017 arXiv   pre-print
Determining functional brain connectivity is crucial to understanding the brain and neural differences underlying disorders such as autism.  ...  Having established W-SIMULE's effectiveness, it links four key areas to autism, all of which are consistent with the literature.  ...  The ABIDE data was released with the goal of understanding human brain connectivity and how it reflects neural disorders (Van Essen et al. 2013) .  ... 
arXiv:1709.04090v2 fatcat:vyera32goventfcynuic5zfkdm
« Previous Showing results 1 — 15 out of 8,256 results