24,216 Hits in 4.5 sec

The State of Sparsity in Deep Neural Networks [article]

Trevor Gale, Erich Elsen, Sara Hooker
2019 arXiv   pre-print
We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50  ...  Together, these results highlight the need for large-scale benchmarks in the field of model compression.  ...  Acknowledgements We would like to thank Benjamin Caine, Jonathan Frankle, Raphael Gontijo Lopes, Sam Greydanus, and Keren Gu for helpful discussions and feedback on drafts of this paper.  ... 
arXiv:1902.09574v1 fatcat:bqnagzhdyjg6rea7kvfsirvdom

Sparseout: Controlling Sparsity in Deep Networks [chapter]

Najeeb Khan, Ian Stavness
2019 Lecture Notes in Computer Science  
Sparseout provides a way to investigate sparsity in state-of-the-art deep learning models. Source code for Sparseout could be found at .  ...  Dropout is commonly used to help reduce overfitting in deep neural networks.  ...  Acknowledgments This work was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC). Canadian AI Conference 2019 Sparseout: Controlling Sparsity in Deep Networks  ... 
doi:10.1007/978-3-030-18305-9_24 fatcat:6kc7pdzm2zhxnc5mxutdiogu3y

TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learning [article]

Riyadh Baghdadi, Abdelkader Nadir Debbagh, Kamel Abdous, Fatima Zohra Benhamida, Alex Renda, Jonathan Elliott Frankle, Michael Carbin, Saman Amarasinghe
2020 arXiv   pre-print
Our demonstration includes a mapping of sparse and recurrent neural networks to the polyhedral model along with an implementation of our approach in TIRAMISU, our state-of-the-art polyhedral compiler.  ...  In this paper, we demonstrate a compiler that can optimize sparse and recurrent neural networks, both of which are currently outside of the scope of existing neural network compilers (sparse neural networks  ...  Exploiting sparsity in deep neural networks has been the subject of multiple projects. Park et al.  ... 
arXiv:2005.04091v1 fatcat:zqeblrvhqjh6xjy6i6nquualza

Deep Neural Network Regularization for Feature Selection in Learning-to-Rank

Ashwini Rahangdale, Shital Raut
2019 IEEE Access  
The proposed model makes use of the deep neural network for learning-to-rank for document retrieval.  ...  It employs a regularization technique particularly suited for the deep neural network to improve the results significantly.  ...  FIGURE 4 . 4 Group sparsity produced for deep neural network. The group is formed at each layer of deep neural network based on the sparse values.  ... 
doi:10.1109/access.2019.2902640 fatcat:tfrgjgusdfbaxhw33zrysmmjca

SparseDNN: Fast Sparse Deep Learning Inference on CPUs [article]

Ziheng Wang
2021 arXiv   pre-print
For a compressed neural network, a multitude of inference frameworks have been designed to maximize the performance of the target hardware.  ...  The last few years have seen gigantic leaps in algorithms and systems to support efficient deep learning inference.  ...  The design of SparseDNN is inspired by other state-of-the-art inference systems for dense neural networks such as TensorRT, OpenVINO and MNN [15, 25, 31] , as well as performance engineering research  ... 
arXiv:2101.07948v4 fatcat:bs6rdifdlvat3hr4n435w65h2y

Deep Adaptive Network: An Efficient Deep Neural Network with Sparse Binary Connections [article]

Xichuan Zhou, Shengli Li, Kai Qin, Kunping Li, Fang Tang, Shengdong Hu, Shujun Liu, Zhi Lin
2016 arXiv   pre-print
Deep neural networks are state-of-the-art models for understanding the content of images, video and raw input data.  ...  To address this challenge, this paper presents a hardware-oriented deep learning algorithm, named as the Deep Adaptive Network, which attempts to exploit the sparsity in the neural connections.  ...  Deep Belief Network and Restricted Boltzmann Machines The DBN is a state of art training algorithm of the deep neural networks widely used for feature extraction.  ... 
arXiv:1604.06154v1 fatcat:wqak66fwmfc6dk43rbsib2mp4i

Speeding up Convolutional Neural Networks By Exploiting the Sparsity of Rectifier Units [article]

Shaohuai Shi, Xiaowen Chu
2017 arXiv   pre-print
In this work, we first examine the sparsity of the outputs of ReLUs in some popular deep convolutional architectures.  ...  Rectifier neuron units (ReLUs) have been widely used in deep convolutional networks.  ...  In this work, we first demostrate the sparsity of deep neural networks whoes activation functions are ReLUs during the process of training.  ... 
arXiv:1704.07724v2 fatcat:7oqafexob5bs5bur3dbaojra64

Equilibrium Propagation for Complete Directed Neural Networks [article]

Matilde Tristany Farinha, Sérgio Pequito, Pedro A. Santos, Mário A. T. Figueiredo
2020 arXiv   pre-print
Artificial neural networks, one of the most successful approaches to supervised learning, were originally inspired by their biological counterparts.  ...  However, the most successful learning algorithm for artificial neural networks, backpropagation, is considered biologically implausible.  ...  Introduction Nowadays, many state-of-the-art approaches to supervised learning rely on artificial neural networks (ANNs).  ... 
arXiv:2006.08798v2 fatcat:6puriq3xf5cwlkcsw4crcs7a6y

Sparsity Increases Uncertainty Estimation in Deep Ensemble

Uyanga Dorjsembe, Ju Hong Lee, Bumghi Choi, Jae Won Song
2021 Computers  
Deep neural networks have achieved almost human-level results in various tasks and have become popular in the broad artificial intelligence domains.  ...  To address this problem, we used model pruning and quantization with a deep ensemble and analyzed the effect in the context of uncertainty metrics.  ...  Acknowledgments: This research was supported by Basic Science Research Program through the NRF of Korea. Conflicts of Interest: The authors declare no conflict of interest.  ... 
doi:10.3390/computers10040054 fatcat:v4iuzxdonrfb7kksfrixyhfeva

Marginal Space Deep Learning: Efficient Architecture for Detection in Volumetric Image Data [chapter]

Florin C. Ghesu, Bogdan Georgescu, Yefeng Zheng, Joachim Hornegger, Dorin Comaniciu
2015 Lecture Notes in Computer Science  
design of Deep Learning (DL) network architectures.  ...  Our method outperforms the state-of-the-art with an improvement of 36%, running in less than one second.  ...  This motivated a further improvement through the introduction of multi-column deep neural networks [8] or state-of-the-art network regularization techniques based on a random dropping of units [7] .  ... 
doi:10.1007/978-3-319-24553-9_87 fatcat:2vkmo2dr7zhuvdqyedaav7fgkq

Neural Network Activation Quantization with Bitwise Information Bottlenecks [article]

Xichuan Zhou, Kui Liu, Cong Shi, Haijun Liu, Ji Liu
2020 arXiv   pre-print
Experiments over ImageNet and other datasets show that, by minimizing the quantization rate-distortion of each layer, the neural network with information bottlenecks achieves the state-of-the-art accuracy  ...  Meanwhile, by reducing the code rate, the proposed method can improve the memory and computational efficiency by over six times compared with the deep neural network with standard single-precision representation  ...  In practice, the BIB operation can be inserted in the macroblock of the deep neural network.  ... 
arXiv:2006.05210v1 fatcat:ncl4xrd7evgprdhx3peqzf5zrm

Neural Network Optimization for Reinforcement Learning Tasks Using Sparse Computations [article]

Dmitry Ivanov, Mikhail Kiselev, Denis Larionov
2022 arXiv   pre-print
It significantly reduces the number of multiplications when running neural networks. We tested different RL tasks and achieved 20-150x reduction in the number of multiplications.  ...  This method combines two ideas: neural network pruning and taking into account input data correlations; it makes it possible to update neuron states only when changes in them exceed a certain threshold  ...  This approach was successfully demonstrated by DeepMind in [7] . The neural network architecture presented in this study is called DQN (Deep Q-Network).  ... 
arXiv:2201.02571v2 fatcat:p5742lfus5f2jgvywsjced2cia

Neural Sparse Representation for Image Restoration [article]

Yuchen Fan, Jiahui Yu, Yiqun Mei, Yulun Zhang, Yun Fu, Ding Liu, Thomas S. Huang
2020 arXiv   pre-print
Inspired by the robustness and efficiency of sparse representation in sparse coding based image restoration models, we investigate the sparsity of neurons in deep networks.  ...  The sparsity constraints are favorable for gradient-based learning algorithms and attachable to convolution layers in various networks.  ...  Sparsity of hidden representation in deep neural networks cannot be solved by iterative optimization as sparse coding, since deep networks are feed-forward during inference.  ... 
arXiv:2006.04357v1 fatcat:pwvwi2jv2bf6zaoguli4stwbg4

Session 15 Overview: Compute-in-Memory Processors for Deep Neural Networks

Jun Deguchi, Yongpan Liu, Yan Li
2021 2021 IEEE International Solid- State Circuits Conference (ISSCC)  
Compute-in-memory (CIM) processors for deep neural networks continue to expand their capabilities, and to scale to larger datasets and more complicated models.  ...  The final paper in the session applies the tensor-train method to decompose and compress neural networks so that they fit within on-chip memory.  ...  neural-network (NN) inference processor based on a 4×4 array of programmable cores combining precise mixed-signal capacitor-based in-memory-computing (IMC) with digital SIMD near-memory computing, interconnected  ... 
doi:10.1109/isscc42613.2021.9365855 fatcat:ngyloi6o7fba3j4rig5m6iy37m

Channel Estimation for Massive MIMO Communication System Using Deep Neural Network [article]

Zohreh Mohades, Vahid TabaTabaVakili
2018 arXiv   pre-print
Here, employing deep neural networks we have provided two new greedy algorithms in order to solve MMV problems.  ...  Recently, ample researches have been conducted to solve this problem and diverse methods are proposed, one of which is deep neural network approach.  ...  In this paper, two different structures of deep neural networks, i.e. deep feed-forward networks and recurrent neural networks are considered. II.1.  ... 
arXiv:1806.09126v1 fatcat:ufjb236msjealchbvvx5riqwve
« Previous Showing results 1 — 15 out of 24,216 results