17,651 Hits in 2.0 sec

Continual Learning via Neural Pruning [article]

Siavash Golkar, Michael Kagan, Kyunghyun Cho
2019 arXiv   pre-print
We introduce Continual Learning via Neural Pruning (CLNP), a new method aimed at lifelong learning in fixed capacity models based on neuronal model sparsification.  ...  CLNP also provides simple continual learning diagnostic tools in terms of the number of free neurons left for the training of future tasks as well as the number of neurons that are being reused.  ...  Main contributions • We introduce Continual Learning via Neural Pruning (CLNP), a simple and intuitive lifelong learning method with the following properties: -Given a network with activation based neuron  ... 
arXiv:1903.04476v1 fatcat:tsaehmr2ujav5oxezasjltl5su

Sparse Flows: Pruning Continuous-depth Models [article]

Lucas Liebenwein, Ramin Hasani, Alexander Amini, Daniela Rus
2021 arXiv   pre-print
Continuous deep learning architectures enable learning of flexible probabilistic models for predictive modeling as neural ordinary differential equations (ODEs), and for generative modeling as continuous  ...  Our empirical results suggest that pruning improves generalization for neural ODEs in generative modeling.  ...  Pruning finds minimal and efficient neural ODE representations. Our framework finds highly optimized and efficient neural ODE architectures via pruning.  ... 
arXiv:2106.12718v2 fatcat:zmb6kyshzrd7hehjsutdf3glce

Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches? [article]

Di Zhang
2021 arXiv   pre-print
This allows the training and pruning process to continue without compromising performance.  ...  Therefore, we propose a strategy that combines the idea of neural network structure search with a pruning algorithm to alleviate this problem.  ...  of convergence and the degradation of test performance when the neural network continues training after pruning.  ... 
arXiv:2109.03862v1 fatcat:wp7a7iph4bf2tcsbioxasvq3za

Neural Network Pruning Through Constrained Reinforcement Learning [article]

Shehryar Malik, Muhammad Umair Haider, Omer Iqbal, Murtaza Taj
2021 arXiv   pre-print
We achieve this by proposing a novel pruning strategy via constrained reinforcement learning algorithms.  ...  Network pruning reduces the size of neural networks by removing (pruning) neurons such that the performance drop is minimal.  ...  Neural network pruning strategies can be grouped into three categories namely i) Offline pruning, ii) Online pruning and iii) Pruning via Reinforcement Learning.  ... 
arXiv:2110.08558v2 fatcat:u3lkgd3pvjcvjcf74uw7hvg3gy

Deep Model Compression Via Two-Stage Deep Reinforcement Learning [article]

Huixin Zhan, Wei-Ming Lin, Yongcan Cao
2021 arXiv   pre-print
., pruning, is achieved via exploiting deep reinforcement learning (DRL) to co-learn the accuracy and the FLOPs updated after layer-wise channel pruning and element-wise variational pruning via information  ...  In particular, this paper focuses on proposing a generic reinforcement learning-based model compression approach in a two-stage compression pipeline: pruning and quantization.  ...  accuracy by employing reinforcement learning to co-learn the layer-wise pruning rate and the element-wise variational pruning via information dropout.  ... 
arXiv:1912.02254v2 fatcat:djllnhd26fampj3fudl33minxi

Adaptive Neural Connections for Sparsity Learning

Alex Gain, Prakhar Kaushik, Hava Siegelmann
2020 2020 IEEE Winter Conference on Applications of Computer Vision (WACV)  
Sparsity learning aims to decrease the computational and memory costs of large deep neural networks (DNNs) via pruning neural connections while simultaneously retaining high accuracy.  ...  We propose Adaptive Neural Connections (ANC), a method for explicitly parameterizing fine-grained neuron-to-neuron connections via adjacency matrices at each layer that are learned through backpropagation  ...  In this sense, neuron to neuron connections can be learned adaptively through backpropagation via this method. We term this layer-wise method, Adaptive Neural Connections (ANC).  ... 
doi:10.1109/wacv45572.2020.9093542 dblp:conf/wacv/GainKS20 fatcat:nflqixfh7nbrzcuqikpl3wehbi

Improving and Understanding Variational Continual Learning [article]

Siddharth Swaroop, Cuong V. Nguyen, Thang D. Bui, Richard E. Turner
2019 arXiv   pre-print
In the continual learning setting, tasks are encountered sequentially.  ...  In this paper, we explore how the Variational Continual Learning (VCL) framework achieves these desiderata on two benchmarks in continual learning: split MNIST and permuted MNIST.  ...  A method similar in flavour is Continual Learning via Network Pruning [6] , where they now regularise during training to automatically prune out a large part of the network after every task.  ... 
arXiv:1905.02099v1 fatcat:q7qzj5qaibfxzhebg4ypa4akxq

Efficient Neural Network Training via Forward and Backward Propagation Sparsification [article]

Xiao Zhou, Weizhong Zhang, Zonghao Chen, Shizhe Diao, Tong Zhang
2021 arXiv   pre-print
We first formulate the training process as a continuous minimization problem under global sparsity constraint.  ...  For the former step, we use the conventional chain rule, which can be sparse via exploiting the sparse structure.  ...  bernoulli distributions are estimated via STE. [50] estimates the gradients via Gumbel-Softmax trick [17] , which is more accurate than STE. • Stochastic Continuous Mask. [28, 21] parameterize the mask  ... 
arXiv:2111.05685v1 fatcat:a3zhdgrdcbgpljwxlles3iggye

Transfer Learning for Mixed-Integer Resource Allocation Problems in Wireless Networks [article]

Yifei Shen, Yuanming Shi, Jun Zhang, Khaled B. Letaief
2018 arXiv   pre-print
In this paper, we propose a transfer learning method via self-imitation, to address this issue for effective resource allocation in wireless networks.  ...  It is based on a general "learning to optimize" framework for solving MINLP problems.  ...  Transfer Learning via Fine-tuning Fine-tuning is the most frequently employed method for transfer learning in neural networks.  ... 
arXiv:1811.07107v1 fatcat:6g3md2ot6faixbza5o7tfsxmde

Sparse Training via Boosting Pruning Plasticity with Neuroregeneration [article]

Shiwei Liu, Tianlong Chen, Xiaohan Chen, Zahra Atashgahi, Lu Yin, Huanyu Kou, Li Shen, Mykola Pechenizkiy, Zhangyang Wang, Decebal Constantin Mocanu
2022 arXiv   pre-print
Pruning plasticity can help explain several other empirical observations about neural network pruning in literature.  ...  Works on lottery ticket hypothesis (LTH) and single-shot network pruning (SNIP) have raised a lot of attention currently on post-training pruning (iterative magnitude pruning), and before-training pruning  ...  However, pruning plasticity drops significantly after the second learning rate decay, leading to a situation where the pruned networks can not recover with continued training.  ... 
arXiv:2106.10404v4 fatcat:p5jb643ykvdj5mqexsx434wv7a

Retraining a Pruned Network: A Unified Theory of Time Complexity

Soumya Sara John, Deepak Mishra, Sheeba Rani Johnson
2020 SN Computer Science  
Fine-tuning of neural network parameters is an essential step that is involved in model compression via pruning, which let the network relearn using the training data.  ...  The time needed to relearn a compressed neural network model is crucial in identifying a hardware-friendly architecture.  ...  -Retraining a network with random connection pruning destroys the sparsity that was introduced via pruning. This is due to the unstructured sparsity, which was absent in clustered node pruning.  ... 
doi:10.1007/s42979-020-00208-w fatcat:gcq6xbadxzfu3b7dkkxi4ykz74

Max-plus Operators Applied to Filter Selection and Model Pruning in Neural Networks [article]

Yunxiang Zhang, Santiago Velasco-Forero
2019 arXiv   pre-print
Experimental results demonstrate that the filter selection strategy enabled by a Max-plus is highly efficient and robust, through which we successfully performed model pruning on different neural network  ...  in layers of conventional neural networks.  ...  Sparsity constraints were imposed in [15] on channel-wise scaling factors and pruning was based on their magnitude, while in [26] group-sparsity was leverage to learn compact CNNs via a combination  ... 
arXiv:1903.08072v2 fatcat:6fqwmggx3jfrpfe3zbl6ycqdzu

DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator Search [article]

Yushuo Guan, Ning Liu, Pengyu Zhao, Zhengping Che, Kaigui Bian, Yanzhi Wang, Jian Tang
2020 arXiv   pre-print
Specifically, DAIS relaxes the binarized channel indicators to be continuous and then jointly learns both indicators and model parameters via bi-level optimization.  ...  In this paper, we introduce Differentiable Annealing Indicator Search (DAIS) that leverages the strength of neural architecture search in the channel pruning and automatically searches for the effective  ...  , and then jointly learn the model parameters and relaxed channel indicators via a differentiable search procedure.  ... 
arXiv:2011.02166v1 fatcat:siwdqg2lvvhqhobcmfnqbkzklq

Winning the Lottery with Continuous Sparsification [article]

Pedro Savarese and Hugo Silva and Michael Maire
2021 arXiv   pre-print
The search for efficient, sparse deep neural network models is most prominently performed by pruning: training a dense, overparameterized network and removing parameters, usually via following a manually-crafted  ...  In addition to setting a new standard for pruning, Continuous Sparsification also offers fast parallel ticket search, opening doors to new applications of the Lottery Ticket Hypothesis.  ...  Sparse transfer learning via winning lottery tickets. arXiv:1905.07785, 2019. [17] Soelen, R. V., J. W. Sheppard.  ... 
arXiv:1912.04427v4 fatcat:ickxgydq4zdrde4em3geiaknta

Robust and Verifiable Information Embedding Attacks to Deep Neural Networks via Error-Correcting Codes [article]

Jinyuan Jia, Binghui Wang, Neil Zhenqiang Gong
2020 arXiv   pre-print
In the era of deep learning, a user often leverages a third-party machine learning tool to train a deep neural network (DNN) classifier and then deploys the classifier as an end-user software product or  ...  We propose to recover the message via adaptively querying the classifier to save queries.  ...  After the neural network is deployed, the attacker can recover the training data points via querying the neural network.  ... 
arXiv:2010.13751v1 fatcat:vvooyux62jajtmsxlq5x4ctaiu
« Previous Showing results 1 — 15 out of 17,651 results