2,516 Hits in 3.8 sec

Winning Lottery Tickets in Deep Generative Models [article]

Neha Mukund Kalibhat, Yogesh Balaji, Soheil Feizi
2021 arXiv   pre-print
In this paper, we confirm the existence of winning tickets in deep generative models such as GANs and VAEs.  ...  Furthermore, we show the practical benefits of lottery tickets in generative models by detecting tickets at very early stages in training called "early-bird tickets".  ...  Early-bird Tickets We evaluate early-bird tickets in generative models, similar to winning lottery tickets using FID as discussed in the previous section.  ... 
arXiv:2010.02350v2 fatcat:sb4wo22epjan3ossyrkjvsfjge

Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP [article]

Haonan Yu, Sergey Edunov, Yuandong Tian, Ari S. Morcos
2020 arXiv   pre-print
Here, we evaluate whether "winning ticket" initializations exist in two different domains: natural language processing (NLP) and reinforcement learning (RL).For NLP, we examined both recurrent LSTM models  ...  The lottery ticket hypothesis proposes that over-parameterization of deep neural networks (DNNs) aids training by increasing the probability of a "lucky" sub-network initialization being present rather  ...  APPROACH GENERATING LOTTERY TICKETS Pruning methods In our experiments, we use both one-shot and iterative pruning to find winning ticket initializations.  ... 
arXiv:1906.02768v3 fatcat:7eb3nb7hyjfonb3nx5q37lnlpq

Exploring Lottery Ticket Hypothesis in Media Recommender Systems [article]

Yanfang Wang, Yongduo Sui, Xiang Wang, Zhenguang Liu, Xiangnan He
2021 arXiv   pre-print
In this paper, we extend LTH to media recommender systems, aiming to find the winning tickets in deep recommender models.  ...  With MF and LightGCN as the backbone models, we found that there widely exist winning tickets in recommender models.  ...  We studied lottery ticket hypothesis in media recommender systems, exploiting IMP algorithm to find winning tickets of the user-item embedding table in two representative deep recommender models -MF and  ... 
arXiv:2108.00944v1 fatcat:misrjzlvb5ft5pos2nght2tazm

On the Transferability of Winning Tickets in Non-Natural Image Datasets [article]

Matthia Sabatelli, Mike Kestemont, Pierre Geurts
2020 arXiv   pre-print
We study the generalization properties of pruned neural networks that are the winners of the lottery ticket hypothesis on datasets of natural images.  ...  Our results show that there are significant benefits in transferring and training sparse architectures over larger parametrized models, since in all of our experiments pruned networks, winners of the lottery  ...  Lottery Tickets VS fine-tuned pruned models So far we have focused our transfer-learning study on lottery tickets that come in the form of f (x; m θ k ), where, as mentioned in Sec. 3, θ k corresponds  ... 
arXiv:2005.05232v2 fatcat:eoxqzi6kjfherhse7qw6oo2rei

Sparse Transfer Learning via Winning Lottery Tickets [article]

Rahul Mehta
2019 arXiv   pre-print
In this paper, we extend the Lottery Ticket Hypothesis to a variety of transfer learning tasks.  ...  The recently proposed Lottery Ticket Hypothesis of Frankle and Carbin (2019) suggests that the performance of over-parameterized deep networks is due to the random initialization seeding the network with  ...  This work was made possible by a generous grant from Amazon Research for our compute infrastructure.  ... 
arXiv:1905.07785v2 fatcat:l5gws4anizbsdlxhinrekf7pbu

Visualizing the Loss Landscape of Winning Lottery Tickets [article]

Robert Bain
2021 arXiv   pre-print
This work vastly reduces the time required to compute such loss landscapes, and uses them to study winning lottery tickets found via iterative magnitude pruning.  ...  We also share results that contradict previously claimed correlations between certain loss landscape projection methods and model trainability and generalization error.  ...  Later in the paper we introduce the lottery ticket hypothesis (LTH) and iterative magnitude pruning (IMP) by (Frankle & Carbin, 2019) , and apply the same loss visualizations to winning lottery tickets  ... 
arXiv:2112.08538v1 fatcat:ufzpaoj3xnbuljonjcgtwyeb74

One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers [article]

Ari S. Morcos, Haonan Yu, Michela Paganini, Yuandong Tian
2019 arXiv   pre-print
However, the generality of winning ticket initializations remains unclear.  ...  Moreover, winning tickets generated using larger datasets consistently transferred better than those generated using smaller datasets.  ...  The lottery ticket hypothesis was first postulated and examined in smaller models and datasets in [7] and analyzed in large models and datasets, leading to the development of late resetting, in [8]  ... 
arXiv:1906.02773v2 fatcat:mkqxlsh6xnbhpip4aak2qjlcsa

Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches? [article]

Di Zhang
2021 arXiv   pre-print
A new winning ticket sub-network with deeper network structure, better generalization ability and better test performance can be obtained in this recursive manner.  ...  of generating winning ticket sub-network when the final network structure is not given.  ...  Conclusion In this paper,we introduced the recursive lottery ticket hypothesis by generalizing the lottery ticket hypothesis for neural networks.  ... 
arXiv:2109.03862v1 fatcat:wp7a7iph4bf2tcsbioxasvq3za

[Re] One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers

Varun Gohil, S. Deepak Narayanan, Atishay Jain
2020 Zenodo  
The lottery ticket hypothesis states that smaller subnetworks within a larger deep network can be trained in isolation to achieve accuracy similar to that of original network, as long as they are initialized  ...  The paper "One ticket to win them all:generalizing lottery ticket initializations across datasets and optimizers" empirically shows that these winning tickets are transferable.  ...  Further, as generating lottery tickets using iterative pruning is computationally expensive and time-consuming, more efficient methods for generating winning tickets are needed.  ... 
doi:10.5281/zenodo.3818618 fatcat:ewhabnvw7zcijhafqrculymzfi

You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership [article]

Xuxi Chen, Tianlong Chen, Zhenyu Zhang, Zhangyang Wang
2021 arXiv   pre-print
The lottery ticket hypothesis (LTH) emerges as a promising framework to leverage a special sparse subnetwork (i.e., winning ticket) instead of a full model for both training and inference, that can lower  ...  Through extensive experiments, we demonstrate the effectiveness of lottery verification in diverse models (ResNet-20, ResNet-18, ResNet-50) on CIFAR-10 and CIFAR-100.  ...  which is a learning process for the lottery ticket model.  ... 
arXiv:2111.00162v1 fatcat:6xi7aotdkvb43bqyqtwqgmbbcq

The Elastic Lottery Ticket Hypothesis [article]

Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, Zhangyang Wang
2021 arXiv   pre-print
Lottery Ticket Hypothesis (LTH) raises keen attention to identifying sparse trainable subnetworks, or winning tickets, which can be trained in isolation to achieve similar or even better performance compared  ...  A natural question that comes in is: can we "transform" the winning ticket found in one network to another with a different architecture, yielding a winning ticket for the latter at the beginning, without  ...  Acknowledgment Z.W. is in part supported by the NSF AI Institute for Foundations of Machine Learning (IFML).  ... 
arXiv:2103.16547v3 fatcat:it24hrwm2zejti5acxpupbabam

Evaluating Lottery Tickets Under Distributional Shifts

Shrey Desai, Hongyuan Zhan, Ahmed Aly
2019 Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019)  
Our experiments show that sparse subnetworks obtained through lottery ticket training do not simply overfit to particular domains, but rather reflect an inductive bias of deep neural networks that can  ...  The Lottery Ticket Hypothesis (Frankle and Carbin, 2019) suggests large, overparameterized neural networks consist of small, sparse subnetworks that can be trained in isolation to reach a similar (or better  ...  Our experiments show that "winning tickets" can indeed be identified in a sentiment task formulated from noisy, user-generated datasets.  ... 
doi:10.18653/v1/d19-6117 dblp:conf/acl-deeplo/DesaiZA19 fatcat:2ojb6dvmhvemxdjrd3rwoyqvki

Towards Practical Lottery Ticket Hypothesis for Adversarial Training [article]

Bai Li, Shiqi Wang, Yunhan Jia, Yantao Lu, Zhenyu Zhong, Lawrence Carin, Suman Jana
2020 arXiv   pre-print
Recent research has proposed the lottery ticket hypothesis, suggesting that for a deep neural network, there exist trainable sub-networks performing equally or better than the original model with commensurate  ...  The high cost incurred limits the applications of the lottery ticket hypothesis.  ...  In Figure 3a , we calculate the relative 2 distance between the full and the pruned model weights w f and w p generated using winning and boosting tickets 2 = w f −wp 2 w f 2 .  ... 
arXiv:2003.05733v1 fatcat:zrzuwlgvmzfwxlspws34y5zhqq

The Search for Sparse, Robust Neural Networks [article]

Justin Cosentino, Federico Zaiter, Dan Pei, Jun Zhu
2019 arXiv   pre-print
We perform an extensive empirical evaluation and analysis testing the Lottery Ticket Hypothesis with adversarial training and show this approach enables us to find sparse, robust neural networks.  ...  Orthogonal to pruning literature, deep neural networks are known to be susceptible to adversarial examples, which may pose risks in security- or safety-critical applications.  ...  Furthermore, winning lottery tickets constantly outperform all other approaches at the highest level of sparsity of 3.6% and 1.8%. In some cases even finding robust lottery tickets as in Figure 7 .  ... 
arXiv:1912.02386v1 fatcat:kgzxdrftlvdwvfj6zalwmhjnea

Playing Lottery Tickets in Style Transfer Models [article]

Meihao Kong, Jing Huo, Wenbin Li, Jing Wu, Yu-Kun Lai, Yang Gao
2022 arXiv   pre-print
and SANet, other models such as LST, MANet, AdaAttN and MCCNet can also play lottery tickets, which shows that LTH can be generalized to various style transfer models.  ...  Recently, the lottery ticket hypothesis (LTH) has shown great potential in finding extremely sparse matching subnetworks which can achieve on par or even better performance than the original full networks  ...  of the overall process for playing lottery tickets in style transfer models.  ... 
arXiv:2203.13802v2 fatcat:5vc3fpas5jc5rp3s2xqgwjbjhe
« Previous Showing results 1 — 15 out of 2,516 results