Filters








4,731 Hits in 3.0 sec

Neural reparameterization improves structural optimization [article]

Stephan Hoyer, Jascha Sohl-Dickstein, Sam Greydanus
2019 arXiv   pre-print
In this paper, we propose using the implicit bias over functions induced by neural networks to improve the parameterization of structural optimization.  ...  Rather than directly optimizing densities on a grid, we instead optimize the parameters of a neural network which outputs those densities.  ...  Then we can reparameterize the problem by optimizing the weights and inputs (θ and β) of a neural network which outputsx. Structural optimization.  ... 
arXiv:1909.04240v2 fatcat:kvunrdz63rhoroigrxqnr4hppe

Parameter Efficient Training of Deep Convolutional Neural Networks by Dynamic Sparse Reparameterization [article]

Hesham Mostafa, Xin Wang
2019 arXiv   pre-print
Modern deep neural networks are typically highly overparameterized. Pruning techniques are able to remove a significant fraction of network parameters with little loss in accuracy.  ...  Rather, effective learning crucially depended on the continuous exploration of the sparse network structure space during training.  ...  Thus, simultaneous exploration of network structure and parameter optimization through gradient descent are synergistic. Structural exploration improves the trainability of sparse deep CNNs.  ... 
arXiv:1902.05967v3 fatcat:yr7oed7kzzewnipb3h22c5h4ae

Automatically Learning Compact Quality-aware Surrogates for Optimization Problems [article]

Kai Wang, Bryan Wilder, Andrew Perrault, Milind Tambe
2020 arXiv   pre-print
fail to improve solution quality due to non-smoothness issues that arise when training through a complex optimization layer.  ...  Empirically, we demonstrate these improvements on a non-convex adversary modeling task, a submodular recommendation task and a convex portfolio optimization task.  ...  Overall, adaptively adjusting the surrogate model allows us to extract the underlying structure of the optimization problem using few meta-variables.  ... 
arXiv:2006.10815v2 fatcat:ixvcqrjgh5gfhgpg7ew6jdy43q

Design space reparameterization enforces hard geometric constraints in inverse-designed nanophotonic devices [article]

Mingkun Chen, Jiaqi Jiang, Jonathan Fan
2020 arXiv   pre-print
global optimizers.  ...  As a proof-of-concept demonstration, we apply reparameterization to enforce strict minimum feature size constraints in local and global topology optimizers for metagratings.  ...  permittivity values in a manner that improves a Figure of Merit (FoM).  ... 
arXiv:2007.12991v1 fatcat:kvpwowvspvcjnhw3a4en7j5ntq

Efficient Re-parameterization Operations Search for Easy-to-Deploy Network Based on Directional Evolutionary Strategy [article]

Xinyi Yu, Xiaowei Wang, Jintao Rong, Mingyang Zhang, Linlin Ou
2022 arXiv   pre-print
Structural re-parameterization (Rep) methods has achieved significant performance improvement on traditional convolutional network.  ...  Most current Rep methods rely on prior knowledge to select the reparameterization operations. However, the performance of architecture is limited by the type of operations and prior knowledge.  ...  From left to right, it represents the reparameterization structure from the first layer to the last layer. Fig. A3 A3 Fig.  ... 
arXiv:2204.06403v2 fatcat:64syd5qtbndanmsroejjk23gsa

KF-LAX: Kronecker-factored curvature estimation for control variate optimization in reinforcement learning [article]

Mohammad Firouzi
2018 arXiv   pre-print
A key challenge for gradient based optimization methods in model-free reinforcement learning is to develop an approach that is sample efficient and has low variance.  ...  In this work, we apply Kronecker-factored curvature estimation technique (KFAC) to a recently proposed gradient estimator for control variate optimization, RELAX, to increase the sample efficiency of using  ...  In particular case where the state-value function is approximated with a compatible function approximator, the natural gradient tends toward the optimal greedy policy improvement (Appendix C shows the  ... 
arXiv:1812.04181v1 fatcat:bq7nm6dkkfawpke4ctb53zsbze

Dynamic Model Pruning with Feedback [article]

Tao Lin, Sebastian U. Stich, Luis Barba, Daniil Dmitriev, Martin Jaggi
2020 arXiv   pre-print
Deep neural networks often have millions of parameters.  ...  sparsity pattern and (ii) incorporating feedback signal to reactivate prematurely pruned weights we obtain a performant sparse model in one single training pass (retraining is not needed, but can further improve  ...  A.3.4 IMPLICIT NEURAL ARCHITECTURE SEARCH DPF can provide effective training-time structural exploration or even implicit neural network search.  ... 
arXiv:2006.07253v1 fatcat:5t5ijn7hzfh3xiuy6hbsvjvdju

Improving Discrete Latent Representations With Differentiable Approximation Bridges [article]

Jason Ramapuram, Russ Webb
2019 arXiv   pre-print
We also observe an accuracy improvement of 77% in neural sequence sorting and a 25% improvement against the straight-through estimator [5] in an image classification setting.  ...  The DAB network is not used for inference and expands the class of functions that are usable in neural networks.  ...  Most state of the art neural networks [21, 46, 15] rely on some variant of Robbins-Monroe [44] based stochastic optimization.  ... 
arXiv:1905.03658v3 fatcat:kui3s7x6njdvvdwizka7pw4zxm

Understanding symmetries in deep networks [article]

Vijay Badrinarayanan and Bamdev Mishra and Roberto Cipolla
2015 arXiv   pre-print
Recent works have highlighted scale invariance or symmetry present in the weight space of a typical deep network and the adverse effect it has on the Euclidean gradient based stochastic gradient descent optimization  ...  Our empirical evidence based on the MNIST dataset shows that the proposed updates improve the test performance beyond what is achieved with batch normalization and without sacrificing the computational  ...  Consequently, optimization trajectories can vary significantly for different reparameterizations [2] .  ... 
arXiv:1511.01029v1 fatcat:evijefp6jjdp7gij6umwq7hwfy

Slice Sampling Reparameterization Gradients

David M. Zoltowski, Diana Cai, Ryan P. Adams
2021 Neural Information Processing Systems  
We evaluate the method on synthetic examples and apply it to a variety of applications with reparameterization of unnormalized probability distributions. 35th Conference on Neural Information Processing  ...  Here we describe how to differentiate samples from slice sampling to compute slice sampling reparameterization gradients, enabling a richer class of Monte Carlo objective functions to be optimized.  ...  With reparameterized slice sampling we can use fully reparameterized gradients to optimize the VCD objective.  ... 
dblp:conf/nips/ZoltowskiCA21 fatcat:bjs5o66v6jat5ieriw3kisgw6a

Backpropagation through the Void: Optimizing control variates for black-box gradient estimation [article]

Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, David Duvenaud
2018 arXiv   pre-print
Our method uses gradients of a neural network trained jointly with model parameters or policies, and is applicable in both discrete and continuous settings.  ...  Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy.  ...  Ideally, we will take advantage of any known structure in f .  ... 
arXiv:1711.00123v3 fatcat:5vevkhhclbbtpo4zzyf2vzcewi

Relaxed Multivariate Bernoulli Distribution and Its Applications to Deep Generative Models

Xi Wang, Junming Yin
2020 Conference on Uncertainty in Artificial Intelligence  
To optimize the variational objective of VAE, the reparameterization trick is commonly applied to obtain a lowvariance estimator of the gradient.  ...  Recent advances in variational auto-encoder (VAE) have demonstrated the possibility of approximating the intractable posterior distribution with a variational distribution parameterized by a neural network  ...  ., 2015) , we show that: (1) modeling label dependencies can improve classification accuracy; and (2) our model is able to well capture the underlying class structure of the data.  ... 
dblp:conf/uai/WangY20 fatcat:sh7wpgilynd2pngyw3ub4m3nxi

Inference by Reparameterization in Neural Population Codes [article]

Rajkumar Vasudeva Raju, Xaq Pitkow
2016 arXiv   pre-print
Unfortunately, a subtle feature of LBP renders it neurally implausible. However, LBP can be elegantly reformulated as a sequence of Tree-based Reparameterizations (TRP) of the graphical model.  ...  The neural network represents uncertainty using Probabilistic Population Codes (PPCs), which are distributed neural representations that naturally encode probability distributions, and support marginalization  ...  It may therefore be possible to develop reparameterization algorithms with all the convenient properties of LBP but with improved performance on loopy graphs.  ... 
arXiv:1605.06544v1 fatcat:fvo3hci4vjh3jnvbscrahvvngu

Soft Actor-Critic With Integer Actions [article]

Ting-Han Fan, Yubo Wang
2022 arXiv   pre-print
Our key observation for integer actions is that their discrete structure can be simplified using their comparability property.  ...  Hence, the proposed integer reparameterization does not need one-hot encoding and is of low dimensionality.  ...  Thereby, in this section, we will discuss the structure of integer variables and propose a simple linear map trick to reparameterize an integer variable from STGS.  ... 
arXiv:2109.08512v2 fatcat:b3zkknhk5ratjoqwtqxk4cdakm

Variational Neural Machine Translation [article]

Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, Min Zhang
2016 arXiv   pre-print
Experiments on both Chinese-English and English- German translation tasks show that the proposed variational neural machine translation achieves significant improvements over the vanilla neural machine  ...  In order to perform efficient posterior inference and large-scale training, we build a neural posterior approximator conditioned on both the source and the target sides, and equip it with a reparameterization  ...  Typically, these models utilize an neural inference model to approximate the intractable posterior, and optimize model parameters jointly with a reparameterized variational lower bound using the standard  ... 
arXiv:1605.07869v2 fatcat:5raucswrxvgxnedtxwcfgzproy
« Previous Showing results 1 — 15 out of 4,731 results