Filters








4,490 Hits in 4.9 sec

Size and depth of monotone neural networks: interpolation and approximation [article]

Dan Mikulincer, Daniel Reichman
2022 arXiv   pre-print
We study the interpolation problem for monotone data sets: The input is a monotone data set with n points, and the goal is to find a size and depth efficient monotone neural network, with non negative  ...  On the other hand, we prove that for every monotone data set with n points in ℝ^d, there exists an interpolating monotone network of depth 4 and size O(nd).  ...  We are grateful to David Kim for implementing our construction of a monotone neural network and testing it over several monotone data sets.  ... 
arXiv:2207.05275v1 fatcat:ose2algqdfe3nmbmogpz4kmsde

Analyzing Monotonic Linear Interpolation in Neural Network Loss Landscapes [article]

James Lucas, Juhan Bae, Michael R. Zhang, Stanislav Fort, Richard Zemel, Roger Grosse
2021 arXiv   pre-print
This Monotonic Linear Interpolation (MLI) property, first observed by Goodfellow et al. (2014) persists in spite of the non-convex objectives and highly non-linear training dynamics of neural networks.  ...  Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective  ...  Varying depth and hidden size. We explored the effect of varying depth and hidden size on the MLI property.  ... 
arXiv:2104.11044v2 fatcat:ovo6c3tkxbdcxgi6hoczugnffy

On Monotonic Linear Interpolation of Neural Network Parameters

James Lucas, Juhan Bae, Michael R. Zhang, Stanislav Fort, Richard S. Zemel, Roger B. Grosse
2021 International Conference on Machine Learning  
This Monotonic Linear Interpolation (MLI) property, first observed by Goodfellow et al. (2014) , persists in spite of the nonconvex objectives and highly non-linear training dynamics of neural networks  ...  Linear interpolation between initial neural network parameters and converged parameters after training with stochastic gradient descent (SGD) typically leads to a monotonic decrease in the training objective  ...  Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute (www.vectorinstitute.ai  ... 
dblp:conf/icml/LucasBZFZG21 fatcat:bwumvqj4ubaa7do65m5m6fyty4

Certified Monotonic Neural Networks [article]

Xingchao Liu, Xing Han, Na Zhang, Qiang Liu
2020 arXiv   pre-print
Our method allows us to train neural networks with heuristic monotonicity regularizations, and we can gradually increase the regularization magnitude until the learned network is certified monotonic.  ...  monotonic neural networks with arbitrary model structures.  ...  The exact value of R(f ) is intractable, and we approximate it by drawing samples of size 1024 uniformly from the input domain during iterations of the gradient descent.  ... 
arXiv:2011.10219v1 fatcat:tlmeyymzxnbe3kq2ca6hskvei4

Increasing Depth Leads to U-Shaped Test Risk in Over-parameterized Convolutional Networks [article]

Eshaan Nichani, Adityanarayanan Radhakrishnan, Caroline Uhler
2021 arXiv   pre-print
For neural networks, however, model capacity can also be increased through depth, yet understanding the impact of increasing depth on test risk remains an open question.  ...  In this work, we demonstrate that the test risk of over-parameterized convolutional networks is a U-shaped curve (i.e. monotonically decreasing, then increasing) with increasing depth.  ...  Acknowledgements The authors were supported by the National Science Foundation (DMS-1651995), Office of Naval Research (N00014-17-1-2147 and N00014-18-1-2765), MIT-IBM Watson AI Lab, and a Simons Investigator  ... 
arXiv:2010.09610v2 fatcat:4tylqykcnnh3hbpdbpg3fzi5zy

A Convolutional Neural Network Approach to Predicting Network Connectedness Robustness

Yang Lou, Ruizi Wu, Junli Li, Lin Wang, G. Chen
2021 IEEE Transactions on Network Science and Engineering  
In the present paper, an efficient method based on convolutional neural network (CNN) is proposed to train for estimating the network connectedness robustness.  ...  Extensive experimental studies on directed and undirected, as well as synthetic and real-world networks suggest that: 1) the proposed CNN-based methodology performs excellently in the prediction of the  ...  In configuration, the upper and lower bounds of the LCC size are imposed onto the output of CNN-RP, and logically unreasonable data are replaced by interpolated values.  ... 
doi:10.1109/tnse.2021.3107186 fatcat:h5uh3f6ifnf3do2kn3iost2n5m

Why Deep Neural Networks for Function Approximation? [article]

Shiyu Liang, R. Srikant
2017 arXiv   pre-print
First, we consider univariate functions on a bounded interval and require a neural network to achieve an approximation error of ε uniformly over the interval.  ...  Our results are derived for neural networks which use a combination of rectifier linear units (ReLUs) and binary step units, two of the most popular type of activation functions.  ...  ACKNOWLEDGMENTS The research reported here was supported by NSF Grants CIF 14-09106, ECCS 16-09370, and ARO Grant W911NF-16-1-0259.  ... 
arXiv:1610.04161v2 fatcat:2s6w3rtulvbrlmsad77s4wtmve

Quantum Neural Networks

Sanjay Gupta, R.K.P. Zia
2001 Journal of computer and system sciences (Print)  
It is shown that QNNs of logarithmic size and constant depth have the same computational power as threshold circuits, which are used for modeling neural networks.  ...  QNNs of polylogarithmic size and polylogarithmic depth can solve the problems in NC, the class of problems with theoretically fast parallel solutions.  ...  A quantum neural network QNN(s(n), d(n)) of precision p(n) is a circuit of size s(n) and depth d(n), constructed from the gates D and U of precision p(n).  ... 
doi:10.1006/jcss.2001.1769 fatcat:ciuh4h4eubbvjgelze5hd43g2u

On the Universal Approximability and Complexity Bounds of Quantized ReLU Neural Networks [article]

Yukun Ding, Jinglan Liu, Jinjun Xiong, Yiyu Shi
2019 arXiv   pre-print
To the best of our knowledge, this is the first in-depth study on the complexity bounds of quantized neural networks.  ...  Then we provide upper bounds on the number of weights and the memory size for a given approximation error bound and the bit-width of weights for function-independent and function-dependent structures.  ...  The memory size of this network then naturally serves as an upper bound for the minimal network size.  ... 
arXiv:1802.03646v4 fatcat:y4qjs2hirfhfnbvemctnhhwspm

Quantum Neural Networks [article]

Sanjay Gupta, R.K.P. Zia
2002 arXiv   pre-print
It is shown that QNNs of logarithmic size and constant depth have the same computational power as threshold circuits, which are used for modeling neural networks.  ...  QNNs of polylogarithmic size and polylogarithmic depth can solve the problems in , the class of problems with theoretically fast parallel solutions.  ...  Acknowledgements: The author is indebted to Harald Hempel for carefully reading the first draft of the paper and suggesting numerous improvements.  ... 
arXiv:quant-ph/0201144v1 fatcat:jkd6gsmqyrhthcdgpz54b7baq4

Neural networks and rational functions [article]

Matus Telgarsky
2017 arXiv   pre-print
Neural networks and rational functions efficiently approximate each other.  ...  of size O(polylog(1/ϵ)) which is ϵ-close.  ...  Adam Klivans and the author both thank Almare Gelato Italiano, in downtown Berkeley, for necessitating further stimulating conversations, but now on the topic of health and exercise.  ... 
arXiv:1706.03301v1 fatcat:sinwakabengojg5h6alxggu7uq

Kinetics-Informed Neural Networks [article]

Gabriel S. Gusmão, Adhika P. Retnanto, Shashwati C. da Cunha, Andrew J. Medford
2021 arXiv   pre-print
We present an algebraic framework for the mathematical description and classification of reaction networks, types of elementary reaction, and chemical species.  ...  Under this framework, we demonstrate that the simultaneous training of neural nets and kinetic model parameters in a regularized multi-objective optimization setting leads to the solution of the inverse  ...  John Kitchin and acknowledge his early seminal work on the utilization of neural networks for the solution of simple coupled forward kinetics ODEs available via his blog (kitchingroup.cheme.cmu.edu/blog  ... 
arXiv:2011.14473v2 fatcat:nylge6xokzc2jhukhypgdr2r3q

Variable-Input Deep Operator Networks [article]

Michael Prasthofer, Tim De Ryck, Siddhartha Mishra
2022 arXiv   pre-print
VIDON is invariant to permutations of sensor locations and is proved to be universal in approximating a class of continuous operators.  ...  We address this issue by proposing a novel operator learning framework, termed Variable-Input Deep Operator Network (VIDON), which allows for random sensors whose number and locations can vary across samples  ...  Using the results on function approximation by tanh neural networks from [4] we find that there exists a tanh neural network U M of width O(M d ) and depth O(n) that maps {U 0 j } j to { U n j } j for  ... 
arXiv:2205.11404v1 fatcat:3umodvrul5h4pntlzm7m5dvmli

Multi-level Residual Networks from Dynamical Systems View [article]

Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, David Begert
2018 arXiv   pre-print
Deep residual networks (ResNets) and their variants are widely used in many computer vision applications and natural language processing tasks.  ...  In this paper, we adopt the dynamical systems point of view, and analyze the lesioning properties of ResNet both theoretically and experimentally.  ...  This is similar to the gradient exploding/vanishing problem for deep neural networks or recurrent neural networks.  ... 
arXiv:1710.10348v2 fatcat:r2ytol3wijcvjcfzoxi5q3vnoq

Size and Depth Separation in Approximating Benign Functions with Neural Networks [article]

Gal Vardi, Daniel Reichman, Toniann Pitassi, Ohad Shamir
2021 arXiv   pre-print
When studying the expressive power of neural networks, a main challenge is to understand how the size and depth of the network affect its ability to approximate real functions.  ...  We call functions that satisfy these conditions "benign", and explore the benefits of size and depth for approximation of benign functions with ReLU networks.  ...  Arora and B. Barak. Computational complexity: a modern approach. Cambridge University Press, 2009.  ... 
arXiv:2102.00314v3 fatcat:v56bn7msrfe6ngltmxkhiqggr4
« Previous Showing results 1 — 15 out of 4,490 results