7,521 Hits in 5.6 sec

Neural Tangent Kernel: Convergence and Generalization in Neural Networks [article]

Arthur Jacot, Franck Gabriel, Clément Hongler
2020 arXiv   pre-print
vectors) follows the kernel gradient of the functional cost (which is convex, in contrast to the parameter cost) w.r.t. a new kernel: the Neural Tangent Kernel (NTK).  ...  At initialization, artificial neural networks (ANNs) are equivalent to Gaussian processes in the infinite-width limit, thus connecting them to kernel methods.  ...  The last author acknowledges support from the ERC SG Constamis, the NCCR SwissMAP, the Blavatnik Family Foundation and the Latsis Foundation.  ... 
arXiv:1806.07572v4 fatcat:vqgyqhbr4vfnrbgyfcq6uz75h4

The Recurrent Neural Tangent Kernel [article]

Sina Alemohammad, Zichao Wang, Randall Balestriero, Richard Baraniuk
2021 arXiv   pre-print
The study of deep neural networks (DNNs) in the infinite-width limit, via the so-called neural tangent kernel (NTK) approach, has provided new insights into the dynamics of learning, generalization, and  ...  In this paper we introduce and study the Recurrent Neural Tangent Kernel (RNTK), which provides new insights into the behavior of overparametrized RNNs.  ...  Neural tangent kernel: Convergence and generalization in neural networks.  ... 
arXiv:2006.10246v4 fatcat:xvwj2giikjebnbz6g6nz7pgsue

Exact Convergence Rates of the Neural Tangent Kernel in the Large Depth Limit [article]

Soufiane Hayou, Arnaud Doucet, Judith Rousseau
2022 arXiv   pre-print
Tangent Kernel (NTK).  ...  in deep neural networks.  ...  Acknowledgements The project leading to this work has received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (grant agreement  ... 
arXiv:1905.13654v11 fatcat:cgo3sxljajgl5olua2yucey6om

A Neural Tangent Kernel Perspective of GANs [article]

Jean-yves Franceschi
2022 arXiv   pre-print
To this end, we leverage the theory of infinite-width neural networks for the discriminator via its Neural Tangent Kernel.  ...  We characterize the trained discriminator for a wide range of losses and establish general differentiability properties of the network.  ...  We acknowledge financial support from the DEEPNUM ANR project (ANR-21-CE23-0017-02), the ETH Foundations of Data Science, and the European Union's Horizon 2020 research and innovation programme under grant  ... 
arXiv:2106.05566v4 fatcat:gjkntcee6vgptpiek3uc6azsoi

Neural Tangent Kernel: A Survey [article]

Eugene Golikov, Eduard Pokonechnyy, Vladimir Korviakov
2022 arXiv   pre-print
., 2018] demonstrated that training a neural network under specific parameterization is equivalent to performing a particular kernel method as width goes to infinity.  ...  The present survey covers key results on kernel convergence as width goes to infinity, finite-width corrections, applications, and a discussion of the limitations of the corresponding method.  ...  In addition to experiments with a small network, we experimented with a variant of Resnet50 [He et al., 2016] .  ... 
arXiv:2208.13614v1 fatcat:65jlpyh7tzdsnfrvmdwomruiyy

Single-level Adversarial Data Synthesis based on Neural Tangent Kernels [article]

Yu-Rong Zhang, Sheng Yen Chou, Shan-Hung Wu
2022 arXiv   pre-print
Generative adversarial networks (GANs) have achieved impressive performance in data synthesis and have driven the development of many applications.  ...  This is done by modeling the discriminator as a Gaussian process with a neural tangent kernel (NTK-GP) whose training dynamics can be completely described by a closed-form formula.  ...  function k L : R d × R d → R, called the neural tangent kernel (NTK).  ... 
arXiv:2204.04090v5 fatcat:q5hmqozdkzbxrn4hxg5f5d5n3a

Deep Neural Tangent Kernel and Laplace Kernel Have the Same RKHS [article]

Lin Chen, Sheng Xu
2021 arXiv   pre-print
We prove that the reproducing kernel Hilbert spaces (RKHS) of a deep neural tangent kernel and the Laplace kernel include the same set of functions, when both kernels are restricted to the sphere 𝕊^d-  ...  Additionally, we prove that the exponential power kernel with a smaller power (making the kernel less smooth) leads to a larger RKHS, when it is restricted to the sphere 𝕊^d-1 and when it is defined on  ...  Lee, and Iosif Pinelis for helpful discussions and thank Mikhail Belkin and Alexandre Eremenko for introducing to us the works [19, 22] and [17] , respectively.  ... 
arXiv:2009.10683v5 fatcat:ln5pljmkfbeivjylimpml5zgmi

Neural Tangent Kernel Empowered Federated Learning [article]

Kai Yue, Richeng Jin, Ryan Pilgrim, Chau-Wai Wong, Dror Baron, Huaiyu Dai
2022 arXiv   pre-print
Meanwhile, recent advances in the interpretation of neural networks have seen a wide use of neural tangent kernels (NTKs) for convergence analyses.  ...  We further develop a variant with improved communication efficiency and enhanced privacy.  ...  Related Work Neural Tangent Kernel.  ... 
arXiv:2110.03681v3 fatcat:qjmeefwjhvajvik7myfbrymhka

Tensor Programs II: Neural Tangent Kernel for Any Architecture [article]

Greg Yang
2020 arXiv   pre-print
We prove that a randomly initialized neural network of *any architecture* has its Tangent Kernel (NTK) converge to a deterministic limit, as the network widths tend to infinity.  ...  Our material here presents the NTK results of Yang (2019a) in a friendly manner and showcases the *tensor programs* technique for understanding wide neural networks.  ...  Jeffrey Pennington for feedback and discussions.  ... 
arXiv:2006.14548v4 fatcat:h4oqfmmylfg6hnj42zjakaymxa

Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks [article]

Zhou Fan, Zhichao Wang
2020 arXiv   pre-print
We study the eigenvalue distributions of the Conjugate Kernel and Neural Tangent Kernel associated to multi-layer feedforward neural networks.  ...  In an asymptotic regime where network width is increasing linearly in sample size, under random initialization of the weights, and for input samples satisfying a notion of approximate pairwise orthogonality  ...  We would like to thank John Lafferty and Ganlin Song for helpful discussions regarding the Neural Tangent Kernel.  ... 
arXiv:2005.11879v3 fatcat:p5nlnglu2ve7zhxp7nka2icfle

Analyzing Finite Neural Networks: Can We Trust Neural Tangent Kernel Theory? [article]

Mariia Seleznova, Gitta Kutyniok
2022 arXiv   pre-print
Neural Tangent Kernel (NTK) theory is widely used to study the dynamics of infinitely-wide deep neural networks (DNNs) under gradient descent.  ...  We also describe a framework to study generalization properties of DNNs, in particular the variance of network's output function, by means of NTK theory and discuss its limits.  ...  In particular, it has been shown that the evolution of NN's output during gradient flow training can be captured by a so-called Neural Tangent Kernel (NTK) Θ t [Jacot et al., 2018 , Arora et al., 2019  ... 
arXiv:2012.04477v3 fatcat:svxmiqzacrdb3d6mt764uk5pdi

On the Random Conjugate Kernel and Neural Tangent Kernel

Zhengmian Hu, Heng Huang
2021 International Conference on Machine Learning  
We investigate the distributions of Conjugate Kernel (CK) and Neural Tangent Kernel (NTK) for ReLU networks with random initialization.  ...  For the residual network, in the limit that number of branches m increases to infinity and the width n remains fixed, the diagonal elements of Conjugate Kernel converge in law to a log-normal distribution  ...  Conclusion We derive the precise distributions and moments of diagonal elements of Conjugate Kernel (CK) and Neural Tangent Kernel (NTK) for ReLU networks.  ... 
dblp:conf/icml/HuH21 fatcat:zgcqkw2mdrad7d4k625h2aubmy

Neural Tangent Kernel Maximum Mean Discrepancy [article]

Xiuyuan Cheng, Yao Xie
2021 arXiv   pre-print
We present a novel neural network Maximum Mean Discrepancy (MMD) statistic by identifying a new connection between neural tangent kernel (NTK) and MMD.  ...  for kernel MMD.  ...  Acknowledgement The authors thank Alexander Cloninger and Galen Reeves for helpful discussion on the initial version of the paper, and the anonymous reviewers for helpful feedback.  ... 
arXiv:2106.03227v2 fatcat:wz3tj5tekbgwdfet5vvqjwoppi

Graph Neural Tangent Kernel: Fusing Graph Neural Networks with Graph Kernels [article]

Simon S. Du, Kangcheng Hou, Barnabás Póczos, Ruslan Salakhutdinov, Ruosong Wang, Keyulu Xu
2019 arXiv   pre-print
Compared to graph kernels, graph neural networks (GNNs) usually achieve better practical performance, as GNNs use multi-layer architectures and non-linear activation functions to extract high-order information  ...  The current paper presents a new class of graph kernels, Graph Neural Tangent Kernels (GNTKs), which correspond to infinitely wide multi-layer GNNs trained by gradient descent.  ...  Moreover, under a random initialization of parameters, the random matrix H(0) converges in probability to a certain deterministic kernel matrix, which is called Neural Tangent Kernel (NTK) Jacot et al  ... 
arXiv:1905.13192v2 fatcat:wr3k326c4fbq5kxtt2pz77gl5i

On the Neural Tangent Kernel Analysis of Randomly Pruned Wide Neural Networks [article]

Hongru Yang, Zhangyang Wang
2022 arXiv   pre-print
We study the behavior of ultra-wide neural networks when their weights are randomly pruned at the initialization, through the lens of neural tangent kernels (NTKs).  ...  Further, if we apply some appropriate scaling after pruning at the initialization, the empirical NTK of the pruned network converges to the exact NTK of the original network, and we provide a non-asymptotic  ...  Discussion and Future Direction In this paper, we apply neural tangent kernel analysis on randomly pruned neural network at the initialization.  ... 
arXiv:2203.14328v2 fatcat:d4an3enxqzeutpgfkq7s62sowm
« Previous Showing results 1 — 15 out of 7,521 results