Implicit Regularization via Neural Feature Alignment [article]

Aristide Baratin, Thomas George, César Laurent, R Devon Hjelm, Guillaume Lajoie, Pascal Vincent, Simon Lacoste-Julien
2021 arXiv   pre-print
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. This can be interpreted as a combined mechanism of feature selection and compression. By extrapolating a new analysis of Rademacher complexity bounds for linear models, we motivate and study a heuristic complexity measure
more » ... at captures this phenomenon, in terms of sequences of tangent kernel classes along optimization paths.
arXiv:2008.00938v3 fatcat:xtcsbf4kcnbn3itixjq3ddrwhy