Sharper Bounds for Regularized Data Fitting *

Haim Avron, Kenneth Clarkson, David Woodruff
We study matrix sketching methods for regularized variants of linear regression, low rank approximation , and canonical correlation analysis. Our main focus is on sketching techniques which preserve the objective function value for regularized problems, which is an area that has remained largely unexplored. We study regularization both in a fairly broad setting, and in the specific context of the popular and widely used technique of ridge regularization; for the latter , as applied to each of
more » ... pplied to each of these problems, we show algorithmic resource bounds in which the statistical dimension appears in places where in previous bounds the rank would appear. The statistical dimension is always smaller than the rank, and decreases as the amount of regulariza-tion increases. In particular, for the ridge low-rank approximation problem min Y,X Y X − A 2 F + λY 2 F + λX 2 F , where Y ∈ R n×k and X ∈ R k×d , we give an approximation algorithm needing O(nnz(A)) + ˜ O((n + d)ε −1 k min{k, ε −1 sd λ (Y *)}) + poly(sd λ (Y *) −1) time, where s λ (Y *) ≤ k is the statistical dimension of Y * , Y * is an optimal Y , ε is an error parameter, and nnz(A) is the number of nonzero entries of A. This is faster than prior work, even when λ = 0. We also study regularization in a much more general setting. For example, we obtain sketching-based algorithms for the low-rank approximation problem min X,Y Y X − A 2 F + f (Y, X) where f (·, ·) is a regularizing function satisfying some very general conditions (chiefly, invariance under orthogonal transformations).