An Equivalence Between Sparse Approximation and Support Vector Machines

Federico Girosi
1998 Neural Computation  
This paper shows a relationship between two di erent approximation techniques: the Support Vector Machines (SVM), proposed by V . V apnik (1995), and a sparse approximation scheme that resembles the Basis Pursuit De-Noising algorithm (Chen, 1995 Chen, Donoho and Saunders, 1995). SVM is a technique which can be derived from the Structural Risk Minimization Principle (Vapnik, 1982) and can be used to estimate the parameters of several di erent approximation schemes, including Radial Basis
more » ... adial Basis Functions, algebraic/trigonometric polynomials, B-splines, and some forms of Multilayer Perceptrons. Basis Pursuit De-Noising is a sparse approximation technique, in which a function is reconstructed by using a small number of basis functions chosen from a large set (the dictionary). We show that, if the data are noiseless, the modi ed version of Basis Pursuit De-Noising proposed in this paper is equivalent t o SVM in the following sense: if applied to the same data set the two techniques give the same solution, which is obtained by solving the same quadratic programming problem. In the appendix we also present a derivation of the SVM technique in the framework of regularization theory, rather than statistical learning theory, establishing a connection between SVM, sparse approximation and regularization theory.
doi:10.1162/089976698300017269 pmid:9698353 fatcat:gcsqwivvx5hirncjq7cki6swci