Geometric insights into support vector machine behavior using the KKT conditions

Iain Carmichael, J. S. Marron
2021 Electronic Journal of Statistics  
The support vector machine (SVM) is a powerful and widely used classification algorithm. This paper uses the Karush-Kuhn-Tucker conditions to provide rigorous mathematical proof for new insights into the behavior of SVM. These insights provide unexpected relationships between SVM and two other linear classifiers: the mean difference and the maximal data piling direction. For example, we show that in many cases SVM can be viewed as a cropped version of these classifiers. By carefully exploring
more » ... ese connections we show how SVM tuning behavior is affected by data characteristics including: balanced vs. unbalanced classes, low vs. high dimension, separable vs. non-separable data. These results provide further insights into tuning SVM via cross-validation by explaining observed pathological behavior and motivating improved cross-validation methodology. MSC2020 subject classifications: Primary 62H99.
doi:10.1214/21-ejs1902 fatcat:ted7hrzu35c5xpz5n4bf5jud6a