Variable Selection via A Combination of theL0andL1Penalties

Yufeng Liu, Yichao Wu
2007 Journal of Computational And Graphical Statistics  
Variable selection is an important aspect of high-dimensional statistical modeling, particularly in regression and classification. In the regularization framework, various penalty functions are used to perform variable selection by putting relatively large penalties on small coefficients. The L 1 penalty is a popular choice because of its convexity, but it produces biased estimates for the large coefficients. The L 0 penalty is attractive for variable selection because it directly penalizes the
more » ... ectly penalizes the number of nonzero coefficients. However, the optimization involved is discontinuous and nonconvex, and therefore it is very challenging to implement. Moreover, its solution may not be stable. In this article, we propose a new penalty that combines the L 0 and L 1 penalties. We implement this new penalty by developing a global optimization algorithm using mixed integer programming (MIP). We compare this combined penalty with several other penalties via simulated examples as well as real applications. The results show that the new penalty outperforms both the L 0 and L 1 penalties in terms of variable selection while maintaining good prediction accuracy.
doi:10.1198/106186007x255676 fatcat:ilies6udlfdqfhgmpouafwjyzq