Stochastic Variance-Reduced Cubic Regularization for Nonconvex Optimization [article]

Zhe Wang, Yi Zhou, Yingbin Liang, Guanghui Lan
2018 arXiv   pre-print
Cubic regularization (CR) is an optimization method with emerging popularity due to its capability to escape saddle points and converge to second-order stationary solutions for nonconvex optimization. However, CR encounters a high sample complexity issue for finite-sum problems with a large data size. complexity. In this paper, we propose a stochastic variance-reduced cubic-regularization (SVRC) method under random sampling, and study its convergence guarantee as well as sample complexity. We
more » ... ow that the iteration complexity of SVRC for achieving a second-order stationary solution within ϵ accuracy is O(ϵ^-3/2), which matches the state-of-art result on CR types of methods. Moreover, our proposed variance reduction scheme significantly reduces the per-iteration sample complexity. The resulting total Hessian sample complexity of our SVRC is (N^2/3ϵ^-3/2), which outperforms the state-of-art result by a factor of O(N^2/15). We also study our SVRC under random sampling without replacement scheme, which yields a lower per-iteration sample complexity, and hence justifies its practical applicability.
arXiv:1802.07372v2 fatcat:ftxphrtjirdoflxqfnr5v2q45a