Training Structured Neural Networks Through Manifold Identification and Variance Reduction [article]

Zih-Syuan Huang, Ching-pei Lee
<span title="2022-03-18">2022</span> <i > arXiv </i> &nbsp; <span class="release-stage" >pre-print</span>
This paper proposes an algorithm (RMDA) for training neural networks (NNs) with a regularization term for promoting desired structures. RMDA does not incur computation additional to proximal SGD with momentum, and achieves variance reduction without requiring the objective function to be of the finite-sum form. Through the tool of manifold identification from nonlinear optimization, we prove that after a finite number of iterations, all iterates of RMDA possess a desired structure identical to
more &raquo; ... hat induced by the regularizer at the stationary point of asymptotic convergence, even in the presence of engineering tricks like data augmentation and dropout that complicate the training process. Experiments on training NNs with structured sparsity confirm that variance reduction is necessary for such an identification, and show that RMDA thus significantly outperforms existing methods for this task. For unstructured sparsity, RMDA also outperforms a state-of-the-art pruning method, validating the benefits of training structured NNs through regularization.
<span class="external-identifiers"> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2112.02612v3">arXiv:2112.02612v3</a> <a target="_blank" rel="external noopener" href="https://fatcat.wiki/release/zaslevsawzdovke3oggvcfdsvy">fatcat:zaslevsawzdovke3oggvcfdsvy</a> </span>
<a target="_blank" rel="noopener" href="https://web.archive.org/web/20220325013942/https://arxiv.org/pdf/2112.02612v3.pdf" title="fulltext PDF download" data-goatcounter-click="serp-fulltext" data-goatcounter-title="serp-fulltext"> <button class="ui simple right pointing dropdown compact black labeled icon button serp-button"> <i class="icon ia-icon"></i> Web Archive [PDF] <div class="menu fulltext-thumbnail"> <img src="https://blobs.fatcat.wiki/thumbnail/pdf/b8/91/b89142e8831436ac72bbe3d513ded32035fb4dcf.180px.jpg" alt="fulltext thumbnail" loading="lazy"> </div> </button> </a> <a target="_blank" rel="external noopener" href="https://arxiv.org/abs/2112.02612v3" title="arxiv.org access"> <button class="ui compact blue labeled icon button serp-button"> <i class="file alternate outline icon"></i> arxiv.org </button> </a>