ROME: Robustifying Memory-Efficient NAS via Topology Disentanglement and Gradients Accumulation [article]

Xiaoxing Wang and Xiangxiang Chu and Yuda Fan and Zhexi Zhang and Xiaolin Wei and Junchi Yan and Xiaokang Yang
2020 arXiv   pre-print
Single-path based differentiable neural architecture search has great strengths for its low computational cost and memory-friendly nature. However, we surprisingly discover that it suffers from severe searching instability which has been primarily ignored, posing a potential weakness for a wider application. In this paper, we delve into its performance collapse issue and propose a new algorithm called RObustifying Memory-Efficient NAS (ROME). Specifically, 1) for consistent topology in the
more » ... h and evaluation stage, we involve separate parameters to disentangle the topology from the operations of the architecture. In such a way, we can independently sample connections and operations without interference; 2) to discount sampling unfairness and variance, we enforce fair sampling for weight update and apply a gradient accumulation mechanism for architecture parameters. Extensive experiments demonstrate that our proposed method has strong performance and robustness, where it mostly achieves state-of-the-art results on a large number of standard benchmarks.
arXiv:2011.11233v1 fatcat:a7eup6odhjgspbmkeyhnddyide