Direct Zero-Norm Minimization for Neural Network Pruning and Training [chapter]

S. P. Adam, George D. Magoulas, M. N. Vrahatis
2012 Communications in Computer and Information Science  
Designing a feed-forward neural network with optimal topology in terms of complexity (hidden layer nodes and connections between nodes) and training performance has been a matter of considerable concern since the very beginning of neural networks research. Typically, this issue is dealt with by pruning a fully interconnected network with "many" nodes in the hidden layers, eliminating "superfluous" connections and nodes. However the problem has not been solved yet and it seems to be even more
more » ... evant today in the context of deep learning networks. In this paper we present a method of direct zero-norm minimization for pruning while training a Multi Layer Perceptron. The method employs a cooperative scheme using two swarms of particles and its purpose is to minimize an aggregate function corresponding to the total risk functional. Our discussion highlights relevant computational and methodological issues of the approach that are not apparent and well defined in the literature.
doi:10.1007/978-3-642-32909-8_30 fatcat:pc64552wn5fnpjojsiqletiqvi