Learning the Structure of Deep Architectures Using L1 Regularization

Praveen Kulkarni, Joaquin Zepeda, Frederic Jurie, Patrick Pérez, Louis Chevallier
2015 Procedings of the British Machine Vision Conference 2015  
Figure 1 : Proposed deep processing pipeline. Given an image representation, e.g., the output of the convolutional part of a pre-trained stateof-art DNN, J fully connected layers, each involving a diagonal matrix that controls its effective dimensions, are jointly learned with final linear SVM classifiers. Our proposed approach is illustrated in Fig. 1 . The architecture we consider consists of a sequence of fully-connected layers, with a diagonal matrix between them. We present a method that
more » ... tomatically selects the size of the weight matrices inside fully-connected layers indexed by j = 1, . . . , J. Our approach relies on a regularization penalty term consisting of the 1 norm of the diagonal entries of diagonal matrices inserted between the fully-connected layers. Using such a penalty term forces the diagonal matrices to be sparse, accordingly selecting the effective number of rows and columns in the weights matrices of adjacent layers. We present a simple algorithm to solve the proposed formulation and demonstrate it experimentally on a standard image classification benchmark. We can express the architecture in Fig.
doi:10.5244/c.29.23 dblp:conf/bmvc/KulkarniZJPC15 fatcat:5446jqhc4jesnkh27axnhb5vmi