In-place Activated BatchNorm for Memory-Optimized Training of DNNs

Samuel Rota Bulo, Lorenzo Porzi, Peter Kontschieder
2018 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition  
In this work we present In-Place Activated Batch Normalization (INPLACE-ABN) -a novel approach to drasti cally reduce the training memory footprint of modern deep neural networks in a computationally efficient way. Our solution substitutes the conventionally used succession of BatchNorm + Activation layers with a single plugin layer, hence avoiding invasive framework surgery while providing straightforward applicability for existing deep learning frameworks. We obtain memory savings of up to
more » ... by dropping intermediate results and by recovering required information during the backward pass through the inversion of stored forward results, with only minor increase (0.8-2%) in computation time. Also, we demonstrate how frequently used checkpointing approaches can be made computationally as efficient as INPLACE-ABN. In our experiments on image classification, we demonstrate on-par results on ImageNet-1k with state-of-the-art approaches. On the memory-demanding task of semantic segmentation, we report competitive results for COCO-Stuff and set new state-of-the-art results for Cityscapes and Mapillary Vistas. Code can be found at https://github.com/ mapillary/inplace_abn.
doi:10.1109/cvpr.2018.00591 dblp:conf/cvpr/BuloPK18 fatcat:inqltbivvjcurecpbc7fca24cy