Learning 3D Semantic Reconstruction on Octrees

Xiaojuan Wang, Marc Pollefeys, Martin Oswald, Ian Cherabier
We present a fully convolutional neural network for semantic 3D reconstruction by using octree representation. Our network embeds variational regularization and octree splitting together to predict the semantic label for each voxel in a coarse-to-fine manner. In each resolution level, the network performs numbers of unrolled iterations of variational optimization with shared weights and then learns to propagate and split set of certain voxels for optimization in next resolution level. In
more » ... t to previous networks that work on dense voxel grids, our network is much more efficient in terms of memory consumption and inference efficiency, while achieves almost equivalent reconstruction performance. This allows a high resolution reconstruction in case of limited memory. We perform experiments on the synthetic SUNCG dataset and the recently released ScanNetv2 dataset, and our network shows comparable reconstruction results compared with the corresponding dense network while consuming less memory. i Acknowledgement First and foremost, my great thanks go to Dr. Martin R. Oswald and Ian Cherabier. They always give me any resources I need for my research without any reservation. Because of the memory requirement of this project, I can have two Nvidia Quadro GV100 cards at the same time! Most of all, they taught me three important lessons about research: 1) Learn to schedule your research progress efficiently. Though research is to explore uncharted territory, it does not mean brute-force traversal. A well-trained researcher should use his priors and heuristics to reduce search space, so that it can be done much more efficient. 2) Have the faith to solve it. This master thesis project involves almost every aspect of the 3D neural network, from low-level implementation such as convolution on octree in GPU to high-level reconstruction performance evaluation. When I encounter some problems I feel I could not solve them, Martin taught me to analyze what happened and gave me the faith that this can be solved, and just give a try. This help me release my anxieties and gain confidence to continue working and concentrate when get stuck. 3) Keep thinking and never give up. Sometimes we might be frustrated for no progress and have no idea of what to do next. However, there might be some corners we forgot, keep thinking and you will later find those corners which might lead to solutions. I also would like to give big thanks to my boyfriend Tong He for his unconditional love and support. He always believes in me and encourages me to pursue my dream. When I was stuck in the research and became anxious, he always can feel the empathy. I remember once when I was stuck on the GPU implementation of convolution on octree, he offered me useful suggestions which were consensus in system area but were ignored by me. He also gave me huge encouragement and confidence on finishing the framework. In a word, I thank him for everything he has done for me. ii
doi:10.3929/ethz-b-000332863 fatcat:2j3ywucwlbb4dihks5ijlqwo3m