Multi-scale fully convolutional neural networks for histopathology image segmentation: from nuclear aberrations to the global tissue architecture [article]

Rüdiger Schmitz, Frederic Madesta, Maximilian Nielsen, Jenny Krause, René Werner, Thomas Rösch
2021 Medical Image Analysis   accepted
Histopathologic diagnosis relies on simultaneous integration of information from a broad range of scales, ranging from nuclear aberrations ($\approx \mathcal{O}(0.1{\mu m})$) through cellular structures ($\approx \mathcal{O}(10{\mu m})$) to the global tissue architecture ($\gtrapprox \mathcal{O}(1{mm})$). To explicitly mimic how human pathologists combine multi-scale information, we introduce a family of multi-encoder FCNs with deep fusion. We present a simple block for merging model paths with
more » ... differing spatial scales in a spatial relationship-preserving fashion, which can readily be included in standard encoder-decoder networks. Additionally, a context classification gate block is proposed as an alternative for the incorporation of global context. Our experiments were performed on three publicly available whole-slide images of recent challenges (PAIP 2019, BACH 2020, CAMELYON 2016). The multi-scale architectures consistently outperformed the baseline single-scale U-Nets by a large margin. They benefit from local as well as global context and particularly a combination of both. If feature maps from different scales are fused, doing so in a manner preserving spatial relationships was found to be beneficial. Deep guidance by a context classification loss appeared to improve model training at low computational costs. All multi-scale models had a reduced GPU memory footprint compared to ensembles of individual U-Nets trained on different image scales. Additional path fusions were shown to be possible at low computational cost, opening up possibilities for further, systematic and task-specific architecture optimization. The findings demonstrate the potential of the presented family of human-inspired, end-to-end trainable, multi-scale multi-encoder FCNs to improve deep histopathologic diagnosis by extensive integration of largely different spatial scales.
doi:10.1016/j.media.2021.101996 pmid:33647783 arXiv:1909.10726v3 fatcat:n6wt6uj3urc7zcfn6qf6juzade