Adaptive Biharmonic In-Painting for Sparse Acquisition Using Variance Frames

Andrew Barnum, Jun Jiao
2017 Microscopy and Microanalysis  
Dynamical adaption of sparse image acquisition processes in (scanning) transmission electron [(S)TEM] microscopes remains an important avenue for developing improved performance at reduced dose rates and acquisition times. Though more advanced algorithms, such as Bayesian dictionary in-painting or compressed sensing [1], produce high-fidelity reconstructions, they suffer from long reconstruction times unsuitable user feedback expected during on-line image acquisition. Demonstrated here is an
more » ... y-to-implement method for adaptive sparse sampling of images, aimed at providing sub-Nyquist sampling. Faster performance can be achieved through a combination of relaxed biharmonic functions over a grid weighted by acquired values and adaptive updating of the sampling matrix based on the results, allowing sparse sampling of an image without a priori knowledge of the image contents. The biharmonic equation is a fourth-order partial differential equation common in continuum mechanics, but has found use in image restoration when filling in unknown values. A routine is available for Python through scikit-image [2], and modern microscopes with scripting interfaces are capable of using this function. The developed procedure begins with 2% sampling of the frame pixels followed by an initial reconstruction is performed through biharmonic in-painting. A Sobel filter is applied twice to the resulting image, since it has been found that weighting the new values directly by the variance led to spurious artifacts and slow convergence. The second Sobel filter shifts the matrix values to regions from regions with the sharpest contrast and towards those with higher complexity. The resulting double-variance image is normalized such that the sum of all components equals unity, providing a probability matrix to weight the selection of the next 2% of the signal. All previously sampled pixel locations are set to 0 in the second Sobel matrix, preventing repeat acquisition from the same location by zeroing the pixel probabilities. Testing of the routine is shown in Figure 1 on a sample bright-field image [4] of a silicon finFET structure, providing both crystalline and amorphous structures. The test image used was 256x256 pixels, with a pixel size of 0.06 nm chosen to just permit resolution of the <110> lattice. To assess the quality of the reconstructions with regard to the fully sampled image, three metrics were computed: peak signal-to-noise ratio, structural similarity, and normalized root mean-squared error [3]. As can be seen in Fig. 1 (B) and (D), the adaptive method steadily diverges from the random sampling method after a 10% for both signal-to-noise and in the normalized mean-square error from the fully sampled image. Fig. 1 (C) shows no divergence between the sampling methods with regard to the structural similarity between the reconstructions and the fully sampled image, despite the detailed Si[110] lattice visible in the 40% frames for 1 (E) and (F).An advantage of this technique when acquiring images containing atomic lattice is that once the number of sampled pixels passes 15% of the total frame area, detectable patches of lattice become discernible. Assigning a cut-off parameter based on the intensity of lattice reflections in the image FFT allows the acquisition process to be automatically stopped once a sufficient quantity of atomic patching information has been recorded. The data can then be passed off to more powerful 148
doi:10.1017/s1431927617001428 fatcat:uawba6tjhrf25bjsi7igvwo3ca