A Review of Image Fusion Algorithms Based on the Super-Resolution Paradigm

Andrea Garzelli
2016 Remote Sensing  
A critical analysis of remote sensing image fusion methods based on the super-resolution (SR) paradigm is presented in this paper. Very recent algorithms have been selected among the pioneering studies adopting a new methodology and the most promising solutions. After introducing the concept of super-resolution and modeling the approach as a constrained optimization problem, different SR solutions for spatio-temporal fusion and pan-sharpening are reviewed and critically discussed. Concerning
more » ... ssed. Concerning pan-sharpening, the well-known, simple, yet effective, proportional additive wavelet in the luminance component (AWLP) is adopted as a benchmark to assess the performance of the new SR-based pan-sharpening methods. The widespread quality indexes computed at degraded resolution, with the original multispectral image used as the reference, i.e., SAM (Spectral Angle Mapper) and ERGAS (Erreur Relative Globale Adimensionnelle de Synthèse), are finally presented. Considering these results, sparse representation and Bayesian approaches seem far from being mature to be adopted in operational pan-sharpening scenarios. Introduction Recent trends in image fusion, including remote sensing applications, involve the super-resolution (SR) paradigm and, more generally, apply constrained optimization algorithms to solve the ill-posed problem of spectral-spatial (pan-sharpening) and spatio-temporal image resolution enhancement. Specifically, pan-sharpening denotes the merging of a monochrome image acquired by a broadband panchromatic (Pan) instrument with a multispectral (MS) image featuring a spectral diversity of bands and acquired over the same area, with a spatial resolution greater for the former. This can be seen as a particular problem of data fusion, in which the goal is to combine the spatial details resolved by the Pan instrument, but not by the MS scanner, and the spectral diversity of the MS image, against the single band of Pan, into a unique product. The most commonly-encountered case is when both the MS and Pan datasets are available at the two dates. However, multitemporal pan-sharpening denotes the process by which MS and Pan datasets that are used to perform the data fusion task are acquired from the same platform, but at different times or from different platforms. In the latter case, we may talk of multi-platform pan-sharpening. A typical application scenario is when either of the platforms mounts only one of the MS and Pan instruments, for example CartoSat-1 (Pan geocoded at 2.5 m) and RapidEye (MS geocoded at 5 m). In this case, pan-sharpening is multi-platform and is most likely to be also multitemporal [1] . The majority of pan-sharpening methods may be labeled as spectral or spatial. In spectral methods, geometric details are extracted from the Pan image by subtracting from it an intensity image obtained by a spectral transformation of the MS bands. In spatial methods, geometric details are extracted from the Pan image by subtracting from it a low-pass version of Pan obtained by means of linear shift-invariant digital filters. Finally, for both approaches, the geometric details are injected into the MS bands interpolated at the scale of the panchromatic band. Spectral methods [2-10] are traditionally known as component-substitution (CS), though explicit calculation of the spectral transform, and its inverse may not be necessary. Spatial methods [11] [12] [13] [14] [15] [16] [17] [18] may be contextualized within multiresolution analysis (MRA), though in most cases, a unique low-pass filter is required [19] . This hard categorization is brought back to previous studies [20, 21] , in which it is proven that there exists a duality between the classes of spectral and spatial methods featuring complementary properties of robustness to spatial and spectral impairments, respectively. Super-resolution fusion methods form a new third class of spectral-spatial (pan-sharpening) and spatio-temporal image resolution enhancement algorithms. Conventional approaches to generating an SR image normally require inputting multiple spatial/spectral/temporal low-resolution images of the same scene. The SR task is cast as the inverse problem of recovering the original high-resolution image by fusing the low-resolution images, based on reasonable assumptions or prior knowledge about the observation model that maps the high-resolution image into the low-resolution ones. The fundamental reconstruction constraint for SR is that the recovered image, after applying the same generation model, should reproduce the observed low-resolution images. However, SR image reconstruction is generally a severely ill-posed problem because of the insufficient number of low-resolution images, ill-conditioned registration and unknown blurring operators, and the solution from the reconstruction constraint is not unique. Various regularization methods have been proposed to further stabilize the inversion of this ill-posed problem [22] . A similar approach considers image fusion as a restoration problem. The aim is therefore to reconstruct the original scene from a degraded observation, or, equivalently, to solve a classical deconvolution problem [23, 24] . As an example of possible application fields, these methods may solve the classical strip-line degradation problem in satellite optical imagery, e.g., Landsat 7ETM+, MODIS, etc. [25]. Prior knowledge is required on the nature of the two-dimensional convolution that models the band-dependent point spread function of the imaging system. There is a spectral model between the Pan channel and the MS channels of the same sensor, notwithstanding that the corresponding images feature different spatial resolutions, that is spatial frequency contents. Such a model is well embodied by the plots of the spectral responsivities of the individual channels of the complete sensor (MS and Pan instruments mounted on the same platform) or, in the most general case, the spectral responses of different MS + Pan sensors. While individual narrowband channels (e.g., B, G, R and NIR) approximately cover the same wavelength intervals, the bandwidth of Pan may significantly vary from one instrument to another. Older instruments, like SPOT 1-3 and 5, featured narrowband Pan (approximately spanning through 500-700 nm). Modern very high resolution (VHR) and extremely high resolution (EHR) MS scanners are generally equipped with a broadband Pan instrument covering the wavelengths from 450 nm-800 nm or even 900 nm [26] . Bayesian methods and variational methods have been also proposed in the last decade, with different possible solutions that are based on specific assumptions that make the problem mathematically tractable [27] [28] [29] [30] [31] . The paper reviews the concept of super-resolution in an image fusion framework, by resorting to the theoretical interpretation of image super-resolution as a constrained optimization problem. Different SR solutions for spatio-temporal fusion and pan-sharpening are reviewed and critically discussed. The distinctive feature of the paper is the reviewed methodology, i.e., the super-resolution paradigm, which includes constrained-optimization solutions, sparse representation methods and Bayesian restoration approaches. The broad application field is remote sensing image fusion, while specific applications, i.e. spatio-temporal fusion, fusion with missing data (destriping/denoising) and pan-sharpening have been reviewed within the common framework of the adopted methodology. Finally, pan-sharpening has been selected for objective assessment on SR-based methods with respect to the simple, classical, widespread proportional additive wavelet in the luminance (AWLP) method, which serves as a benchmark for clear and immediate comparisons.
doi:10.3390/rs8100797 fatcat:5hyaoamdx5gcbj3dvwp2iv4oc4