A Generic Framework for Assessing the Performance Bounds of Image Feature Detectors

Shoaib Ehsan, Adrian Clark, Ales Leonardis, Naveed ur Rehman, Ahmad Khaliq, Maria Fasli, Klaus McDonald-Maier
2016 Remote Sensing  
Since local feature detection has been one of the most active research areas in computer vision during the last decade and has found wide range of applications (such as matching and registration of remotely sensed image data), a large number of detectors have been proposed. The interest in feature-based applications continues to grow and has thus rendered the task of characterizing the performance of various feature detection methods an important issue in vision research. Inspired by the good
more » ... actices of electronic system design, a generic framework based on the repeatability measure is presented in this paper that allows assessment of the upper and lower bounds of detector performance and finds statistically significant performance differences between detectors as a function of image transformation amount by introducing a new variant of McNemar's test in an effort to design more reliable and effective vision systems. The proposed framework is then employed to establish operating and guarantee regions for several state-of-the art detectors and to identify their statistical performance differences for three specific image transformations: JPEG compression, uniform light changes and blurring. The results are obtained using a newly acquired, large image database (20,482 images) with 539 different scenes. These results provide new insights into the behavior of detectors and are also useful from the vision systems design perspective. Finally, results for some local feature detectors are presented for a set of remote sensing images to showcase the potential and utility of this framework for remote sensing applications in general. Remote Sens. 2016, 8, 928 2 of 35 the current-voltage characteristics of a diode are also dependent upon temperature. Thus, looking at the datasheets of the required electronic components for this power supply would be a logical step for finding devices that operate reliably in extremely low temperatures. Only those components would be selected which show stable operating characteristics across the required range of temperatures to ensure that the final output of the power supply would satisfy the initial design specifications. Now come back to the computer vision world and design a simple toy car tracking system with local feature detection as its primary stage while expecting only 20% uniform decrease in illumination. Looking at the repeatability results presented in [1] (which are widely considered the most comprehensive) for the Leuven dataset (which involves uniform changes in light) [2], MSER detector [3] appears to be the best option for achieving a reasonable value of repeatability (more than 60%) for this small transformation amount. Now consider two sample images shown in Figure 1 which the designed vision system would encounter when deployed in the actual environment. The first image is the reference image and the second image has undergone 20% uniform decrease in light relative to the reference. Theoretically speaking, the feature detection unit (based on MSER) of the designed vision system would achieve high repeatability score for this negligible image transformation. As it turns out, MSER only manages a repeatability value of 28.17% for the image pair shown, which is much less than what is expected of the feature detection unit and highlights its unreliable behavior-a stark contrast to the power supply design example.
doi:10.3390/rs8110928 fatcat:zsznn53a3bbftghyndwgvnrt5u