Subject : Evaluation of Image Interpretation
PhD defence on December 2, 2009, PhD thesis manuscript (French)
Jury Members :
Image processing algorithms include a set of methods that process the image from its acquisition by a sensor (camera, satellite, ultrasound…) to the extraction of useful information for a given application (detection of a particular object, quantitative measure…). Among these algorithms, some are dedicated to detect, locate and identify one or more objects in an image. Given the challenges associated with extracting this information (including military or medical applications), it is particularly important that results provided by image interpretation algorithms are as relevant as possible. The problem addressed in this thesis is the evaluation of interpretation results of an image or a video, given the associated ground truth. The challenges are multiple such as the comparison of algorithms for a particular application, the evaluation of an algorithm during its development or its optimal setting. There are two major steps in image interpretation which are the localization and recognition. Many methods and metrics have been proposed in the literature to evaluate localization or recognition results in several competitions (Technovision, Pascal…) or conferences (PETS, ECCV…). However, it remains difficult to estimate the relevance of a metric and even to determine which should be advocated for a given application.
We propose in this thesis a formalization of the expected properties of a localization metric. We perform a rigorous comparative study of localization metrics of the state from the art considering these properties. This study has allowed a precise characterization of the metrics behavior. We perform a similar work on recognition methods using a local representation of objects in order to quantify a recognition error: a result leading to an erroneous object recognition as an object of class cat instead of class dog must, for example, be considered better compared to the same object affected to the class car.
We have also developed an evaluation method of an image interpretation result making use of the lessons from these comparative studies. The advantage of the proposed method is to evaluate an image interpretation result taking into account both the quality of localization, recognition and detection of objects of interest in the image. A matching process between the ground truth and the evaluated interpretation result is realized. Each matched object contributes to the overall score by taking into account localization and recognition errors. The overall score takes into account over-detection and under-detection of objects. The behavior of this method was tested on a benchmark (from the PASCAL database) and presents some interesting results. While respecting the properties used in the comparative study, this new method, thanks to several settings, obtains an appropriate behavior in the intended application, taking into particular account the matching mode or the relative importance of localization and recognition in the computation of the overall score.