Differential Edit Distance: A metric for scene segmentation evaluation
In this work a novel approach to evaluating video temporal decomposition algorithms is presented. The evaluation measures typically used to this end are non-linear combinations of Precision-Recall or Coverage-Overflow, which are not metrics and additionally possess undesirable properties, such as nonsymmetricity. To alleviate these drawbacks we introduce a novel uni-dimensional measure that is proven to be metric and satisfies a number of qualitative prerequisites that previous measures do not. This measure is named Differential Edit Distance (DED), since it can be seen as a variation of the well-known edit distance. After defining DED, we further introduce an algorithm that computes it in less than cubic time. Finally, DED is extensively compared with state of the art measures, namely the harmonic means (F-Score) of Precision-Recall and Coverage-Overflow. The experiments include comparisons of qualitative properties, the time required for optimizing the parameters of scene segmentation algorithms with the help of these measures, and a user study gauging the agreement of these measures with the users’ assessment of the segmentation results. The results confirm that the proposed measure is a uni-dimensional metric that is effective in evaluating scene segmentation techniques and in helping to optimize their parameters.