S18 - Session P1 - Exploratory analysis on pixel-wise evaluation metrics for plant segmentation

Authors: Paul Melki *, Jean-Pierre Da Costa, Lionel Bombrun, Estelle Millet, Boubacar Diallo, Hakim ElChaoui

A considerable number of metrics can be used to evaluate the performance of machine learning algorithms. While much work is dedicated to the study and improvement of data quality and models' performance, much less research is focused on the study of these evaluation metrics, their intrinsic relationship, the interplay of influence among the metrics, the models, the data and the environments and conditions in which they are to be applied. While some works have been conducted on general machine learning tasks like classification, fewer efforts have been dedicated to more complex problems such as object detection and image segmentation, in which the evaluation of performance can vary drastically depending on the objectives and domains of application. Working in an agricultural context, specifically on the problem of automatic detection of plants using image segmentation models, we present and study 12 common and less-common evaluation metrics, which we use to evaluate three image segmentation models on the same train and test sets of images. Within an exploratory framework, we study the relationship among these 12 metrics using correlation analysis, factorial analysis, specifically Principal Component Analysis, and clustering of variables. We identify three distinct groups of highly linked metrics, each emphasizing a different aspect of the quality of segmentation, which are in alignment with both the theoretical definitions of the metrics, and human visual inspection. Finally, we provide interpretations of these metrics in the context of precision agriculture and some clues to practitioners for understanding and choosing the metrics that are most relevant to their agricultural task.