S18 - Session O1 - Combining 3D structural features and multimodal fusion data to correct illumination effect of plant multispectral point clouds

Authors: Pengyao Xie *, Haiyan Cen

Plant spectral response and three-dimensional topological relationships can be obtained by analyzing the morphological structure and physiological information contained in the three-dimensional multispectral point cloud of plants through close-range remote sensing. This information can be used to monitor plant growth dynamics, evaluate plant agronomic traits and provide reference indicators, as well as reveal the interactions among plant phenotypes, genotypes and environmental factors. Currently there is no commercially available and mature plant-oriented multispectral point cloud sensor. In this study, we proposed a device and a supplementary method for obtaining single-frame plant multispectral point clouds based on multimodal image registration. To eliminate the local differences in plant multispectral reflectance images caused by the combination of illumination inhomogeneity and different positions of plant leaves, we designed an artificial neural network model with three-dimensional structural features as input, and determines the optimal number of input features for this neural network. The results show that compared with some classical image registration algorithms tested in this study, the proposed joint registration algorithm has a better registration performance for plant RGB images and multispectral images acquired by different sensors, with an average SSIM of 0.931, which is 4.72% better than the average registration performance of other classical algorithms (SSIM = 0.889). The spatial distribution of the reference object reflectance is calculated with high accuracy using artificial neural networks (R 2 = 0.962, RMSE = 0.036). Compared with the ground truth, the average RMSE of the spectra before and after correction at different leaf measurement points decreased from 0.182 to 0.040, a decrease of 78.0%; the spectral range(Max-Min) obtained from multi-view measurements decreased from 0.140 to 0.055, a decrease of 60.7%. The proposed method of combining three-dimensional structure features for correction of multispectral images in this study is optimized for proximally acquired multispectral images of plants.