A Multimodal Multi-Task Fusion Framework For Simultaneous Rice Paddy Disease And Nutrient Deficiency Detection

Abstract

Rice (Oryzasativa L.) is the primary caloric staple for over 3.5 billion people globally; however, annual yield losses attributable to fungal, bacterial, and viral diseases combined with macro- and micro-nutrient deficiencies consistently reduce global production by 20–40%. Existing deep learning approaches address either disease classification or nutrient deficiency detection in isolation, precluding joint field diagnosis and increasing deployment overhead for resource-constrained agricultural settings. In this paper, we propose AgroMTL-Rice, a multimodal multi-task learning framework that simultaneously addresses both diagnostic challenges within a unified architecture. The system fuses an EfficientNetB3 deep visual encoder with a 32-dimensional handcrafted multimodal sensor feature branch comprising RGB-derived vegetation indices (ExG, ExR, NGRDI, VARI, GLI), HSV physiology descriptors, chlorosis and necrosis proxy masks, and Gray-Level Co-occurrence Matrix (GLCM) texture statistics. Dual attention gating mechanisms enable symptom-disentangled feature routing to task-specific classification heads. It achieves 97.1% accuracy (F1 = 0.970) with paddy disease classification on the combined dataset (4,800+ rice leaf images) from two public benchmark repositories, 4.2 percentage points better than the next best single-task baseline (ViT-B/16) with nutrient deficiency classification (F1 = 0.965). The pseudo-spectral and texture fusion gives a cumulative accuracy improvement of 5.9% on top of the baseline model trained with only RGB data, confirmed by comprehensive ablation analysis. The findings show that AgroMTL-Rice is a promising solution for integrating an in situ diagnostic system for precision agriculture in the field.