AGROMTL-RICE: A SELF-SUPERVISED, GRAPH-FUSED MULTIMODAL MULTI-TASK FRAMEWORK FOR JOINT RICE DISEASE, NUTRIENT DEFICIENCY, AND SEVERITY CLASSIFICATION

Abstract

Building on our previously published AgroMTL-Rice framework (Maharajan et al., 2026), which fused a supervised EfficientNetB3 encoder with handcrafted pseudo-spectral features for joint rice disease and nutrient-deficiency classification, this study extends the architecture along four methodological axes: (i) a self-supervised Vision Transformer visual encoder, initialised from DINO pretraining and further adapted via domain-specific SimCLR contrastive pretraining on unlabeled rice-leaf images; (ii) a trainable neural spectral encoder that complements the original hand-engineered vegetation-index features; (iii) a cross-modal Graph Attention Network that replaces simple feature concatenation with learned inter-modality attention across four representation nodes (deep visual, handcrafted-spectral, learned-spectral, and texture); and (iv) an uncertainty-weighted dynamic multi-task loss that extends the original two-task formulation to a third task, lesion/deficiency severity, generated via a disclosed weak-supervision heuristic. The extended framework, AgroMTL-Rice, was implemented and evaluated on two publicly available rice-leaf image collections (a three-class disease set, n = 120, and a three-class nutrient-deficiency set, n = 1,159), which are smaller in scale and differ in composition from the source data used in the original publication. On a held-out test split, the model achieved 88.9% accuracy (macro-F1 = 0.889) on disease classification, 97.1% accuracy (macro-F1 = 0.970) on nutrient-deficiency classification, and 88.5% accuracy (macro-F1 = 0.885) on the proxy severity task. A systematic ablation across five configurations showed that the contribution of each architectural component is task-dependent rather than uniformly positive: removing the self-supervised pretraining step improved disease accuracy, while removing the graph-fusion module improved nutrient accuracy, indicating that added architectural capacity does not straightforwardly help on constrained, small-sample data.  The dynamic loss weights did not diverge meaningfully from equal weighting within the training budget used here. We report these outcomes transparently, together with Grad-CAM and SHAP-based interpretability analyses, and discuss the implications for deploying multimodal multi-task diagnostic systems under realistic, small-data smallholder-agriculture constraints.