HISTOMOLECULAR NET: A MULTI-SCALE DISENTANGLED AND CROSS-MODAL FRAMEWORK FOR PRECISION CANCER DIAGNOSIS

Abstract

Cancers are highly heterogeneous and have varying prognostic outcomes, and clear classification is very essential in the patient stratification and to make clinical decisions. Although digital pathology has improved cancer diagnosis, modern day pathology seems to be no longer based on the sole use of histological characteristics but rather on the combination of molecular characterization. In order to meet this paradigm shift, we suggest a new framework of digital pathology that can simultaneously predict histology features and molecular markers and explicitly model the interaction between them. Multi-scale disentangling module is proposed to produce complementary representations with cellular level high magnification and tissue level low-magnification whole slide images. Expanding on these characteristics, an attention based hierarchical multi-task multi-instance learning framework is developed to do simultaneous histology and molecular prediction. Besides, it uses a co-occurrence probability-based label correlation graph and a cross-modal interaction module of dynamical confidence constraint and gradient modulation. The results of experiments show a high level of performance in comparison with state-of-the-art approaches, which indicates the potential for improving diagnostic accuracy in oncology of the framework.