MFISN: Modality Fuzzy Information Separation Network for Disease Classification

Most of the previous machine learning-based models for multi-modal medical diagnosis, primarily designed for unimodal images, usually do not fully leverage the potential of multimodal medical images, leading to limited classification accuracy. These conventional methods typically focus only on the intermodality common information, neglecting the intra-modality specific information and assuming that the common information is more effective in disease diagnosis. Moreover, they do not adequately address the impact of fuzzy information between different medical imaging modalities on diagnostic results. To this end, we propose a Modality Fuzzy Information Separation Network for disease classification, which extracts both common and specific information from fuzzy information to construct a comprehensive representation of multi-modal medical images. Specifically, we extract modality invariant features as common information by explicitly modeling and maximizing loss constraints on mutual information. For specific information extraction, a constraint on feature space independence between specific and common information is imposed on each modality. Above two steps, we concatenate common information and specific information to construct a comprehensive multi-modal representation for separating fuzzy information. Finally, we purposely design a decoder network to reconstruct medical images from uni-modal specific information and common information to demonstrate the effectiveness of the modality fuzzy information separation network. We conducted a validation of the proposed method's performance in classifying cardiomegaly, pneumothorax, edema, and skin disease. The experimental results substantiate the effectiveness of our proposed approach.