This is the official implementation of Cross-scale Multi-instance Learning for Pathological Image Diagnosis. This method is also deployed as Cross-modality attention-based multimodal fusion for patient survival prediction.
Journal Paper
Cross-scale Multi-instance Learning for Pathological Image Diagnosis
Ruining Deng, Can Cui, Lucas W. Remedios, Shunxing Bao, R. Michael Womick, Sophie Chiron, Jia Li, Joseph T. Roland, Ken S. Lau, Qi Liu, Keith T. Wilson, Yaohong Wang, Lori A. Coburn, Bennett A. Landman, and Yuankai Huo
Medical Image Analysis
Conference Paper
Cross-scale Attention Guided Multi-instance Learning for Crohn's Disease Diagnosis with Pathological Images
Ruining Deng, Can Cui, Lucas W. Remedios, Shunxing Bao, R. Michael Womick, Sophie Chiron, Jia Li, Joseph T. Roland, Ken S. Lau, Qi Liu, Keith T. Wilson, Yaohong Wang, Lori A. Coburn, Bennett A. Landman, and Yuankai Huo
MMMI 2022
Multi-modal Fusion Paper
Cross-modality Attention-based Multimodal Fusion for Non-small Cell Lung Cancer (NSCLC) Patient Survival Prediction
Ruining Deng, Nazim Shaikh, Gareth Shannon, and Yao Nie
SPIE 2024
+ We release an inference pipeline for CD classification as a single Docker.sudo docker pull ddrrnn123/cs-mil:2.0
docker run --rm -v [/Data2/CS-MIL_data]/input:/input/:ro -v [/Data2/CS-MIL_data]/output:/output --gpus all -it ddrrnn123/cs-mil:2.0
You may put your WSIs in the "input" folder and change the dirname inside of "[]" to your local root.
You can also refer the source code of the whole pipeline in run_inference.py for the step-by-step process, which are
- Step1. Get tiles (with foreground segmentation);
- Step2. Embedding the patches by SimSiam pre-trained models at different scales;
- Step3. Clustering the features;
- Step4. Get CD classification by pretrained CS-MIL models.
Analyzing high resolution whole slide images (WSIs) with regard to information across multiple scales poses a significant challenge in digital pathology. Multi-instance learning (MIL) is a common solution for working with high resolution images by classifying bags of objects (i.e. sets of smaller image patches). However, such processing is typically performed at a single scale (e.g., 20X magnification) of WSIs, disregarding the vital inter-scale information that is key to diagnoses by human pathologists. In this study, we propose a novel cross-scale MIL algorithm to explicitly aggregate inter-scale relationships into a single MIL network for pathological image diagnosis. The contribution of this paper is three-fold:
(1) A novel cross-scale MIL (CS-MIL) algorithm that integrates the multi-scale information and the inter-scale relationships is proposed;
(2) A toy dataset with scale-specific morphological features is created and released to examine and visualize differential cross-scale attention;
(3) Superior performance on both in-house and public datasets is demonstrated by our simple cross-scale MIL strategy.
In this work, we also propose a cross-modality attention-based multimodal fusion pipeline designed to integrate modality-specific knowledge for patient survival prediction in non-small cell lung cancer (NSCLC). Instead of merely concatenating or summing up the features from different modalities, our method gauges the importance of each modality for feature fusion with cross-modality relationship when infusing the multimodal features. Compared with single modality, which achieved c-index of 0.5772 and 0.5885 using solely tissue image data or RNA-seq data, respectively, the proposed fusion approach achieved c-index 0.6587 in our experiment, showcasing the capability of assimilating modality-specific knowledge from varied modalities.
(1) Run main_mixprecision.py to train SimSiam models at different scales.
(2) Run get_features_simsiam_256.py (20x) (same for 512.py (10x), 1024.py (5x)) to extract features from patches.
Run create_kmeans_features_local_singleresolution.py to get k-mean clustering results from features.
(1) Run MIL_global_Stage1_Training.py to train the model.
(2) Run MIL_global_Stage1_Testing.py to test the model.
To assess the effectiveness of the cross-scale attention mechanism, we evaluated CS-MIL using two toy datasets that represent distinct morphological patterns at different scales in digital pathology. These datasets were selected to simulate different scenarios and test the functionality of our approach.
The figure below shows the patches for training in the two datasets (Micro-anomaly dataset and Macro-anomaly dataset).
(1) The micro white crosses pattern only appear on positive patches at 20x maganification in the micro-anomaly dataset.
(2) The macro anomaly (ellipse) is easily recognized at 5x with larger visual fields in macro-anomaly dataset.
All of the patches are extracted from normal tissue samples in Unitopatho dataset. Two datasets were released to measure the generalization of the cross-scale designs for digital pathology community.
The patches and testing regions are avaliable at here.
The proposed method accurately differentiates distinctive patterns at different scales in a stable manner. Figure below displays the cross-scale attention maps at the instance level and multiple scales.
(1) For the Micro-anomaly dataset, the instance attention successfully highlights positive regions with higher attention scores in corresponding regions at 20x.
(2) For the Macro-anomaly dataset, the instance attention correctly locates ellipses instead of circles with higher attention scores at 5x.
(3) The box plots on the right panel show the attention score distribution at different scales, proving that the cross-scale attention mechanism provides reliable scores at different scales.
Run MIL_bag_generation.py to generate the bags for trainingset and validationset.
(1) Run MIL_main_DeepSurv_dataset1.py (same for dataset2.py) to train the model.
(2) Run MIL_main_DeepSurv_batch_dataset1_getattention.py (same for dataset2.py) to test the model and get the attention scores.
This code is inspired by DeepAttnMISL and AttentionDeepMIL.
@inproceedings{deng2022cross,
title={Cross-Scale Attention Guided Multi-instance Learning for Crohn’s Disease Diagnosis with Pathological Images},
author={Deng, Ruining and Cui, Can and Remedios, Lucas W and Bao, Shunxing and Womick, R Michael and Chiron, Sophie and Li, Jia and Roland, Joseph T and Lau, Ken S and Liu, Qi and others},
booktitle={International Workshop on Multiscale Multimodal Medical Imaging},
pages={24--33},
year={2022},
organization={Springer}
}
@article{deng2023cross,
title={Cross-scale Multi-instance Learning for Pathological Image Diagnosis},
author={Deng, Ruining and Cui, Can and Remedios, Lucas W and Bao, Shunxing and Womick, R Michael and Chiron, Sophie and Li, Jia and Roland, Joseph T and Lau, Ken S and Liu, Qi and others},
journal={arXiv preprint arXiv:2304.00216},
year={2023}
}
@article{deng2023cross,
title={Cross-modality Attention-based Multimodal Fusion for Non-small Cell Lung Cancer (NSCLC) Patient Survival Prediction},
author={Deng, Ruining and Shaikh, Nazim and Shannon, Gareth and Nie, Yao},
journal={arXiv preprint arXiv:2308.09831},
year={2023}
}
