Contrastive visual language pretraining has emerged as a powerful method for either training new language-aware image encoders or augmenting existing pretrained models with zero-shot visual recognition capabilities. However, existing works typically train on large datasets of image-text pairs and have been designed to perform downstream tasks involving only small to medium sized-images, neither of which are applicable to the emerging field of computational pathology where there are limited publicly available paired image-text datasets and each image can span up to 100,000 x 100,000 pixels in dimensions. In this paper we present MI-Zero, a simple and intuitive framework for unleashing the zero-shot transfer capabilities of contrastively aligned image and text models to gigapixel histopathology whole slide images, enabling multiple downstream diagnostic tasks to be carried out by pretrained encoders without requiring any additional labels. MI-Zero reformulates zero-shot transfer under the framework of multiple instance learning to overcome the computational challenge of inference on extremely large images. We used over 550k pathology reports and other available in-domain text corpora to pretrain our text encoder. By effectively leveraging strong pretrained encoders, our best model pretrained on over 33k histopathology image-caption pairs achieves an average median zero-shot accuracy of 70.2% across three different real-world cancer subtyping tasks. Our code is available at: https://github.com/mahmoodlab/MI-Zero.
@inproceedings{lu2023visual,title={Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images},author={Lu*, Ming Y and Chen*, Bowen and Zhang, Andrew and Williamson, Drew FK and Chen, Richard J and Ding, Tong and Le, Long Phi and Chuang, Yung-Sung and Mahmood, Faisal},booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},pages={19764--19775},year={2023},}
2022
Augmented Features Synergize Radiomics in Post-Operative Survival Prediction and Adjuvant Therapy Recommendation for Non-Small Cell Lung Cancer
Lawrence Wing-Chi Chan, Tong Ding, Huiling Shao, Mohan Huang, William Fuk-Yuen Hui, William Chi-Shing Cho, and
5 more authors
Contrastive visual language pretraining has emerged as a powerful method for either training new language-aware image encoders or augmenting existing pretrained models with zero-shot visual recognition capabilities. However, existing works typically train on large datasets of image-text pairs and have been designed to perform downstream tasks involving only small to medium sized-images, neither of which are applicable to the emerging field of computational pathology where there are limited publicly available paired image-text datasets and each image can span up to 100,000 x 100,000 pixels in dimensions. In this paper we present MI-Zero, a simple and intuitive framework for unleashing the zero-shot transfer capabilities of contrastively aligned image and text models to gigapixel histopathology whole slide images, enabling multiple downstream diagnostic tasks to be carried out by pretrained encoders without requiring any additional labels. MI-Zero reformulates zero-shot transfer under the framework of multiple instance learning to overcome the computational challenge of inference on extremely large images. We used over 550k pathology reports and other available in-domain text corpora to pretrain our text encoder. By effectively leveraging strong pretrained encoders, our best model pretrained on over 33k histopathology image-caption pairs achieves an average median zero-shot accuracy of 70.2% across three different real-world cancer subtyping tasks. Our code is available at: https://github.com/mahmoodlab/MI-Zero.
@article{chan2022augmented,title={Augmented Features Synergize Radiomics in Post-Operative Survival Prediction and Adjuvant Therapy Recommendation for Non-Small Cell Lung Cancer},author={Chan, Lawrence Wing-Chi and Ding, Tong and Shao, Huiling and Huang, Mohan and Hui, William Fuk-Yuen and Cho, William Chi-Shing and Wong, Sze-Chuen Cesar and Tong, Ka Wai and Chiu, Keith Wan-Hang and Huang, Luyu and others},journal={Frontiers in Oncology},volume={12},pages={659096},year={2022},publisher={Frontiers}}
