Look, investigate, and classify: A deep hybrid attention method for breast cancer classification

Bolei Xu, Jingxin Liu, Xianxu Hou, Bozhi Liu, Jon Garibaldi, Ian O. Ellis, Andy Green, Linlin Shen, Guoping Qiu

Research output: Chapter in Book or Report/Conference proceedingConference Proceedingpeer-review

25 Citations (Scopus)


One issue with computer based histopathology image analysis is that the size of the raw image is usually very large. Taking the raw image as input to the deep learning model would be computationally expensive while resizing the raw image to low resolution would incur information loss. In this paper, we present a novel deep hybrid attention approach to breast cancer classification. It first adaptively selects a sequence of coarse regions from the raw image by a hard visual attention algorithm, and then for each such region it is able to investigate the abnormal parts based on a soft-attention mechanism. A recurrent network is then built to make decisions to classify the image region and also to predict the location of the image region to be investigated at the next time step. As the region selection process is non-differentiable, we optimize the whole network through a reinforcement approach to learn an optimal policy to classify the regions. Based on this novel Look, Investigate and Classify approach, we only need to process a fraction of the pixels in the raw image resulting in significant saving in computational resources without sacrificing performances. Our approach is evaluated on a public breast cancer histopathology database, where it demonstrates superior performance to the state-of-the-art deep learning approaches, achieving around 96% classification accuracy while only 15% of original image pixels are required for computation.

Original languageEnglish
Title of host publicationISBI 2019 - 2019 IEEE International Symposium on Biomedical Imaging
PublisherIEEE Computer Society
Number of pages5
ISBN (Electronic)9781538636411
Publication statusPublished - Apr 2019
Externally publishedYes
Event16th IEEE International Symposium on Biomedical Imaging, ISBI 2019 - Venice, Italy
Duration: 8 Apr 201911 Apr 2019

Publication series

NameProceedings - International Symposium on Biomedical Imaging
ISSN (Print)1945-7928
ISSN (Electronic)1945-8452


Conference16th IEEE International Symposium on Biomedical Imaging, ISBI 2019


  • Breast cancer classification
  • Deep learning
  • Reinforcement learning
  • Visual attention

Cite this