Customizing SAM for histologic image segmentation of kidney biopsy by a detector approach: det-SAM

In recent years, large models have manifested unparalleled strengths in various fields and became a major trend in AI advancement. In the field of computer vision, the Segment Anything Model (SAM) has surpassed the majority of conventional models with its high generalization ability and profound understanding of the notion of 'object'. However, despite its outstanding capacity in common semantic segmentation tasks, its performance in medical image segmentation is previously proven unsatisfactory by many. For the purpose of unraveling SAM's untapped potentials in medical image segmentation, we introduce an innovative detector approach and a corresponding model: det-SAM. This model is characterized by its detection head in the architecture, which provides additional domain information to SAM through prompt engineering in medical segmentation tasks. This design allows det-SAM to utilize SAM's advantages in graphical discernibility and achieve the foremost accuracy in three segmentation tasks on histological images of kidney biopsy. In addition, our model is proficient in handling multifarious sizes and compositions of medical images, and the computational requirement of the model's training process is significantly lower than other customization approaches of SAM.