A Template-Based 3D Reconstruction of Colon Structures and Textures from Stereo Colonoscopic Images

This article presents a framework for 3D reconstruction of colonic surface using stereo colonoscopic images. Due to the limited overlaps between consecutive frames and the nonexistence of large loop closures during a normal screening colonoscopy, the state-of-art simultaneous localization and mapping (SLAM) algorithms cannot be directly applied to this scenario, thus a colon model segmented from CT scans is used together with the colonosocopic images to achieve the colon 3D reconstruction with high accuracy. The proposed framework includes 3D scan (point cloud with RGB information) reconstruction from stereo images, a visual odometry (VO) based camera pose initialization module, a 3D registration scheme for matching texture scans to the segmented colon model, and a barycentric-based texture rendering module for mapping textures from colonoscopic images to the reconstructed colonic surface. A realistic simulator is developed using Unity to simulate the procedures of colonoscopy and used to provide experimental datasets in different scenarios. Experimental results demonstrate the good performance of the proposed 3D colonic surface reconstruction method in terms of accuracy and robustness. Currently, the framework requires a pre-operative colon model as the template for colon reconstruction and can reconstruct 3D colon maps when the colon has no large deformation and the colon structure is clearly visible. The datasets used in this article and the developed simulator are made publicly available for other researchers to use (https://github.com/zsustc/colon_reconstruction_dataset).