Large Language Models as Causal Reasoners for Robust Medical Segmentation: CAUSAL-SAM-LLM

The clinical utility of deep learning for medical image segmentation is hindered by poor generalization, as models often learn spurious correlations between anatomy and domain-specific styles. To address this, we introduce Causal-SAM-LLM, a framework that leverages Large Language Models(LLMs) for causal reasoning atop a frozen Segment Anything Model (SAM) encoder. Our approach introduces two synergistic components. First, Linguistic Adversarial Disentanglement (LAD) uses a Vision-Language Model to generate textual descriptions of confounding styles and contrastively trains the segmentation features to be style-invariant. Second, Test-Time Causal Intervention (TCI) allows an LLM to interpret a clinician’s natural language commands to modulate decoder features for real-time error correction. On a composite benchmark from four public datasets (BTCV, CHAOS, AMOS, BraTS), we evaluate out-of-distribution (OOD) generalization across scanners, modalities, and anatomies.Causal-SAM-LLM sets a new state-of-the-art, outperforming the strongest baseline by up to 5.2 points in Dice score and 10.0 mm in Hausdorff Distance, while fine-tuning under 8% of the total parameters. Our work charts a path toward robust, efficient, and interactively controllable medical AI.