Grounding Before Reporting: Local-to-Global Decoupled Learning for Medical Vision-Language Models

Vision-Language Models (VLMs) show promise for automated medical report generation, but conventional joint training often couples fine-grained visual grounding and holistic report generation within a single trainable parameter space. This causes optimization conflicts, where models over-rely on dominant global priors and produce generic reports with degraded anatomical grounding. To address this, we propose Local-to-Global Decoupled Learning (LGDL), a staged framework that preserves local anatomical knowledge during global adaptation. LGDL comprises three components: Local Expert Initialization for organ-level grounding, Decoupled Sequential LoRA Adaptation for separate optimization stages, and Local Knowledge Retention for lightweight replay. Evaluated on two grounded benchmarks: FOCUS (fetal ultrasound) and PadChest-GR (chest X-rays), LGDL improves Global and Local F1-scores by + 0.1237 and + 0.0167 on FOCUS, and + 0.0267 and + 0.0249 on PadChest-GR. These results demonstrate that explicit decoupling effectively mitigates supervision conflicts, yielding more accurate and anatomically grounded reports.