SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis: SkinCLIP-VL

Zhixiang Lu; Shijie  Xu; Kaicheng Yan; Xuyue Cai; Chong Zhang; Yulong Li; Angelos Stefanidis; Anh Nguyen; Jionglong Su

SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis: SkinCLIP-VL

Zhixiang Lu
, Shijie Xu
, Kaicheng Yan
, Xuyue Cai
, Chong Zhang
, Yulong Li
, Angelos Stefanidis
, Anh Nguyen^*
, Jionglong Su^*

^*Corresponding author for this work

XJTLU
XJTLU
Xi'an Jiaotong-Liverpool University
Mohamed Bin Zayed University of Artificial Intelligence
Shanghai Artificial Intelligence Laboratory
University of Liverpool

Research output: Chapter in Book or Report/Conference proceeding › Conference Proceeding › peer-review

85 Downloads (Pure)

Abstract

The deployment of vision-language models (VLMs) in dermatology is hindered by the trilemma of high computational costs, extreme data scarcity, and the black-box nature of deep learning. To address these challenges, we present SkinCLIP-VL, a resource-efficient framework that adapts foundation models for trustworthy skin cancer diagnosis. Adopting a frozen perception, adaptive reasoning paradigm, we integrate a frozen CLIP encoder with a lightweight, quantized Qwen2.5-VL via low-rank adaptation (LoRA). To strictly align visual regions with clinical semantics under long-tailed distributions, we propose the Consistency-aware Focal Alignment (CFA) Loss. This objective synergizes focal re-weighting, semantic alignment, and calibration. On ISIC and Derm7pt benchmarks, SkinCLIP-VL surpasses 13B-parameter baselines by 4.3-6.2% in accuracy with 43% fewer parameters. Crucially, blinded expert evaluation and out-of-distribution testing confirm that our visually grounded rationales significantly enhance clinical trust compared to traditional saliency maps.

Original language	English
Title of host publication	The IEEE International Conference on Multimedia & Expo 2026
Subtitle of host publication	ICME 2026
Publisher	IEEE Press
Chapter	1
Pages	1-6
Number of pages	6
Publication status	Published - 5 Jul 2026
Event	The IEEE International Conference on Multimedia & Expo 2026: ICME 2026 - Bangkok, Thailand, Bangkok, Thailand Duration: 5 Jul 2026 → 9 Jul 2026 https://2026.ieeeicme.org/

Conference

Conference	The IEEE International Conference on Multimedia & Expo 2026
Country/Territory	Thailand
City	Bangkok
Period	5/07/26 → 9/07/26
Internet address	https://2026.ieeeicme.org/

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 3 Good Health and Well-being

Access to Document

ICME_SkinCLIP-VLAccepted author manuscript, 7.71 MBLicence: CC BY

Cite this

@inproceedings{53c9ccdb887e49ff83a276d7e8205b29,

title = "SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis: SkinCLIP-VL",

abstract = "The deployment of vision-language models (VLMs) in dermatology is hindered by the trilemma of high computational costs, extreme data scarcity, and the black-box nature of deep learning. To address these challenges, we present SkinCLIP-VL, a resource-efficient framework that adapts foundation models for trustworthy skin cancer diagnosis. Adopting a frozen perception, adaptive reasoning paradigm, we integrate a frozen CLIP encoder with a lightweight, quantized Qwen2.5-VL via low-rank adaptation (LoRA). To strictly align visual regions with clinical semantics under long-tailed distributions, we propose the Consistency-aware Focal Alignment (CFA) Loss. This objective synergizes focal re-weighting, semantic alignment, and calibration. On ISIC and Derm7pt benchmarks, SkinCLIP-VL surpasses 13B-parameter baselines by 4.3-6.2\% in accuracy with 43\% fewer parameters. Crucially, blinded expert evaluation and out-of-distribution testing confirm that our visually grounded rationales significantly enhance clinical trust compared to traditional saliency maps.",

author = "Zhixiang Lu and Shijie Xu and Kaicheng Yan and Xuyue Cai and Chong Zhang and Yulong Li and Angelos Stefanidis and Anh Nguyen and Jionglong Su",

year = "2026",

month = jul,

day = "5",

language = "English",

pages = "1--6",

booktitle = "The IEEE International Conference on Multimedia \& Expo 2026",

publisher = "IEEE Press",

note = "The IEEE International Conference on Multimedia \& Expo 2026 : ICME 2026 ; Conference date: 05-07-2026 Through 09-07-2026",

url = "https://2026.ieeeicme.org/",

}

Lu, Z, Xu, S, Yan, K, Cai, X, Zhang, C, Li, Y, Stefanidis, A, Nguyen, A & Su, J 2026, SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis: SkinCLIP-VL. in The IEEE International Conference on Multimedia & Expo 2026: ICME 2026., 1, IEEE Press, pp. 1-6, The IEEE International Conference on Multimedia & Expo 2026, Bangkok, Thailand, 5/07/26.

TY - GEN

T1 - SkinCLIP-VL: Consistency-Aware Vision-Language Learning for Multimodal Skin Cancer Diagnosis

T2 - The IEEE International Conference on Multimedia & Expo 2026

AU - Lu, Zhixiang

AU - Xu, Shijie

AU - Yan, Kaicheng

AU - Cai, Xuyue

AU - Zhang, Chong

AU - Li, Yulong

AU - Stefanidis, Angelos

AU - Nguyen, Anh

AU - Su, Jionglong

PY - 2026/7/5

Y1 - 2026/7/5

N2 - The deployment of vision-language models (VLMs) in dermatology is hindered by the trilemma of high computational costs, extreme data scarcity, and the black-box nature of deep learning. To address these challenges, we present SkinCLIP-VL, a resource-efficient framework that adapts foundation models for trustworthy skin cancer diagnosis. Adopting a frozen perception, adaptive reasoning paradigm, we integrate a frozen CLIP encoder with a lightweight, quantized Qwen2.5-VL via low-rank adaptation (LoRA). To strictly align visual regions with clinical semantics under long-tailed distributions, we propose the Consistency-aware Focal Alignment (CFA) Loss. This objective synergizes focal re-weighting, semantic alignment, and calibration. On ISIC and Derm7pt benchmarks, SkinCLIP-VL surpasses 13B-parameter baselines by 4.3-6.2% in accuracy with 43% fewer parameters. Crucially, blinded expert evaluation and out-of-distribution testing confirm that our visually grounded rationales significantly enhance clinical trust compared to traditional saliency maps.

AB - The deployment of vision-language models (VLMs) in dermatology is hindered by the trilemma of high computational costs, extreme data scarcity, and the black-box nature of deep learning. To address these challenges, we present SkinCLIP-VL, a resource-efficient framework that adapts foundation models for trustworthy skin cancer diagnosis. Adopting a frozen perception, adaptive reasoning paradigm, we integrate a frozen CLIP encoder with a lightweight, quantized Qwen2.5-VL via low-rank adaptation (LoRA). To strictly align visual regions with clinical semantics under long-tailed distributions, we propose the Consistency-aware Focal Alignment (CFA) Loss. This objective synergizes focal re-weighting, semantic alignment, and calibration. On ISIC and Derm7pt benchmarks, SkinCLIP-VL surpasses 13B-parameter baselines by 4.3-6.2% in accuracy with 43% fewer parameters. Crucially, blinded expert evaluation and out-of-distribution testing confirm that our visually grounded rationales significantly enhance clinical trust compared to traditional saliency maps.

M3 - Conference Proceeding

SP - 1

EP - 6

BT - The IEEE International Conference on Multimedia & Expo 2026

PB - IEEE Press

Y2 - 5 July 2026 through 9 July 2026

ER -