Depression detection via multimodal analysis using a large language model-powered interview platform

The global prevalence of Major Depressive Disorder (MDD) increases annually, establishing it as one of the leading contributors to the global burden of diseases. Traditional symptom-based methods, such as face-to-face interviews between psychiatrists and patients, are widely used for detecting and assessing depression. However, these methods are time-consuming which often require several tens of minutes per interview. While the patient population is growing, there is a significant shortage of qualified clinicians, especially in low- and middle-income communities. Furthermore, the subjectivity in human assessments leads to variability in diagnostic results among different clinicians. In this study, we aim to address these challenges by developing objective, efficient, and reliable Artificial Intelligence (AI) tools and methods for the detection and assessment of MDD. Specifically, we have developed a human–computer interview platform, powered by Large Language Models (LLMs) equipped with knowledge from the Hamilton Rating Scale for Depression (HAMD). Additionally, we proposed multimodal topic-aware deep learning models that are trained on text, audio, and video data to detect MDD, assess depression severity, and predict HAMD scores. Data were gathered through human–computer interviews with 346 participants recruited from psychiatric outpatient clinics and community settings. The experimental results demonstrate that our platform achieves diagnostic performance comparable to that of trained clinicians.