Implications of imbalanced datasets for empirical ROC-AUC estimation in binary classification tasks

Yujian Liu, Dejun Xie*, Yazhe Li

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)


The area under the curve (AUC) is the most popular measure for summarizing a binary classifier's receiver operating characteristic (ROC) curve. Therefore, it is essential to ensure that the AUC estimation is accurate. One straightforward and popular estimation approach is to calculate the empirical AUC from the data. However, one must look closely at the behaviour of this point estimator, particularly its variance. This study demonstrates both analytically and empirically that the empirical AUC estimation could be highly volatile in many circumstances when applied to an imbalanced dataset. To be more specific, we have proved that under some frequently encountered circumstances, variances of the empirical AUC estimator increase with the imbalanced level of the dataset. Hence, under the imbalanced setting, variances could be high. Furthermore, we conduct several simulations and experiments to solidify our findings. Therefore, extra attention must be paid when the empirical ROC-AUC is used to summarize the classifier's performance, especially when the dataset presents high class imbalance.

Original languageEnglish
Pages (from-to)183-203
Number of pages21
JournalJournal of Statistical Computation and Simulation
Issue number1
Publication statusPublished - 23 Jul 2023


  • Area under receiver operating characteristic curve
  • empirical AUC estimator
  • imbalanced dataset


Dive into the research topics of 'Implications of imbalanced datasets for empirical ROC-AUC estimation in binary classification tasks'. Together they form a unique fingerprint.

Cite this