Implications of imbalanced datasets for empirical ROC-AUC estimation in binary classification tasks

The area under the curve (AUC) is the most popular measure for summarizing a binary classifier's receiver operating characteristic (ROC) curve. Therefore, it is essential to ensure that the AUC estimation is accurate. One straightforward and popular estimation approach is to calculate the empirical AUC from the data. However, one must look closely at the behaviour of this point estimator, particularly its variance. This study demonstrates both analytically and empirically that the empirical AUC estimation could be highly volatile in many circumstances when applied to an imbalanced dataset. To be more specific, we have proved that under some frequently encountered circumstances, variances of the empirical AUC estimator increase with the imbalanced level of the dataset. Hence, under the imbalanced setting, variances could be high. Furthermore, we conduct several simulations and experiments to solidify our findings. Therefore, extra attention must be paid when the empirical ROC-AUC is used to summarize the classifier's performance, especially when the dataset presents high class imbalance.