You are here:
Area Under the Curve (AUC) Quality Alert
Area under the curve (AUC) is a performance metric for binary classification. It measures the ability of a model to distinguish between two classes: positive and negative (true and false, yes and no, and so on). The AUC is derived from the receiver operating characteristic (ROC) curve, which plots the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings.
Actions to Consider
- A low AUC indicates that the model doesn’t perform well. Consider improving the model before you activate it.
- A high AUC indicates that the model performs so well that data leakage can be a factor. Investigate and remove any variables from your model that can cause data leakage.
Detection Methodology
Model Builder displays an alert when it detects a high or low AUC based on these values.
- Below 0.6 (low)
- Above 0.95 (high)
Example
A telecom company wants to predict whether customers are likely to churn within the next month. The company builds a binary model with these input variables.
- customer tenure
- monthly charges
- internet service type
- number of complaints
- number of calls
- data usage
After model training, an alert displays because the AUC score was lower than expected (training AUC=0.58). The score is only slightly better than random guessing (AUC=0.5), and it indicates that the model isn't able to distinguish between customers who might leave or continue with the service. To resolve the issue, here are some actions to consider.
- Improve predictive accuracy with additional variables, such as, “customer satisfaction scores", “customer interactions", and “competitor presence".
- Improve imbalanced data by oversampling the smaller group (churners), or use class weighting to give churn data higher importance during model training.
- Clean the database to remove inconsistencies, outliers, and missing values.
- Adjust the AUC threshold to balance precision and recall based on business goals.
- Use more complex modeling algorithms, such as random forest or gradient boosting, if linear models underperform.
- Retrain the model with an updated dataset.
