You are here:
Cross-Validation Failure Alert
Cross-validation evaluates how well a model generalizes to unseen data by splitting the dataset into multiple parts, training the model on some parts, and validating it on others. Failure occurs when training data isn’t large enough, uses too few examples, or has an invalid format, missing values, or outliers.
Actions to Consider
One of the folds in K-fold cross-validation performed very differently from the other folds. Investigate your training data for outliers or anomalous data and filter accordingly.
Detection Methodology
Model Builder displays an alerts when it detects a significant performance difference in one of the folds.
Example
A retail brand wants to predict the weekly demand for products in various stores so it can optimize inventory and improve restocking. To achieve this, the retailer builds and trains a regression model using these input variables.
- store ID
- product ID
- price
- promotion
- stock count
- season
- weekly sales
After model training, an alert displays because cross-validation failed. A lack of product, store, or historical sales data in the dataset or data splits for cross-validation can cause failure. To resolve the issue, here are some actions to consider.
- Retrain the model using attribute-rich input variables, such as “product type” or “store type.” Avoid using “product ID” or “store ID.”
- Remove variables such as “end-of-week stock count” that can leak future information, making the model less reliable.
- Use more data if the dataset is too small (too few rows, columns, or both) for cross-validation.
