You are here:
Evaluate Model Quality
Use training metrics to evaluate the model’s ability to predict an outcome and determine whether it’s ready to activate. Training metrics provide information about how effectively the model understands patterns and relationships within the training data, and indicates its predictive efficacy.
To evaluate model quality, it’s helpful to review key concepts.
- The outcome variable is the goal of the model. It’s the desired prediction or outcome.
- Variables are the input data that the model analyzes to make its predictions.
- Training is the process the model takes to learn patterns and relationships that can be used to make predictions on new, unseen data.
Accuracy
To measure accuracy, Einstein uses r-squared for regressions and area under the curve (AUC) for binary classifications. Based on standard thresholds for performance, Einstein tells you whether the overall accuracy of your model is performant, too low, or too high.
- Performant means the model is generally accurate.
- Too low means the model isn’t much better than random guessing.
- Too high means the model is perfect or nearly perfect, which indicates potential data leakage or overfitting.
Top Predictors
The top predictors are the inputs, or variables that have the greatest impact on predicting the outcome.
Variable Distribution
The distribution of actual, or observed, values in the data for the outcome variable.
- Regression Metrics
Metrics for regressions help evaluate the performance of a model that predicts numerical values for continuous data. - Binary Classification Metrics
Metrics for binary classification help evaluate the performance of a model that categorizes data into two classes.
