You are here:
Predictive Model Monitoring
Maintain the accuracy, stability, and reliability of your deployed predictive models with model monitoring in AI Models (formerly Einstein Studio). This ongoing process provides in-depth insights into model's performance and identifies data quality or connectivity issues. Without consistent monitoring in Data 360, model performance can decline and lead to inaccurate predictions, unexpected outcomes, and reduced confidence in your results.
Model monitoring is crucial for these reasons:
- Maintains reliability and trust of a model for continuous use.
- Monitors how an activated model performs with live data to ensure that it fulfils its goal.
- Provides an understanding of corrective actions such as retraining the model with new data.
- Tracks performance to understand a model’s behavior over time based on these metrics.
- Model usage statistics for live models, including those created from scratch (Einstein- created models) and connected models (Bring Your Own Models—BYOM).
- Connectivity issues (connected models)
- Data quality issues (models created from scratch)
Model Usage (Created from Scratch and Connected Models)
Track the usage of models created from scratch and connected models to get insights about model inferences and statistics using live prediction data. Time series charts offer flexible periodicities, allowing you to view data over the last 24 hours, 7 days, or 30 days.
| STATISTICS | DESCRIPTION | KEY METRICS |
|---|---|---|
| Activity Metrics | Key usage data and errors compared to a previous time period. | Total Inferences, connection errors, time out errors, and authentication errors, often with a trend indicator (for example, +1.3% vs last 30 days). |
| Total Inferences Over Time | Time series chart with the total inference count generated across the selected time period. | Total inferences with a trend indicator. Activated version numbers are represented as vertical markers. |
| Inferences by Integration Channel | Bar chart showing the distribution of predictions based on the integration channel. | Inferences by the integration channels such as predict job, transforms, flows, and APIs. |
Data Quality (Created from Scratch Models Only)
Model performance relies on data quality, and issues can occur if live data differs significantly from training data. Monitor to track data quality issues such as missing values and out-of-bounds values. Alerts for these issues surface when the number of records are affected by an issue for a specific variable.
| ISSUE | DeSCRIPTION | |
|---|---|---|
| Missing Values | Total count of missing values across all variables against the total number of inferences. | Count of missing values, total inferences, and a trend percentage. The trend indicator shows if values are up by 20% or down by 30% in comparison to the selected time period. |
| Missing Values by Variable | Bar chart with variables that contain missing values. A card that also displays metrics by variable and integration channel. | Percentage of missing values for each variable. |
| Out-of-Bound (OOB) Values by Variable | Bar chart with variables that contain values outside the expected range. A card also displays metrics by variable and integration channel. | Percentage of out-of-bounds values for each variable. |
Connectivity (Connected Models Only)
Alerts that are surfaced for connectivity issues include connection, timeout, and authentication errors, which are tracked with live data.
| ERROR | DeSCRIPTION | key metric |
|---|---|---|
| Connectivity Errors | Displays the total number of errors (connection, time out, authentication) against the total number of inferences. | Total connectivity errors with a trend indicator. |
| Connection Errors Over Time | Time series chart with connectivity issues by the cause over a period of time. | Connectivity issues over time with a trend indicator. |
- Drift Monitoring (Beta)
Use Drift Monitoring in Data 360 to track predictive model degradation, which occurs when the distribution of model predictions changes significantly over time. Drift indicates shifts in customer behavior, business conditions, or underlying data patterns.
