Monitor Data Pipeline Health
Identify pipelines that need attention, investigate failed runs, and determine whether to reprocess or retransform data.
Required Editions
The Data Pipelines list provides an overview of each pipeline’s availability and recent run activity. A pipeline can be Active and still have failed runs. Active means that the pipeline is available to run, not that every run succeeded.
| Available in: Developer, Enterprise, Performance, and Unlimited editions. |
| User Permissions Needed | |
|---|---|
| To monitor data pipeline health | Marketing Intelligence Data Specialist |
- In Marketing Intelligence, on the Data Management tab, select Data Pipelines.
-
Find the pipeline and review these columns:
- Last Run Status: If the status is Failed, the latest run failed.
- Last 30 Runs: If the value isn’t 100%, at least one of the last 30 runs failed.
- If Last 30 Runs isn’t 100%, click View Failures to review the failed runs and their date ranges.
- To retry a failed run, find the run and then click Reprocess.
- To review the pipeline’s complete run history, click the pipeline name.
- On the Details tab, review the pipeline information. Expand Data Pipeline Entities to review Status, Last Refreshed, and Rows Processed for each entity.
-
Select the Process Log tab. Expand a run to review its data
streams and transforms, start and end dates, Status,
Run Type, last refreshed date, and rows added.
The Process Log includes scheduled runs and manual reprocesses.
-
For a failed run, hover over the failure indicator to check for an error
message.
Error details aren’t available for every failure.
| Reprocess | Retransform | |
|---|---|---|
| Use when | A run failed or source data is missing. | You changed mappings or calculated fields, or mapped custom fields. |
| What it does | Retrieves data from the provider and processes it through the pipeline. | Applies the current mappings and transformations to previously extracted data. |
| Retrieves data from the provider | Yes | No |
| How to run it | For one of the last 30 runs, click View Failures, find the run, and click Reprocess. For a longer gap, open the pipeline, click Reprocess, and select a date range of up to one year. | Open the pipeline and click Retransform. |
Mapping changes apply automatically only to data processed after the change. To apply the changes to previously processed data, run Retransform.
Before retransforming data, review the mappings on the Data Sets tab. For mapping validation errors during setup, see Troubleshooting Mapping Errors.
