Data Drift in Incremental Runs

When running a batch data transform in incremental mode, data drift can cause output objects to include changed records instead of filtering them out or replacing them. Data drift is corrected in the daily scheduled ‌full run.

Data drift occurs in these situations.

• When filtering on values to remove records from the output, an updated record isn’t filtered out if the value that was changed now meets the removal criteria. The updated record is filtered out in the next full run.
• If a value in a column that’s mapped to a primary key column changes, the updated record is treated like a new record because the updated record’s generated primary key is different from the original key. In this scenario, the output of the incremental run retains one record with the original generated primary key and one record with the updated key until the next full run.

To correct data drift, a full run occurs every 24 hours.