You are here:
Batch Data Transform Run Types
A batch data transform runs in either incremental or full mode. During an incremental run, the transform applies only to input data that has been added, updated, or deleted. These changes can occur through a data stream or from the target of another batch data transform that runs incrementally. A full run processes both new and existing data. You can’t configure the run type. The batch data transform selects the run type based on specific criteria.
Incremental Runs
These criteria are required for an incremental run.
- The transform is scheduled.
- Updates to input data are less than 5% of the total amount of data.
- The data input objects are DLOs or DMOs.
- For DMO inputs, the primary key field is mapped to a primary key field in one of its DLOs.
- When Replace is selected for the output Write Mode
Note When selecting "Replace" as the output Write Mode, an attempt is made to run the transform in incremental mode. When the transform runs in incremental mode, it upserts or deletes only the changes records, it doesn’t replace them. Only changes from input sources from previous successful runs are loaded and the data is then merged to the output node. When the transform doesn’t qualify for incremental, it runs in full mode.
An incremental run supports these functions.
- Buckets
- Edit attributes
- Drop columns
- Filters
- Single-row formulas
- Data type conversion transformations—Text to Date, Text to Number, Number to Text
- Date transformations—Add or Subtract Days or Months, Date Difference, Extract
- Number transformations—Fill
- Text transformations—Trim, Substring, Split, Uppercase, Lowercase, Replace, Format Date
During certain scenarios, data drift can cause duplicate or incorrect records in incremental transform output. Data drift is corrected in the next full run. See Data Drift in Incremental Runs.
Full Runs
A transform with these features requires a full run.
- Manual runs triggered by a flow or initiated by clicking Run Now
- Aggregations
- Multi-row formulas
- Joins
- Updates to more than 5% of the total amount of data
- When Upsert is selected for the output Write Mode
- When external DLOs or DMOs are used as input sources
A full run always occurs after these actions, even if the transform meets the criteria to run incrementally.
- Edit a transform
- Update the input DMO mappings
- Rebuild the input data object
To mitigate drift, a full run always occurs on the first scheduled run of the day for all transforms. Drift occurs when results that previously satisfied filter conditions continue to display after they no longer match the updated conditions. The daily full run corrects this issue.
- Data Drift in Incremental Runs
When running a batch data transform in incremental mode, data drift can cause output objects to include changed records instead of filtering them out or replacing them. Data drift is corrected in the daily scheduled full run.
