You are here:
Workflow of Data Processing Engine Definition Runs
A definition's run consists of different stages. The stages include syncing data sources on the definition's runtime platform, transforming data, and writing back results to target objects. The overall workflow remains the same for definitions on CRM Analytics and Data Cloud. Though there are key differences in the resources used to run definitions of each platform.
Required Editions
| View supported editions. |
Overview of a Definition's Run
A Batch Job record is created for each run. The batch job consists of details of all the stages of the run and provides a unified summary of the runs after they complete. For each stage, one or more Batch Job Part records are created. The number of batch job parts that are created vary based on the definition's design and the runtime platform. Batch job parts are also known as tasks. The tasks are executed in a sequence and if one task fails, the subsequent tasks are automatically canceled.
Workflow of a Definition's Run on CRM Analytics (CRMA)
Here are the stages of a definition's run on CRMA:
| Stage | What Happens | Tasks Created | Resources |
|---|---|---|---|
| Data sync | Data from each of the definition's data sources are synced in CRMA. | A task is created to sync data from each of the definition's data sources. Each task has the same name as the Data Source node that it syncs data for. | To learn about how to track the data sync jobs in CRM Analytics, see . |
| Data transformation | A recipe is created in CRMA to transform the data. After the recipe completes the data transformation, the data is sent back to Data Processing Engine for write back. |
A task named Execution is created. | To learn about how recipes transform data, see |
| Data write back | Transformed data is written back to the target objects and CRMA datasets.
|
A task is created to write back data to each of the definition's target writeback objects. Each task has the same name as the Writeback Object node that it writes back data for. If a definition consists of a Composite Writeback node, a task is created for each of the Writeback Object nodes that are part of the Composite Writeback node and a task is created for the Composite Writeback node. All the tasks have the same name as the Composite Writeback node but the tasks for the individual Writeback Object nodes contain a numerical qualifier. If the writeback target is a CRM Analytics dataset, a task isn't created for write back. |
To learn about the resources that are used to write back results, see: |
After a run completes, a task named Analysis is created. This task prepares the analysis of the completed run. The analysis of the run is available on the Analysis tab of the run's batch job.
Workflow of a Definition's Run on Data 360
Here are the stages of a definition's run on Data 360:
| Stage | What Happens | Tasks Created | Resources |
|---|---|---|---|
| Data cleanup | Transient (in-memory) data that has remained in Data 360 from the definition's previous runs is deleted in preparation for the current run. | A task named Cleanup is created. | N/A |
| Data sync | Data from each of the definition's data sources are synced in Data 360.
For Data 360-native data sources such as data model objects (DMOs), data lake objects (DLOs), and calculated insight objects (CIOs), data sync isn't required. |
|
To learn about the resources that are used to sync data, see . |
| Data transformation | A batch data transform is created in Data 360 to transform the data. | A task named Execution is created. | To learn about batch data transforms in Data 360, see Batch Data Transforms. |
| Data write back | Transformed data is written back to the target objects.
|
A task is created to write back data to each of the definition's target writeback objects.
If the definition consists of a Composite Writeback node, a task is created for each of the Writeback Object nodes that are part of the Composite Writeback node and a task is created for the Composite Writeback node. All the tasks have the same name as the Composite Writeback node but the tasks for the individual Writeback Object nodes contain a numerical qualifier. |
To learn about the resources used to write back results, see: |
Extra Tasks Created for a Run
If you run a definition from the builder and use the Save Count of Node Records or Save Node Output Without Writing Back Results debug mode for the run, extra tasks are created. Each task is named after the node for which you wanted to view the output record count. For more information about how the debug mode, see Debug Mode: Save Count of Node Records.
