You are here:
Data Processing Engine Best Practices
Get to know the best practices for creating and running Data Processing Engine definitions.
Required Editions
| View supported editions. |
Know Your Target Objects
Based on your business process, decide which objects and fields you want to write your results back to. If you want to write your results back to a custom object, create the custom object and the required fields first.
Plan Your Data Processing Engine Definition
Write or draw out all the nodes and input variables that you want to create in your Data Processing Engine definition. Understand the node creation sequence. Decide which input variables you want to define and specify values for while running the definition in Flows.
Know Your Write Back User Permissions
When you define a Writeback Object node, specify a user with the writeback permissions. Results are written back to the target object if the user has permissions to the target object, fields, and related fields. The user needs Create and Edit permissions for object and fields.
Create Your Data Sources First
The data source nodes are the only nodes that determine what data your definition is working with. All your subsequent nodes are based on the data that's extracted from the objects selected in the source nodes.
Use Formula Fields as Data Source
Values stored in Formula data type fields sometimes may not have the latest calculated value in CRM Analytics. To make sure definition runs use the latest calculated field values, avoid using formula fields as data source. Instead, use the Formula node to calculate formula field values.
Alternatively, if formula fields are used as data sources, make sure that the objects that consist of the formula fields have their CRM Analytics' connection mode set to Full Sync.
Create Input Variables Before Creating the Node That Uses It
You can use input variables in Filter and Formula node types. Before you create a node that uses an input variable, create the required input variable.
Create Your Nodes in a Sequence
Each node can use the results of a previously created node as its source. Create nodes in a sequence to ensure that all the nodes other than the Writeback Object nodes are referenced by another node.
Save Your Definition Frequently
Sometimes the Data Processing Engine Builder can have unexpected problems such as losing Internet access. Salesforce doesn’t save your changes automatically, so make sure to save your work often.
Avoid Simultaneous Runs of Your Definition
A Data Processing Engine definition can have two or more runs in progress simultaneously. Such runs process the same data multiple times and write back duplicate results to the target objects. Depending on how you started the Data Processing Engine definition run, you can ensure that there are no simultaneous runs of the same definition.
| Run type | Action |
|---|---|
| Runs started from the Data Processing Engine builder | Check Monitor Workflow Services to ensure that another run of the definition isn't in progress. To cancel a run that's in progress and start a new run, click Cancel Job. |
| Runs started using Flows | Before you run a definition in a flow, check whether the definition has any in-progress runs. Use the Batch Job object to check whether a definition has any runs in either Submitted or In Progress status. |
If you run two or more definitions with the same target object simultaneously, then the results of one definition can affect the results of the other definitions. To avoid this issue, run the definitions one after another.
Make Sure New Data Sources Are Synced to Companion Orgs Before Running Data Cloud Runtime Definitions
It takes time for data model object (DMO) and calculated insight object (CIO) metadata to sync from the home org to companion orgs. If you run a definition before the new DMO or CIO metadata syncs in the companion org, the run can fail.
You must add a pause before running definitions in companion orgs for these scenarios:
- Using a new DMO or CIO: If you create a DMO or CIO in the home org, wait for the metadata to sync to the companion org before running any definition that uses the new DMO or CIO as a data source.
- Running dependent definitions: If one definition writes to a new DMO or CIO and a subsequent definition immediately uses the newly created DMO or CIO as data source, you must configure a sufficient pause between the runs. The pause makes sure that the dependent definition's run can access data from the new DMO or CIO in the companion org.
Base the duration of the pause on the amount of time Data Cloud takes to sync metadata from your home org to the companion org.
- Data Processing Engine Best Practices for Data Sync in CRM Analytics
After a batch job is created to run a Data Processing Engine definition with CRM Analytics runtime, new and updated records from since the last run are synced in CRM Analytics. To optimize the data sync process during definition runs, adopt a data sync strategy that works best of your company.
