Work with Data Processing Engine

Use the Data Processing Engine node canvas to add nodes to your Data Processing Engine definition. You can plan and create your definitions either in the CRMA or Data Cloud runtimes. Activate and run your definitions using the builder or using Flows. Use Monitor Workflow Services to monitor your definition runs.

• Create a Data Processing Engine Definition
  Manage complex data transformations and integrations in your org by using Data Processing Engine definitions.
• Add Data Source Nodes to Your Definition
  Use the Data Source node to define the origin of the data that the definition must transform and process. Data sources can include objects, files, CRM Analytics datasets and the supported data source types differ based on the definition's runtime platform. Before you add transformation nodes in a definition, add the data sources. Every definition must have at least one data source node.
• Add Transformation Nodes to Your Definition
  After you create and configure the source and target nodes, add additional nodes that specify how the source data in the data source node must be processed before the results are written to the writeback object node.
• Use Input Variables to Define Dynamic Processing Rules
  Define flexible data processing rules by using input variables to define default values, filter records, and create formulas. To dynamically control the data filters and formulas at run time, create variables without values and use them to configure Filter and Formula nodes. During a definition run, specify variable values based on your specific requirements. For example, to filter opportunities above a certain value for different fiscal years, set a default value for the input variable and then update it when you run the definition.
• Add Writeback Object Nodes to Your Definition
  Add one or more writeback object nodes to your Data Processing Engine definition to write back the transformed results from your data source. Configure your writeback object node to select the appropriate target object based on the runtime where you plan to run your definition.
• View Reference Nodes
  Select a node and view the details of other nodes that currently reference elements of it, such as node sources and mapped fields. These nodes that reference the selected node are known as reference nodes. If you make changes to a node, it’s helpful to know its reference nodes to anticipate any changes that need to be made to them, too.
• Validate Definition Design on the Data Processing Engine Builder with Data Preview
  Validate definitions in increments as you keep adding nodes to it. Preview how data is transformed and passed on from one node to another without saving, activating, or running definitions multiple times to make them work as per your expectations. If nodes don't transform data as expected, use the data preview results to find and fix node configuration issues. You can simulate different runtime settings to verify whether the definition works for different data.
• Activate Data Processing Engine Definitions
  Validate and enable your definition to run in your org after you've added and configured the data source, transformation nodes, and writeback object nodes
• Run Data Processing Engine Definitions
  Use a flow to run a definition on a schedule or as part of a business process. Developers use invocable actions, Metadata API, and Apex to run definitions.
• Monitor Data Processing Engine Definition Runs
  Monitor definition runs to review the status of your definition runs. On Monitor Workflow Services, you can analyze key details of definition runs such as count of records processed and written back, analysis of latest and previous runs, and reasons for the failure of the latest runs.
• Process Datasets in Near Real Time with On-Demand Processing
  Process small to medium datasets in Salesforce and write back results as new or updated records without waiting for a scheduled window. Run near real-time data validations and ad hoc processing from Salesforce objects, context definitions, and CSV files by using on-demand definitions.
• Data Processing Engine Definition to Calculate Purchase Credit and Bonus (Example)
  Cloud Kicks provides a 5% year-end purchase credit to accounts based on their total effective orders that ended between January 1 and November 30. The salespeople associated with these orders are also rewarded with 0.5% of the total order amount as a bonus. The purchase credit and bonus are granted to the accounts and salespeople on December 1 each year.
• Track Your Data Processing Engine Usage
  Data Processing Engine definition runs can process and write back data up to a specific size. If your Salesforce org reaches a daily or monthly limit, the definitions stop running.
• Share Data Processing Engine Definitions
  You can create Data Processing Engine definitions and share them with other Salesforce orgs using managed and unmanaged packages and change sets. Partners and independent software vendors (ISV) can create and share definitions with their customers.
• Customize Data Processing Engine Template Definitions
  Customize an out-of-the-box Data Processing Engine (DPE) template definition to add data sources or nodes, adjust transformation logic, or support business calculations not included in the template.
• Delete a Data Processing Engine Definition
  Delete inactive Data Processing Engine definitions that you aren’t using. Before you delete a definition, make sure that it’s not referenced in any flow.
• Build Data Processing Engine Definitions in a JSON File
  Build a Data Processing Engine definition in a JSON file and then upload it to the builder. You can also download the definition, make changes, and then upload the updated definition.
• Troubleshoot Data Processing Engine Issues
  Can’t seem to get the Data Processing Engine working for you? To resolve issues with Data Processing Engine, try these solutions.
• Data Processing Engine Developer Resources
  You can create, activate, and run Data Processing Engine definitions programmatically by using APIs.