Data Processing Engine Workflow

Define Your Business Requirements

Plan your Data Processing Engine definition based on the business problem or process that you want to configure the definition for.

• Define the business requirement such as reporting, analytics, data integration, and real-time insights.

  Determine the volume of data that you want to process and the frequency of updates, so you can decide the runtime where you can run your Data Processing Engine definitions.

• Make a list of the data sources needed for your data processing such as Salesforce data, external databases, custom objects, and CSV files.
• Determine the types of data transformations such as aggregations, joins, filters, and calculations.
• Decide the destination where you want to write back the transformed data, such as objects, datasets, or Data Cloud objects.

Design Your Definition

Create your Data Processing Engine definition and configure nodes to select data sources, transform data, and write back the transformed results.

• Add data source nodes for each of the definition's data sources.

  See Add Data Sources to Your Definition.

• Use nodes such as append, filter, formula, and join to transform the selected data.

  See Add Transformation Nodes to Your Definition.

• Use input variables to define default values, filter records and create formulas.

  See Use Input Variables to Define Dynamic Processing Rules.

• Add writeback object nodes to define where the transformed data is written back.

  See Add Writeback Object Nodes to Your Definition.

After you’ve designed your definition, activate it. You can preview the definition and run it from the builder with debug modes to validate whether the definition transforms data as per the business requirements.

Schedule Definition Runs

Run the definition as part of the business process that it transforms and writes back data for. Run the definition based on a schedule or based on triggers by using Salesforce Flow. Or, trigger runs from external systems

• Use the Action element in a flow.

  If an integration requires to run the definition, use the custom invocable action.

• Use Monitor Workflow Services to track the status of definition runs.

  To track the overall status of runs and their tasks, use the Batch Job record that's created for the run. The name of the Batch Job record is the same as the definition.