You are here:
Data Processing Engine Limits and Considerations
When you create and run Data Processing Engine definitions, adhere to the Salesforce org limits and keep the considerations in mind.
Required Editions
| View supported editions. |
Data Processing Engine Definition Limits
When you design a Data Processing Engine definition, stay within these Salesforce org limits. Contact your Salesforce representative if you want to increase any of the Data Processing Engine definition limits.
- A Salesforce org can have up to 50 active definitions.
- An org can have 50 unique data source objects across definitions.
- A definition can have up to 500 nodes.
- A node in a definition can have up to 500 fields.
- A definition can have up to 50 Data Source nodes.
- A Filter node can have up to 50 filter conditions.
- A Group and Aggregate node can group fields by up to ten fields.
- You can upload a CSV file of size up to 75 MB to Salesforce.
- Case statement conditions can’t exceed 1,600 characters cumulatively across all nodes in a definition.
Flow Limits
Keep the Flows limits and considerations in mind when you create a flow to run a definition. For more information, see Flow Limits and Considerations.
Data Pipelines Limits
Track your org's daily and monthly usage limits for Data Pipelines when you run Data Processing Engine definitions using the CRMA runtime. Definitions don't run if your org exceeds these usage limits. Contact your Salesforce representative if you want to increase the Data Pipelines limits.
| Limit | Value | NOTES |
|---|---|---|
| Maximum number of writeback nodes per Data Processing Engine definition | 20 | N/A |
| Maximum number of users assigned the Data Pipelines Base User permission set | 3 | N/A |
| Maximum number of rows in all datasets | 250 million | N/A |
| Maximum number of rows processed | 10 million per month | N/A |
| Maximum processing time per month | 30 hours | The cumulative minutes taken by the recipes of all Data Processing Engine definition runs to transform data . The recipe job runtime doesn't consider the time taken to sync data for recipe transformation and export results for writeback. |
| Maximum number of concurrent recipes | 1 | Data Processing Engine executes only one recipe at a time per license even if multiple jobs are triggered at the same time. Purchase additional licenses if you need to process more than one recipe at a time. |
| Maximum amount of data written back in a rolling 24-hour period to Salesforce: | 1 GB | Data Processing Engine assesses this limit when a job starts. If the limit is exceeded, the engine doesn’t start the job. If the limit isn’t exceeded, the engine starts and finishes the job even if the job writes data that results in the limit being exceeded. When the limit is exceeded, jobs fail until the amount of data written in the preceding 24 hours is below the limit. The amount of data that's written back by Data Processing Engine runs counts against Data Pipeline's Output to CSV limit and not against Data Pipeline's Output to Salesforce limit. If your org reaches the 1 GB Output to CSV limit, work with your Salesforce account executive to purchase add-ons. Each add-on adds 1 GB to your org's Output to CSV limit. If your org requires 5 GB each day, you must get 4 add-ons. |
| Maximum number of recipes | 1000 | N/A |
| Maximum number of objects enabled for data sync | 50 | N/A |
| Maximum number of concurrent data sync runs | 2 | N/A |
| Maximum average number of values per multivalue field | 100 | N/A |
Data Cloud Limits
Track your org's daily and monthly usage limits for Data Cloud when you run Data Processing Engine definitions using the Data Cloud runtime. Definitions don't run if your org exceeds these usage limits. Contact your Salesforce representative if you want to increase the Data Cloud limits.
| Limit | Value |
|---|---|
| Maximum number of CSV data source nodes in a Data Processing Engine definition | 5 |
| Maximum number of writeback nodes in a Data Processing Engine definition | 5 |
| Maximum size of CSV file that can be uploaded | 150 MB |
Notification Alerts
Admins receive alerts in the notification tray whenever the following Data Pipelines limits exceed 80% of the usage limit.
- Rows in All Datasets
- Recipe Job Runtime
- Output to CSV
- Recipe Rows Out
Review your Data Processing Engine definitions and take action.
Writeback Limits
Data Processing Engine writes results back to your Salesforce object and fields by using APIs. Writebacks are counted against your org's Bulk API 2.0 limits.
